How the definition of quality evolves as the project progresses?

We all have an interest in doing our best work. But sometimes it feels quite difficult to determine what the best possible quality is, given time and budget constraints.

The meaning of quality and project goals

What is quality? There are many definitions. In general, it is the level of satisfaction with meeting the expectations, both implicit and explicit, of all stakeholders. By stakeholders, we mean that everyone is affected by the state of the product and project: your customers, your users, your developers, or your suppliers.

The definition of quality must be consistent with the purpose and context of the project. Common sense dictates that the development and management of a prototype project cannot be approached in the same way as a long-term or high-risk domain project. And you cannot expect the same level of quality in these two types of projects.

How to define test approach
  • First, discuss the business goals of the project, assess the risks of the project, and determine quality expectations from all stakeholders. Depending on the project objectives, you need to understand stakeholder expectations of the project and determine what quality means to you.
  • Based on the context of the project, you should select applicable test activities that will help reduce risks and meet the definition of quality and achieve the business objectives. Then develop an approach that will help you achieve the required level of quality. You should also develop a testing strategy.
  • Discuss and approve the testing activities along with the development activities. They should be included in the project framework as they can be quite time consuming.
  • Once approved, put them in the project framework and in the definition of work completed.
  • After that, establish the testing process within the team.
  • But here is the trick: later, as the project evolves, the project goals may change. The basic test strategy and approach should be updated along with the project evolution.
How the definition of quality evolves with the project

As a rule, many changes are made to the project during its development. New requirements are identified, both functional (new features) and non-functional (e.g., accessibility, performance or localization requirements). The goal is to keep these expectations in mind as you develop your testing strategy and share the new quality vision with the team as it evolves. For example, if you're introducing new accessibility requirements, it's not enough to just change the code and test the changes once. It's important to add an accessibility criterion to the definition of "done" for all subsequent features.

We suggest regularly scheduled test strategy reviews with key project stakeholders (e.g., quarterly or biannually) to ensure that the quality definition and testing approach are aligned with current project goals. Below you will find three examples of quality definition vs. project goals to give you an idea of how the approach to testing can vary from project to project:

  • Proof of а concept
  • Short-term project (without further support)
  • Long-term project
Proof of а concept

Project Objectives: Understand the value or suitability of the solution. The constraints are time and budget.

Quality Definition: The product must be good enough to understand the value of the solution.

Possible test approach:
  • Manual ad-hoc testing
  • User acceptance testing (UAT)
  • UX testing (if applicable)
  • No time investments in dev support activities (unit tests, peer reviews, etc).

Risks: All project participants must clearly understand that this type of project has no release quality and is not designed for maintainability. Functional and non-functional requirements are not applicable to this type of project. Its only purpose is to determine the value of the solution.

Short-term project (without further support)

Project Objectives:Develop a short-term solution for a specific purpose. Usually the main constraint is time.

Quality definition:The product must meet functional and non-functional requirements such as reliability, performance, etc. (if applicable). However, maintainability may not be part of these requirements.

Possible test approach:
  • Manual and instrumental testing methods are used to ensure that the product meets requirements. The team uses exploratory testing and test development techniques.
  • UAT is used to make sure that the product meets business need)
  • The team may practice paired programming, peer code reviews, and team design reviews to prevent defects, but the team is not focused on continuous testing and code base coverage.
  • There is no temporary investment in continuous development support activities (e.g., unit tests or CI mechanism). A minimally viable set of tests can be implemented.
  • A task management system is chosen for short-term support (e.g., Trello may be used).
  • Defects are either fixed or quickly rejected.

Risks:: All project participants must clearly understand that this type of project has the quality of release, but because the main constraint is time, maintainability was not the focus of product development, so it may not respond well to change, and further support and development of such a project can be costly.

Long-term project

Project goals: Develop a long-term reliable solution. Volume is usually divided into iterations, and cost can be a major constraint. In addition, the project is expected to undergo a number of changes along the way.

Definition of quality: After each iteration, the product must meet functional and non-functional requirements. In addition, the approach to design, development and testing must prioritize maintainability and responsiveness to change from the beginning.

Possible test approach: The team focus is on continuous development, testing and support, so continuous automation testing is used when possible:

  • Automated regression test suites are created to provide quick feedback after any change is implemented.
  • Automated test suites include unit, integration, end-to-end, and acceptance tests.
  • Test coverage requirements may be introduced.
  • Automated contract tests may also be included in the suite.
  • Some non-functional requirements may be covered by continuous on-demand automated tests (e.g., continuous performance tests, automated security checks, or visual regression tests).
  • The team practices pair programming, peer code reviews, team design reviews, etc. to prevent defects, along with continuous automated testing.
  • Manual testing is used to ensure the product meets high-level business goals, appeal and usability. The team uses exploratory testing and test design techniques.
  • Instrumental non-functional testing is used to ensure the product meets non-functional requirements such as reliability, performance, etc. (if defined).
  • A long-term task management system is selected.

Risks: All project participants should understand that developer support, the creation of automated test suites, and careful design require more time, but only in short-term projects. Implementing changes without a continuous feedback mechanism, such as automated regression test suites, unit tests, or CI, can lead to a waste of time in long-term projects.

To conclude

It is critical for a project to update the definition of quality and the testing approach as the project goals are achieved and new requirements are identified. There are many types and techniques of testing to cover different product expectations, and you must choose the ones that fit the context of your project to ensure that all stakeholder expectations are met at any stage of the project life cycle.