When to Stop Testing

Several factors can influence the decision on when to stop testing. These can include budget, time constraints, levels of testing required, addressing critical issues, or even the type of testing to be performed. Sometimes, if there is a simple code change with clear acceptance criteria or a project with a specific budget for testing, the time to stop can be self-evident. However, most of the time, choosing when to stop testing is far more complex.

Why We Test

Before we discuss when to stop testing, let’s first discuss why we test. Testing provides opportunities to improve the quality of a product. For example, defects and integration issues can be found and resolved when testing an application before it reaches a live environment. The sooner a defect is found, the cheaper it is to fix, so testing saves money. Also, delivering a quality product improves overall reputation, leading to increased sales. Thus, one could conclude that a prime objective of testing is to ensure a quality product is delivered that will benefit the business and the customer.

Common Reasons to Stop Testing

There are many reasons why testers stop testing. These include:

• End of the sprint – The sprint often dictates the time available for testing. Once the sprint ends, new features are delivered, so testing for the current code must conclude then. This doesn’t mean items can’t carry over to the next sprint, but it may be necessary to skip any extra testing, like exploratory sessions.
• End of the project – A project can end as planned, be paused indefinitely, or canceled without notice. No matter the reason, when a project concludes and priorities reset, it’s time to stop testing and move to the next priority. Management usually gives this instruction.
• Metrics are met – Some projects have metric goals as part of the requirements. For example, the test schedule may end when code coverage is at 85 percent, or no further bugs are discovered after a set amount of time.
• Exit Criteria are met – A good test plan will specify exit criteria detailing activities or conditions to meet before testing can conclude. Having these criteria makes the decision easier on when to stop testing. However, if time and budget allow, the tester may want to perform extra testing if they have concerns about a specific area.
• The end-user is satisfied – Depending on the issue, sometimes testing can be signed off at the point when the end-user or customer is satisfied. An example would be when the user has requested a simple change with existing code that has a low risk of breaking.
• GETMO – The term GETMO, or ‘Good Enough To Move On’ is frequently used when a tester has to make a decision affecting the business. When faced with the choice of approving or holding up a release, sometimes testers have to understand business priorities and judge when code is ‘good enough.’ Alternatively, if a code is not ‘good enough,’ a tester is duty-bound to reject passing it and be able to explain their decision.

It’s important to note that testing should not stop until bugs have been evaluated and prioritized. Critical bugs must be addressed and resolved before a release, but bugs assessed as being less impactful can either be resolved or logged to be fixed in an upcoming release. Testers must be diligent about reporting and following up on bugs found in the system.

Wrong Reasons to Stop Testing

Sometimes, testing can stop for poor reasons such as:

• Frustration – Testers can feel frustrated when there aren’t enough resources, they have to deal with poorly written code, teammates aren’t cooperating, or numerous other reasons. Frustration is an inevitable part of the job, but it is not a proper reason to stop testing.
• A hold up with development – Sometimes, development takes longer than anticipated. If this happens and there is no new code to test, some testers would consider it a nice break. However, the better option is to continue with writing test cases, performing exploratory sessions, updating regression tests, or any other test activities until the product is ready to be tested.
• Lack of requirements – While a lack of requirements (or poorly written ones) is frustrating and can prevent the tester from working on that feature, it shouldn’t cause them to stop testing other areas. The situation can also present a coaching opportunity. Show the ticket’s author why you need this information. Explain to them how you can test more efficiently with a well-defined and usable set of requirements.
• Too many defects – Occasionally, there are more bugs than one tester feels they can handle, leading the tester to give up until issues are resolved. This is not a good reason to stop testing. Instead, it’s a great time to dig in to ensure there’s proper documentation of each issue.
• Poor management – When management doesn’t seem interested in the success of a project, it can make it hard to care about the quality. If this situation occurs, remember that you can’t control what others do, only how you react to it. Take pride in your work and be a motivation to others.

Balance GETMO vs. Delaying a Release

An everyday conundrum in QA is judging if the code is GETMO or whether a delay is preferable. Deciding can be quite a difficult problem and often requires input from management, but you need to have the right information to pass along so others can make the correct decision. At this stage, there is typically a lot of pressure from the client, product team, and management. As a tester, you have to determine the risk vs. reward of releasing. While a tester must understand the business’s point of view, it is also their duty to ensure a quality product is released.

A good tester needs to have specific characteristics to make the right decisions. They need good judgment and to be bold when making a decision that they know will not be popular. They must understand the uniqueness of each project, and they also need to have empathy. Knowing that delaying a release is not their fault, but it will cause problems for others, the tester should be empathetic to the situation.

Risks

There are always risks involved when testing stops too early or too late. If testing stops too early, it’s possible to miss defects that would otherwise have been found. When this happens, it can cause the customer to lose faith in the product. Alternatively, stopping too late can cause the project to go over budget or delay releases unnecessarily, both of which can cause the business to have a poor reputation.

Conclusion

The decision to stop testing can be complicated, and it takes specific wisdom to make the right call. Testers often have to make tough decisions, and they should do so with integrity and empathy. The final decision to stop testing should always center on providing a quality product that benefits the business and the customer.