White Box Testing – Guide, Tools, and Techniques

White box testing is a software testing approach based on an analysis of the internal structure of the component or system. Internal structure may include code, architecture, integrations, and data flows of a system.

Why is White Box Testing Performed?

Testers perform white box testing for several reasons. The main reason to include it in your test plan is that it provides much more test coverage than black box testing alone. The more code is tested, the less likely it is to have defects, and any defects found will have less severity. Suppose developers perform unit testing before delivering code to testers. In that case, they may have already addressed issues that the testers would have found.

Another reason for white box testing is to find defects earlier. It’s commonly known that the sooner a defect is discovered, the less time and money it will cost to resolve. For example, imagine finding a conflict in a work item’s requirements and acceptance criteria. Addressing that at the start of development will avoid having to handle a more significant defect later. White box testing also builds confidence in a system. A team can be confident they are delivering a higher quality product by testing more code, with multiple techniques, on multiple levels.

White Box Testing Techniques 

It’s permissible to do white box testing on any of the four testing levels, but taking place during unit/component and integration testing is the most common. Many companies have a specific code coverage requirement for unit tests, reported as a percent. In many CI/CD environments, it’s necessary for code coverage to be complete before any code can move to the next environment within the pipeline. Integration testing assesses components against each other which often requires an understanding of the system’s architecture. Finally, white box testing techniques are applied to ensure internal workings are performing as expected.

There are several different techniques for white box testing. The selected one or more approach is typically based on the system under test and project requirements.

Statement Coverage – A statement is a line of code that performs a specific task. Statement coverage is the most commonly used white box technique. It creates tests for any executable statement within the code (IF, IF-ELSE.) This allows the calculation of the test coverage percentage using the number of executed statements divided by the total number of executable statements. The number is then multiplied by 100 to get the percentage. A common goal is to have between 80-90% coverage.

Decision Coverage – Decision coverage is similar to statement testing as it tests executable code based on decisions within the code (Think IF or CASE statements). Since there can be multiple outcomes per decision, there will be multiple tests for each decision. Calculate the coverage by taking the number of executed outcomes and dividing it by the total number of possible outcomes. Multiply your answer by 100 to get the percentage.

Basis Path Testing – Similar to statement and decision coverage, basis path testing involves creating a test case for every path within the code. Based on a flow graph, these test cases aim to make the smallest number of test cases while covering every statement.

API Testing – Application Program Interfaces (API) allow a system to connect and share specified data to another system. API testing involves checking the communication, security, functionality, and reliability between system APIs.

Fault Injection – Fault injection testing involves purposefully entering erroneous data or creating errors and observing how the system responds and recovers. Doing this helps with code coverage because it shows possible vulnerabilities in the system. Sometimes this type of testing is performed by a security team.

Mutation Testing – Mutation testing is a way of challenging previously created component tests. Testing involves making small changes (mutations) in the source code and evaluating if the tests detect the error. It improves code quality by creating more effective tests.

Examples

Let’s first consider decision coverage. In our example, the system tries to determine if the user is over 18. Our code looks something like this:

We can create 2 test cases, where X = 17 and X=19, to test the two decisions. To calculate test coverage, we would use this equation:

Test Coverage = (Number of outcomes executed / Number of outcomes possible) x 100

(2/2) x 100 = 100%

A simple way to perform mutation testing on the example above would be to change the operators.

White Box Testing and the SDLC

No matter the technique used, white box testing always starts with test planning. The software development lifecycle (SDLC) explains the software creation process from ideation to how to support it when in production. Planning is arguably the most important step because it sets the foundation for the remainder of the cycle.

Requirements are described and clarified during the analysis phase, and flow graphs and decision paths are mapped. Consider white box testing options while discussing the system and infrastructure during the design phase. This is an excellent opportunity to discuss code coverage and unit testing responsibilities.

Depending on the techniques you have included in your test plan, most white box testing takes place during the development and testing phase. Finally, the deployment and maintenance phases may require white box testing as issues arise. Good test documentation comes in very handy at this time because issue resolution is time-sensitive.

White Box Testing Tools

Coverage and Path Testing Tools

The following tools are generally open source and historically would have been downloaded and installed in a development or testing environment. Your selection of tool will depend on the programming language you are using. Below are a couple of examples for Java and Python:

• Cobertura – Cobertura is a free tool that analyzes the code to determine which parts need coverage. It is only for use with Java programs and works with Maven.
• Coverage.py – Coverage.py measures code coverage and supplies feedback on code that could have been executed but wasn’t. It has a free version and an Enterprise subscription package. It is only for use with Python.

With the rise of CI servers and GitHub actions, these tools have evolved to analyze the code whenever it is committed to a central repository.

Many tools are available, and again, your choice will depend on your code’s language. To discover what’s currently available, search the GitHub Marketplace for code coverage, which will likely reveal a host of relevant tools.

API Testing Tools

• Postman – Postman is a popular platform for building, using, and testing APIs. It has both free and paid versions. It works with HTTP, REST, SOAP, GraphQL, and WebSockets.
• Swagger – Swagger is a suite of tools that helps to design, document, and test APIs. It has free and paid versions and works with REST, SOAP, and GraphQL.
• Insomnia – Insomnia offers a tool for designing APIs and another for testing and deploying CI/CD pipelines. It has free and paid versions and works with REST, SOAP, and GraphQL.

Fault Injection Testing Tools

• OneFuzz – OneFuzz is Microsoft’s fuzzing-as-a-service platform that integrates with CI pipelines. It is free and works with Windows and Linux.
• Kraken – Kraken (sometimes krkn), injects deliberate failures into clusters to check their resiliency. It is free and works for Kubernetes and OpenShift.

Mutation Testing Tools

• PIT – PIT is a free tool that runs unit tests against automatically modified code versions. It is for use with Java, JUnit, and TestNG.
• Stryker – Stryker is a free tool that performs automated mutation testing, supporting over 30 mutation types. It works with JavaScript, C#, and Scala and will run on any test runner.

Limitations

White box testing does have some limitations. It can be costly because the work is tedious and lengthy. Developers bound by code coverage requirements have less time to code for the project. They can also become distracted by the pressure of the requirement instead of remaining focused on the product. White box testing can cause friction among the QA team when some testers are knowledgeable about the internal structures while others are not. When a situation like this happens, some testers may feel their knowledge is inadequate or of lesser value. It can also cause friction between developers and testers if testing responsibilities are vague or incorrectly assumed. Discussing expectations and responsibilities during planning is best practice to prevent these limitations.

Conclusion

White box testing requires specific knowledge of the system’s internal structure. It also supports the strategy of ‘test early and test often’ when enforcing code coverage requirements. There are many techniques for white box testing, although statement coverage is the most widely used. Testing tools are widely available to support the technique used. White box testing has the potential to build confidence in a system by providing more and better test coverage.