Structural testing encompasses three critical phases of software development and testing; yet, one or more of these phases is often deliberately bypassed, overlooked, or performed in a less than rigorous manner because either the technical advantages are not fully considered or, more often, the cost and schedule benefits are not appreciated. While structural testing is required by the FDA for medical devices of moderate and major level of concern,1 this testing should be done for all software, regardless of the level of concern. Also, it should be noted that there is no fundamental difference between structural testing of software used in a medical device and that used in a manufacturing process or a manufacturer’s quality system (or, for that matter, in any other software). An understanding of these considerations and, thus, the importance of performing structural testing, are discussed below.

Definition of Software Structural Testing

   Software structural testing is meant to challenge the decisions made by the program with test cases based on the structure and logic of the design and source code. Complete structural testing exercises the program’s data structures (such as configuration tables) and its control and procedural logic at the test levels discussed below.
   Structural testing should be done at the unit, integration, and system levels of testing.2 (As used herein, a “unit” is the smallest separately compilable – or equivalent – element of code, such as a procedure, subroutine, class, method, or database table.) Structural testing assures the program’s statements and decisions are fully exercised by code execution. For example, it confirms that program loop constructs behave as expected at their data boundaries. For configurable software, the integrity of the data in configuration tables are evaluated for their impact on program behavior. At the unit level, structural testing also includes the identification of “dead code,” which is code that cannot be reached for execution by any code pathway.
   Integration structural testing should be performed after all verification testing of the units involved and before system-level structural testing. Figure 1 illustrates the general relationship between software verification and validation. Software verification confirms that the output of each phase of software development is true to (i.e., is consistent with) the inputs to that phase. Performance qualification testing confirms the final software product, running in its intended hardware and environment, is consistent with the intended product as defined in the product specifications and software requirements.

 Unit-Level Structural Testing

   Figure 2 illustrates a typical configuration for a structural unit test. The unit is compiled and linked with a driver and stubs, as needed. The driver is a substitute for any actual unit that will eventually call the unit-under-test, and if the driver passes data to the unit-under-test, it is set up to pass test case variable values such as maximum, minimum, and other nominal and “stress-test” values. The stubs are substitutes for any units called by the unit-under-test. As with the driver, if the stubs return data to the unit-under-test, they also pass “stress test” and nominal data values, as appropriate. The interface of the drivers and stubs, including their names, are the same as the true units’ interfaces, allowing the set of units to be linked without altering the unit-under-test.

   Unit-level structural tests can be conducted on the actual target hardware, on an emulator or simulator of the actual hardware, or, if the situation requires, on a totally different processor. The latter case may occur, e.g., if the actual hardware has no provision for determining the results of a unit’s tests, but where the code is written in a higher order language. Thus, the higher order source code (such as C or C++) can be compiled and linked to run on another computer that supports reading the test results where the target computer (for example, an embedded microprocessor) could not support the tests.
   Structural testing (a.k.a. white-box testing) is performed with the item-under-test, in this case the unit, being viewed internally for purposes of determining how the item should behave – for example, in determining all possible code branches. The primary purpose of unit-level structural testing is to verify the code complies with the design, including logic, algorithms’ correctness, and accuracy (versus parallel code or hand calculations), and the correctness of the data’s engineering units. This requires, for each unit, complete branch tests, complete code tests (including verifying there is no dead code), and stress tests (to detect, e.g., overflow and underflow conditions as well as nominal and maximum loop-control data values). The detailed design is used to develop the acceptance criteria.

The Environment for Both Integration-Level and System-Level Structural Tests

   It is best to set up the integration and system structural tests using the actual hardware and environment to the extent practical. There are several reasons for this, but the two most significant are (a) the software may have subtle conditions, both good and bad, that will only show up when running on the actual hardware, and (b) the final computerized system, including the intended hardware and software, must be qualified running on that hardware, and the structural tests should advance the software development towards that end. However, there are also good and sufficient reasons to perform structural tests partially or wholly in a simulated environment. In considering establishing simulation capabilities, the two most common configurations are to either emulate the computer and simulate the environment (used most often when the actual computer is an embedded microprocessor and it is difficult to stimulate known inputs and/or read the outputs of a test) or to simulate both the computer and the environment (used, e.g., if an emulator of the target computer is not available). The principal advantages, then, in using a simulation of the environment and, at times, the computer include the following: (a) the ability to set up absolutely known input values, such that the results can be predetermined to establish the acceptance criteria of each test; (b) a simulator makes it easy to establish inputs that are over, under, and at the exact limits of critical data values; (c) it is easy to set up illegal inputs to test all error and failure conditions; and, finally, (d) the results of each test can be readily seen.

Integration-Level Structural Testing

   Integration structural testing combines functionally cohesive units of verified code (which includes unit-level structurally tested code) by compiling and/or assembling and linking the code, along with any drivers and stubs needed. The structure is then loaded into the actual or simulated environment for execution. This allows the tester to focus on that one functional package to confirm its correct operation, including all internal and external interfaces. Following completion of each functional package’s test, the next functional package may be either separately tested or added to (i.e., linked with) the previously tested package(s). Regression testing (i.e., running a selected subset of previous, successfully run test cases) must be performed on the previously tested packages to confirm they are not adversely affected by the newly introduced functional package.
   Figure 3 illustrates a building-block, or incremental, approach to structural testing. While the figure’s illustration is related to a structured design, the same approach is used for all other design methods, including flowchart and object-oriented design. In Figure 3, the first function needs two stubs, “Get Formatted ABC” and “Output Y to Device X,” (where a stub is a simple software dummy needed to link successfully, but is not part of the software being tested). The stubs in the figure have no calls to additional units. The second and third functions require no driver (because they use the actual “Compute Y”) and they require no stubs (because they use the actual “Output Error Messages” unit).

The Incremental Approach

   The incremental approach to integration-level structural tests is the best for software developers (as opposed to third party, validation testers – see below), especially if the program is large or complex. In this approach, selected small, functionally cohesive portions of the software are compiled, linked, and tested. This approach is used regardless of the software life cycle development method being employed, including any of the following three methods. In the waterfall method, all of the requirements are developed, then the design is completed, and, finally, selected “threads” are coded and structurally tested. In the spiral method, a major element of the software system is discussed and then the requirements, design, and code are developed, and the element’s structural test is performed prior to going on to discuss and develop the next major element. Finally, in the incremental software development method, all of the specifications and requirements may be developed, but the design and implementation are developed one function at a time.
   In any case, if the operating system was uniquely developed for the system-under-test, that operating system should be structurally tested first. This portion of the code itself should be broken down into functionally cohesive packages if it is large and/or complex; otherwise, it can be structurally tested as an entity. The second portion of the code to be tested is normally the unique input/output section. If there are diverse input/output devices, these may be structurally tested separately. But it is often best to select a thread that includes both the ability to input data and to see the resulting output for each structural test. The third step is to select and structurally test a functionally cohesive portion of the application and the utilities needed to support that application. (Remember: The units selected for integration structural testing, and this is especially true for the utilities, should be verification tested first – which includes unit testing – prior to any integration-level structural testing.) Then, select the next functionally cohesive portion of the application software and associated application utilities for the next structural test, and so on. All previously tested functions should be regression tested, as appropriate...