GreenDart Inc. Team Blog

The key test optimization opportunity of an effective T&E effort is the design and execution of Design of Experiments (DOE). DOE is a systematic method to determine the relation between inputs or factors affecting a process or system and the output of that system. The system under test may be a process, a machine, a natural biological system or many other dynamic entities. This discussion concerns use of DOE for testing a software intensive system (a standalone program, or integrated hardware and software).

Test planners have a range of software test strategies and techniques to choose from in developing a detailed test plan. The choices made will depend on the integration level (i.e., unit test to system of systems T&E) of the target test article, as well as the specified and generated test requirements. Typically, the complete test plan will involve a combination of these techniques. Most of them are commonly known, but applying DOE for testing software intensive systems may not be as familiar. In this context, the design in “DOE” is a devised collection of test cases (experimental runs) selected to efficiently answer one or more questions about the system under test.  This test case collection may comprise a complete software test plan, or a component of that plan.

DOE can save significant test time within the overall  DT&E and/or OT&E efforts. In one particular instance GreenDart achieved 66% DT&E schedule savings through the successful application of DOE.

Agile system development has emerged as an alternative to the long-standing “waterfall” software development process. Agile development involves the rapid development of incremental system capabilities. These incremental development activities are called “sprints”. At the start the program selects requirements from the overall system requirements specification, builds user stories around those requirements, and allocates those user stories to specific sprint events. During each sprint event the developers go through a mini-waterfall effort of requirements, design, code, and test of a very small segment of the overall system. Once the sprint is complete the resultant product is typically integrated into the evolving overall system. Any unfulfilled sprint requirements go into a requirements holding ledger called the Product Backlog (PBK) for reassignment to future sprints. Sprint re-planning occurs, as needed, based on the accomplishments of previous sprints.

Testing within each sprint roughly resembles a very short waterfall testing effort, as described in our T&E description earlier. However, for this discussion, we focus on the unique Agile T&E activities that occur outside of each sprint. These activities include requirements verification, trace, and test results assessments.

T&E validates the user stories and associated critical technical parameters against top level requirements, identifying any issues early in the sprint development cycles. As sprints are executed and various levels of story “completion” are achieved, the PBK is updated to re-capture and re-plan those requirements that were not completed during the sprint or that underwent vital user-driven updates. Perturbations to delivered incremental capabilities and changes to go-forward strategies are quite common. These changes are captured by a dynamic T&E planning effort. Finally, because the programs are typically on a rapid 2-week sprint cadence, the T&E engagement and reporting cycles are quite short. This creates additional T&E/developer coordination opportunities, which improves T&E planning timeliness.

At the end of each sprint, the T&E team assess the achieved sprint requirements, now integrated into the evolving target system. Tests are performed for those specific requirements, and regression tests on existing capabilities, with each sprint integration. Successful T&E results drive capability acceptance while T&E failures drive PBK updates and future sprint re-planning efforts.