GreenDart Inc. Team Blog

To maximize the V&V initiative impact the V&V team benefits from a process that focuses precious resources towards the most important challenges of the system development effort. The Criticality Analysis and Risk Assessment (CARA) methodology is a useful approach to identify those challenges, including for enterprise V&V assessments of program priorities and execution budget. Specifically, the CARA method is used to map key enterprise challenges and risk to targeted V&V activities. Once the candidate and prioritized list of verification activities are identified, cost estimates are developed for each verification activity. Budget considerations define the end of the V&V tasking list. The results are provided to the Management team for final approval and coordination with the customer, development community, stakeholders, and external users.

The key to driving V&V activities on risk and program priorities is the application of the CARA process. Specifically, for targeted priority areas, the criticality of each function/requirement is defined by assessing performance, operations, cost, and schedule implications for each function. Those assessments are then rated from Catastrophic to Low. All of these assessments are quantifiable and repeatable, ensuring consistent ratings/rankings across the enterprise.

In parallel the risk of each function/requirement is defined for complexity of the design, maturity of the applied technology, stability of the underlying requirements, testability of the function, and the developer’s experience in this particular functional area. As with the criticality assessment, these risk assessments are very quantifiable and repeatable, ensuring consistent ratings/rankings across the enterprise.

Finally, we multiply the criticality and risk ratings to develop the CARA score. The CARA score defines both the priority of the function/requirements and the associated verification tasking required to successfully assess the requirements verification activities of the selected function/requirements. Higher CARA scores drive more robust assessment activities.

GreenDart has specific lists of tasking for each of the Low, Medium, and High CARA scores that cross the development lifecycle (requirement development through sustainment). We are prepared to apply the CARA process as part of our V&V offering. How can we help enable your V&V program success within your limited resources?

The transition from system development into operations, and the subsequent system maintenance and sustainment activities, present organizations with many challenges. V&V addresses these challenges and reduces the potential that anomalies will arise.

During the transition phase the V&V team reviews the transition and activation products, ensuring they meet system needs.  Developers are responsible for developing the transition and activation products needed for deploying the system.  The products include the preparation of the system for each user site, manuals for each user site, source files, system descriptions for the maintenance site, as-built system data for the maintenance site, and transition plans for transitioning to the maintenance site.

During sustainment, both the cost and risk of defect correction and system upgrades is high.  Drivers for such modifications include hardware and/or operating system changes, user errors occurring during operations, other software enhancements, and problems with or replacement of components developed outside of the development plan.

During this phase, V&V performs an assessment of the developer products, which supports the transition of the modifications into the operational phase.  Proposed modifications are analyzed to ensure that the changes to be made are correct and do not interfere with other parts of the system or jeopardize the mission.  The V&V staff independently assesses such potential modifications and are tasked with identifying risks of introducing new errors into the system and/or other options to achieve the needed changes.  These V&V assessments avoid costly rework.

GreenDart has experience performing all levels of Transition, Sustainment, and Maintenance V&V. How can we help you?

Most system test efforts focus on achieving stated program requirements. However, frequently, critical systems require a level of capability validation that goes beyond the developer product assessment activities.  For these critical systems, the V&V effort expands to include the development and application of an independent test environment, used to isolate and separately execute the targeted critical software.  The independent team is completely segregated from the developer test efforts and must not be compromised by any of the following nominal developer/IV&V -engagement tendencies:

• Assessments of developer test products and/or test strategies
• Use of any developer test hardware and/or software simulators

A major focus of independent testing is to prove the robustness of the software.  Initially, the V&V team generates a test plan based on potential risks in the software from the V&V risk and hazards analysis.  The plan identifies the effort required to support an independent testing activity.  The V&V team develops test cases that stress software requirements at their limits (i.e., throughput, boundary conditions, etc.) to verify that such cases are dynamically handled properly by the deliverable software.  After the plan and test cases are approved, the V&V team executes the testing and analyzes the results.  The final report includes evidence of the software performance during this testing exposing any defects, risks, or limitations.

Because of the isolated nature of independent testing, unique test environments are required to achieve the assessment objectives. Therefore, the independent test team comprises of both test and software team members.

GreenDart has successful experience developing and executing independent test efforts. How can we help you?

System development typically concludes with a series of test events that culminate in the delivery of the desired capabilities. The test events start with the system element integration as pieces of the end-system are brought together. The corresponding test activity is called Integration Test. Once all the system elements are integrated, System Test is performed on the complete system. Finally, once all the System Test objectives are met then System Acceptance Test is performed to assure the final deliverable capability meets all customer requirements.

Correspondingly, there are Verification and Validation (V&V) activities that assure successful Integration, System, and Acceptance testing. These V&V activities follow a fairly similar activity of assessing test plans, procedures, and, once completed, test results against planned outcomes. This step is critical to assure the quality, completeness and accuracy of the developer’s test efforts.

INTEGRATION AND SYSTEM TEST V&V
During this phase, the developer integrates and tests the integrated system components which could include hardware components and reuse code.  Integration testing confirms functional requirements compliance after the software sub-elements are integrated and directs attention to internal software interfaces and external hardware and operator interfaces.  Subsequent system testing validates the entire program against system requirements and performance objectives.

V&V assesses the sufficiency and completeness of the developer's integration and system test program, identifying weaknesses, and focusing needed V&V test assessments accordingly.  Previous validation activities have prepared this phase for developer software integration and incremental system testing.

V&V also assesses changes to the following developer products:  system development and design documents, interface control documents, source code, software executables, test plans and procedures, test results and other products as described in the developer's Test Plan.  As the developer's integration and system testing proceed, the V&V team carefully monitors critical testing activities and validates any changes that are made to the code or documentation.

So, we’ve completed the first two critical steps of the overall V&V effort, Requirements V&V and Design V&V. Through those two sequential V&V efforts we have identified errors early, which has saved on the total cost of developing the system (early error discoveries are less expensive to fix than later error detections). But, by far, the Code V&V portion of the V&V effort offers significant variations in V&V investment that deliver different ROI results. In this blog we explore the various code V&V strategies, their impact on program cost, and their impact on the target program’s development cost and schedule.

Code V&V efforts range from an oversight activity up to the actual execution of developed code within a V&V simulated environment.  The former code V&V strategy is relatively inexpensive and is consistent with the effort put into requirements and design V&V efforts. The latter is significantly more expensive, as will be seen below, and is performed for those systems where critical mission performance challenges are identified early without impact to the actual target system. A good example of target systems for detailed code V&V are mission critical systems that, if they fail, can cause significant loss of life. So, the investment in detailed code V&V creates a Return On Investment (ROI) that is measured in both dollars and safety/performance considerations.

OVERSIGHT CODE V&V

For oversight code V&V, the V&V engineer reviews code artifacts (flat files, tool-enhanced code representations, etc.) and performs trace analysis to design specifications to ensure what was planned to be developed is actually what is developed. This top-level assessment is critical to maintain traceability, assuring delivered code addresses all of the program requirements. However, this effort does not address the actual performance attributes of the developed code.

The next stage of overall V&V effort is Design V&V. Once the initial V&V investment is made during the requirements development phase, that investment pays off during the design phase V&V effort. Specifically, the foundational V&V priorities identified in the requirements phase become design V&V decision drivers, leading to desired assessments of critical system capabilities.

System design transforms the developer's system requirements into discrete items that implement those requirements.  The outputs of the system design phase are used to support the detailed design activity.  Detailed design lays out the specific system units comprising each item including a unit's interfaces with other system units and each unit's internal structure, processes and data needs.

V&V is conducted during system design and detailed design phases.  Assessed developer products include:  System Requirements Specification, Interface Requirements Specification, System Design Document, Interface Design Document, System Test Plan, System Test Description (STD), and System Architecture Description (SAD).  V&V assesses these developer artifacts and those presented during program reviews.

Design phase V&V solidifies the preliminary and final design specifications, and performs risk and security evaluations.  Validating the design artifacts ensures that the coding and the unit testing which follow are secured by a sound design traceable to the driving requirements.

Developer errors found in the design phase of the effort by V&V have a significant impact on total system development costs. So, design V&V is a critical next step in the overall V&V process and has the potential to deliver excellent return on the V&V investment.

Requirements Verification & Validation

During the requirements phase the developer generates program requirements needed to support development. The developer also traces these requirements to their parent Customer system requirements. Verification & Validation assesses developer artifacts generated during this phase. The V&V team may analyze the Customer/developer project documents, requirements, traceability data, the Software/System Requirements Specifications, the Interface Requirements Specifications, and Interface Control Documents for interfacing to external and internal systems.

V&V of the developer's requirements ensures that the design phase is working from a mature base of requirements and satisfies the users' needs. Furthermore, these tasks also ensure that the requirements are testable and traceable.

If developers define their test approach, testing plans and test data during the requirements phase, then the developer's test planning effort is also reviewed by the IV&V team during the requirements phase. 

Interestingly, developer errors found in this phase of the effort by V&V have the greatest positive impact on total system development costs. So, requirements V&V is a significantly critical step in the overall V&V process and has the potential to deliver the best return on the V&V investment.

INTRO

Verification and Validation (V&V) drives the developmental process for introducing the latest product. GreenDart understands efficiently applying V&V to assess the correct requirements, design, code, and test for an upcoming product is critical to successfully deploying new capabilities. Ensuring the product’s success is a factor of quality and capability assessments. Successful V&V employs V&V tools to optimize the V&V effort. Assessment tools are utilized throughout all development stages and are capable of supporting V&V efforts by providing co-verification, sim-modeling, software debugging, testing, code analysis, and test-verification. 

APPROACH

Applying our V&V assessment tools library allows GreenDart to be versatile in its approach to reaching an optimal solution. GreenDart selects the appropriate V&V assessment tools for the different phases of development based on the unique assessment requirements of the program. In a system’s development cycle, its design can be divided into different stages of experimentation and refinement. For earlier stages of development, co-verification and sim-modeling tools and practices allows GreenDart to perform early design assessment and analyze failure effects on virtual prototypes. Subsequently, proper code analysis and software debugging tools are applied to provide diagnostics and generate actionable results for the latter phases of the development cycle. This enables GreenDart to identify system issues and to enforce compliance with industry standards. Furthermore, the importance of valid testing and test-verification is applied, which includes AI technology and unique visual test modeling mechanisms for faster test-creation and execution, while cost-efficiently analyzing any application or data.

BENEFITS

 Reducing Software Cost Through Verification and Validation

Many programs risk cost growth and schedule delays because of software development issues. Collecting, tracking and containing software issues/defects in the development phase where they occur is an excellent cost-control management practice. Programs surveyed indicated that an average of 31 percent of defects corrected were detected after the development phase in which they were inserted. Capturing software defects in phase is critical because detecting defects out of phase results in expensive program rework. The best mechanism for capturing defects in-phase, and thereby reducing software development cost is Verification and Validation (V&V). 

 V&V is an engineering technical discipline using proven techniques to detect defects in software and/or hardware and avoid risks in developmental programs. Independent V&V (IV&V) is V&V performed by an independent authority, which is managerially,  technically, and financially independent from the product developer. IV&V is the unbiased process of assessing system development performance against key technical, programmatic, and financial objectives ensuring the user needs are satisfied. Proper and most efficient V&V starts on or before the beginning of the product developer’s requirements phase. The purpose of  V&V is to help a product developer build quality into the product throughout its life-cycle phases: Requirements, Design, Development, Test, and post-delivery Sustainment. Additionally, a good V&V program benefits from the use of V&V Tools and the application of advanced V&V processes such as GreenDart’s Criticality Analysis and Risk Assessment (CARA). Finally, the V&V agent may also be responsible for Independent Testing of the pre-deployed system. We cover all these V&V efforts in our complete V&V discussion.

V&V includes V&V planning and re-planning as system development evolves. It includes the review of developers’ requirements definition, design analysis, unit code and testing, test planning and test results assessment, and conduct of independent V&V testing where appropriate.

IV&V plans are based on government direction and funding, the developer’s System Development Plan, the developers’ processes, and other development planning data.