GreenDart Inc. Team Blog

GreenDart Official Press Release: GreenDart-Awarded-DHS-CBP-IVV-Subcontract

GreenDart has been a contributor to the IEEE-1012 Verification and Validation standard since 2012. For the past years we have strongly promoted the addition of an Agile V&V annex to that standard. Good news – if all goes well the next release of the standard will have an Agile V&V annex. GreenDart, along with a select group of IEEE-1012 members, is authoring that annex. The following figure contrasts traditional Waterfall and Agile system development methodologies, and is expected to be included in the upcoming IEEE-1012 release:

Agile V&V shares many strategies with the traditional V&V method. However, the development tempo of Agile, and the artifacts created during that process, require a different approach to achieve the desired V&V outcomes. As an example, the Sprint V&V process focuses on the pre-Sprint artifacts, to include analysis and assessments of the defined User Stories, the alignment with current development Product Backlog (PBL) and the metrics used to drive both the User Stories and PBL strategies.

GreenDart has significant experience performing Agile V&V for the most critical systems. We are experts in the Agile life cycle development strategy and have received awards for the planning, execution and reporting of complex Agile V&V efforts. How can we help you?

If you liked this post, please rate it below using the star icons!

 Author: Bedjanian

To maximize the V&V initiative impact the V&V team benefits from a process that focuses precious resources towards the most important challenges of the system development effort. The Criticality Analysis and Risk Assessment (CARA) methodology is a useful approach to identify those challenges, including for enterprise V&V assessments of program priorities and execution budget. Specifically, the CARA method is used to map key enterprise challenges and risk to targeted V&V activities. Once the candidate and prioritized list of verification activities are identified, cost estimates are developed for each verification activity. Budget considerations define the end of the V&V tasking list. The results are provided to the Management team for final approval and coordination with the customer, development community, stakeholders, and external users.

The key to driving V&V activities on risk and program priorities is the application of the CARA process. Specifically, for targeted priority areas, the criticality of each function/requirement is defined by assessing performance, operations, cost, and schedule implications for each function. Those assessments are then rated from Catastrophic to Low. All of these assessments are quantifiable and repeatable, ensuring consistent ratings/rankings across the enterprise.

In parallel the risk of each function/requirement is defined for complexity of the design, maturity of the applied technology, stability of the underlying requirements, testability of the function, and the developer’s experience in this particular functional area. As with the criticality assessment, these risk assessments are very quantifiable and repeatable, ensuring consistent ratings/rankings across the enterprise.

Finally, we multiply the criticality and risk ratings to develop the CARA score. The CARA score defines both the priority of the function/requirements and the associated verification tasking required to successfully assess the requirements verification activities of the selected function/requirements. Higher CARA scores drive more robust assessment activities.

GreenDart has specific lists of tasking for each of the Low, Medium, and High CARA scores that cross the development lifecycle (requirement development through sustainment). We are prepared to apply the CARA process as part of our V&V offering. How can we help enable your V&V program success within your limited resources?

The transition from system development into operations, and the subsequent system maintenance and sustainment activities, present organizations with many challenges. V&V addresses these challenges and reduces the potential that anomalies will arise.

During the transition phase the V&V team reviews the transition and activation products, ensuring they meet system needs.  Developers are responsible for developing the transition and activation products needed for deploying the system.  The products include the preparation of the system for each user site, manuals for each user site, source files, system descriptions for the maintenance site, as-built system data for the maintenance site, and transition plans for transitioning to the maintenance site.

During sustainment, both the cost and risk of defect correction and system upgrades is high.  Drivers for such modifications include hardware and/or operating system changes, user errors occurring during operations, other software enhancements, and problems with or replacement of components developed outside of the development plan.

During this phase, V&V performs an assessment of the developer products, which supports the transition of the modifications into the operational phase.  Proposed modifications are analyzed to ensure that the changes to be made are correct and do not interfere with other parts of the system or jeopardize the mission.  The V&V staff independently assesses such potential modifications and are tasked with identifying risks of introducing new errors into the system and/or other options to achieve the needed changes.  These V&V assessments avoid costly rework.

GreenDart has experience performing all levels of Transition, Sustainment, and Maintenance V&V. How can we help you?

Most system test efforts focus on achieving stated program requirements. However, frequently, critical systems require a level of capability validation that goes beyond the developer product assessment activities.  For these critical systems, the V&V effort expands to include the development and application of an independent test environment, used to isolate and separately execute the targeted critical software.  The independent team is completely segregated from the developer test efforts and must not be compromised by any of the following nominal developer/IV&V -engagement tendencies:

• Assessments of developer test products and/or test strategies
• Use of any developer test hardware and/or software simulators

A major focus of independent testing is to prove the robustness of the software.  Initially, the V&V team generates a test plan based on potential risks in the software from the V&V risk and hazards analysis.  The plan identifies the effort required to support an independent testing activity.  The V&V team develops test cases that stress software requirements at their limits (i.e., throughput, boundary conditions, etc.) to verify that such cases are dynamically handled properly by the deliverable software.  After the plan and test cases are approved, the V&V team executes the testing and analyzes the results.  The final report includes evidence of the software performance during this testing exposing any defects, risks, or limitations.

Because of the isolated nature of independent testing, unique test environments are required to achieve the assessment objectives. Therefore, the independent test team comprises of both test and software team members.

GreenDart has successful experience developing and executing independent test efforts. How can we help you?

System development typically concludes with a series of test events that culminate in the delivery of the desired capabilities. The test events start with the system element integration as pieces of the end-system are brought together. The corresponding test activity is called Integration Test. Once all the system elements are integrated, System Test is performed on the complete system. Finally, once all the System Test objectives are met then System Acceptance Test is performed to assure the final deliverable capability meets all customer requirements.

Correspondingly, there are Verification and Validation (V&V) activities that assure successful Integration, System, and Acceptance testing. These V&V activities follow a fairly similar activity of assessing test plans, procedures, and, once completed, test results against planned outcomes. This step is critical to assure the quality, completeness and accuracy of the developer’s test efforts.

INTEGRATION AND SYSTEM TEST V&V
During this phase, the developer integrates and tests the integrated system components which could include hardware components and reuse code.  Integration testing confirms functional requirements compliance after the software sub-elements are integrated and directs attention to internal software interfaces and external hardware and operator interfaces.  Subsequent system testing validates the entire program against system requirements and performance objectives.

V&V assesses the sufficiency and completeness of the developer's integration and system test program, identifying weaknesses, and focusing needed V&V test assessments accordingly.  Previous validation activities have prepared this phase for developer software integration and incremental system testing.

V&V also assesses changes to the following developer products:  system development and design documents, interface control documents, source code, software executables, test plans and procedures, test results and other products as described in the developer's Test Plan.  As the developer's integration and system testing proceed, the V&V team carefully monitors critical testing activities and validates any changes that are made to the code or documentation.

So, we’ve completed the first two critical steps of the overall V&V effort, Requirements V&V and Design V&V. Through those two sequential V&V efforts we have identified errors early, which has saved on the total cost of developing the system (early error discoveries are less expensive to fix than later error detections). But, by far, the Code V&V portion of the V&V effort offers significant variations in V&V investment that deliver different ROI results. In this blog we explore the various code V&V strategies, their impact on program cost, and their impact on the target program’s development cost and schedule.

Code V&V efforts range from an oversight activity up to the actual execution of developed code within a V&V simulated environment.  The former code V&V strategy is relatively inexpensive and is consistent with the effort put into requirements and design V&V efforts. The latter is significantly more expensive, as will be seen below, and is performed for those systems where critical mission performance challenges are identified early without impact to the actual target system. A good example of target systems for detailed code V&V are mission critical systems that, if they fail, can cause significant loss of life. So, the investment in detailed code V&V creates a Return On Investment (ROI) that is measured in both dollars and safety/performance considerations.

OVERSIGHT CODE V&V

For oversight code V&V, the V&V engineer reviews code artifacts (flat files, tool-enhanced code representations, etc.) and performs trace analysis to design specifications to ensure what was planned to be developed is actually what is developed. This top-level assessment is critical to maintain traceability, assuring delivered code addresses all of the program requirements. However, this effort does not address the actual performance attributes of the developed code.

A team lead by GreenDart has been awarded a $300M/7yr DHS Blanket Purchase Agreement contract. Under this contract GreenDart provides Test and Evaluation/Cybersecurity Resilience services across all DHS components including Customs and Border Protection, US Coast Guard, Transportation Security Agency, US Citizenship and Immigration Services, and Cybersecurity and Infrastructure Security Agency. Domains include Detection, Information Technology, Watercraft, Aviation, Vehicles, Unmanned Aerial Systems (UAS), and Counter-UAS Systems.

With this BPA prime contract, the GreenDart team can be expected to perform ITA services including Developmental and Operational Test and Evaluation (DT&E and OT&E) planning, execution, and reporting for DHS major acquisition programs. This also includes complex tests and assessments of various IT and non-IT system technologies into the DHS operational domain, and spans all DHS mission functional areas. GreenDart’s efforts also include cybersecurity and cyber resilience assessments. The technologies and systems that are approved through this BPA include Information Technology, Cyber Resilience, Technology Systems, Watercraft, Aviation, Unmanned Aerial Systems (UAS), Counter UAS, Vehicles, and Detection Systems. This DHS ITA BPA has a 7 year ordering period with a total estimated value of up to $300,000,000. Work is to be performed in the Washington, D.C. metro area as well as across the continental United States.

The ordering period for this contract is May 2021-May 2028. 

The next stage of overall V&V effort is Design V&V. Once the initial V&V investment is made during the requirements development phase, that investment pays off during the design phase V&V effort. Specifically, the foundational V&V priorities identified in the requirements phase become design V&V decision drivers, leading to desired assessments of critical system capabilities.

System design transforms the developer's system requirements into discrete items that implement those requirements.  The outputs of the system design phase are used to support the detailed design activity.  Detailed design lays out the specific system units comprising each item including a unit's interfaces with other system units and each unit's internal structure, processes and data needs.

V&V is conducted during system design and detailed design phases.  Assessed developer products include:  System Requirements Specification, Interface Requirements Specification, System Design Document, Interface Design Document, System Test Plan, System Test Description (STD), and System Architecture Description (SAD).  V&V assesses these developer artifacts and those presented during program reviews.

Design phase V&V solidifies the preliminary and final design specifications, and performs risk and security evaluations.  Validating the design artifacts ensures that the coding and the unit testing which follow are secured by a sound design traceable to the driving requirements.

Developer errors found in the design phase of the effort by V&V have a significant impact on total system development costs. So, design V&V is a critical next step in the overall V&V process and has the potential to deliver excellent return on the V&V investment.