In my blog Keeping Pace with Human Spaceflight: An Agile Approach to Independent Verification and Validation, I explored how Agile independent verification and validation (IV&V) emerged as a means to keep pace with the development of complex software for NASA’s Orion spacecraft. By applying concepts from Agile software development to the realm of evaluation, we are seeing a lot of success in quickly assessing whether things do what they are supposed to do and don’t do what they are not supposed to do.
Agile IV&V has been received very positively, earning praise from the NASA IV&V facility's director and the Orion software's architect. We have assisted NASA in developing briefs for applying this concept to other IV&V work.
In the best traditions of scientific collaboration, we offer some summary bullets on our findings:
1. Keep the main thing the main thing. Assurance is our primary product, in addition to issues and risks. With complexity comes distraction. Keep focused.
2. Capability-based assurance focuses on software in the context of mission capabilities, which requires system understanding captured in the IV&V Technical Reference while continuing to apply the IV&V “three questions”:
- Does the system and its software do what it is supposed to do?
- Does the system and its software not do what it is not supposed to do?
- Does the system and its software behave appropriately under adverse conditions?
3. Risk-targeted IV&V, also known as “Follow the Risk,” considers both functional and non-functional risk categories across the entire Orion risk landscape and addresses both mitigated risk and residual risk.
4. Adaptive IV&V incorporates critical thinking about what makes sense when performing analysis and what is the right thing to do. This includes leveraging existing assets such as the IV&V Technical Framework and the IV&V Catalog of Methods, but they do not constrain it.
5. Integration assurance integrates appropriate behaviors—especially emergent behaviors—and behaviors involving interfaces within and between software and systems.
By applying Agile concepts to the realm of evaluation, we can quickly assess critical systems on which human life depends.