Assessing Risk in Complex Software Development
Here are 8 tips as part of an Agile independent verification and validation approach
12-17-2019
SAIC
SOFTWARE
In my blog Keeping Pace with Human Spaceflight: An Agile Approach to Independent Verification and Validation, I explored how Agile independent verification and validation (IV&V) emerged as a means to keep pace with the development of complex software for NASA’s Orion spacecraft. By applying concepts from Agile software development to the realm of evaluation, we are seeing a lot of success in quickly assessing whether things do what they are supposed to do and don’t do what they are not supposed to do.
Agile IV&V has been received very positively, earning praise from the NASA IV&V facility's director and the Orion software's architect. We have assisted NASA in developing briefs for applying this concept to other IV&V work.
In the best traditions of scientific collaboration, we offer some summary bullets on our findings:
1. Keep the main thing the main thing. Assurance is our primary product, in addition to issues and risks. With complexity comes distraction. Keep focused.
2. Capability-based assurance focuses on software in the context of mission capabilities, which requires system understanding captured in the IV&V Technical Reference while continuing to apply the IV&V “three questions”:
- Does the system and its software do what it is supposed to do?
- Does the system and its software not do what it is not supposed to do?
- Does the system and its software behave appropriately under adverse conditions?
3. Risk-targeted IV&V, also known as “Follow the Risk,” considers both functional and non-functional risk categories across the entire Orion risk landscape and addresses both mitigated risk and residual risk.
4. Adaptive IV&V incorporates critical thinking about what makes sense when performing analysis and what is the right thing to do. This includes leveraging existing assets such as the IV&V Technical Framework and the IV&V Catalog of Methods, but they do not constrain it.
5. Integration assurance integrates appropriate behaviors—especially emergent behaviors—and behaviors involving interfaces within and between software and systems.
6. Cybersecurity focuses on the threats and vulnerabilities relevant to software as the anchor or starting point for analysis, and it evaluates other system components as necessary. This includes appropriately handling adverse cyber threats via preventative or adaptive behaviors.
7. Dynamic analysis, including independent test, chooses an appropriate dynamic analysis method when that is the most efficient or practical means of generating the needed assurance evidence.
8. Agile principles are tailored for the Orion IV&V team's use by adopting useful parts of Agile while ignoring the non-useful parts. A few examples include:
- Fast, integrated learning cycles where the IV&V team learns quickly from mistakes and failures as well as successes. It applies continual planning and maintenance cycles for closed-loop feedback.
- Retrospectives, which identify what is working well and what is not working well, in order to spread successes to other teams and identify what improvements need to be made.
- Small, self-organizing teams that are empowered to innovate and succeed or fail quickly within “safe” boundaries without negative consequences, to make their work more effective and efficient.
- Daily 15-minute standups that briefly cover what each analyst has accomplished since the last standup, what each analyst plans to accomplish by the next standup, and what, if any, obstacles they need help overcoming.
NASA is often seen on the leading edge of technology and computing, but by applying Agile concepts, it is also on the leading edge of IV&V methodology. And when we consider what’s at stake—human life and multibillion-dollar systems—IV&V is a big deal. We can’t wait to see how these concepts continue to mature and how other engineering programs apply them.
