Today, waterfall and agile are the two predominantly used project management approaches in IT. But with the world of software development moving away from waterfall-type approaches to more agile approaches like scrum, kanban, and SAFe, IT infrastructure teams face a seemingly difficult challenge of how to stay relevant in this new world. The answer is simple — do what’s right to provide the best value to your customer. These lean approaches can unlock efficiencies and boost your infrastructure capabilities.

So, what tools can infrastructure managers and teams use to determine the best approach for their projects? And more specifically, how can infrastructure providers apply agile approaches to infrastructure services and projects?

Below, we explain how agile can benefit your infrastructure development and offer a practical application of transitioning a manual infrastructure process to a streamlined, automated process through agile methodology.

When does waterfall work?

Waterfall is an excellent option when 1) project ‘knowns’ are defined as requirements; 2) are tied to the actual work to be performed; and 3) align to a scheduled release date. Traditionally, this approach worked well for infrastructure projects, because they started with a detailed list of requirements that pre-defined the service capabilities/components and aligned to operational needs.

For example: An infrastructure project team is responsible for deploying a new server farm. Due to the nature of the work, the server team:

• Will have a single release
• Needs a complete list of requirements that likely tie in with operational processes (like change and configuration management)
• Must fully understand each requirement, its current and future activities, and its downstream impacts prior to design and development (the server farm needs to connect to the existing network, connect to the existing storage platform, draw power for a limited source, and store its log data to a centralized log aggregation)
• Has project dependencies that cannot be broken (like integrations with existing, expensive infrastructure)

In this scenario, waterfall works well, because the server team has a full understanding of requirements and objectives at project inception.

But what happens when there are unknowns in the infrastructure project? This is common in greenfield projects, in infrastructure services that are truly service- or product-oriented. What if there are limited dependencies in a brownfield project, or even dependencies that can be ignored? Can the team benefit from agile? The answer—yes!

Is waterfall still right for me?

The Cynefin framework (shown below) is an excellent tool for selecting your development approach by quantifying your project’s known and unknown activities/outcomes into obvious, complicated, complex, chaotic, or disorder domains.

• If your project is complex, complicated, or chaotic, agile is likely the best approach
• If your project is obvious, waterfall is an appropriate option
• If your project falls in the center (disorder), objectives or desired outcomes should be clarified by breaking projects down into smaller components and reclassifying them within the domains

How can agile help my infrastructure?

Agile offers the flexibility to produce customer value quickly while 1) beginning build activities with what you know at the time; 2) iteratively refining requirements based on actual user and internal feedback; and 3) aligning decisions to the overall product objectives. This approach benefits the user and team through:

• Early delivery of the service through minimum viable products (MVPs)
• Iterative improvements to the product and features through user feedback
• Rapid decision-making by the local team doing the work
• Quick recovery from bad decisions using fail-fast methods
• Accurate documentation of the product with each iteration (documenting the build instead of building to the document)

Infrastructure projects tend to be repeated over time—take technical refreshes of data center equipment, deployment of new servers or storage, or addition of network capacity. Because some of the activities, or even desired outcomes, might not be known the first time you do one of these projects (chaotic domain), an agile approach can help you deliver initial needs while fine-tuning your delivery. As you continue to repeat the projects and optimize your people, processes, and tools, you can move through the Cynefin spectrum (chaotic to complex to complicated domain) until your functions are simple (obvious domain), anticipated, and hopefully highly automated.

Does your project already fall into the obvious domain? No problem! Even tasks that are well-suited to traditional waterfall methodologies can benefit from agile, especially when it comes to boosting efficiency through automation.

An automated agile infrastructure in practice

Increasingly, as automation becomes a priority and cloud computing options become more prevalent, projects that were previously “known” and addressed through waterfall are turning to agile. The wide array of cloud computing service choices (e.g., containers vs. servers, APIs vs. web services, block vs. blob storage), and the shift from manual to automated processes introduce new unknowns, even for established, smoothly executed projects.

Take, for instance, automating an operating system’s gold images through a continuous integration/continuous deployment (CI/CD) pipeline.

• Background: Traditionally, this is a monolithic system that requires hundreds of man hours each month to manually build and validate. Despite conducting the same activities, each subsequent release follows the same manual build and validation process, increasing cost and the risk of human error.
• Cynefin classification: The team’s current understanding of the process falls into the obvious domain (waterfall-appropriate) but needing to automate the gold image “product” through a CI/CD pipeline reclassifies it as complex or complicated (Agile-oriented), depending on the existing DevOps maturity.
• How agile can help: Even though the objectives remain the same, an agile approach (shown below) allows the team to 1) break the monolithic service and process into unique constituent parts; 2) treat each part as a microservice within a modernized IT system; and 3) build each part in increments.

In this scenario, iterative agile releases allow the team to incrementally automate the gold image process as a group of microservices, building products faster while reducing cost and improving product quality.

• Release 1: As an MVP, patches are added to the base image through the CI/CD pipeline, while agents are added manually afterward
• Release 2: Automate security tool agents
• Release 3: Automate monitoring agents
• Release 4: Automate security configurations
• Release 5: Automate validation testing

After these releases, the gold image is fully automated as a group of microservices, with each agent/configuration becoming a “microservice” integrated into the base operating system. When one agent is upgraded to the next version (a change in requirements), only the “micro-service” is impacted.

Realized benefits:

• Significantly reduced cost from eliminated manual hours
• Users receive a consistent image that passed automated validation/integration testing
• Users receive a documented manifest (e.g., release notes) detailing changes from the previous “product” release
• Enables a highly governed release and roll-back process across your IT stacks
• Reduced mean time to recovery (MTTR)

Key takeaways

Modernizing infrastructures through agile provides more responsive and higher quality products for your customer. As you begin to plan or implement your agile methodology, remember to:

1. Define and understand your project’s objectives (e.g., building a gold image is a different objective than automating a gold image) and needed customer value
2. Use the Cynefin framework to understand and validate the known and unknown factors making your project disordered, chaotic, complex, complicated, or obvious. Your project is likely more complicated than you realize—expect to discover additional unknown factors along the way!
3. Start in small increments to provide immediate value to the consumer of the product through prioritized features, and keep them informed on what’s coming next
4. Use agile principles to iterate, retrospect, and constantly improve the product