5 min read
In this blog post, I take a look at modern IT governance by applying the classic “Three Ways” of DevOps principles originally introduced by Gene Kim in his seminal 2012 article.
“We assert that the Three Ways describe the values and philosophies that frame the processes, procedures, practices of DevOps, as well as the prescriptive steps.”
Here’s a quick reminder of the three ways set out by Gene:
- The First Way: Flow/Systems Thinking
- The Second Way: Amplify Feedback Loops
- The Third Way: Culture of Continual Experimentation and Learning
For Gene, all DevOps patterns can be derived from these three principles. I’m going to extend his idea to argue that we can use these same principles to map out the patterns for modern IT governance that aligns with our DevOps.
Change Management hasn’t scaled with our DevOps automation
Since 2012 the IT world has undergone a quiet (and largely peaceful 😅) revolution. Most companies have embraced the cloud, value stream and product thinking. Even highly regulated companies like banks and healthcare organizations now talk of empowering development teams with continuous delivery and DevOps tooling.
However, as development and operations started speeding up, a new bottleneck emerged: governance.
Traditionally, change management gates were introduced as the governance enforcement point in the handoff from dev to ops.
- Release documentation
- Change advisory boards
- External approvals
The thing is, there is now a mountain of research that proves this approach does not work. It makes you slower and riskier. Even regulators think this is a mistake.
Manual ways of enforcing governance put developers in a “build trap” where they can’t deliver their changes. It causes batching, slow feedback loops, and ultimately riskier changes. It also reinforces the silos between development and operations, and the gatekeepers in this mode are typically 3rd parties who have little ownership of the risks involved.
The first way: Flow/Systems Thinking
The way out of the build trap is to make every change compliant by default.
The key idea is to replace the change management gate with automation that enforces compliance for every change. Where a working definition of compliance is:
1 Have a process that manages risk.
- You need to write this down. It is what the auditor will ultimately test you against
- Map risks to controls. For example, say a given risk is that an insider in your company delivers code or infrastructure changes that are not in the interest of the company. This is called insider threat. Typical controls for insider threat are enforced code review and deployment approvals.
- Check out our free SDLC template if you’d like a starting point)
2 Actually follow the process! <- this is compliance 🫣
- Good news! DevOps tooling, developer platforms, golden paths, etc. are a great way to achieve this!
3 Have the proof you followed the process (for auditors, regulators, risk, security)
- Have the pipelines and automation produce the audit trail for every control
- This System of Record is a ideally in a centralized database (we have some opinions on this 🙂, but the important point is to make sure you have a queryable data source for this)
Let’s say your stated definition of compliance is that a change must have a code review, binary provenance, managed dependencies, security scans, and a deployment approval. Instead of using retrospective, manual processes to establish that these things happened, simply enforce them in the pipelines and automate the recording of the facts as they happen. Then voila! …every change is compliant by default, and able to be shipped. This is the first way.
The Second Way: Amplify Governance Feedback Loops
The first way gives you a secure golden path to production. But compliance isn’t that simple. You also need to verify that only compliant workloads actually run. When the auditor asks “how can you be sure nobody has gone around the golden path to deploy software?” you need to be able to prove the negative. This is the stream/reservoir problem.
You can solve this by creating a provable forensic record of what’s running in your production environment, and connecting it to the proof that workloads are compliant and approved to run.
This is a governance feedback loop that provides a real-time audit of running workloads.
With this real time monitoring of production against the compliance system-of-record you can automatically detect:
- Undocumented or non-compliant workloads
- Drift from desired state
- Breaches
It is a “shift-left” approach for audit, moving it to a real time and continuous process.
The Third Way: Culture of Continual Experimentation and Learning
With the first and second ways, now you have put repetitive compliance work on auto-pilot. But we’re not done yet! Let’s slightly reframe Gene’s third way in the context of governance:
“The Third Way is about creating a culture that fosters two things: continual experimentation, taking identifying risks and learning from failure; and understanding that repetition and practice is the prerequisite to mastery.”
The essence of the third way is exploration. How do we find new and better ways to test the effectiveness of controls (think chaos engineering for compliance) and how could we introduce faults and game-days for compliance.
This process encourages us to find new ways to collaborate across disciplines to identify risks and introduce new automated controls, ultimately improving our resilience.
Conclusion
There is an old saying in systems thinking:
“95% of performance is governed by the system”
Which is interesting in its own right, but in the context of governance a new corollary emerges:
->“95% of governance is performed by the system”
The solution to the governance problem in DevOps has always been there for us to discover: systems thinking and three ways!
Further reading:
