Kosli devops change management secure change of custody

How to automate a secure chain of custody across your pipelines in 5 steps

Mike Long
Mike Long
Published June 7, 2021 in technology

Imagine you’re a Fintech CTO 🤓 with several teams and tens of microservices. Do you know what’s currently running in prod? How about yesterday? A week ago? Last month? And if you do know what’s in prod, do you also know how it got there? 🤔

Getting answers to these questions isn’t straightforward, but for teams in regulated industries it’s essential. You have to know what has changed, when it changed, and who changed it. You have to know what was running in prod at 2.30am last Thursday, or indeed at any other given time. ⏰

In this post we’ll show you how to answer these questions - and get rid of a bunch of paperwork - by automating a secure chain of custody across your pipelines. In only 5 steps. 👍

  1. Build a “black box recorder” for the pipelines
  2. Ensure binary provenance
  3. Automate risk controls
  4. Segregation of duties and approvals
  5. Environment change logs

Ready? Let’s dive in! 🐬

1. Build a “black box” recorder for the pipelines

The first step to compliance automation is to build a “black box”◼️ to automatically record every event in our software delivery process - from the initial git commit to the final push to prod.

I’m using the black box analogy because our software process needs to have an automated record of every change that is both infallible and indestructible. 💪

It should also be based on an append-only journal where every change is non-modifiable. Without this feature we will not be able to prove compliance in a way that will satisfy regulators.

Now we can eliminate a lot of the manual change documentation with automation. 🤖

2. Ensure binary provenance

Keep in mind our overall objective - we want to know what’s running in prod and how it got there. To be cryptographically sure of this we need to rewind 🔙 all the way back to the build process to establish binary provenance.

Binary provenance is a fancy term, but it boils down to two simple concepts:

  • How to identify software
  • How to connect this identity to its ingredients

For software identification, we really favour cryptographic fingerprints 🐾 (Kosli uses SHA256 sums for example). This is a unique identifier for any sequence of bytes.

Forget about file names, versions, strings, schema, metadata, or anything else that can be easily changed. You want the SHA. 👈

You can then know with certainty that any time you see this same collection of bytes - regardless of where you find it 🔍 - it’s the same binary because the prints match. If they don’t match it’s either a completely different artifact or, less likely, someone has modified the original.

And that means you can’t qualify one thing and then deploy another without realising the discrepancy. 🕵️🕵️

To establish binary provenance we take the SHA256 fingerprint of every artifact we build, record it securely in our black box with the source code that created it, and any other relevant meta-information, such as links to build logs, etc. 📊

3. Automate Risk Controls

Risk controls of different descriptions are usually defined by your Secure Software Development Lifecycle (SSDLC). For example, in the process documentation we might have things like:

  • Code must be unit tested
  • All merges to master must have approved pull request
  • Code must be pass integration & contract tests
  • Code must have passed security analysis

These controls can be anything we like, but whatever they are we should do them in our pipeline. Remember, implementing tasks as gates 🚧 is a mistake and these are no different. That’s why we automate all of these processes and make sure they’re executed for every change. Whatever our risk controls are, we automate them! 🤖🤖🤖🤖🤖

The task of conforming to our process is now automatic, and we can proceed to automate the collection of all the evidence that these tasks have been done against the relevant binary in our black box. 😎

So, we start with the SHA of our binary, which is our initial fingerprint, and as we qualify it and pass it through the risk controls in our pipeline we add the evidence to the binary and record it in our black box.

This gives us a complete history of where our binaries came from and all of the changes and events that they have been subject to. And because our black box is append-only we’re only ever adding ➕➕➕ new evidence.

That means we can go back in time and look at any point in the history of our pipeline. We can re-run stages of the pipelines or overwrite existing data, but these will be new commits in the database and the black box will capture them.

We now have a ready made audit trail that’s secure 🔐 and incorruptible. 🥳

4. Segregation of duties and approvals

“But what about the human in the loop?” I hear you ask. 🙋🏽

In high security environments there’s usually a requirement that a person with a designated role is tasked with approving the risk of changes to production. It’s a way to mitigate insider threats and ensure accountability by giving a human👨🏻‍💼 responsibility for looking at proposed changes and taking responsibility for them.

But, as we’re already following steps 1-3, so this step can be automated from our CI system, from Slack, from Git, or from inside the black box itself 👍 Remember, we have a record of every change, so our designated human can predefine all the criteria needed for a release candidate to pass to production. ➡️

For example, if we can verify binary provenance and check for all of the necessary risk controls in the pipeline, the deployment step can be automated because all the required evidence is there. The key point is that even if people are required to approve, it doesn’t have to be a high ceremony as long as you manage risk and documentation automatically.

5. Environment Change Logs

The final step is to log 📊 every change made to every service in production. By doing this for every change we can know what’s currently in production, what was in production on a given date, what the last 10 deployments were, etc. 👀

And that’s basically all there is to it. A secure chain of custody is all about creating a trail of breadcrumbs by recording where our binaries come from, and how they change as they make their way through our pipeline to production. ⏩⏩⏩⏩⏩⏩✅

Cool or what? 😎

If you have any questions about this blog or want to talk about doing DevOps in a regulated space, feel free to book some time with me. I’d love to hear👂 about the specific challenges in your industry.

TVM! Mike. 🙏


About this article

Published June 7, 2021 in technology

About the author

Mike Long

Mike Long


ABOUT THIS ARTICLE

Published June 7, 2021, in technology

AUTHOR
Mike Long
Mike Long
Live in Git Blame? Don’t spend hours searching for the change that broke your application! Query, search and discover all the changes in one place

Latest articles

Kosli announces Innovation Partnership with DNB and Firi

We are pleased to announce that Innovasjon Norge has awarded Kosli an innovation grant of 3.4 million NOK to pursue a R&D project with DNB and Firi. In this blog we’ll give you an overview of the …

The Ultimate Guide to git blame: A How To with Examples

Source control tools give users many powers and one of the big ones is traceability. With traceability tools you can know exactly who made each change and when they made it. In Git, you use the git …

Git Blame in VS Code: The 4 Best Options

Most production projects have a team collaborating on them, so even in a single file there can be multiple contributors. When things go wrong, it’s useful to understand how and why certain changes …

Sign up to our newsletter

We'll let you know about the Kosli launch, product news, features and updates
Kosli is committed to protecting and respecting your privacy. By submitting this newsletter request, I consent to Kosli sending me marketing communications via email. I may opt out at any time. For information about our privacy practices, please visit Kosli's privacy policy.
Kosli team reading the newsletter

Let’s chat!

Got a question about Kosli? An idea for a new feature? Join Kosli Slack and talk to us.

Join
Developers using Kosli