Kosli loop with error signs and documents

5 reasons why your CI system is a terrible Compliance System of Record

Mike Long
Mike Long
Published February 9, 2022 in technology

“Why can’t we use our CI system for our Compliance System of Record (CSoR)?”

This is a question we get asked a lot when we’re talking about compliance with regulated DevOps teams. And it’s a perfectly reasonable question to ask.

If Jenkins, GitLab, GitHub, or CircleCI is the engine for your DevOps it will contain a lot of information relevant to maintaining a CSoR. However, your CI system shouldn’t form the basis for your CSoR and in this article we’ll give you 5 reasons why.

If you want to know what a successful CSoR looks like you can check out our previous article on the topic. For now, let’s consider why it shouldn’t be stored in your CI system.

Compliance System of Records diagram Carl Nygard Kosli

1. CI systems have lots of mutable data

The first objection to using a CI system concerns the single biggest challenge a CSoR should address: preventing modification and deletion of information. A compliance audit trail must stand up to scrutiny. Auditors need to know if process evidence can be modified by internal or external bad actors.

The problem with Continuous Integration systems is that they tend to have lots of mutable information. For example, it’s often trivial to delete stuff like pipeline runs and repositories. Data and evidence can also be lost in less obvious ways if pipeline reruns overwrite previous information, or if log retention policies delete old data. A robust CSoR doesn’t modify compliance data.

2. CI systems are hard to secure

A frequent counter argument to the mutable data objection is that you can apply security controls. However, “We could just lock down the CI server” isn’t the winning argument you might think.

Having a highly rigid security posture around your CI system makes it harder for developers to get work done because it imposes heavy processes around how CI changes. Things that used to be easy - like creating a new microservice or renaming a repository - now involve paperwork and risk theater. This leads to a lot of disillusionment in the tech teams and establishes security as an enemy.

But, beyond this cultural anti-pattern, this is still a poor compliance solution. By turning a compliance problem into a security problem you’re getting weak compliance and rigid pipelines - the worst of both worlds.

3. CI systems are not databases

The third reason why your CI system should not be your CSoR relates to use cases beyond audit trails. Compliance data is used in many contexts. It can be used in reports to regulators, in security monitoring systems, and in further operational automation such as deployment controls.

CI systems don’t have a structured data layout and query language in which to use this information. It is also difficult to add your own specific data into the historical record without abusing these systems.

4. CI systems provide poor change observability

CI systems are source-code centric with user interfaces often scoped at the git repository level. This makes sense for a developer working on software, but the narrow focus makes it hard to reason about change in large distributed systems. Could your CI system give you good answers to the following questions?

  • Which versions of different microservices are running in production?
  • What are the recent changes to production?
  • What was in production last Wednesday evening?
  • What deployments are missing approvals?

Beyond this, most CI systems don’t typically use reliable software identification methods like Content Addressable storage, so it’s hard to correlate the CI events with actual running software.

5. CI-based CSoR’s == DevOps lock-in

Finally, using a CI system as a System of Record relates to overall technology strategy for your development teams. Forcing everyone into a locked CI ecosystem hurts your ability to adopt better and more distributed tooling.

If teams want to use different CI systems, or would naturally benefit from doing so, basing your audit trail on distributed and incompatible systems can make audits time consuming and hard to navigate.

Tl;dr - Don’t do it!

It’s tempting, and seemingly logical, to base your CSoR in your CI system. But, as you can see, there are very good reasons to avoid it.

For one thing, there’s absolutely no guarantee that an auditor will be satisfied with a mutable CI system as the record of compliance proof. And if that wasn’t disqualifying enough, you risk undermining developer productivity with a rigid system that offers poor observability and locks in your tools. It’s a lose/lose/lose situation.

Succeeding with DevOps requires sharp tools for specific jobs - you always want to have round pegs in round holes. Building a successful CSoR in a DevOps way is no different. It shouldn’t come as a surprise that using your CI system for this task is full of inefficiency and risk.

why your ci system is a terrible compliance system of records list


About this article

Published February 9, 2022 in technology

About the author

Mike Long

Mike Long


ABOUT THIS ARTICLE

Published February 9, 2022, in technology

AUTHOR
Mike Long
Mike Long
Live in Git Blame? Don’t spend hours searching for the change that broke your application! Query, search and discover all the changes in one place

Latest articles

Kosli announces Innovation Partnership with DNB and Firi

We are pleased to announce that Innovasjon Norge has awarded Kosli an innovation grant of 3.4 million NOK to pursue a R&D project with DNB and Firi. In this blog we’ll give you an overview of the …

The Ultimate Guide to git blame: A How To with Examples

Source control tools give users many powers and one of the big ones is traceability. With traceability tools you can know exactly who made each change and when they made it. In Git, you use the git …

Git Blame in VS Code: The 4 Best Options

Most production projects have a team collaborating on them, so even in a single file there can be multiple contributors. When things go wrong, it’s useful to understand how and why certain changes …

Sign up to our newsletter

We'll let you know about the Kosli launch, product news, features and updates
Kosli is committed to protecting and respecting your privacy. By submitting this newsletter request, I consent to Kosli sending me marketing communications via email. I may opt out at any time. For information about our privacy practices, please visit Kosli's privacy policy.
Kosli team reading the newsletter

Let’s chat!

Got a question about Kosli? An idea for a new feature? Join Kosli Slack and talk to us.

Join
Developers using Kosli