Kosli devops change management git for compliance audit trail

Using Git for a compliance audit trail

Mike Long
Mike Long
Published November 6, 2020 in technology

Kosli is a DevOps change management platform for storing a record of compliance controls. It helps financial institutions, medical device manufacturers, automotive and other mission-critical development teams to prove conformance to their software process.

A key concept in Kosli is the immutable, append-only journal which provides a tamper proof audit trail. This blog explains how we came to design the durable storage technology in Kosli.

Persisting Compliance Data

Kosli DevOps Change Management

As a tool for recording software process data automatically, Kosli works by providing a REST API to DevOps pipelines. These pipelines can then communicate with Kosli via scripting languages, curl commands, etc.

Ultimately, Kosli is a database and its data must be stored in a durable manner. Given the mission critical nature of the domain, trust in the integrity of the data forms a key part in the value proposition.

The options

You might think that the unique requirements of audit would lead us to implement a custom persistence solution. But writing databases is hard, so first we surveyed the proven persistence technologies that might fit for our purpose:

  • Relational Databases
  • Document databases
  • Blockchain
  • Git

RDBMS

When adding a database to an application the natural default is to use a relational database such as Postresql or MySQL. These tried and trusted technologies are very well suited to the record-based data required by many applications.

However, compliance data has some unique requirements that are hard to implement in relational databases.

The purpose of an audit trail is to prove that a process has been followed by providing evidence. We want the proof to be immutable, i.e. it shouldn’t be possible to change or tamper with the evidence without knowing what changed and when.

Secondly, as a compliance database, we don’t know the structure of evidence users will want to record as proof. Shoehorning user data into table structures is not a good fit.

So, the strengths of RDBMS, namely table-based mutable data, are actually weaknesses in this domain.

Document Databases

Moving closer to the evidence needs, there is an alternative database approach that might be more appropriate: the Document-oriented database. Popular examples are MongoDB and Amazon DocumentDB.

Document-oriented databases are great for storing the kind of semi-structured data we use for compliance, but they still lack the immutability we need. To add versioning and immutability to a document database solution is hard to secure and complex to implement.

Blockchain

Going further down the rabbit hole, a new technology which solves the problem of immutable journaling has arrived: blockchain.

In fact, the most popular applications of blockchain are to record transactions of different types in a distributed, decentralized public ledger.

Its strength is in proving transactions in a zero-trust model. However, in the case of compliance data, what we would like is a centralized record of truth of evidence rather than transactions.

What we’d really like is a centralized, efficient, ledger designed for versioning documents….

Git

Enter git. Git provides the same append-only cryptographic basis as blockchain, but it also has a document oriented foundation for schema-less data generated by our domain.

Sidenote: Similar to blockchain, git uses Merkle Trees to encrypt an immutable append-only journal. However, we represent projects in Kosli using a git repository with a single branch so technically we are using Hash Chains rather than Merkle Trees, but hey-ho ¯()

In addition, git is an open standard that integrates well with other development tools and leads to interesting possibilities for further integrations. As an added bonus, developers understand git. As a development tool it fits neatly into everyone’s existing knowledge.

By mapping our domain model onto filesystem and git operations, we can get an immutable, append-only journal, together with versioned updates, all in an open standard format.

Kosli app environment screenshot

Summary

Persistence technologies all have a context in which they provide the best solution. Each of the database options we looked at are best-in-class for a given problem domain. However, for our application, git was the best fit.

Since developing Kosli, we have found git to be an extremely secure and flexible foundation for our solution.

RDBMS DocumentDB Blockchain Git
Immutable
Schemaless
Centralized
Open Standard
Simple

In other news: we will soon unveil our cloud beta, and if you are interested in participating drop me a line at mike@kosli.com and we’ll notify you as soon as we are live…


About this article

Published November 6, 2020 in technology

About the author

Mike Long

Mike Long


ABOUT THIS ARTICLE

Published November 6, 2020, in technology

AUTHOR
Mike Long
Mike Long
Live in Git Blame? Don’t spend hours searching for the change that broke your application! Query, search and discover all the changes in one place

Latest articles

Kosli announces Innovation Partnership with DNB and Firi

We are pleased to announce that Innovasjon Norge has awarded Kosli an innovation grant of 3.4 million NOK to pursue a R&D project with DNB and Firi. In this blog we’ll give you an overview of the …

The Ultimate Guide to git blame: A How To with Examples

Source control tools give users many powers and one of the big ones is traceability. With traceability tools you can know exactly who made each change and when they made it. In Git, you use the git …

Git Blame in VS Code: The 4 Best Options

Most production projects have a team collaborating on them, so even in a single file there can be multiple contributors. When things go wrong, it’s useful to understand how and why certain changes …

Sign up to our newsletter

We'll let you know about the Kosli launch, product news, features and updates
Kosli is committed to protecting and respecting your privacy. By submitting this newsletter request, I consent to Kosli sending me marketing communications via email. I may opt out at any time. For information about our privacy practices, please visit Kosli's privacy policy.
Kosli team reading the newsletter

Let’s chat!

Got a question about Kosli? An idea for a new feature? Join Kosli Slack and talk to us.

Join
Developers using Kosli