9 min read
There is a huge amount of hype around AI. Companies are growing faster than ever, IT budgets are being redirected, and product roadmaps everywhere are being redrawn.
There is no doubt LLM’s are a transformative technology. At the same time, as with any early technology cycle we are far from understanding the patterns of success. And for sure, mis-steps and bad takes abound.
Like any technology company, we’ve been deeply investigating various AI technologies - looking at the possibilities AI brings, but also taking a critical look at what is possible with today’s approaches.
This article lays out our vision for how AI will disrupt a key area of software delivery governance: the audit process. We believe there is a clear and obvious application of AI in this space that has the potential to completely disrupt a multi-billion dollar industry.
This is the first article in a series in which will share our AI journey as we bring these powerful new features to market. It is offered in the spirit of building-in-the-open, and we’ll be sharing what we learn along the way.
Audits are painful for developers AND compliance teams
Auditing software delivery today is a painful and inefficient process that frustrates both development and compliance teams. Developers feel a massive amount of pain when it comes to gathering evidence. They spend countless hours taking screenshots, filling out spreadsheets, and documenting that they did tests, security scans, etc. .
But compliance folks then have to evaluate all of that evidence, prepare it for audit, and the quality is often poor. Screenshots can be easily manipulated, spreadsheets may contain errors, and documentation frequently becomes outdated. It’s frustrating for compliance officers as individuals, and incurs great cost to the business.
Developers and compliance teams also frequently get in each other’s way. Devs get pulled away from development work to answer questions and gather evidence. The back-and-forth between compliance and development drags on as each side tries to clarify requirements and provide missing information.
To make matters worse, this entire process has to be repeated over and over again - as frequently as every month for FedRAMP. This creates a continuous drain on resources and makes it difficult to improve processes since so much time is spent just satisfying basic compliance requirements. Banks spend millions of dollars every year to meet their audit requirements. It’s a huge cost center.
We’ve solved audits for evidence collection. With AI we’ll solve it for evidence evaluation
Until now we have been focused on solving the audit problem for developers. With Kosli, their evidence gathering work is automated - there’s no need for taking screenshots and spreadsheets.
All evidence needed for process compliance is shipped automatically to Kosli, and this frees up a huge amount of value in the software delivery process. It allows teams to deploy faster, safer changes in smaller and more frequent batches.
But the compliance teams are still outside the loop and audits are still a huge source of manual work for the enterprise. The job of evaluating the evidence sent to Kosli is still manual and potentially confusing for non-technically minded employees. They have to ask developers to interpret the evidence for them.
They need to help to answer questions like - show me every unit test for deployments to this application. That’s difficult to answer if they’re looking at a bunch of screenshots or technical data. But, what if they could evaluate the data in Kosli using AI? What if they could simply query Kosli with a simple prompt like - “Kosli, show me every unit test for deployments to this application?”
This article lays out Kosli’s vision of how AI can transform the auditing landscape. Agentic auditing promises to enhance audit efficiency, reduce costs, and elevate audit quality.
What is the point of an SDLC audit?
The purpose of an SDLC audit in banking is to provide independent assurance that an organization’s software development practices are:
- Controlled: Development activities follow a structured, documented process that manages risks and ensures quality
- Compliant: The development process meets regulatory requirements, including those from bodies like Federal Reserve, OCC, FDIC and State banking regulators
- Consistent: Development practices are standardized and repeatable across projects and teams
- Secure: Security controls are properly integrated throughout the development lifecycle, from requirements through deployment
All this sounds like a positive thing! Especially if you make software that moves billions of dollars every day…
But,
Audits are a slow and expensive governance loop
- Engineers produce evidence manually (screenshots, spreadsheets, change tickets)
- Auditors evaluate the evidence manually
This process has a lot of problems! As well as being very manual, it is:
Retrospective
- Problems can be undiagnosed for long periods of time
- Annual at best
Patchy
- The volumes of change are so great that audits on look at samples of evidence, providing limited confidence in the efficacy
High Trust
- A lot of trust has to be placed in humans, because the auditors are often not technical and the engineers are producing the evidence
Expensive
- IT audits can cost millions of dollars
This is a painful loop for compliance and engineering, and it is becoming more painful because organizations are trying to ship increasing numbers of changes
As Software speeds up, audit evidence explodes
The modern software delivery process has significantly accelerated over time.
This will only get faster with the adoption of AI coding assistants and autonomous systems, and distributed architectures.
The increasing speed of development and complexity in the software delivery process produces more changes. And increasing changes creates ever increasing volumes of evidence and audit data. This increases the time and cost of audits because all of that evidence has to be verified.
A quick sidebar on how Kosli works…
Across your software delivery tooling and processes, Kosli collects evidence of the key events you need to prove compliance to your processes.
Kosli gathers the evidence for developers automatically across the software build, release and run areas of the SDLC, and connects the dots to create a system of continuous compliance.
This fundamentally transforms the audit process in several key ways:
- Audits shift focus from evidence collection to analysis
- Compliance checks that were point-in-time become real-time
- Quality improves with the move from high trust to zero trust automated evidence
Navigating a Sea of Evidence with an Audit Co-pilot
So with Kosli, teams go faster, compliance evidence increases, which eventually leads to new challenge: how do we navigate, contextualize and summarize a vast dataset of very technical data, especially with non-technical stakeholders?
We’ve gotten quite far now without mentioning AI, but this problem statement is the crux of the opportunity for AI in software delivery govenance.
Essentially, the governance loop gets closed by engineering teams providing evidence to governance stakeholders. This evidence is extremely technical, and requires a lot of interpretation. In fact, auditors would most likely not be able to understand or validate the evidence without the help of engineering guidance.
What if we instead allowed governance professionals to ask questions of the data set directly, in language they are already familiar with? This is the perfect application for modern AI technologies, as an interpreter and guide.
And actually, many of the questions auditors already ask can very easily translate into a copilot interface.
PROMPT = "Give me a list of changes that happened on Thursdays"
PROMPT = "Set up a deployment control that blocks deployments to the payments platform on Friday afternoons"
PROMPT = "How many pull requests shipped this week"
PROMPT = "List the pull requests shipped this week for the auth service"
PROMPT = "Create a report containing the github issues worked on recently. Group by repo"
PROMPT = "Which production workloads don't have an associated vuln scan"
We’ve been working to make this vision a reality through the use of a new MCP interface in Kosli, where you can bring our LLM to query the data in Kosli, and honestly we’ve been struck by how effective it is:
One of the most exciting use cases is presenting responses in a format auditors can easily understand and relate to—whether that means incorporating data from other MCP interfaces, exporting datasets to CSV or PDF, or investigating production incidents with clear, traceable evidence.
Moving from Continuous Collection to Autonomous Evaluation with Agentic
Agentic software is AI that can autonomously pursue goals and take actions in a software environment - essentially programs that can understand tasks, make decisions, and execute them independently rather than just following predefined scripts. Think of it like having a smart assistant that can actually do things in your computer systems rather than just analyze or respond.
The key theme across all these is that the software can understand context, make decisions, and take appropriate actions rather than just following fixed rules.
So imagine agents that can provide continuous oversight?
In a compliance context, this means the software could:
- Monitor systems and events continuously
- Identify compliance issues proactively
- Take automated remediation steps where appropriate
- Escalate complex issues to humans
- Learn from past incidents to improve detection
An AI solution like this creates a compliance officer that’s always on duty - it can handle routine monitoring and basic issues independently, while knowing when to involve human experts for more complex situations.
What could agentic audit interfaces look like?
PROMPT = "When a change re-uses a jira ticket, contact the user and request an explanation, and open an approval ticket from their manager. If not complete within 48 hours, create a JIRA ticket and escalate to the compliance team"
PROMT = "Monitor deployments for age. When a workload has been in production for more than 30 days notify the team. Unless it is being scanned on a timely basis, invalidate the vulnerability scan 24 hours later"
PROMPT = "Send me an email when a pull request has more than 15 commits, or 200 lines changed"
Why AI is so transformational
- AI is a step change in complexity of solutions that can be made
- Applying the power of AI to business problems can literally change the org chart
- Makes it possible to create software that is more like a canvas than a calculator
Are you interested in applying AI to software governance? We are actively exploring the area and would love to hear from you.
Kosli already opens the floodgates to production for developers. It allows them to deploy fast, safe, compliant changes to production, adding huge amounts of value and removing toil.
Kosli AI allows auditors to validate all of these changes without using manual processes. It automates a huge cost center to the business
