7 min read
Since November, Kosli’s own engineering team has been running a live experiment: what happens to code review when the thing generating the code - and increasingly, the thing reviewing it - is an AI, not a person. Alex Kantor, Kosli’s Director of Technology, walked through that experiment in this webinar: what broke, what it cost to fix, and what four “obvious” assumptions in a standard code review control turned out not to hold once you took the human out of the loop.
Below is a chapter-by-chapter guide to the talk, with timestamps.
00:00 - Why “on top of the bread” is a control problem
Alex opens with a video of a father giving his son painfully literal instructions for making a peanut butter sandwich - “put the butter knife in the peanut butter,” “put it on top of the bread” - and the son doing exactly what was said, not what was meant. It’s the same failure mode as a control written for humans: the words work because a person fills in the gaps. An AI doesn’t fill them in, it just executes.
The team’s review process was originally social - mob sessions and Slack threads - which worked because humans were doing the reviewing. As AI-assisted coding accelerated from November onward, that stopped scaling. Kosli tried GitHub Copilot’s review product first, then ran a side-by-side test against Claude in January and switched. But even a good AI reviewer left an open question: was it actually checking the things the team’s control said it should check, or just producing plausible-looking approvals.
05:13 - What we built: a panel of specialist reviewers
Rather than one generic review prompt, Kosli built 16 specialist reviewer personas - security, architecture, test quality, and others - each tuned on Kosli’s own codebase and standards rather than generic best practice. Each persona runs twice, once on Claude and once on Gemini, with a moderator that reconciles disagreement and reasons through conflicts. Every step is logged: scripts control the process and produce the evidence, policies (written in Rego) decide whether the process was actually followed. It went live on Kosli’s production repo in April, built largely through Alex “vibe coding” his way through and running retrospectives whenever the reviewer’s output looked off.
08:37 - Four assumptions we didn’t know we were making
Kosli’s code review control reads like most teams’: change reviewed by a peer, reviewer isn’t the author, evidence recorded, branch protected. None of that changes because the reviewer is an AI - on paper. But underneath those five lines sit assumptions nobody had ever written down, because with a human reviewer they were either obvious or too expensive to check. The retrospectives on the new reviewer broke in four different ways, and all four pointed at the same control.
Assumption 1: the reviewer read the whole change (09:35) - A 15-file, 715-line pull request went through six rounds of review, with later rounds still surfacing issues that had been there since round one. The retro showed the reviewer had actually only read about five of the fifteen files - a context window overflow - while claiming it had reviewed everything. In one case it reported 70% coverage and 100% coverage in the same breath. The fix: every reviewer now has to produce a per-chunk acknowledgement of exactly what it read, with line numbers, checked deterministically against the actual diff.
Assumption 2: the findings are real (12:35) - A one-line config change came back with 12 findings, none of them legitimate - the reviewer was describing code that wasn’t even in the diff. Classic hallucination: it wanted to be useful, so it invented something to be useful about. The fix: a finding only counts if it cites a specific file and line number that can be checked against the real diff, and a second model has to independently confirm it by actually reading the file and using tools, not just asserting “confirmed.”
Assumption 3: the reviewer knows and applies the standards (14:20) - Kosli deliberately ran a pull request with a known cross-site scripting risk through six specialist reviewers. All six summarized the code correctly. None of them flagged the vulnerability, even though the relevant standard was loaded into their context. Loading a standard isn’t the same as applying it. The fix: reviewers now have to check standards one at a time and report a finding for each, rather than being trusted to reason it through the way a human would.
Assumption 4: one good reviewer is enough (16:47) - With a second AI opinion costing almost nothing, Kosli expected the two review models to converge over time. They didn’t: measured over the last two months, the two models’ findings overlap somewhere between a fifth and a third - each has its own blind spots, just like humans do. The value of running two models isn’t agreement, it’s coverage. A policy check now confirms every specialist actually ran and flags humans when the process degrades.
19:00 - What it costs, and what it returns
Most reviews cost one to two dollars, with every run’s cost logged rather than estimated. The first pass typically comes back around 15 minutes after a pull request opens. Since July 2025, Claude has assisted with roughly 80% of Kosli’s pull requests, and the team is now merging about 155 PRs a month, up from around 80 a year earlier, without any comparable growth in headcount. Despite that near-doubling in volume, code quality has held and merge time has stayed at just under two hours. The reviewer is what let the team absorb that volume without the controls quietly slipping.
21:34 - Pushing further: a fully agentic pipeline
Following a team discussion in Edinburgh in March, Alex spiked a pipeline that goes from ticket to merge with no human in the loop until the final merge decision: an agent picks up the ticket, plans it, writes the code, runs it through the reviewer panel, fixes what comes back, and stops just short of merging. One ticket-to-merge run generated over 80 pieces of audited evidence. Two new requirements stood out: the ticket itself becomes the most safety-critical artefact, so it gets cryptographically fingerprinted the moment a human approves it, to prove nothing changed underneath during development; and prompt injection scanning becomes a form of vulnerability scanning, checking what goes into the AI and not just what comes out.
24:56 - No new controls, just sharper ones
Alex’s early read on the growing list of extra requirements was that Kosli had somehow ended up with 28 new controls to manage. A conversation with FINOS’s SDLC controls working group changed his mind: none of it was new. Every requirement uncovered along the way was a sub-requirement that existing controls already implied but never spelled out, because with humans it was either obvious or too expensive to check. The intent of the controls doesn’t change - the detail underneath them just has to get far more specific.
26:55 - Who’s accountable, and the closing thought
The harder question isn’t who’s accountable when an AI-assisted change goes right, it’s who’s accountable when it goes wrong. Alex’s framing: if the team followed every control and the evidence proves it, and something still breaks, the questions point at whoever defined the controls - they weren’t specific enough. If the controls were well-defined but poorly implemented, the questions point at whoever operates the process. Evidence is what tells you which conversation you’re actually having, instead of every incident turning into archaeology and opinion. Alex closes by arguing the controls regulated engineering teams need already exist - they’ve just been living in people’s heads. The job now is to write them down precisely enough that machines can prove they’re being followed, in real time.
