← Back to home

Architecture

review-harness

Automatic pull request review. Free checks first, AI only where the risk justifies the cost, and the AI can never approve anything.

1. The problem

AI agents write more code than a person can read line by line. Reading everything does not scale. Reading nothing is unsafe.

The obvious fix, running a model over every pull request, has two problems. The first is money: it costs on every PR, and most PRs do not need it. The second is more serious: it puts the least trustworthy component in the system in the position of deciding what enters the codebase.

The engineering problem is this: decide which PRs deserve human attention, spend money only where the risk justifies it, and make sure the AI can never approve anything.

The basis is the process described in Anthropic's October 2026 article on how the Claude Code team does code review, adapted for a small budget.

2. The numbers

Counted from how the tool actually operates.

5
steps in a fixed order: tests, risk, security, AI and decision
3
of those steps cost nothing: tests, risk and security
1
step costs money, and it ships turned off
3
possible results: pass, needs human, fail
3
rounds maximum per PR in the AI step
0
whole files read by the AI. Only the changed lines
0
keys handed to a PR coming from a fork
2
GitHub permissions: read the code and comment on the PR. Nothing else

3. The design

It is a funnel ordered by cost. Each free step reduces how much work is left for the expensive one, and the expensive one only receives what the free steps already flagged as risky.

Pull request opened1. Teststhe commands you configuredfree2. Riskwhich files changedfree3. Securitygitleaks · cargo-auditfree.review-harness.tomlread from main,never from the PRONLY MEDIUM AND HIGH RISK CONTINUE FROM HERE4. AI reviewoff by default · changed lines only · 3 roundsfindings outside the PR discarded · never approvescosts money5. One comment on the PRpass · needs human · fail

The configuration enters from the side on purpose: it is always read from the main branch, never from the PR under review. If a PR could bring its own rules, the first thing a bad PR would do is mark its own files as low risk and switch the checks off.

The three results

The middle result is a product decision, not the system being unsure.

Pass

Tests passed, no security problems, and only low-risk files changed.

Merge it.

Needs human

Everything passed, but the PR touches sensitive files.

Read it carefully before merging.

Fail

Tests failed or a security problem was found.

Fix it. Merging stays blocked.

4. The decisions

Every decision has a price. Below is what I chose, what I rejected, why, and what it costs.

01. Free checks first, AI last and off by default

Chose
Tests, the changed-file list and the security scan run first. Only medium- and high-risk PRs reach the AI, and the AI ships turned off.
Over
Sending every pull request to a model.
Because
Most PRs do not need an AI to be judged. A failing test and a list of changed files answer nearly everything for free. Putting the paid step last means paying only where the free steps already pointed at risk.
Price I pay
The AI never sees a low-risk PR. A subtle bug in a file nobody marked sensitive will not be caught by it.

02. The AI can report problems and can never approve

Chose
The AI step only reports. Approval is decided by the deterministic steps.
Over
Letting the model give the verdict.
Because
Approval is the only decision that matters, and a model that can be steered by the content it is reading must not hold that pen. A malicious PR writing instructions into its own diff would be instructing the reviewer to approve it.
Price I pay
A PR that is genuinely fine still carries whatever the AI flagged, and someone has to dismiss it by hand.

03. A finding pointing at a line outside the PR is discarded

Chose
If the model comments on a line that is not in the diff, the finding is thrown away.
Over
Showing everything the model said.
Because
A model reading a diff will confidently comment on code it inferred rather than read. A finding that does not anchor to a changed line is a finding nobody can verify, and noise in a review tool is what gets the tool switched off within two weeks.
Price I pay
A real problem caused by the PR but visible only in an unchanged line is dropped silently.

04. The configuration is read from main, never from the PR

Chose
The .review-harness.toml file always comes from main, whatever the PR carries.
Over
Reading the configuration from the branch under review.
Because
If a PR could bring its own rules, it would only need to declare itself low risk to escape everything. A rule has to live somewhere the PR cannot reach, otherwise it is not a rule.
Price I pay
Changing the rules needs its own PR to main, reviewed under the old rules. Tightening the config is slower than it looks.

05. The third result is "needs human", not a second kind of failure

Chose
Three results instead of two.
Over
Pass or fail only.
Because
A PR can be entirely correct and still deserve a careful read, because of where it lands rather than what it does. Collapsing that into fail teaches the team to merge over failures, and collapsing it into pass hides the case.
Price I pay
Someone has to actually read those. The tool moves the work, it does not remove it.

06. Two permissions: read the code and comment

Chose
The tool cannot change anything in the repository.
Over
Giving it write access so it could fix things or merge on its own.
Because
A review tool with write access is a new door into the repository, and it runs on every PR, including ones from strangers. Enforcement belongs to GitHub's branch protection, not to the tool.
Price I pay
The tool cannot enforce itself. Without the ruleset configured, everything it says is advisory and can be ignored, including by an agent pushing straight to main.

07. Action pinned to a commit, credentials not persisted, no key to forks

Chose
The checkout is pinned by commit hash rather than tag, with persist-credentials off, and a PR from a fork never receives the API key.
Over
Using the action's tag and the default settings.
Because
A tag can be moved. An action referenced by tag can be swapped under you, and it runs inside a job that holds a token. And a fork PR is written by someone you do not know: a key handed to a fork is a key given away.
Price I pay
A pinned hash has to be updated by hand, and an outdated pin is a security problem of its own.

5. What breaks

What is unresolved, including what the README states itself.

The tool cannot enforce itself

Protecting the branch is a manual step in GitHub's settings, and the README says so plainly: without the ruleset, anyone, including an AI agent, pushes straight to main and skips the whole review. The most important part of the system is the one part it does not control.

The risk list is written by hand

A sensitive path nobody listed is a low-risk path. The quality of the entire tool rests on a list someone has to remember to maintain, and an outdated list does not complain.

The scans catch what is already known

gitleaks finds known secret patterns and cargo-audit finds already-published vulnerabilities. A new secret format and an undisclosed vulnerability both pass, and a green result does not distinguish between having no problem and having found none.

Discarded findings leave no trace

Throwing away findings that point outside the PR is the right default and it drops real ones along with the noise. With no record of what was discarded, there is no way to know how often it happens or whether the rule is calibrated.

cargo-audit only works for Rust

In a project that is not Rust, that step does nothing and the tool does not say so. The report looks complete while one of the checks never really ran.

A large PR gets a partial review that looks complete

The AI step has a token limit and three rounds. A large PR hits the limit, and the final comment does not distinguish between a review that covered everything and one that stopped halfway.

Nothing guarantees the "needs human" was read

The middle result only works if someone actually reads carefully. The tool has no way of knowing whether that happened, and a team under pressure merges the yellow the same way it merges the green.

6. What I would change at 100x

Scenario: 100 repositories, hundreds of pull requests a day, and several teams on the same installation.

What survives unchanged

The order of the steps and the ban on the AI approving. Both are rules rather than infrastructure: they hold the same with one repository or a hundred, and they are exactly the parts that stop the bill exploding and stop the system being fooled.

What breaks first

The hand-written risk list. With 100 repositories there are 100 lists, each maintained by whoever remembers, all ageing at different rates. The tool keeps working and starts judging wrongly without saying so.

In the order I would change them

  1. 1

    Derive risk instead of declaring it. A path that has shown up in a past incident, or that a lot of code depends on, is high risk whether or not it is written in a list. The declared list becomes the exception, not the source.

  2. 2

    Centralise the configuration with per-repository overrides, so a security rule can be changed in one place rather than across a hundred pull requests.

  3. 3

    Budget the AI step at organisation level, not per PR. Three rounds per PR is a reasonable limit for one repository and means nothing across hundreds: the real control is a spending ceiling, with the expensive review going to the riskiest PRs first.

  4. 4

    Measure what the tool misses. Today there is no feedback at all: a bug that reached production after a green result leaves no mark on the tool. Linking an incident back to the PR that caused it is what turns the risk list from a guess into data.

This page describes the system as it stands today, including what is still unresolved. The tool reads the code and comments on the pull request, and cannot change anything in the repository.