Tenth Chair

Code review

AI code review prompts that actually find bugs

Most AI review prompts return a page of plausible concerns and nothing you can act on. The fix is not a better persona — it is a rule about what counts as a finding.

Here is the prompt almost everyone starts with:

Act as a senior engineer and review this code for bugs
and security issues.

It returns something that looks excellent. Six or seven well-organised concerns, each phrased with confidence, each touching a real category of risk. Input validation. Error handling. Potential race conditions. Consider adding logging.

Then you go and check them, and most of them are not reachable in your codebase. The race condition is on a path that only ever has one writer. The validation concern is handled two frames up the stack. By the third review you are spending an hour to find nothing, and quietly stop running it.

This is the central problem with AI code review, and it is not a model capability problem. A frontier model can find genuinely subtle defects. The problem is that nothing in the prompt distinguishes a defect from a worry, so you get both, formatted identically, with equal confidence.

The rule that fixes it

One rule does most of the work:

No finding blocks without a concrete reachable failure.

Enforced as an output format on every review:

trigger  →  failure  →  cause        (with file:line)

The reviewer must name the exact input or state sequence that reaches the problem, what observably goes wrong, and the line responsible. If it cannot produce that chain, the concern is advisory — worth reading once, never worth a day of investigation.

Compare the two forms:

Not a findingA finding
"This could be risky under concurrent load." "When two refunds for the same customer are in flight and the first is abandoned, the attempt counter is never cleared, so the second refund starts at the escalated path and skips the safe route — retrier.py:88."

The second one you can act on in ten minutes. The first costs you a day and usually ends in a shrug.

There is a stricter version of the rule that is worth adopting once you are comfortable: a blocking finding requires a failing test. If the reviewer cannot describe a test that fails before the fix and passes after it, the finding advises rather than blocks. This is aggressive, and it is the single most effective noise filter available.

Why this matters more than the persona. In one measured cleanup of an AI review board's output, three findings out of roughly twenty were real and blocking. The other seventeen were defensible-sounding noise that would each have cost about a day to investigate and dismiss. Seventeen days of work, to protect against nothing. The rule above is what separates the three from the seventeen.

Give the reviewer probes, not just a diff

The second-biggest lever, and the one people skip because it requires thinking before prompting.

A reviewer handed only a diff returns vibes — general observations, standard categories, a checklist rendered against your code. A reviewer handed named attack surfaces returns evidence, because it goes and traces the specific path you pointed at.

Defects hide in three places. Logic bugs inside a single function are the easy case; your tests and your own reading catch most of those. What survives to production lives here:

Seams — where components hand off

State — what persists between operations

Timing — what changed between the read and the write

Pick three to six that actually apply to your change. Resist listing ten — a reviewer with ten probes gives each one a sentence.

A prompt you can paste today

This runs four opposed reviewers in a single chat window. No tooling, no agent framework, works in any capable model.

You are running the four BLOCKING seats of a review board sequentially
over one change, returning all four verdicts in one report:
architecture-compliance, security, black-hat, red-team.

THE CHANGE: [paste diff, or describe precisely: files, before/after
behaviour]

CONTEXT: [2-4 sentences: why the change exists, what incident or
requirement motivated it, what the system does]

PROBE HARD (this list is where reviews are won):
1. [probe]
2. [probe]
3. [probe]
4. [probe]

DOCTRINE: a finding blocks ONLY with a concrete reachable failure —
state the exact inputs/state sequence and the defect line.
Plausible-but-unreachable concerns are advisory notes, clearly
separated.

Return: four titled verdict sections + a one-line chair summary.

The doctrine paragraph is not decoration. Remove it and the output degrades immediately back into a list of worries. It is the cheapest four lines in the prompt.

Why four reviewers rather than one

Because they hunt different things, and a single general reviewer averages across all of them badly.

SeatAttacks
Black HatThe design's logic — hidden assumptions, quiet wrongness, second-order effects
Red TeamIts life in production — crash windows, restarts, concurrency, leaked state
SecurityReal exploitable risk in this specific stack
ComplianceSilent deviation from the written architecture

The Black Hat charter is the one to start with, and it is short enough to include in full:

You are the Black Hat on the [PRODUCT] review board (de Bono's black
hat: disciplined critical judgment, not cynicism). Your mandate: find
why this change is a mistake before it ships. Assume it is flawed and
prove it.

Review the change (start from the diff) and hunt for:
- HIDDEN ASSUMPTIONS. What must be true for this to work that isn't
  guaranteed? (ordering, timezones, schema stability, single-writer
  locks, memory headroom, clock skew, retry semantics.)
- FAILURE MODES. What breaks under scale, partial data, restart,
  concurrency, or a late-starting dependency? [Add the specific
  classes of bug your codebase has actually been bitten by.]
- CORRECTNESS. Look for the quiet wrongness: silent zero-fill hiding
  missingness, a counter that survives a code path it shouldn't, an
  estimate presented as a measurement.
- SECOND-ORDER EFFECTS. What downstream consumer does this quietly
  break? What does it make harder to change later?
- THE WORST REALISTIC OUTCOME. State it plainly.

Be specific and adversarial; vague worry is not a finding. Every
finding needs a concrete trigger ("when X happens, Y breaks because Z").

Output: `Lens verdict:` BLOCK / CONDITIONAL / CLEAR, then findings
(severity, file:line, the trigger→failure→cause chain, and the fix or
guard required).

The bracketed line matters more than the rest. Replace it with the failure classes your system has actually been bitten by — go through your incident history and your postmortems. Ten minutes, once, and every review afterwards is sharper. That list is the difference between a generic reviewer and one that knows where your bodies are buried.

A worked example

A payments service adds a fast-void watchdog. When a card authorization must be voided, a background check now fires the void within seconds instead of waiting for the nightly batch cycle. On success it writes an audit row to the compliance journal.

Sixty lines. Tests pass. CI green. It reads as entirely reasonable.

Six probes went with the change. The one that mattered:

(6) Does the journal writer tolerate the lean row the watchdog passes?
    Trace every field's type into the serializer.

Three of the four seats passed it. Compliance confirmed the placement reused the existing journal seam. Security found no new surface. Red Team probed the crash windows and double-void scenarios and documented why the batch cycle safely no-ops afterwards.

The Black Hat failed it, with one must-fix:

Finding — Black Hat seat
Trigger
Any successful watchdog void.
Failure
Compliance journal row silently dropped, on 100% of successful voids.
Cause
Audit row passes None for three numeric fields; serializer.py:142 casts unconditionally (float(None) → TypeError); swallowed by the broad except at watchdog.py:88.
Verdict: FAIL — one must-fix. Pass zeros matching the batch path's convention, or guard the casts. Add a regression test that asserts the row lands.

The exact visibility the feature existed to provide would have shipped non-functional. Nothing crashes. Nothing logs at error level. The test suite is green because the test stubs the journal and never asserts the row lands. An auditor would have found it, months later, by asking why the journal had no watchdog rows.

One-line fix, merged the same day.

Worth noticing what did the work here. Not a cleverer model — the same model wrote the code. It was one seat whose entire job is to assume the change is broken, and a probe that told it to trace data types through the serializer rather than reason about the design.

Five ways this goes wrong

Confident theater

Beautiful verdicts, no finding that ever changes anything. Cause: personas without the doctrine rule. Fix: put the doctrine paragraph in every prompt and enforce trigger→failure→cause.

Death by advisory noise

Fifteen findings per review, so you stop reading them. Cause: accepting findings with no reachable failure sequence. Fix: reclassify anything without a trigger sequence as advisory and do not action it.

Vibes reviews

The reviewer tells you things you already knew. Cause: no probes. Fix: ten minutes, three to six probes, targeting seams, state and timing.

Abdication

"The AI reviewed it" appears in a pull request description. Cause: treating the reviewer as an authority rather than an adversary. Its job is to attack the change; your job is to judge the attack. A confident model has been wrong before and will be again.

Reviewing ideas instead of changes

Boards review diffs. "Review my architecture" produces a review of your architecture-shaped opinions. If there is no code yet, write the design down as a concrete proposal with named components and interfaces, and treat that document as the diff.

Start here

Take the last non-trivial change you shipped. Get the diff. Write three probes — one seam, one state, one timing. Paste the Black Hat charter, the diff, the probes, and the doctrine line. Then judge the attack yourself.

If it finds nothing, that is information about your probes, not about your code. Rewrite them to target state and timing rather than logic, and run it again.

Go further

The complete kit

This article covers one seat of nine and one law of eight. The full kit adds the Chair, Red Team, Security and Compliance charters plus five advisory seats; the remaining seven laws with the failures that produced them; five orchestration patterns including parallel multi-agent and re-verdict; a sixteen-pattern probe catalogue; a second worked review; and two fill-in worksheets.