AI Code Review That Produces Evidence You Can Check
Give the reviewer a contract, require a reproduction, and measure the comments you reject as carefully as the defects you catch.
Contents
- Decide what the reviewer should establish
- Prepare the refund exercise
- Run deterministic checks before judging the reviewer
- Capture the exact production diff
- Use a review prompt that permits no findings
- Turn comments into decisions
- Measure missed defects and false positives separately
- Run the workflow in Agentastic
- Frequently asked questions
A useful AI code review workflow gives a fresh reviewer the requirements and the exact change, asks for reproducible findings, and sends confirmed problems back to the author before a human makes the merge decision. The output you want is evidence that changes your decision about the code.
This guide builds that workflow around a small refund helper. You can run its checks locally, then use the same contract, prompt, and finding rubric on your own repository. The examples were checked with Python 3.14.4 and Git 2.52.0 on October 15, 2026; the Python files use only the standard library. No model review experiment was performed for this article.
Decide what the reviewer should establish#
An implementation agent answers “How can I make this work?” A reviewer should answer “Under what concrete condition does this change violate the requirement?”
Those are different tasks. A fresh session helps separate them: the reviewer sees the code and contract without having spent an hour defending the implementation. However, two sessions can still share the same mistaken assumption, especially when the requirements are ambiguous. Using different models is a hypothesis to evaluate, not proof of independence.
GitHub's own responsible-use documentation identifies both missed problems and false positives in AI review. That makes a clean change an essential part of a review exercise: a reviewer that always finds something can appear useful when every example is deliberately broken.
Write a review contract before choosing tools:
| Review input | What to include |
|---|---|
| Requirement | Observable behavior and relevant non-goals |
| Change identity | Base commit, candidate commit, and diff |
| Surrounding code | Callers, data contracts, and relevant tests |
| Check results | Commands, exit status, and failures |
| Reviewer authority | Whether it may run checks, create tests, or edit files |
| Expected output | Evidence-backed findings, including an empty result when appropriate |
For a production review, give the agent enough source access to follow a call chain. A small diff can violate a contract several files away. Keep the change fixed while reviewing it; otherwise the comments and test output may describe different versions.
Prepare the refund exercise#
Download the candidate helper, requirements, clean control, and fixture grader.
The function receives integer cents:
remaining_after_refund(paid_cents, refunded_cents, request_cents)The caller guarantees a valid original payment and prior refund total. This helper must reject a negative request or one larger than the remaining balance. It allows a zero request and a refund of the entire remainder.
This deliberately narrow contract matters. Asking the reviewer to invent payment-service authentication, currency conversion, or database locking would produce irrelevant comments: the exercise contains no payment API or stored state.
The candidate contains two seeded errors:
def remaining_after_refund(paid_cents, refunded_cents, request_cents):
if request_cents > paid_cents:
raise ValueError("refund exceeds payment")
return paid_cents - refunded_cents - request_centsFor example, a payment of 1,000 cents with 600 already refunded has 400 available. A request for another 500 must fail. The candidate instead compares 500 with the original 1,000 and returns a negative balance.
The second error is a negative request. Refunding minus 100 cents should be rejected; the candidate adds 100 to the balance.
These answers are public teaching material. For a review trial, copy only the requirements and one candidate into a fresh workspace, with neutral filenames. Keep the clean implementation, grader, this article, and other trial outputs outside the reviewer's accessible environment. Use private examples for a serious comparison; a public fixture cannot establish absence of training contamination.
Run deterministic checks before judging the reviewer#
Save the files together and run:
python3 grade_fixture.py review_target.py
python3 grade_fixture.py review_control.pyThe grader checks five behaviors. It exits with status 1 for the deliberately broken candidate and status 0 for the clean control:
| Check | Seeded candidate | Clean control |
|---|---|---|
| Partial refund | Pass | Pass |
| Zero refund | Pass | Pass |
| Refund the entire remaining balance | Pass | Pass |
| Reject a negative request | Fail | Pass |
| Reject a request above the remaining balance | Fail | Pass |
These are results from executing the fixture functions, not findings produced by an AI reviewer. The grader imports the selected Python file, so use it with these trusted examples or run unfamiliar candidates in your evaluation sandbox.
For normal development, share existing test failures with the reviewer: finding a second cause or a missing edge case can still be useful. For an experiment measuring detection of these two defects, keep the answer-revealing grader output private until after the review. State which information was available.
A test suite is one layer of harness engineering. A reviewer can suggest a missing test, but should not redefine success after seeing which assertions failed.
Capture the exact production diff#
On a committed feature branch, record the candidate and its comparison point:
git rev-parse HEAD
git merge-base main HEAD
git diff --stat main...HEAD
git diff --no-ext-diff main...HEAD -- > review.diffThe three-dot comparison uses the merge base with the named branch, as documented by Git. It does not include uncommitted work. Commit the intended candidate, or deliberately capture staged and unstaged changes through your review tool. Confirm that new files are included.
Record the resolved commit IDs alongside the review, rather than relying on a branch name that may move. If the patch changes after review, retain the old result and perform a new review for the new candidate.
For larger changes, divide review by behavior: data validation, state transitions, persistence, and error handling. Give every reviewer the shared contract, but let each inspect the surrounding code needed for its assignment. Our parallel coding workflow explains when work can safely be divided.
Use a review prompt that permits no findings#
Adapt this prompt to the repository:
Review the candidate against the attached requirements.
Read relevant callers and tests before alleging a defect.
Do not modify the candidate. You may propose a reproduction separately.
For each confirmed finding, provide:
- file and line;
- the requirement or established contract that is violated;
- the smallest realistic input or execution sequence that triggers it;
- expected and actual behavior;
- evidence, and whether you actually executed the reproduction;
- severity based on user impact.
Separate unanswered questions from confirmed defects.
Avoid style preferences unless they violate an explicit project rule.
If you cannot establish a defect, say "No confirmed findings" and
state any material coverage limits.Do not ask for “at least five problems.” A quota rewards speculation. Likewise, avoid supplying another reviewer's answer before the first pass is complete. Collect each initial response separately, then compare them.
An agent may need to create a temporary reproduction. Give it a separate location for that work, and preserve the candidate hash. Reproduction code should not silently become part of the proposed application change.
Turn comments into decisions#
For each comment, a human or independent check assigns one outcome:
| Outcome | Meaning | Next action |
|---|---|---|
| Confirmed defect | A contract violation with evidence | Fix and add a regression check |
| False positive | The reported defect does not occur under the contract | Record why and reject |
| Needs investigation | Evidence or a requirement is missing | Ask a focused question |
| Duplicate | Same root defect as an existing finding | Link it to the original |
| Optional improvement | A useful suggestion outside acceptance criteria | Consider separately |
A finding about the example might read:
Location: refund.py, validation before return
Trigger: paid=1000, refunded=600, request=500
Expected: ValueError because only 400 cents remain
Actual: returns -100
Evidence: reproduced with the three integer arguments
Impact: accepts an over-refund request under the function contractThe finding earns its place by connecting input, contract, and result. “There may be edge cases” does not supply enough information to fix anything.
After a repair, run the reproduction against the new candidate and rerun the existing tests. Review the repair diff too: a patch that rejects all refunds could make one negative test pass while breaking the function.
Measure missed defects and false positives separately#
Start with both broken and clean changes, repeat each review configuration in fresh sessions, and keep every result. Three repetitions can expose obvious inconsistency; they do not establish a universal ranking.
A blank recording table is more useful than an invented benchmark:
| Candidate | Reviewer configuration | Trial | Confirmed defects | False positives | Review minutes |
|---|---|---|---|---|---|
| Seeded | Record client, model, settings | 1 | Fill after adjudication | Fill after adjudication | Record |
| Clean | Same configuration | 1 | Fill after adjudication | Fill after adjudication | Record |
Count unique defects, not repeated comments about the same defect. For the seeded exercise, defect recall is detected seeded defects divided by two. Across adjudicated findings, precision is confirmed findings divided by confirmed findings plus false positives. An empty denominator is undefined; it is not perfect precision.
Also record the fraction of clean trials that received at least one false alarm. That captures the cost of distracting a developer even when a comment count looks small. Include the human minutes spent reproducing and rejecting comments in your cost per accepted change.
The two-defect exercise verifies your workflow. Extend it with realistic historical changes using the repository evaluation guide before selecting a default reviewer.
Run the workflow in Agentastic#
Agentastic's Code Review action captures branch comparison context, the unified diff, commit history, and review criteria. Configure the primary Review Diff action in Settings → Tasks, and select additional reviewers in Settings → Code Review. Inspect their output and adjudicate findings before applying changes.
Use the Diff Viewer to connect comments to the exact change. For experimental comparisons, launch fresh review sessions and preserve their outputs yourself; the review UI does not automatically supply a benchmark dataset or calculate false-positive rates.
Start with one implementation and one separate review. Download Agentastic, open a small change, and run the contract-and-reproduction process before adding more reviewers.
Frequently asked questions#
What makes an AI code review independent?#
Start a fresh review session with the requirements, exact diff, and relevant repository context. Avoid feeding it the author's persuasive explanation or another reviewer's conclusions. This separates evidence and context, but does not guarantee that two models will make independent mistakes.
Should I use a different model to review AI-generated code?#
A different model is one configuration worth testing. Compare confirmed defects, missed defects, false positives, and review time against a fresh session of your existing model. Model diversity alone does not establish better review quality.
How do I reduce false positives in AI code review?#
Ask for a location, violated requirement, concrete trigger, and reproducible evidence for each finding. Include clean changes in your evaluation, and let reviewers report no confirmed findings instead of requiring a fixed number of comments.
Can an AI reviewer replace tests?#
Use tests to verify executable behavior and an AI reviewer to propose additional problems or missing cases. A review comment is a hypothesis until it is checked. Passing tests also leave gaps, so retain human judgment for requirements and risks the tests do not cover.
Does this guide benchmark review models?#
No. It supplies a small Python fixture with two seeded defects, a clean control, and deterministic checks. The fixture checks were run; no agent review results or model rankings are claimed.