Evaluate Coding Agents on the Work Your Team Actually Does

To evaluate a coding agent on your repository, give it a fixed starting snapshot and a clearly specified task, run it under recorded limits, and grade the resulting code with independent checks. Repeat the process across representative tasks and fresh trials. Compare completed work, failure modes, human effort, and cost.

The object you are evaluating is a workflow: model, client, instructions, tools, environment, and review policy together. A model name alone does not describe what ran. Anthropic's evaluation guide makes this distinction and emphasizes repeatable trials and outcome-based grading.

This is a practical protocol with downloadable templates, checked on October 15, 2026. The snapshot commands were exercised with Git 2.52.0 in a temporary repository. It contains no measured agent scores. Use it to collect your own evidence before changing the workflow your team depends on.

Start with a decision you can actually make#

Write the decision in one sentence. For example: “Should our small API bug fixes use the current workflow or a lower-cost first attempt with one escalation?”

That question specifies a workload and a policy. “Which agent is best?” leaves both undefined.

Keep the comparison narrow enough to interpret. If you change the model, instruction files, tool permissions, and time allowance together, you can compare the complete workflows, but cannot attribute the difference to one component. Record that limitation explicitly.

For a first evaluation, choose two configurations and freeze them. Avoid switching models halfway through the run because an early example looked disappointing. Comparing coding agents introduces the selection dimensions; this guide turns a particular choice into a reproducible exercise.

Select 10–20 tasks that resemble your backlog#

Start with resolved issues your team understands. Include enough variety to reveal where a workflow helps or fails:

These are suggested pilot categories, not a representative sample of every repository. Choose their proportions from your recent work. An evaluation containing only tiny Python functions cannot establish suitability for a large Swift application.

For each historical task, identify the last commit before the fix, the requirement as it was understood then, and a human-verified solution. Keep the solution private. It proves the task is solvable, but the agent's patch does not need to resemble it.

Exclude tasks that require unavailable production credentials or an unexplained product decision. If the issue says “make pagination work” but the grader secretly requires a particular cursor format, fix the task description before running anything.

Freeze a task manifest and a grading rubric#

Download the task manifest template, acceptance rubric, and empty trial ledger.

The manifest records the starting commit and snapshot digest, prompt, toolchain, model configuration, instruction hashes, execution limits, and grader identity. Fill every placeholder. Give changed protocols new version numbers so that incompatible runs do not silently enter the same comparison.

A minimal acceptance contract for the refund task is:

Freeze the checks before comparative runs. Establish that the original snapshot fails the relevant checks and the known-good repair passes them. If both pass, the checks may not distinguish a repair. If both fail, the task or environment may be broken.

Grade maintainability separately from execution. Useful criteria include unnecessary dependencies, duplicated business rules, changes outside the task, and weakened existing tests. Give the reviewer concrete evidence requirements. “I prefer the old patch” is not a reproducible criterion.

Remove the historical answer from the environment#

Historical tasks are especially easy to contaminate. A pre-fix checkout can still contain the fix in another branch, a remote-tracking reference, a later commit, a release note, or an issue attachment.

A Git worktree is convenient for everyday development, but it shares repository objects and references. Starting at an old commit in a worktree does not make newer objects inaccessible.

For a simple repository without submodules or Git LFS assets, export the chosen tree from a trusted preparation environment:

Run this from the source repository after replacing the placeholder. Git archive exports tracked content from the selected tree. Audit the archive: export attributes can omit files, and submodule contents or LFS payloads need their own pinned preparation.

The exported files have no original .git directory. If the agent requires Git, initialize a new repository inside the task directory and create a baseline commit containing only the exported snapshot. Keep the original commit mapping in the evaluator's manifest.

This removes one leakage path; it does not create an access boundary by itself. Mount only the trial directory and approved dependencies into the execution environment. Do not give the agent access to the original repository, evaluator files, previous attempts, shell history, or a shared home directory containing old conversations.

Provision pinned dependencies first, then enforce the intended network policy. If network access is required, record allowed destinations and inspect whether issue pages, release archives, or package sources expose the solution. A prompt saying “do not look at the answer” is not a substitute for an actual access restriction. The sandbox permissions guide provides a way to test those boundaries.

Even careful preparation cannot prove a public issue was absent from a model's training data. Disclose that risk and favor private or newly constructed tasks when it matters.

Separate the agent workspace from the evaluator#

The agent should see the task and normal development checks. It should not control the final grade.

Keep private acceptance checks outside the accessible workspace. After the run, copy the candidate artifact into a separate evaluator environment and run the trusted checks there. Ensure the candidate has not changed test configuration to skip required checks.

Hidden implementations are useful; hidden requirements are unfair. The agent should know that negative refunds are invalid even if it cannot read the evaluator's exact negative-refund test.

A container image can make dependencies repeatable, as demonstrated by the SWE-bench evaluation harness. Still record image digests, architecture, service versions, environment variables that affect behavior, and resource limits. “Docker was used” is not enough to reproduce a trial.

Keep credentials and external effects out of the exercise unless they are essential to its stated purpose. For a database task, use a resettable local fixture database with no production data.

Define attempts, trials, and repetitions#

Use these terms consistently in your ledger:

• Task: the fixed problem and acceptance contract.
• Trial: one fresh execution of a workflow on that task.
• Attempt: a permitted stage or retry inside a trial.
• Repetition: another trial of the same task and configuration.

Suppose workflow A permits one attempt and workflow B permits a small-model attempt followed by one escalation. Each completed workflow is one trial; workflow B may consume two attempts. Charge both attempts to B.

Start with three repetitions per task and configuration if your budget permits. Ten tasks, two workflows, and three repetitions means 60 trials. This is a pilot budget, not a claim that 60 is statistically sufficient for every decision.

Reset code, generated files, service data, chat context, and writable caches between trials. Do not resume the previous session. Randomize or interleave configuration order so a provider outage or warm machine does not affect only one workflow. Record shared incidents when they occur.

Choose the maximum duration, token or money ceiling, allowed human help, and retry policy before execution. A developer silently rescuing one agent creates a different workflow.

Record the artifact and classify the outcome#

For each trial, preserve the prompt, configuration, candidate patch, tool output, check results, elapsed time, usage records, and reviewer decision. Redact credentials before sharing traces.

A useful outcome vocabulary is:

Do not erase a failed run because a retry passed. Link the replacement to the original and apply the same predeclared rule to both configurations.

Report two views when infrastructure matters: acceptance over all scheduled trials, and code success conditional on valid execution. The first describes the complete operating experience. The second helps diagnose model or implementation behavior. Show the counts behind both.

For a discovered grader bug, preserve the old results, revise the protocol, and rerun the affected configurations equally. Do not repair the grade only for your preferred model.

Read paired results before calculating a winner#

Compare configurations on the same tasks. A total can hide a serious regression in one category.

A purely hypothetical example: workflow A accepts 24 of 30 valid trials, while B accepts 25. That is 80% versus about 83.3%, a difference of one accepted trial. It does not by itself establish that B is better. Inspect which task changed, whether repeated runs disagree, and whether review effort rose.

Three repetitions of ten tasks still contain only ten distinct tasks. They help describe run-to-run variation; they do not turn a narrow task bank into thirty independent examples of your whole backlog. For consequential decisions, increase task diversity and use uncertainty analysis that accounts for grouping by task.

Also inspect unacceptable patches. Did the agent misunderstand the requirement, fail to run tests, or spend its whole budget repairing the environment? The next improvement may belong in your harness, not in model selection.

Use the independent review workflow for adjudication, and the cost workbook to include retries and review time. Preserve a held-out task set for the final decision; repeated tuning against every visible failure turns the evaluation into training material.

Use Agentastic as a workspace, with a separate evaluation record#

Agentastic can help organize isolated task work, inspect diffs, and run review passes. For historical evaluations, import prepared snapshots or use suitably configured containers; ordinary worktrees alone do not meet the leakage controls above.

The Usage panel can inform local token tracking, but its list-rate estimates are not invoices or a task-level benchmark database. Maintain the manifest and trial ledger separately.

Download Agentastic to organize a small pilot. Begin with two tasks whose expected behavior you can explain completely, validate the grading process, then expand to the representative set.

Frequently asked questions#

How many tasks do I need to evaluate a coding agent?#

Start a pilot with 10–20 representative tasks, grouped by the work you actually do, and repeat each configuration several times. This can expose practical failures but is too small for broad claims about model superiority. Expand the sample before making close or high-impact comparisons.

Are Git worktrees sufficient isolation for a coding benchmark?#

Worktrees separate working files but share Git objects and references. An agent may discover later commits or previous solutions. For historical-task evaluation, export the starting tree into an isolated environment without the original repository history, private graders, or network access to the answer.

Should an agent see the acceptance tests?#

Give the agent the behavioral requirements and ordinary developer checks. Keep independent acceptance implementations outside its writable and readable environment during the trial, then grade afterward. Private tests must verify disclosed requirements rather than introduce hidden product expectations.

How should failed environments count in an evaluation?#

Retain every scheduled trial and classify failures. Report end-to-end completion over all scheduled trials, plus code success conditional on valid execution when useful. Apply a predeclared rerun policy equally across configurations, and never silently discard inconvenient failures.

Does this article include measured agent benchmark results?#

No. It provides a protocol, manifest, rubric, and empty results ledger. The examples explain how to collect evidence; no agent was run through this repository evaluation and no comparative scores are claimed.