A coding agent is useful when its changes survive a check of the resulting repository. A fluent explanation, a plausible patch, and a passing run are different observations. An evaluation should say which one it measured.
Define the finish line before the run
SWE-bench made repository issues an executable evaluation: an agent receives an issue and codebase, edits the code, and its patch is checked against tests. This is a stronger question than asking whether a model can suggest code in isolation. It still depends on the quality of the issue, tests, and environment. OpenAI’s SWE-bench Verified review found ambiguous issues, overly narrow tests, and setup problems in the original pool; the resulting subset contains 500 human-screened tasks.
For a team’s own workload, write acceptance checks before seeing an agent’s answer. Include the expected behavior, regression tests, build or type-check requirements, and any review that cannot be reduced to a test. Keep the task’s starting commit and hidden checks fixed. Record a pass only when the final artifact clears that gate. A paper on apparently solved SWE-bench issues illustrates why this matters: tests can accept patches that still diverge from the intended behavior. That study is evidence about the evaluated patches, not a universal error rate for every benchmark.
Compare complete configurations
An observed score belongs to a configuration: model version, agent instructions, tools, sandbox image, time limit, retry rule, and grader. Change several of these at once and the difference cannot be assigned to one component. Run candidate configurations on the same task set, with the same starting state and verification rule. Preserve logs for failures and reruns so that a broken environment is visible instead of silently counted as model behavior.
The published Graff and FrontierHarness comparison is useful here because it separates terminal tasks from patch-verification tasks and states where instructions, model, and runtime differ. Those rows describe recorded configurations; they are not a controlled estimate of the effect of changing only the harness. A public leaderboard is a starting point for choosing candidates, not the final answer for a different workload.
Report the cost of finishing
For each task, retain completion, model spend, tool or sandbox spend, elapsed time, retries, and failure type. Report the number of verified completions over all assigned tasks. For cost per completion, divide total spend across all attempts, including failures, by verified completions; otherwise a cheap-looking system can hide expensive misses. Show the task count and denominator beside every rate. Latency needs a distribution, including median and slow-tail time, since a few timeouts can dominate the experience of using an agent.
Finally, split results by task class—small bug fix, cross-file change, test repair, dependency work—without selecting classes after seeing the outcome. Keep a held-out slice for routing decisions. The most useful result is not a single winning agent: it is a repeatable account of where each configuration completes work, what it consumes, and where confidence is still thin.
Sources
- Jimenez et al., SWE-bench — repository issue benchmark and test-based grading.
- OpenAI, SWE-bench Verified — task review, construction, and limitations.
- Wang et al., Are “Solved Issues” Really Solved Correctly? — patch validation study.
- Codegraff, Graff on FrontierHarness — recorded conditions and comparison limits.
- Codegraff, Verify code changes — a practical record for reviewing agent patches.