Routing is the choice of a full execution path for a task: model, instructions, tools, context strategy, sandbox, budget, and retry policy. The right route is the one that finishes the work under the constraints that matter for that task.
Describe the task before choosing the route
Classify a request using information available at dispatch time: repository and language, estimated change scope, available tests, dependency setup, whether commands must run, and the cost of an incorrect answer. A one-file documentation edit and an ambiguous cross-file bug may warrant different budgets. Do not classify a task using the agent’s eventual success; that leaks the answer into the routing decision.
SWE-bench shows why repository issues need more than code generation: an agent must inspect a codebase, change it, and satisfy a verifier. Its task mix is still narrower than a particular organization’s work. Use public benchmarks to build a candidate set, then test those candidates on held-out examples of the work you actually assign.
Measure the complete route
For each task class, measure verified completion first. Then compare total spend per completion and wall time, including the retry and failure paths. A cheaper first attempt can be more expensive after repeated misses. Track the median and slow-tail latency, since users experience the full wait and not an average token rate. Apply a budget ceiling before dispatch and specify when to stop, retry with the same route, or escalate to a stronger one.
Execution conditions are part of the route. SWE-ReX documents a sandboxed interface for running agent commands in reproducible environments. The lesson for routing is to record the environment image, command access, setup failures, and timeouts with the model choice. A task that failed because dependencies could not install should be marked as an infrastructure fault and inspected separately; silently treating it as a model miss corrupts the comparison.
Keep the comparison honest
Log the task identifier and start commit, route version, verification result, elapsed time, spend, retry count, and failure category. Hold the task and grader constant when testing route alternatives. The Graff and FrontierHarness comparison publishes conditions that differ across some rows, including instructions and runtime. That makes it useful evidence about those configurations, but not proof that one changed component caused every observed difference.
Route on enough tasks to see uncertainty. Show both counts and rates, and avoid promoting a class-specific winner after a handful of examples. Keep a held-out set for decisions, rerun it when a model or harness version changes, and watch for regressions in correctness and cost. If the evidence is thin, choose a conservative default and collect more outcomes. A routing policy should be a versioned, testable decision rule that can improve as the workload changes.
Sources
- Jimenez et al., SWE-bench — repository issue tasks and executable grading.
- SWE-agent, SWE-ReX — sandboxed command execution interface.
- Codegraff, Graff on FrontierHarness — configuration details and comparison limitations.
- Codegraff, Graff CLI guide — an example of inspecting a model route before execution.