BenchBeta

Bench runs a suite of real tasks against a baseline and a candidate. Your tests decide each trial, and Bench reports the pass rates, the cost and how far the difference can be trusted. It reports what happened and never calls a change better or safe.

Bench is in beta. It runs on your machine or in your CI, through the vendors’ own CLIs, with your sign-in or your API keys.

Write a Suite

cedr bench init

This writes .cedr/bench/suite.toml. Edit it to describe what you are comparing and the tasks to try:

version = 1
trials = 3

[baseline]
harness = "claude"
model = "claude-sonnet-5-5"
ref = "main"

[candidate]
harness = "claude"
model = "claude-sonnet-5-5"
ref = "HEAD"

[[tasks]]
id = "fix-parser"
prompt = "Fix the failing parser test."
setup = "npm ci"
test = "npm test -- parser"
test_files = ".cedr/bench/fix-parser"

To test a change to your rules or skills, compare two git refs. To test a model, change model or effort. Only committed work is measured, and Bench needs the repo’s full history: it refuses a shallow clone.

An agent can edit the tests it is judged by. Keep your tests in test_files and they are copied over its work before test runs.

Each step’s leftover processes are stopped when it ends on Linux and macOS, so a server started in setup is gone by test. Start it in the test command.

A suite is code. Its commands run with your permissions, so run only suites you trust. cedr bench plan shows what a suite would do without starting anything.

Run It

cedr bench plan
cedr bench run

bench suite: complete, $1.27 of $2.00 (b-20261101T193000Z-1a2b3c)
              passed  rate   95%          median  cost
  baseline    2/3     67%    21%-94%      41s     $0.12
  candidate   3/3     100%   44%-100%     38s     $0.09
  fix-parser  baseline 2/3, candidate 3/3
pass rate +33 points; the intervals overlap, so this run does not separate them

Overlapping intervals mean the run does not tell the two apart, not that they are equal. More trials narrow them. Each result also records the models, CLI versions, commits and a digest of your agent files, so you can tell what was measured.

cedr bench history lists saved runs and cedr bench show <id> prints one again. Results stay on your machine.

Press Ctrl-C to stop a run on Linux or macOS: the agent stops too, and what the run measured so far is saved. On Windows, Ctrl-C ends the run without saving it.

Command	Options
cedr bench init	--name NAME writes .cedr/bench/NAME.toml instead of suite.toml
cedr bench plan [SUITE]	--task ID, --trials N, --format text|json
cedr bench run [SUITE]	--budget-usd N, --task ID, --trials N, --fail-on never|lower|separated, --allow-unpriced, --no-record, --format text|json
cedr bench history	--limit N, --format text|json
cedr bench show <ID|PATH>	--fail-on never|lower|separated, --format text|json
cedr bench budget [USD]	--unset, --pull, --org ORG, --config PATH, --format text|json

bench run and bench show exit 0 when the run is complete and the --fail-on condition did not hold, 1 when it held, 3 when the run is incomplete, whatever the counts say, and 2 on a usage, configuration or file error.

Set a Budget

Every run has a cap in US dollars at API prices, including on a plan where you are not billed per call. The default is $2.00.

cedr bench budget 5        # set it on this machine
cedr bench budget --unset  # back to the default
cedr bench run --budget-usd 10   # for one run

Owners and admins can also set a budget for the organization on the Settings page of the app. Machines and CI pick it up with cedr bench budget --pull --org <organization ID>.

A run stops before a trial that could pass the cap, and reports itself as incomplete. Claude Code is also held to what is left during a trial. Other agents are checked between trials, so a trial can go over by what it spends before its timeout. An agent stopped at its timeout has an unknown cost, so the run stops there too unless you passed --allow-unpriced.

Use Other Agents

claude and codex are built in. Run any other agent with the command harness:

[candidate]
harness = "command"
model = "my-model"
command = ["my-agent", "--model", "{model}", "--prompt-file", "{prompt_file}"]

The budget cannot rely on what these spend, so a run with one always needs --allow-unpriced. If the command’s last line of output is JSON such as {"cost_usd": 0.12}, that cost is counted and reported. The same flag is needed for a codex model Bench has no price for. A relative path in command starts from the repo root.

Run It in CI

- uses: actions/checkout@v5
  with:
    fetch-depth: 0
- uses: Sheernox/cedr@vX.Y.Z
  with:
    bench: .cedr/bench/suite.toml
    bench-budget-usd: "5"
    bench-fail-on: lower
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Install the agent’s CLI in an earlier step, and use API keys rather than a signed-in plan. lower fails the check when the candidate passes less often; separated fails it only when the intervals do not overlap as well. An incomplete run always fails. Set bench-allow-unpriced: "true" for a variant that needs --allow-unpriced.

Never point this at a suite from an untrusted pull request.