Reliability of the roles

codev’s reviewing roles are measured, not assumed. This page shows how they scored the last time the evaluation suite ran, on the run committed in the repository as its baseline. It is generated from that baseline and changes with it.

What the suite measures

The suite is a set of small projects, each with a planned change. A seeded case hides defects a role should find, each with its severity, its location and its category of issue; a clean control hides none. Claude Code runs one skill step on each case with a single role, and the suite scores the review record the role writes: the defects found and missed, the findings that match no defect (false positives), whether each quoted evidence is really at the lines it cites, and whether the severity agrees with the expected one. Measuring the roles gives the rules.

What it does not measure

It is not a benchmark of models, and not a promise that a role finds every defect of your project. It does not measure how useful a correct finding is to you. The plan is measured only as a whole, on the planning cases below, by what it modifies, not role by role. A score describes these cases, on this model, on the day of the run.

The run

• Date: 2026-10-15
• Model: Claude Code’s default
• Claude Code: 2.1.289
• codev: 0.6.1
• Cases: 65 (25 seeded, 15 clean controls, 25 planning)
• Repetitions of each case: 5
• Cost: 276.46 USD for the 5 runs of each case (55.07 USD for one run of the suite)

At a glance

Recall

1.0039 of 39 seeded defects found

Evidence validity

1.0069 of 69 quotes found at the cited lines

False positives

90.2 per case; a role fails above 2

Roles within the thresholds

8 of 9▲ 1 fail at least one

The run fails its own thresholds. 1 of 9 roles report more than 2 false positives: skeptic. Precision is the roles’ weak point: they report plausible issues that the case does not hold. Reducing these false positives without losing what the roles find is the next work planned on the roles. Every seeded defect was found: on this model, recall does not tell the roles apart, and the differences show in false positives and severity agreement.

False positives per role

Recall and severity agreement per role

Every role

The table holds every value the charts draw. Cases: seeded and clean controls; found: seeded defects found over those expected; evidence: the share of checked quotes found at the lines they cite; severity: found defects whose severity is the expected one.

Planning

25 planning cases replay a request against the repository as it was before an archived change, and score the plan /codev-propose writes against what that change really did: the primary recall is the share of the requirements the request is about that the plan modifies, the collateral recall the share of those the real change modified in passing; a probable duplicate is a requirement added where one should have been modified. Each figure is the median over the repetitions, with its range.

What a run consumes

• One run of the suite: 55.07 USD
• Input per call, in tokens: 63465
• Output per call, in tokens: 696

Every call to the model is recorded. The tokens a call writes to the cache — what entered its context since the previous call, the bulk of a run’s cost — are shared among what brought them in: the skill read, each codev subcommand, the files, the searches, the model’s own output; prompt is Claude Code’s own prompt and the request. Per case the median over its runs, summed over the cases.

The thresholds

A role fails when its recall is under 0.50, when it reports more than 2 false positives, or when less than 0.80 of its checked evidence is valid — the suite’s defaults, --min-recall, --max-false-positives and --min-evidence. A role with no seeded case has no recall, and one with no checked finding no evidence validity: neither fails on it.

• ▲ skeptic fails: false positives 4 > 2
• ✓ Within every threshold: accessibility, migration, newcomer, ops, performance, privacy, security, tracer

The limits

• 5 runs per case. A model does not answer the same way twice: each score is the median of 5 runs of every case, and a change is compared with the range they span, on the same model, not across days or models.
• Only what the cases cover. 65 cases, 25 of them seeded with 39 defects. A kind of defect that no case seeds is not measured: a recall of 1.00 means these defects were found, not every defect.
• A controlled context. Each case is a small project reviewed by one role alone, within a fixed budget and time. Your project is larger, and the roles review it together.
• Not a guarantee. The scores describe the roles on this model and this version of Claude Code. Another model may score differently. Read what the roles report before acting on it.
• Planning replays a public history. The planning cases replay requests against codev’s own past states, which the model may have read: their figures compare two versions of the skills on the same cases, they do not tell how well codev plans a project it has never seen.

Run it yourself

From a clone of the repository, with Claude Code installed and signed in. A run calls a model: this one cost 276.46 USD for the 5 runs of each case (55.07 USD for one run of the suite).

cargo build --bin codev
cargo run -p codev-evals -- run --out target/evals/mine
cargo run -p codev-evals -- score target/evals/mine

The observations, the reports and the run’s metadata are in evals/baseline. The contributing guide explains how a change to a role is measured.