Rubrics

The standard a run is measured against, written once and pointed at.

A rubric is a name and a list of criteria. It lives in your library rather than on a task, because “the diff is minimal and the tests prove it” is a standard you want to apply to twenty different tasks — and two copies of a standard are two standards.

Writing one

On Rubrics, press New rubric. Give it a name short enough to pick from a list, then one check per box. Enter on the last box adds another.

The checkout spec passes twenty runs in a row
No bare timeout or sleep was added to the spec
The diff touches only the spec and the code it covers

Each criterion is scored on its own, so each one should be a single fact. See Rubrics and judging for what the evaluator can and cannot establish, and how to phrase a criterion so it can fail.

How it reaches a run

A rubric is chosen on the experiment, at launch — not on a task. At that moment its criteria are read once and copied onto every run in the experiment.

Leaving the rubric as None is a valid choice: the run is then judged only on whether the agent’s own process succeeded, has no score, and needs no evaluator key.

Reading the page

Each rubric card lists its criteria in order, which experiments have used it, and how many runs it has judged. Any rubric no experiment has used yet is called out underneath the list, with a link to the builder — an unused rubric is usually one somebody wrote and then forgot to pick.