Rubrics and judging
Evidence, or it did not happen.
A run is judged by an agent, not a checklist
When the agent’s process exits, a second agent takes over: our evaluator. It is the same kind of thing as the agent it is grading — a model, a set of tools, and a loop that keeps going until it has an answer — because a single completion over a pasted transcript can only grade what the agent said, and the entire point of a rubric is to find out what it did.
So the evaluator gets the machine, still running, exactly as the agent left it.
What it can look at
Two things, and it can go through both as many times as it needs:
The record. Everything the agent produced — its reasoning, every tool call, what each call returned, and its final answer. It can search that record and open any single entry in full, which matters when a verdict turns on the exact content of something the agent wrote.
The machine. The same box, as the agent left it. It can list directories, read files, search the workspace and run commands there — including the project’s own tests.
Its investigation is bounded: if it has not answered after forty steps of looking, it must decide on what it has established. A criterion it could not establish fails, with a note saying what was left unproven. It never runs out of room and silently passes something.
The rules it works under
Evidence or it did not happen
The agent’s account of its own work is a claim, including a confident one with a plausible explanation attached. A criterion passes on something the evaluator saw: a tool call with a real result, a file it opened, a command it ran.
Go and look
If a criterion is about a file, open it. About behaviour, run it. About a change, diff it. If the project has its own tests, run them — the machine is destroyed the moment judging ends, so a test suite costs nothing and settles the question completely.
No partial credit
Each criterion is true or false. If it is borderline, it is false: a rubric of near-misses cannot compare two agents, and comparing agents is what this is for.
Judge what the criterion asks
Some criteria are about the result and some are about the process — which tools it reached for, whether it verified its own work, whether it took a shortcut that happens to look right. The criterion is read as written, and standards it does not state are not imported.
Be fair about what was possible
If the task or the machine made a criterion unachievable — a missing credential, a repository that could not be cloned — it still fails, but the note says plainly that this is why. A failure the agent caused and a failure it walked into are different findings.
What you get back
- One verdict per criterion, pass or fail, each with a note citing the concrete thing that was found — a path, a command, a line of output. Notes never restate the criterion back at you.
- A summary of what the agent actually did.
- The evaluator’s own transcript, in its own section on the run page: every question it asked, every command it ran, and what came back. Its verdicts are auditable rather than pronounced.
- A score: criteria passed over criteria judged.
A criterion the evaluator skipped is failed with the reason stated, so a score can never be flattered by an answer that ran out of room.
Writing criteria that work
The difference between a useful rubric and a useless one is whether a criterion can be established by looking.
| Write this | Not this |
|---|---|
| hello.txt contains exactly “redline” | The file is correct |
| The checkout spec passes twenty runs in a row | The flake is fixed |
| node_modules exists inside the plugin’s directory | The plugin is installed properly |
| The diff touches only the spec and the code it covers | The change is minimal |
| The agent called the search tool before answering | The agent used its tools well |
Few and checkable beats many and vague. Five criteria that each name one fact will tell you more about two agents than twenty that each need interpreting.
Running without a rubric
Leaving the rubric as None is a legitimate choice, and the builder says what it means: the run is judged only on whether the agent’s own process succeeded. No evaluator runs, no key is needed, and the run has no score — useful for smoke-testing a new agent connection before you care about quality.
What the evaluator thinks with
An OpenRouter key. Either the one this Redline was started with, or one connected in the project — connect it on Agents using OpenCode’s OpenRouter key method. Ask Redline uses the same key.
If neither exists, a run with a rubric stops with errored and says exactly
that, rather than guessing at a verdict.