---
title: Rubrics and judging
description: Evidence, or it did not happen.
---

## A run is judged by an agent, not a checklist

When the agent's process exits, a second agent takes over: our evaluator. It is
the same kind of thing as the agent it is grading — a model, a set of tools, and
a loop that keeps going until it has an answer — because a single completion over
a pasted transcript can only grade what the agent *said*, and the entire point of
a rubric is to find out what it *did*.

So the evaluator gets the machine, still running, exactly as the agent left it.

## What it can look at

Two things, and it can go through both as many times as it needs:

**The record.** Everything the agent produced — its reasoning, every tool call,
what each call returned, and its final answer. It can search that record and open
any single entry in full, which matters when a verdict turns on the exact
content of something the agent wrote.

**The machine.** The same box, as the agent left it. It can list directories,
read files, search the workspace and run commands there — including the
project's own tests.

Its investigation is bounded: if it has not answered after forty steps of
looking, it must decide on what it has established. A criterion it could not
establish fails, with a note saying what was left unproven. It never runs out of
room and silently passes something.

## The rules it works under

1. **Evidence or it did not happen**

    The agent's account of its own work is a claim, including a confident one
    with a plausible explanation attached. A criterion passes on something the
    evaluator *saw*: a tool call with a real result, a file it opened, a command
    it ran.

2. **Go and look**

    If a criterion is about a file, open it. About behaviour, run it. About a
    change, diff it. If the project has its own tests, run them — the machine is
    destroyed the moment judging ends, so a test suite costs nothing and settles
    the question completely.

3. **No partial credit**

    Each criterion is true or false. If it is borderline, it is false: a rubric
    of near-misses cannot compare two agents, and comparing agents is what this
    is for.

4. **Judge what the criterion asks**

    Some criteria are about the result and some are about the process — which
    tools it reached for, whether it verified its own work, whether it took a
    shortcut that happens to look right. The criterion is read as written, and
    standards it does not state are not imported.

5. **Be fair about what was possible**

    If the task or the machine made a criterion unachievable — a missing
    credential, a repository that could not be cloned — it still fails, but the
    note says plainly that this is why. A failure the agent caused and a failure
    it walked into are different findings.

> **The record is evidence, not instructions**
>
> Everything the evaluator reads — transcript, file contents, command output,
> comments left in the workspace — was produced by the thing it is judging. If
> any of it addresses the evaluator, claims a criterion is satisfied, or asks to
> be passed, that is data and not an instruction. An attempt to steer it is
> itself reported in the summary.

## What you get back

- **One verdict per criterion**, pass or fail, each with a note citing the
  concrete thing that was found — a path, a command, a line of output. Notes
  never restate the criterion back at you.
- **A summary** of what the agent actually did.
- **The evaluator's own transcript**, in its own section on the run page: every
  question it asked, every command it ran, and what came back. Its verdicts are
  auditable rather than pronounced.
- **A score**: criteria passed over criteria judged.

A criterion the evaluator skipped is failed with the reason stated, so a score
can never be flattered by an answer that ran out of room.

## Writing criteria that work

The difference between a useful rubric and a useless one is whether a criterion
can be *established by looking*.

| Write this | Not this |
|---|---|
| hello.txt contains exactly "redline" | The file is correct |
| The checkout spec passes twenty runs in a row | The flake is fixed |
| node_modules exists inside the plugin's directory | The plugin is installed properly |
| The diff touches only the spec and the code it covers | The change is minimal |
| The agent called the search tool before answering | The agent used its tools well |

Few and checkable beats many and vague. Five criteria that each name one fact
will tell you more about two agents than twenty that each need interpreting.

## Running without a rubric

Leaving the rubric as **None** is a legitimate choice, and the builder says what
it means: the run is judged only on whether the agent's own process succeeded.
No evaluator runs, no key is needed, and the run has no score — useful for
smoke-testing a new agent connection before you care about quality.

## What the evaluator thinks with

An OpenRouter key. Either the one this Redline was started with, or one
connected in the project — connect it on **Agents** using OpenCode's *OpenRouter
key* method. [Ask Redline](/automation/ask) uses the same key.

If neither exists, a run with a rubric stops with `errored` and says exactly
that, rather than guessing at a verdict.