How a run works
Everything that happens between pressing Launch and seeing a score.
A run is executed in a fixed order, and the order is the design. Knowing it makes the transcript readable and makes most failures diagnosable at a glance.
The order of operations
The credential is resolved first
Before an image is built or a container is started, Redline checks that the agent for this run has a connected credential in this project. Failing five minutes into a setup for a reason that was knowable in the first second is the most expensive kind of error.
The transcript says which one it used: Using your Anthropic credential.
The machine is provisioned
A container of the shape the experiment chose, from the image the agent’s
CLI is installed in. It is given a handful of environment variables that
tell the agent nobody is watching — CI=1, TERM=dumb, NO_COLOR=1 — so
it behaves like an unattended process rather than a terminal session.
The attachments are made real
Everything the experiment attached becomes something on disk before the agent’s first token: repositories cloned, uploads written, MCP servers configured, skills installed, CLIs and packages installed, plugins fetched and linked. Each of these is a labelled step in the transcript.
A summary of all of it is written to CONTEXT.md in the workspace and
appended to the task’s instructions, so the agent knows what it has been
given without having to discover it.
A step that is required — a clone, an upload — stops the run if it fails. A run whose repository never arrived is not a run about the agent.
The agent runs
Its own CLI is invoked with the task as an argument, never concatenated into a shell string, so a task containing a quote or a semicolon is data and never syntax. Everything it writes is parsed as it arrives and appears in the transcript live.
A run that exceeds the time limit has its machine killed. The default limit is twenty minutes per attempt.
The machine is inspected
When the agent’s process exits, Redline takes a first look at what it left
behind — a directory listing, and git status / git diff --stat for any
repository in the workspace. This is not the verdict; it is the opening
brief handed to the evaluator.
The evaluator judges
Our own agent, on the same box, still running. It reads the record, then goes and checks: opens files, searches the workspace, runs commands. It answers every criterion in the rubric and submits verdicts with notes.
See Rubrics and judging for how it decides.
The machine is destroyed
After the evaluator, never before. Tearing the box down when the agent finishes would leave only the agent’s word for what it did, which is exactly what a rubric exists not to take.
Where a run can fail, and what happens next
Redline classifies every failure by whose it was, because that is the only thing that makes retrying sensible.
| Cause | Example | What happens |
|---|---|---|
| The agent’s | It ran and did not satisfy the rubric, or its process exited non-zero. | This is the result you asked for. It spends one trial and is retried if you have trials left. |
| The platform’s | The image build, the container, the evaluator being rate-limited. | Retried up to twice without spending a trial — the attempt counter is put back. |
| Configuration | The agent has no connected credential; the evaluator has no key. | Not retried. Nothing about it resolves in four seconds; the run stops and says exactly what is missing. |
When a run is requeued, its verdicts are cleared first. A criterion the last attempt failed has not been judged for this one, and a stale verdict must never decide a fresh run.
Runs by your own agent
An agent connected with redline dev runs in your process, so the shape is
slightly different:
- Redline does not install anything for it and does not drive a CLI. Your worker claims the run and calls your function.
- Skills reach it as text in the brief. MCP servers reach it as tools, if your
agent asks for them with
redlineTools(task). - If the experiment attached something that has to physically exist — a
repository, an upload, a CLI, a package, a plugin — the run still gets a
container, and your agent reaches it through the
machine_runtool. See Machines. - The evaluator has no box to open unless one was built, so for a plain SDK run it judges the record and the final answer closely instead.