---
title: Experiments and runs
description: The cross product, what you can change while it runs, and what every status means.
---

## The cross product

An experiment is *every selected task, attempted by every selected agent*. Two
tasks and three agents is six runs, and the builder shows that count in its
footer as you tick things.

All of them are created and queued the moment you press Launch — all of them, or
none: a launch that is refused leaves nothing behind. Nothing has started yet;
runs are picked up as capacity allows. But the full set exists from the first
second, which is what makes the progress honest: *12 of 48* means the same thing
five minutes in as it does at the end.

> **Note**
>
> Trials do **not** multiply the count. Three trials on six runs is still six
> runs; it is permission to attempt each of them up to three times.

## Launching

Everything is checked before a single row is written, so a rejected launch
leaves nothing behind:

- every task, asset, file, rubric and machine must belong to this project
- every catalog agent must be connected and enabled
- every one of your own agents must be **online** — `redline dev` connected —
  because nothing else can execute it

If a rubric is chosen, its criteria are read once at launch and copied onto every
run. Editing the rubric tomorrow cannot rewrite what these runs were judged
against.

## What you can change while it runs

Two settings stay live on the experiment page while it is queued or running:

| Setting | What it does |
|---|---|
| **Attempts each** (trials, 1–10) | How many times a *failing* run may be tried before it is called a failure. |
| **At once** (max concurrency, 1–32) | How many of *this experiment's* runs may occupy the machine simultaneously. Per experiment, so a long sweep cannot starve a quick check launched beside it. |

Changes apply to runs that have not finished.

> **Everything else is frozen**
>
> The tasks, the agents, the attachments, the rubric and the machine shape
> cannot be changed after launch. Those runs were produced under a particular
> configuration, and a page whose settings no longer explain its own numbers is
> a page nobody can read.
>
> To change any of it, press **Edit & re-run**. That opens the whole
> configuration in the builder, ready to change; launching from there creates a
> *new* experiment and the original keeps its results.

## Statuses

### A run

| Status | Means |
|---|---|
| `queued` | Created, waiting for a slot. |
| `running` | An agent is working on it. |
| `judging` | The agent has finished; the evaluator is reading the record and checking the machine. |
| `passed` | Judged, and every criterion was met. |
| `failed` | Judged, and at least one criterion was not. |
| `errored` | The run did not complete — a crash, a missing credential, a timeout, a required setup step that failed. |
| `cancelled` | Stopped by hand. |

`judging` is a live state, not a finished one. A run by your own agent is
reported complete by your worker before our evaluator has read the record, and
calling it *passed* in that window would publish a verdict nobody had reached
yet.

### An experiment

`queued` until the first run starts, `running` while anything of its is
outstanding, `complete` when nothing is. Cancelling an experiment cancels its
queued runs immediately; a run already in progress stops as soon as it notices,
which is a few seconds rather than instant.

## Scoring

A run's score is the share of its criteria that passed — criteria met over
criteria judged, as a percentage. It is derived from the verdicts rather than
reported alongside them, so it cannot disagree with what is underneath it.

A run with no rubric has no score. Its verdict is whether the agent's own
process succeeded.

## Reading the comparison

The experiment page answers two questions:

- **Is it working?** — the tiles: runs settled, pass rate, average duration,
  spend and tokens.
- **For whom?** — the *By agent* bars when more than one agent is in it, and the
  table underneath with one row per run: task, agent, attempts used, status,
  score, duration, cost.

Click any row to open [the run](/results/run).

## Deleting

An experiment can be deleted when it is not running, and deleting it takes its
runs with it. A schedule that produced it is unaffected, and deleting a schedule
never deletes what it already launched.