Experiments and runs
The cross product, what you can change while it runs, and what every status means.
The cross product
An experiment is every selected task, attempted by every selected agent. Two tasks and three agents is six runs, and the builder shows that count in its footer as you tick things.
All of them are created and queued the moment you press Launch — all of them, or none: a launch that is refused leaves nothing behind. Nothing has started yet; runs are picked up as capacity allows. But the full set exists from the first second, which is what makes the progress honest: 12 of 48 means the same thing five minutes in as it does at the end.
Launching
Everything is checked before a single row is written, so a rejected launch leaves nothing behind:
- every task, asset, file, rubric and machine must belong to this project
- every catalog agent must be connected and enabled
- every one of your own agents must be online —
redline devconnected — because nothing else can execute it
If a rubric is chosen, its criteria are read once at launch and copied onto every run. Editing the rubric tomorrow cannot rewrite what these runs were judged against.
What you can change while it runs
Two settings stay live on the experiment page while it is queued or running:
| Setting | What it does |
|---|---|
| Attempts each (trials, 1–10) | How many times a failing run may be tried before it is called a failure. |
| At once (max concurrency, 1–32) | How many of this experiment’s runs may occupy the machine simultaneously. Per experiment, so a long sweep cannot starve a quick check launched beside it. |
Changes apply to runs that have not finished.
Statuses
A run
| Status | Means |
|---|---|
queued |
Created, waiting for a slot. |
running |
An agent is working on it. |
judging |
The agent has finished; the evaluator is reading the record and checking the machine. |
passed |
Judged, and every criterion was met. |
failed |
Judged, and at least one criterion was not. |
errored |
The run did not complete — a crash, a missing credential, a timeout, a required setup step that failed. |
cancelled |
Stopped by hand. |
judging is a live state, not a finished one. A run by your own agent is
reported complete by your worker before our evaluator has read the record, and
calling it passed in that window would publish a verdict nobody had reached
yet.
An experiment
queued until the first run starts, running while anything of its is
outstanding, complete when nothing is. Cancelling an experiment cancels its
queued runs immediately; a run already in progress stops as soon as it notices,
which is a few seconds rather than instant.
Scoring
A run’s score is the share of its criteria that passed — criteria met over criteria judged, as a percentage. It is derived from the verdicts rather than reported alongside them, so it cannot disagree with what is underneath it.
A run with no rubric has no score. Its verdict is whether the agent’s own process succeeded.
Reading the comparison
The experiment page answers two questions:
- Is it working? — the tiles: runs settled, pass rate, average duration, spend and tokens.
- For whom? — the By agent bars when more than one agent is in it, and the table underneath with one row per run: task, agent, attempts used, status, score, duration, cost.
Click any row to open the run.
Deleting
An experiment can be deleted when it is not running, and deleting it takes its runs with it. A schedule that produced it is unaffected, and deleting a schedule never deletes what it already launched.