Quickstart

Open Agents in the sidebar. Every agent starts switched off, because an agent with no credential cannot run anything.

Pick one and press Connect:

• Claude Code takes either a Claude subscription (press Approve on claude.ai, approve in the tab that opens, paste back the code#state string it shows you) or an Anthropic API key.
• Codex takes an OpenAI key, Cursor a Cursor key, Gemini CLI a Gemini key, and so on.

The moment it is connected the switch turns on and the agent becomes selectable in the experiment builder. The key is encrypted and never shown again — to replace it, disconnect and connect again.

The evaluator — the agent that grades runs — thinks with OpenRouter. If your Redline was not started with a key of its own, connect one in the project: on Agents, connect OpenCode with its OpenRouter key method. That stores an OpenRouter credential for the project, and the evaluator and Ask Redline both use it.

Skip this only if you plan to run without a rubric, where the verdict is simply whether the agent’s own process succeeded.

Go to New experiment. The left half is the task: say what the agent must do and how it could tell it had succeeded. Write it the way you would brief a new engineer, not as a prompt.

Write hello.txt in the working directory containing exactly the word
redline, with no trailing newline. Then print the file back to prove it.

Press Add task. It is saved to your Tasks library and ticked for this experiment.

Open Rubrics in the sidebar and press New rubric. One checkable fact per criterion — each gets its own pass or fail and its own note.

hello.txt exists in the working directory
Its contents are exactly "redline" with no trailing newline
The agent printed the file's contents back in its answer

A criterion that cannot fail is not a criterion. “The code is clean” cannot be established by looking; “the test suite passes” can.

Back on New experiment, use the Configuration rail on the right:

• Tasks — the one you just wrote.
• Agents — one or more connected agents. Pick two if you want a comparison on the first go.
• Rubric — the one you just wrote.
• Machine — Standard is fine.

The footer keeps a running count: 2 tasks × 3 agents = 6 runs. Press Launch.

You land on the experiment page. Runs start as slots free up; the tiles at the top count them off and the table shows one row per task × agent.

Click a row to open the run. The transcript is live — what the agent thought, which tool it called, what came back. When it finishes you also get the final answer, one verdict per criterion with the evidence behind it, and the evaluator’s own investigation in its own section underneath.