How do you see what a travel booking agent will do before it books anything real?

What a travel booking agent actually changes, and where

A reply like "I've booked you on the 08:40 to Amsterdam and moved the hotel to the 14th" is the last step of the work, not the work. Anthropic's guide to agent evals draws that line exactly: a transcript is "the complete record of a trial, including outputs, tool calls, reasoning, intermediate results", while "the outcome is the final state in the environment at the end of the trial" — "a flight-booking agent might say 'Your flight has been booked' at the end of the transcript, but the outcome is whether a reservation exists in the environment's SQL database."

For a booking agent, that means six or seven separate things have to be right at once:

Behind all of it sits the policy: cabin class, budget, advance-purchase window, who may approve, which hotel rate is refundable. None of that is visible in the wording of a reply.

Two properties make travel harder to eyeball than most agent work. Money moves: an order is created with a payment of type balance or card, and a cancellation returns a refund to balance, card or airline credits — Duffel documents all three and its test environment has a route for each. And search is not stable: "Flight search results can change without notice, even for the same routes on the same dates," Duffel warns, so today's case is not tomorrow's unless the environment is pinned or reset.

What Duffel's test mode gives you, and what it leaves out

Duffel's test mode is a genuine sandbox and the right first place to build. Its own description: "a 'sandbox' which lets you use the Duffel API risk-free, with no danger of spending any money or booking flights you don't want." You create a testing access token (duffel_test_…), and "with a testing access token, you'll only be able to access resources created in test mode" — test and live resources are separate worlds.

Duffel also publishes scenario routes, so a specific behaviour is reproducible instead of hoped for. Each of these is a city pair you search in test mode:

Duffel is honest about the limits of its own sandbox, and they matter for anyone testing an agent rather than an integration:

• Many airline sandboxes are outside Duffel's control. "While using the Duffel API in test mode, you'll likely be interacting with airlines' sandbox environments," its documentation says, and "we are not able to provide strong guarantees about their behaviour" — an airline sandbox "might be out of action due to maintenance, or flight availability might be 'used up' due to other users' bookings." That is correlated failure across teams, the kind of shared state Anthropic's eval guidance warns inflates or depresses results.
• The reliable airline is not realistic. Duffel Airways (ZZ) is the sandbox airline Duffel guarantees, and the trade-off is stated plainly: "you won't see realistic flight schedules or prices."
• Cards are Duffel's own. "You cannot use real card details when confirming a Payment Intent in test mode"; test cards are listed, "only for use with Duffel Payments".
• It is Duffel's world, not your company's. The token reaches test resources. Your finance policy, the employee who is travelling, their mailbox, their calendar and their loyalty preferences are not in it, and nothing in test mode tells you whether your agent picked a cabin your company allows.

Booking.com's sandbox, and the one requirement that decides how far you can go

Booking.com does publish a non-production environment for the Demand API. "The Demand API Sandbox allows you to safely experiment with accommodation and attractions API collections, so you can validate search and availability, and bookings, modifications and cancellations for accommodation orders without affecting live production data." It "mirrors the structure of the live production environment", uses your regular credentials rather than sandbox-specific keys, and sits at https://demandapi-sandbox.booking.com/3.2 (matching the version you integrate with).

It also ships scenario inventory, which is the useful part: named test accommodation IDs, each with its own testing purpose — "Demand API Sandbox Hotel Orion" for "multiple rate plans, meal options, per-person extra charges, and different prepayment requirements", "Ray's Lyon Property" for "low-availability behaviour", "Demand API Test Hotel" for multi-room bookings, and others covering non-refundable rates, minimum-stay restrictions and pay-at-property. Its documented limitations tell you where the sandbox stops:

• Payments need a real card. "A real credit card is required for testing accommodation booking flows including payments. Charges are temporary and automatically cancelled every Monday; refunds are processed automatically." That is exactly the thing an agent test is supposed to avoid, and Booking.com is explicit that the workaround is a live card and a weekly sweep.
• Accommodation only. "Reservations - Only supported on accommodation sandbox inventory."
• Cars are out. "Car rental API collection is currently not available in sandbox. Use the production environment instead."
• Rate limited. "Maximum 50 requests per minute" in the sandbox; against the API generally, exceeding your account's limit returns "429 - Too many requests".
• Test inventory is test-only. Sandbox accommodation or attraction IDs used in production "will return an error or empty results".

Like Duffel's, this is the right environment to validate your own Booking.com integration against the provider's real answers. What it does not hold is your company: not your travellers, your negotiated rates, your travel policy, your approvers, or the mailbox and calendar the confirmation goes to.

Mocked APIs and graded transcripts: what each one can and cannot see

Mocked flight and hotel APIs

A mock is a server you write or record that answers in place of the provider. WireMock's documentation describes both mechanisms: record-and-playback, where "recorded stubs will start being served immediately after recording is stopped", and scenarios, where "a scenario is essentially a state machine whose states can be arbitrarily assigned" and can be reset.

This is the cheapest way to test your own code: the adapter that builds an offer request, the retry wrapper around a 202 that resolves later, the branch when an order creation fails while the seat is held. It runs in CI in seconds with no credentials.

What a mock cannot do is tell you whether the agent's work was correct. The order it "creates" is a stub, so no order exists to contradict a claim that the booking went through. Search results do not change between runs the way Duffel warns they do. Nothing has history — was this trip already booked? was this hotel already cancelled? — and nothing checks the answer against the provider's published schema, so a stale stub keeps passing after the API moves.

Grading the transcript

Transcript grading scores what a run did, and Anthropic's guide is the clearest public description of the practice. It recommends combining code-based, model-based and human graders, and warns against two traps that travel agents walk into:

• Do not grade the path. There is "a common instinct to check that agents followed very specific steps like a sequence of tool calls in the right order", and Anthropic has "found this approach too rigid and results in overly brittle tests, as agents regularly find valid approaches that eval designers didn't anticipate." Their own example is a travel one: Opus 4.5 solved a τ2-bench task about booking a flight by finding a loophole in the policy — it "failed" the eval as written.
• Score consistency, not just best-of. pass^k, "the probability that all k trials succeed", matters "for customer-facing agents where users expect reliable behavior every time": at a 75% per-trial success rate, three trials all pass about 42% of the time. A booking agent gets one shot at a trip, not ten.

Anthropic also asks for isolated trials: each one should start "from a clean environment", because "unnecessary shared state between runs (leftover files, cached data, resource exhaustion) can cause correlated failures", and shared state "can also artificially inflate performance". Evals give you the score; they do not build the environment the score refers to. That is the gap the other routes are about. How API replicas, mocks and provider sandboxes differ goes through the same distinction for integrations generally.

The mailbox and the calendar have no test mode

Half of a booking run happens outside the travel APIs, and this is where a team's plan usually runs out. Google publishes no Gmail or Calendar sandbox: the Gmail API documentation sets out scopes and a verification regime, not a test environment. An agent that reads threads and sends confirmations needs gmail.send, listed among the sensitive scopes, and gmail.readonly, gmail.compose, gmail.insert, gmail.modify or gmail.metadata, all listed as restricted. Restricted scopes "require restricted scope OAuth App Verification", and "if you store restricted scope data on servers (or transmit), then you must go through a security assessment".

The verification regime is the practical constraint. Google's verification requirements ask for an end-to-end demonstration video that "must show the end-to-end flow of your app including the OAuth grant process", a narrowest-scope justification ("If the requested scope(s) goes beyond the usage needed, you will be directed to request a narrower scope"), limited use of the data, and a rule that "humans are not allowed to read the data" unless the user agreed to that specific review. So the email leg of an agent test is either a real mailbox holding real travellers' mail, or a copy of the mailbox that the agent can be pointed at. Microsoft Graph mail and calendar — for teams on Microsoft 365 — has the same shape: an agent needs a real tenant, with the tenant's real meetings and messages in it.

What running the agent against copies of the APIs looks like

Redline serves copies of the provider APIs, built to answer like the real ones from each provider's published spec, with a watcher that reports when that spec or its documentation pages change. Only the data is fake. Every company gets its own copy of every app, so the records in them belong to that company alone — another company's runs cannot inflate or contaminate yours.

The copies a booking agent touches, and what the repo's own test suites report:

Two useful details for an agent that books. Duffel's documented test routes behave as documented: the copy knows the PVD → RAI no-flights route, the LCY → STN failure webhook, the LTN → SYD refund to airline credits. And the Booking.com copy picks the spec by API key — keys marked beta are served by the beta spec (which adds car order creation), the others by the stable one — and reproduces the documented sandbox rate limit, so an agent's behaviour under a 429 is something you watch instead of assume.

The records are your company's, and made up

The data agent writes a company's records as files in the company folder, one folder per app and one file per record kind, in exactly the format that app's API returns. Volumes are parameters rather than guesses. A shared world file keeps the facts the apps must agree on: the trip and its travellers, the airports and airlines, the hotels, the people with mailboxes and calendars, the meetings with attendees, the threads that refer to the booking. Everything is made-up: people and companies on .example domains, never a real person or a real company presented as the business or the traveller.

Every step is recorded, and the state can be put back

The agent sends its run to the company's project with @redlineai/sdk, whose tracing API is Langfuse's SDK: observe() wraps a model call or a tool as a span, and propagateAttributes({ sessionId }) keeps one trip's conversation together. The console shows each model call's request, the tools it called, the reply, tokens and cost per run. Rules are separate and written by you — on the site's own description of the monitor, "Write what your agent must not do. Each run is read against it, and every reply that breaks it is kept with its words and the reason." So "never book outside the approved cabin", "never pay with anything but the corporate balance" and "never cancel below 24 hours of departure" become records attached to runs rather than a line in a prompt you hope holds. Approval gates and scoped credentials covers the controls that stop an agent acting outside its remit in production.

Meanwhile POST /_twin/reset returns a copy to its seed and GET /_twin/coverage says which operations it serves, so the same case runs again from the same starting state — the isolated trial Anthropic asks for, per case rather than per refresh window. The Duffel copy also exposes its webhook deliveries and a flush endpoint, so an agent's handling of order.created or order.creation_failed can be watched.

How the routes compare

A first week of runs

1. List the actions, not the prompts. Every write the agent may make: create an order, pay, change, cancel, invite, send. Name the API each one lands in, including the ones outside travel.
2. Write the cases from real failures. Anthropic's starting point is 20-50 tasks drawn from real failures; for travel, that is missed connections, a traveller who changes plans on a Friday, a non-refundable rate, a hotel that is full at the price quoted.
3. Fill the company's records. Give the data agent the business, the domain and the apps; the world file keeps the trip, the traveller and the hotel consistent across Duffel, Booking.com, mail, calendar and chat.
4. Write the rules before you run. "Never pay with anything but the corporate balance", "never book a cabin above the policy", "never cancel without confirming", "one confirmation email per booking", "no calendar entry without the traveller as an attendee".
5. Run each case from a reset copy, with the SDK instrumenting the agent, and keep the trace.
6. Grade the state first, then the transcript. Order exists, amount, fare brand, hotel rate plan and cancellation policy, refund type, attendees, recipients, then a rubric for tone and grounding.
7. Re-run the set on every prompt or model change, so this week's result is comparable with last week's.

What the copies do not do

Fair limits, from the copies' own README and the repo's notes:

• They are not the vendors. No airline or hotel reservation is made, no card is charged, no email leaves the copy, no invitation reaches a real calendar. Keep Duffel's test mode and Booking.com's sandbox for validating your own integration against the providers' real answers, and the sandbox's payment flow for Booking.com-side payment work.
• The Booking.com copy does not serve the whole beta surface. 21 beta operations answer 501 "not in this sandbox copy yet": smart search, attractions, messages, transfers, autocomplete, car availability and terms, order modify preview, and some car order detail endpoints. Stable 3.2 is served in full.
• Some answers are not verified against signed-in traffic. Booking.com's recorded unauthenticated requests all returned the same empty 401 — including unknown paths — so the recordings cannot show which paths exist. The fee for a partially refundable product is the copy's choice (50% of the total, one of the amounts Booking.com's own policy page lists). Duffel documents no error for an undefined body field, so the copy ignores it, and what the real API answers to @duffel/api's orders.update body shape is not verified.
• Not every travel app is replicated. Redline covers Duffel and Booking.com for travel, with Gmail, Google Calendar, Microsoft Graph, Slack, Stripe and others around them. It does not replicate a GDS such as Amadeus or Sabre, an expense system, or a hotel chain's own reservation API, so an agent that books through those leaves those calls outside the run.
• The records are made up. If one real case matters — the traveller with a mobility requirement, the negotiated corporate rate — write that record into the company folder by hand rather than exporting production records.

Common questions

Can't we just point the agent at Duffel's test mode and Booking.com's sandbox?

For the provider half of the run, they are the right places, and both are documented well enough to be worth using. The gap is everything the provider cannot know: your travellers and policy, the mailbox the confirmation is sent from, the calendar the trip has to fit into, the approval in chat, the card that gets charged. Booking.com's sandbox also asks for a real credit card for accommodation booking flows, with charges swept every Monday, and does not run cars at all. Duffel's test mode is one token in Duffel's world, with airline sandbox behaviour it says it cannot guarantee and a reliable airline whose schedules and prices are not realistic.

Is a mock of Duffel and Booking.com enough if we grade the agent's calls?

No, because the calls are not the work. Anthropic's definition is that the transcript is the record and "the outcome is the final state in the environment at the end of the trial". With a mock, the order is whatever the stub says it is, so "did the agent book the right fare at the approved price" has no answer, and there is no history to contradict a claim that the trip was already changed. Mocks remain the fastest way to test your adapters and your error handling in CI.

How do we test the confirmation email and the calendar entry?

Point the agent at copies of Gmail and Google Calendar (or Microsoft Graph, for Microsoft 365 tenants) that hold the company's own mailboxes and meetings. The alternative is a real account, and Gmail's scopes make that expensive: gmail.send is classified sensitive, gmail.readonly, gmail.compose, gmail.insert and gmail.modify are restricted, restricted scopes require verification, and storing or transmitting restricted-scope data means a security assessment.

Does the run cover changes, cancellations and things the airline does?

Yes on the agent's side, and the copies reproduce the scenarios: Duffel order changes and cancellations, refunds to balance, card or airline credits, and airline-initiated changes, which Duffel's test environment creates on each API call for the LHR → LTN test route. On Booking.com's side, order modify and cancel are served. What a copy will not do is tell you how a real airline handles a disruption on the day; that is what Duffel's sandbox and, eventually, your production monitoring are for.

Do we still need evals once the agent runs against copies?

Yes. Evals are how you score decisions across many runs, catch regressions after a prompt or model change, and read transcripts when a case fails. The copies change what the score refers to: a state check against a copy that actually created an order is a claim about a booking, not about a response body. Anthropic's warning about shared state is easier to honour here too, since POST /_twin/reset puts a copy back to its seed between cases and each company has its own copy. If your agent is customer-facing, score pass^k rather than best-of runs: a trip gets one attempt.

Sources

• Duffel, Test mode — test mode as a sandbox, duffel_test_ tokens, test and live resources kept apart, airline sandboxes outside Duffel's control ("we are not able to provide strong guarantees about their behaviour"), Duffel Airways (ZZ) reliable but without realistic schedules or prices, test card numbers for Duffel Payments.
• Duffel, Test your integration — the scenario routes for offer requests, offers, orders, payments, airline-initiated changes and airline credits; 200 and 202 card responses and the 30-second webhooks; the warning that search results can change without notice.
• Booking.com, Demand API sandbox — the sandbox base URL, regular credentials, accommodation and attraction flows, the test accommodation IDs and their scenarios, and the limitations: 50 requests a minute, reservations only on accommodation sandbox inventory, cars unavailable in sandbox, and a real credit card for accommodation booking flows with charges cancelled every Monday.
• Booking.com, Rate limiting — per-account request limits and the 429 "Too many requests" response.
• Anthropic, Demystifying evals for AI agents — transcript and outcome defined, the flight-booking example, grader types, grading what the agent produced rather than the path, partial credit, isolated trials and shared state, pass^k and the 42% three-trial figure, 20-50 tasks from real failures, and the τ2-bench flight-booking loophole.
• WireMock, Record and playback — recorded stubs served immediately after recording stops.
• WireMock, Stateful behaviour — scenarios as state machines whose states can be assigned and reset.
• Google, Choose Gmail API scopes — gmail.send as sensitive, gmail.readonly, gmail.compose, gmail.insert, gmail.modify and gmail.metadata as restricted, restricted-scope verification and the security assessment for storing or transmitting restricted-scope data.
• Google Cloud, Verification requirements — the demonstration video requirement, narrowest-scope justification, limited use of API data, and the rule on humans reading data.