Guide
How do you see what an AI agent will do in an online store's back office before it touches the real store?
Run an order and refund agent against BigCommerce, Stripe and Gmail replicas holding your own orders and customers, and compare sandboxes, mocks and evals.
Table of contents
In short
Run the agent against replicas of the three systems its work lands in — the store platform (BigCommerce), payments (Stripe) and email (Gmail) — holding one company's own products, orders, customers and message threads, and record every step of the run: each model call, each tool call with its arguments and result, the cost, the failed calls, and every reply that broke a rule you wrote. Then read the state the run left behind: the order at the status you expected, the refund created once for the right amount, the stock level moved by the right delta, the message in the right thread and nowhere else.
Redline builds that environment. Its BigCommerce copy serves 286 operations taken from the platform's own pinned API specifications, its Stripe copy 612, its Gmail copy 85, and each company gets its own copy of every replica holding that company's records. The four routes teams use today instead — the store platform's own sandbox or trial store, payment test mode, mocked APIs, and evaluating transcripts — each answer a different question. This page compares them from each vendor's and tool's own documentation, and says where a replica run does not help.
What a store back-office agent actually changes
The reply is the last step of the work, not the work. An agent that writes "I've refunded the £48 and put the jumper back in stock" has changed nothing yet: the money moves in a refund call, the stock moves in another call, the order status moves in a third, and the customer's inbox is written to in a fourth. Each of those is a separate system with its own rules, and the wording of the reply tells you nothing about whether they happened.
Order status, refund, stock and email are four different jobs
| What the agent does | Where it lands | What the vendor's docs say is easy to get wrong |
|---|---|---|
| Moves an order to a new status | The store's Orders V2 / Orders V3 endpoints | A status change to Refunded or Partially Refunded can email the shopper; BigCommerce's answer to "Will this trigger an email to the shopper?" is "Yes, if you configure the store to send an email when an order status changes to Refunded or Partially Refunded" (Order Refunds) |
| Refunds an order | POST /v3/orders/{id}/payment_actions/refund_quotes, then a refund request |
"Refunding an order consists of two API requests" — and refunds are refused before payment: "You must receive payment of the order before you can issue a refund" |
| Adjusts stock | The inventory endpoints | The refund does not do it for you: "you cannot return items to inventory that you refunded via API" |
| Emails the customer | Gmail messages.send |
Gmail has no test mailbox; gmail.send is a sensitive scope and gmail.modify is restricted (Choose Gmail API scopes) |
Two details from BigCommerce's refund page change how a scenario has to be built. The quote step exists to "make sure you are refunding to the correct payment provider in the correct amount", so the useful test is not "can the agent refund" but "did it refund to the right provider, for the right lines, once". And the gateway has conditions of its own: "If you use a payment gateway, you must settle payments. Some gateways require a certain amount of time to pass before you can process refunds."
The money underneath moves in stages
Stripe's refund documentation describes a chain the agent's final message hides. Refunds are paid from your available balance, and "if your available balance doesn't cover the amount of the refund, Stripe holds the refund as pending for card transactions ... until your Stripe balance becomes sufficient". A refund can also fail after it was created — "a refund can fail if the customer's bank or card issuer can't process it" — and Stripe notifies you with the refund.failed event, leaving failure_reason on the Refund object. For some payment methods Stripe emails the customer itself to collect bank details before the refund can complete (Refund and cancel payments).
So an agent's refund can be correct at the moment of the call and wrong an hour later, and the customer may hear about the refund from Stripe as well as from the agent.
The four ways teams test a store agent today
Each of these is a real practice with a real purpose. Read together they show what a replica run adds.
The store platform's own sandbox or trial store
BigCommerce gives developers sandbox stores made in the Partner Portal: "Sandbox stores are intended for developing and testing apps. They cannot process any live transactions" (Create a Sandbox Store). It is the real API, with the vendor's current behaviour, on a store of your own — which is the strongest argument for it: nothing else is quite the vendor's own truth.
Its economics are documented too, and they matter when an agent starts calling in a loop. In a real store "each API request consumes one quota unit", and "all apps accessing the store share the store's quota. This prevents a single app from using the entire quota". A Pro store gets 60k requests an hour (450 per 30 seconds), Plus and Standard 20k an hour (150 per 30 seconds), and sandbox quotas are counted by resource rather than by those plan numbers; the response headers (X-Rate-Limit-Requests-Quota, X-Rate-Limit-Requests-Left, X-Rate-Limit-Time-Reset-Ms) tell you where you stand (API rate limits).
What it does not bring is your company's work. The catalogue, the customers, the order that was half refunded last Tuesday and the email thread that started it are not in the sandbox; you create whatever the scenario needs, and you keep it clean afterwards. It also covers one vendor. An agent that refunds touches the payment gateway too, and the customer's mailbox.
Payment test mode and sandboxes
Payment providers have their own no-money environments, and they are precise about the awkward cases. In a Stripe sandbox, "the payments you create aren't processed by card networks or payment providers", data and settings are isolated from live mode, and test cards simulate declines, disputes and refunds; Stripe recommends separate sandboxes for local development and CI "so automated tests don't affect your settings or data". The documented limits are worth reading before you plan a suite: you can't test IC+ pricing in a sandbox, and you can't create connections between a Connect platform's sandbox and connected account sandboxes (Sandboxes, Testing).
What it does not bring is the order being refunded. The sandbox holds vendor fixtures plus the objects you create in it, so the customer with a partial refund and a dispute deadline is yours to build and yours to clean up. And the run itself is not recorded as an agent run: the sandbox tells you what the API did, not what the agent did.
Mocked store, payment and mail APIs
A mock is a server you write or record that answers in place of the vendor. WireMock describes the two mechanisms teams use: a scenario is "a state machine whose states can be arbitrarily assigned", so one stub answers first and another after a state change, and the scenario can be reset between tests; record-and-playback turns real responses into stubs, where "'Playback' doesn't require any explicit action. Recorded stubs will start being served immediately after recording is stopped" (Stateful Behaviour, Record and Playback).
This is the cheapest way to test your own code: the tool that builds an order-status payload, the retry wrapper around a Gmail send, the branch that handles a 422 from the store. It runs in CI in seconds, with no accounts and no keys.
What it cannot do is tell you whether the agent's work was correct. A mocked refund changes no balance and no ledger, so nothing can contradict the agent. A mocked catalogue has no history, so "is there enough of this SKU left after last week's returns?" has no answer. And each answer is only as accurate as the person who wrote the stub — nothing checks it against the vendor's published schema.
Evaluating transcripts and simulated shoppers
Transcript grading scores what a run did. Anthropic's guide to agent evals gives the vocabulary that makes this precise: a transcript is "the complete record of a trial, including outputs, tool calls, reasoning, intermediate results", while "the outcome is the final state in the environment at the end of the trial" (Demystifying evals for AI agents).
Its worked example is close to this article's subject: a support agent handling a refund for a frustrated customer, graded by an LLM rubric, by a state check (refunds: {status: processed}), by required tool calls (verify_identity, process_refund with an amount limit, send_confirmation) and by a transcript limit of ten turns. The same guide is honest about the constraints: "each trial should be 'isolated' by starting from a clean environment", and grading a fixed sequence of calls is brittle — "it's often better to grade what the agent produced, not the path it took". It recommends starting with "20-50 simple tasks drawn from real failures", building partial credit in, and using pass^k — the chance that all k trials succeed — for customer-facing agents, since a 75% per-trial success rate passes three trials in a row only about 42% of the time. It also lists the failure mode: automated evals "can create false confidence if it doesn't match real usage patterns".
An eval suite is the right way to judge an agent's choices across many runs. What it cannot do is conjure the environment those choices ran in. A graded refund against a stub shows the agent decided to refund; it cannot show that the refund was possible, correct, single and reflected in a balance.
What each route can see
| Route | Whose records are in it | Is the agent's run recorded? | Order, payment and email in one world? | Reset for the next scenario |
|---|---|---|---|---|
| Store sandbox or trial store | the ones you create in that store | no — its API answers, the agent's steps are not kept | no | not a documented feature; you clean up yourself |
| Payment test mode or sandbox | vendor fixtures plus objects you create | no | no | separate sandboxes for CI, data persists per sandbox |
| Mocked APIs | only what you wrote or recorded | no | only what you model | yes, scenario state is resettable |
| Transcript evals | whatever environment you point them at | the transcript, and the end state if the environment has one | as much as the harness has | depends on the harness |
| Replicas plus recorded runs (Redline) | one company's own records, written for that company | yes: model calls, tool calls, arguments, results, cost, failures, rule breaks | yes: store, payments and email in the same company world | yes: the copy is replayed back to its seed |
What running the agent against replicas looks like
Redline serves copies of provider APIs and gives each company its own copy of every fake API, so the records in them belong to that company and to no other. The build rule in the repo is that a copy must be an exact replica of the real API — "same URLs, parameters, bodies, answers, errors, headers, IDs and behaviour. Only the data is fake" — generated from the provider's published specification, which is pinned in the repo, with a watcher that reports when the provider's spec or documentation changes.
The copies and what they reproduce
The BigCommerce copy covers 286 operations across catalogues, products, variants, customers, carts, orders, order operations, inventory, metafields, store information and webhooks, and it reproduces the platform's front door rather than only its happy path. Its gate replays answers recorded from the real api.bigcommerce.com byte for byte: an unknown store hash, an unlisted method, a non-numeric path id, a missing or unknown X-Auth-Token, an uppercase path, a double slash. Alongside the operation calls it enforces what the real API enforces — scopes (403), request validation against the spec (422 on v3, 400 on v2), and the rate limit, answering 429 with the x-rate-limit headers and x-retry-after, with the plan switchable between Standard (150 per 30 seconds), Pro (450) and unlimited.
That last part is worth having before an agent runs unattended: an agent that fans out over 40 orders can walk into the shared store quota, and you want to see it retry responsibly rather than discover it in production. The copy's own test suite calls every one of the 286 operations and checks each answer against the spec.
The Stripe copy serves 612 operations and the Gmail copy 85, so the refund behind the order and the email to the customer are in the same run as the store call, with the real vendors' request shapes and error answers in between.
The records are the company's own
The data comes from the same place your reviews and failures do: made-up records written for one company, in each app's real record kinds and formats, so the agent meets a shop rather than an empty store. Volumes are parameters, not guesses. For a store the defaults are 40 orders a month, 45 products with one to four variants, 50 customers over a 90-day window, an average order value of $150 and a return rate of 10%; payments follow the orders (70% of them by card, an 8% refund rate, weekly payouts); the mailbox holds four messages per person per working day, 30% of them from customers, in threads of one to five messages. Everything is made-up data on .example domains, so no production record is anywhere near a test.
One shared world file keeps the apps agreeing with each other, which is what makes a cross-app scenario possible: the order the agent refunds is the order named in the customer's email, the customer keeps one email address across the store, the payment system and Gmail, and a refund that lands in Stripe matches the order status the store now shows.
Every step is recorded, and the state is left behind
The agent sends its runs to the company's Redline project with the @redlineai/sdk, whose tracing API is Langfuse's, so an agent instrumented for Langfuse is instrumented here: observe() wraps a model call or a tool as a span, and propagateAttributes({ sessionId }) keeps one conversation together. What the console reports for each run is cost, latency, tokens, failed calls and the rules those runs broke — rules you write, checked on every run. On Redline's own demo company, one logged failure is a retried bigcommerce_list_orders call, which is the shape of thing an agent that shops for data does under a rate limit.
Because the run happened against a copy, the state it left is still there to read: the same GETs the agent made now return the new order status, the refund and the adjusted stock, and POST /_twin/reset replays the copy back to its seed so the next scenario starts clean. When a decision needs explaining, the session view reads the recorded calls into a graph — per model call, the claims with the row they quote, and each decision traced back to its root — and nothing is re-run.
A first pass worth running
Take five tasks from what actually goes wrong, not from what is easy to script: "refund the wrong jumper from order 4101, tell the customer, and put one unit back", "the customer wrote twice about an order that was already partly refunded", "the gateway has not settled yet, so the refund must be refused and escalated rather than retried", "the same refund request arrives twice", "the email must go to the address on the order and nowhere else". Run each against the copies, read the transcript and the end state, then write the rules that the first pass showed you needed — the ones an agent breaks are rarely the ones you predicted.
Where replicas are the wrong tool
- The vendor's own behaviour is still the vendor's. A copy is built from the published specification and checked against recorded answers; when you need to know what BigCommerce or Stripe changed this month, their sandbox and changelog are the source of truth, not a copy. The repo's watcher reports when a spec or documentation page changes for exactly this reason.
- Real mail is real mail. The Gmail copy answers the API; it does not deliver to a mailbox, get filtered as spam, or trigger out-of-office replies. Deliverability is only tested with a real mailbox.
- Anything outside those APIs is out of scope: carriers and shipping rates, tax services, storefront themes and checkout, and the store's installed apps and extensions.
- Card networks and issuers decide the money. Stripe documents that a refund can come back as failed weeks later; a replica can model that outcome as a step you choose to trigger, but it cannot decide it.
- It is not a load test. The copy reproduces the quota so you can see how the agent behaves at the limit; it tells you nothing about the platform's capacity under your traffic.
- Replicas do not decide whether the agent's choices are good. That is what the eval suite is for — the replica is simply the environment the transcript and the outcome can be graded against.
Common questions
Can I use a BigCommerce sandbox store instead of replicas?
For the store platform's own behaviour, yes — and for some questions it is the better answer, because it is the vendor's live implementation on a store that cannot process transactions. What it will not give you is a per-scenario world: your catalogue, the half-refunded order, the customer's email thread and the payment record are all yours to build inside it, one store at a time, and the agent's run is not recorded. Teams that need both usually use a sandbox store for integration acceptance and a replica with company data for scenario runs.
Do I still need Stripe's test mode if the refund runs through the store?
Yes, if your agent calls Stripe directly or you need the payment provider's own error cases, test cards and sandbox isolation. The two answer different questions: Stripe's sandbox tells you what Stripe does with a request, and a replica run tells you what your agent did across the order, the payment and the email. Where the store's refund endpoint talks to the gateway for you, the store-side request and the gateway's settlement state are both part of the scenario you need to reproduce.
How is this different from mocking the BigCommerce and Gmail APIs?
A mock is only as complete as the person who wrote it: no balance, no catalogue history, no rate limit, no recorded answers from the real API to check it against. A replica is generated from the vendor's published specification and its front-door behaviour is replayed from recordings, so the agent's mistakes show up as the vendor's real errors — a 403 for a missing scope, a 422 for a payload the spec rejects, a 429 when it calls too fast — instead of as whatever your stub happened to return.
Does an eval suite replace a recorded run?
No, and the two fit together. Anthropic's guide is explicit that an agent's outcome is "the final state in the environment at the end of the trial", and that grading should look at what the agent produced rather than the path it took. A recorded run against a copy supplies both the transcript and that final state, so the graders have something real to score. An eval without an environment grades intentions.
Where does the test data come from, and is it safe?
It is written by a data agent as files in the company's folder, one folder per app and one file per record kind, in exactly the format that app's API returns, with the volumes taken from the company's size file rather than guessed. Domain names are .example addresses and the people in it are invented, so no real customer's name, address or order is involved.
Sources
Replicas of Slack, Gmail, Jira and more, filled with your company's data, to test your agent before production.
Run a simulationRelated