How do you see what a payment-handling AI agent will do before it touches real payment systems?

Why a money-handling agent has to be judged on the run, not the reply

An agent that says "your refund is on its way" has done nothing yet; the money moves in the tool call behind that sentence. Stripe's documentation describes why the result of that call is not a single instant: "In live mode, refunds are asynchronous: a refund can appear to succeed and later fail, or can appear as pending at first and later succeed."

Disputes are asynchronous in the same way, and the money is decided by evidence you submit. Stripe's test cards create the dispute as part of the payment — fraudulent, product not received, inquiry, early fraud warning, multiple disputes — and the evidence value decides the outcome: winning_evidence "closes the dispute as won and credits your account for the amount of the charge and related fees", losing_evidence closes it as lost without crediting, and escalate_inquiry_evidence turns an inquiry into a full dispute and debits the account.

A retry is the other trap. Stripe's idempotency documentation says a request made with an idempotency key saves the first result and "retries with the same key return the saved response, including 500 errors", and for API v1 those keys can be removed "after they're at least 24 hours old" — so a refund issued once is one refund only if the agent sent a key. None of that is visible in the agent's final message.

The questions worth answering before production are therefore about state and sequence:

• Did the refund happen once, for the right amount, on the right charge?
• Did the balance end where it should, and is a failed refund visible as failed?
• Was the dispute answered with evidence before its deadline?
• Did the payout go to the intended recipient, and what did the agent say while doing it?

Anthropic's guide to agent evals says the same thing about environments: "Each trial should be 'isolated' by starting from a clean environment", and graders should look at what the agent produced rather than a fixed tool order, because "it's often better to grade what the agent produced, not the path it took." LangChain's 2026 survey of 1,300+ practitioners found 57.3% with agents already in production, nearly 89% with observability implemented (ahead of 52% for evaluations), and a third naming quality the top barrier to production.

The four ways teams test a payment agent today

Each of these is a real practice with a real purpose. Read together they show what a replica run adds.

Provider test modes and sandboxes

A sandbox is the provider's own no-money environment. Stripe: "when testing in a sandbox, the payments you create aren't processed by card networks or payment providers", and test cards simulate declines, disputes, refunds and 3D Secure. PayPal's sandbox is "a self-contained, virtual testing environment that simulates the live PayPal production environment" where apps process requests "without touching any live PayPal accounts".

What it gives you: the vendor's own behaviour, with a published way to trigger the awkward cases. Stripe's test cards cover asynchronous refunds (a refund that starts pending and later succeeds, or starts succeeded and later fails), 3DS outcomes, and disputes, inquiries and early fraud warnings, with evidence values deciding win, loss or escalation. Stripe also provides "test primitives to simulate payout success and failure", and Simulations "let you move time forward in the sandbox so resources such as subscriptions change state and trigger webhook events". PayPal's negative testing makes sandbox errors reproducible: the PayPal-Mock-Response header "configures the sandbox into a negative testing state for transactions that include the merchant" for Payments and Orders methods, and test values do the same for Disputes, Invoicing, Payouts and Subscriptions.

What it does not give you: your company's records, and a record of the agent's steps. The sandbox holds the vendor's test fixtures plus whatever objects you create in it. It is also shared state by design — Stripe's testing page warns that "card-level state from one test can affect later tests that use the same card", and tells you to use a different card for each independent end-to-end scenario. It also warns against using test environments for volume: "Don't use testing environments to load test your integration because you might hit rate limits."

Two further details matter for a money agent. In Stripe's test-mode sandbox, settings are shared with live mode: "Share settings between live mode and the test mode sandbox. You can't test many settings independently." Stripe recommends general sandboxes for new integrations and separate ones for local development and CI "so automated tests don't affect your settings or data". And objects never cross modes: "API objects in one mode aren't accessible to the other."

Mocked payment APIs

A mock is a server you write, or record, that answers in place of the provider. WireMock calls a scenario "a state machine whose states can be arbitrarily assigned", so stub A answers first and stub B after a state change, and the state can be reset between tests. Record-and-playback captures real responses into stubs, and "'Playback' doesn't require any explicit action. Recorded stubs will start being served immediately after recording is stopped."

This is the route LangChain's own documentation for an evaluatable refund agent takes. The tutorial makes the refund node configurable so that when it runs the evaluation, config={"env": "test"}, "we would mock out the refunds without actually updating the DB".

What it gives you: speed, determinism, no accounts, no keys, and a suite that runs on every commit.

What it does not give you: state you did not model. A mock has no balance, no ledger, no dispute deadline and no history of last month's refunds; the refund it "processes" changes nothing, so the outcome cannot be checked. And nothing verifies that a hand-written response matches the provider's published schema — the mock is exactly as accurate as the person who wrote it.

A staging account on test-mode keys

Staging means your own deployment, pointed at the provider's test environment with test keys, so the paths you actually ship — env vars, webhooks, retry logic — are exercised end to end. Stripe supports it directly: general sandboxes "isolate settings and data from live mode and let you create separate environments for local development, CI, and staging". PayPal supports the account side: personal and business sandbox accounts, created with generated credentials, and existing accounts you can "duplicate, or clone" when you need several buyers or merchants.

What it gives you: confidence in the wiring, and a place to load the objects a scenario needs.

What it does not give you: the objects themselves. A useful payment scenario needs a customer with history, an invoice from last month, a charge that was already partially refunded, a dispute due on Friday. In a sandbox you must create all of that first, per sandbox, and keep it clean afterwards — deleting test data can't be undone, and "sandboxes are temporarily unusable while the deletion process occurs". Staging also runs on Stripe's shared test environment, so it inherits the same test-card state problem as the sandbox above.

Evaluating transcripts and eval suites

Trace grading scores what a run did. OpenAI's documentation: "A trace captures the end-to-end record of model calls, tool calls, guardrails, and handoffs for one run. Graders let you score those traces with structured criteria so you can find regressions and failure modes at scale", and datasets turn graded runs into repeatable eval runs. LangSmith's tutorial splits agent evaluation into the final response, the trajectory, and single steps, where a trajectory evaluation "compares the actual sequence of steps the agent took against an expected sequence". Anthropic's guide is blunt about the manual version: reading transcripts builds intuition for failure modes but is "time-intensive", and it "doesn't scale"; automated evals, meanwhile, "can create false confidence if it doesn't match real usage patterns".

What it gives you: judgement about the agent's choices, over many recorded runs, without a model change shipping blind.

What it does not give you: the environment the run happened in. A transcript of a refund issued against a mock shows the agent decided to refund; it cannot show that the refund was possible, correct, single, and reflected in a balance. This is where eval suites and replicas fit together: the replica is the environment the transcript can be graded against.

What running against a replica of the real payment APIs looks like

Redline serves copies of provider APIs and gives each company its own copy of every fake API, so the records in them belong to that company and to no other. The build rule for every copy is stated in the repo: "The API must be an exact replica of the real one: same URLs, parameters, bodies, answers, errors, headers, IDs and behaviour. Only the data is fake." Each copy is generated from the provider's published spec, pinned in the repo, with a watcher that reports when the provider's spec or documentation changes.

The two payment copies and what their own test suites report:

Three things make that copy useful for a money agent.

The records are the company's own. The data agent writes a company's records as files in the company folder, one folder per app, one file per record kind, in exactly the format that app's API returns. Volumes are parameters, not guesses: for Stripe the size includes payments per month, average payment and refund rate. A shared world file keeps the facts the apps must agree on, so the orders carry their payment method and amounts and the customers keep the same emails and addresses across the shop, Stripe and PayPal. It is all made-up data on .example domains — never a real person's name, address or account — which is the point: you get the shape of the company's payment work without putting production records anywhere near a test.

Every step is recorded. The agent sends its runs to the company's Redline project with the @redlineai/sdk, whose tracing API is Langfuse's, so an agent is instrumented as it would be for Langfuse: "every model call, tool call and step". The console measures tokens, cost, failures, tracing and a projection for a company's recorded runs, so you can see what a run costs and which calls failed. Rules are separate and simple: write what your agent must not do, and each run is read against it, with every reply that breaks it kept together with its words and the reason. When a decision needs explaining, the Replay page's Root cause tab reads the recorded calls into a graph of what the agent observed, believed and decided, each claim tied to the row of the context it quotes, and traces the decision back to where it started; nothing is re-run.

The state is there to act on, and you can put it back. A refund in the copy moves the balance and writes the ledger entries; an asynchronous refund card behaves as Stripe documents it, with STRIPE_TWIN_ASYNC_MS (2 seconds by default) standing in for "some time later"; disputes can be won or lost with the evidence values above; test clocks advance subscriptions; idempotent replays carry Stripe's Idempotent-Replayed header. Sandbox controls cover the parts a browser would normally do: the Stripe copy has its own 3DS page and Checkout page for agents without a browser, and the PayPal copy lets you approve an order or a subscription as the payer (POST /_twin/paypal/orders/{id}/approve), lets a payer open a dispute, and implements PayPal's own PayPal-Mock-Response negative testing. POST /_twin/reset returns a copy to its seed and GET /_twin/coverage says what the copy serves, so the same case can be run again from the same starting state.

How the routes compare

What the replicas do not do

Fair limits, from the copies' own READMEs:

• They are not the provider, and no money moves. The copies answer like the real APIs using made-up data. Keep using the provider's sandbox to validate the integration itself before go-live; Stripe and PayPal both document that route above, and Stripe advises general sandboxes for new integrations.
• Some answers are not checked against real keyed traffic. No request with a valid Stripe key was sent to real Stripe, so parameter and state error messages with a valid key, card error messages for several codes, and fee amounts are not verified against real answers — the card fee of 2.9% + 30¢ and the dispute fee are the copy's own assumptions, not values from Stripe's docs pages. On PayPal, validation errors (which need a working token) and which rule PayPal reports first are unverified.
• Not everything the providers offer is in the copies. Stripe's rate limits "are not enforced" and webhook deliveries are not retried; with a key, /v2/… is an unknown URL; Connect onboarding, Issuing, Treasury, Terminal, Tax, Payment Links, the customer portal and non-card payment methods answer through the spec-driven engine only. PayPal APIs outside the 14 served are absent (Referenced Payouts publishes no schema), and webhooks are tried once where PayPal's own schedule is "up to 25 times over 3 days".
• It is not production data. The records are made up for the company, so they will not reproduce one specific real customer's case unless you write that record by hand. If you need a case that exists in production, write it into the company's records rather than exporting real data — Stripe's own testing page says "Don't use real card details. The Stripe Services Agreement prohibits testing in live mode using real payment method details."

Common questions

Is running against a replica the same as Stripe's sandbox or PayPal's sandbox?

No, they answer different questions, and most teams should keep both. The provider's sandbox is the vendor's own environment, and it is the right place to confirm your integration behaves as the vendor documents. A replica is a separate service that answers like the provider with one company's records in it, so you can run the agent itself — its model calls, its tool arguments, its retries, its wording — against state that changes. Redline's Stripe copy, for instance, serves 612 operations and answers recorded unauthenticated requests identically to api.stripe.com, status, body and headers, but it is Redline's server on a local port, not Stripe's.

Do mocked APIs and evals become unnecessary once there is a replica?

No. Mocks stay the cheapest way to unit-test your own code paths and to run fast in CI, and eval suites stay the way to grade the agent's choices across many runs — Anthropic's guidance is to treat the methods as complementary, each covering what the others miss. What changes is what the eval can grade: with a replica underneath, "the agent refunded correctly" is a statement about a balance and a ledger rather than about a response body someone wrote.

Can our production payment data be loaded into the copies?

No. The data agent creates the company's records as made-up records in the apps' own formats, using .example domains and never a real person's or company's details, and the copies are not connected to your live accounts: the agent signs in on the copy's own account chooser and the copy's keys are listed on it. If you want a case modelled exactly, write that record by hand in the company folder; do not export production records or live card details, which Stripe's Services Agreement prohibits using in live-mode testing.

What does one recorded run contain?

The trace of the run: every model call, tool call and step, with the arguments and results, plus tokens, cost and failed calls. Sessions are grouped per conversation and per agent, rule breaks are kept with the words that broke them and the reason, and the Replay page traces a decision back through what the agent read and concluded, each step tied to the row of the context it came from.

How do we stop two scenarios from interfering with each other?

Reset between runs. The copies have POST /_twin/reset to return to the seeded state, and GET /_twin/coverage to see what a copy serves. Each company also gets its own copy of every fake API, so one team's runs do not touch another company's data. This replaces the sandbox habit of never reusing state within one run — the same habit Stripe recommends in its own test mode, where "card-level state from one test can affect later tests that use the same card".

Related reading on this site: how to test an AI agent for security before deployment covers the pre-release testing method, approval gates and scoped credentials covers the controls that stop a payment agent acting outside its remit, and testing with staging, planted secrets and traces covers the red-team side of the same environments.

Sources

• Stripe, Testing — sandbox test values, test cards, asynchronous refunds, dispute evidence values, card-level state, load-testing note, real card details prohibition.
• Stripe, Sandboxes — isolated environments, payments not processed by card networks, separate sandboxes for local development and CI.
• Stripe, Testing use cases — test mode sandbox versus general sandboxes, settings isolation, objects not shared between modes, simulations, deleting test data.
• Stripe, Idempotent requests — idempotency keys, saved responses on retry, key removal after 24 hours.
• PayPal, PayPal sandbox testing guide — what the sandbox simulates and which features do not apply in it.
• PayPal, Sandbox accounts — personal and business sandbox accounts, generated credentials, duplicating accounts.
• PayPal, Simulate negative responses with request headers — the PayPal-Mock-Response header, mock_application_codes and the methods it covers.
• PayPal, Simulate negative responses with test values — test values for Disputes, Invoicing, Payouts and Subscriptions.
• WireMock, Stateful behaviour — scenarios as state machines and resetting them.
• WireMock, Record and playback — recording responses into stubs and serving them after recording stops.
• LangChain, Evaluate a complex agent — the refund agent's test configuration, and final response, trajectory and single-step evaluations.
• Anthropic, Demystifying evals for AI agents — isolated trials, grading the produced outcome rather than the path, limits of transcript review.
• OpenAI, Evaluate agent workflows — traces as the end-to-end record of model calls, tool calls, guardrails and handoffs, trace grading, datasets and eval runs.
• LangChain, State of Agent Engineering — survey of 1,300+ practitioners: production adoption, observability and eval adoption, quality as the top barrier.