How is an API replica for testing AI agents built and checked, and how does it differ from a mock server or the provider's sandbox?

What an API replica for testing agents is

Redline keeps one folder per provider under backend/apis/ and starts each copy on its own port (npm run apis). A copy is a server, not a fixture file: it keeps state, answers OAuth and token flows, paginates, sorts, returns the provider's error shapes, and resets to its seeded data on request. "Only the data is fake" is the rule the build notes state, after listing what must match: URLs, parameters, bodies, answers, errors, headers, IDs and behaviour.

The running copies' index currently reports these implemented endpoint counts for a sample of the apps:

Shopify is listed in the index but is not runnable, and one Booking.com API (demand-beta) has 21 operations missing. Every other app above reports zero missing operations in its spec.

Why this shape suits agent testing rather than application testing: a unit test asserts on a call you wrote, while an agent decides which tool to call and with what arguments. The value of the run comes from the answers it gets back. A copy exists so those answers are the provider's, and so the calls an agent makes across several apps land in one plausible company's world, with every step recorded. The pages on testing a payment agent before production and testing a sales agent against a CRM walk through that in practice.

How a copy is generated from the provider's published API spec

Each provider has specs/<provider>/sources.json, listing what the watcher follows, and the copy loads the pinned result: Google's Discovery documents for Gmail, Calendar, Drive and Sheets; OpenAPI files for Stripe, PayPal, Zendesk, BigCommerce and others; GraphQL introspection where that is the provider's shape; and reference pages for providers that publish no spec. Slack's reference pages are the source for all 340 Web API methods (its machine-readable spec repo was archived in 2020), HubSpot's 216 reference pages each embed the OpenAPI description of their operation (HubSpot's own spec repository is not licensed for external use), Salesforce's routes come from the REST API Developer Guide and Object Reference, and Zendesk's two OpenAPI files plus 134 reference pages cover 673 operations, of which 111 exist only in the pages.

Two build rules follow from that.

Route every operation in the spec, and nothing else. An unknown URL gets the real provider's own unknown-URL answer, recorded, not a generic 404 page. Where an API is too large to copy whole — Microsoft Graph, Shopify's Admin GraphQL — the copy serves the surface an agent uses for that product, with the spec cut to it using a select, and the notes list exactly what is in and out.

Watch the spec, because providers change it. npm run watch-specs compares each live spec with the pinned one and reports what changed, marked breaking, not breaking, or "read" (documentation text and inline schemas). Recent watcher runs on Stripe, Slack, PayPal and the CRM pages reported no change to any source; Stripe's run pinned its docs pages and found every one identical to the saved copy.

How the copy answers with the same URLs, parameters, errors and headers

The build notes list seven exactness layers, and they are the answer to this question.

1. Routing from the source of truth. Every operation in the pinned spec is routed; unknown URLs get the provider's own answer.
2. Recorded real answers. Every operation is called on the real API with no credentials and with an invalid credential, one request at a time, at least 300 ms apart, using no accounts and no keys, and those answers are replayed byte for byte. A compare script reports identical: N, different: M; values that change on every real call (dates, request ids, trace ids, ULIDs) are compared by format and generated in that same format, and the copy names which fields it masked.
3. Validation errors in the provider's own format — status, body and headers.
4. Behaviour from the reference docs: pagination, ID formats, rate-limit headers, ordering, side effects between users. The pages used are saved with the copy.
5. Tests: a conformance suite calls every operation and checks each answer against the spec; a behaviour suite covers the documented behaviour; where the provider ships an official SDK, a suite drives the copy with it.
6. Sign-in the tooling can use: the OAuth or API-key flow the tool expects, with a sandbox account chooser labelled "Agent Sandbox" that must never look like the provider's real sign-in page.
7. Sandbox controls the real API does not have, starting with POST /_twin/reset and GET /_twin/coverage.

The parameters and headers come out of the same work. The Stripe copy parses form-encoded bodies with a[b] and a[0] keys, answers idempotent requests with the recorded Idempotent-Replayed header, echoes Stripe-Version and Stripe-Account, and signs webhook deliveries with Stripe-Signature: t=…,v1=…. The PayPal copy serves the token endpoint, PayPal-Request-Id idempotency and PayPal's own negative-testing header, PayPal-Mock-Response. The Google copy generates Gmail and Calendar IDs in Google's formats and answers push notifications. Where Slack's method pages and scope pages disagree about which token types a scope allows, the copy's installs follow the scope pages and say so.

Where a provider's docs and its spec disagree, the copy picks one and writes the choice down rather than blending them. Examples in the copies' READMEs: Drive's spec marks Permission.emailAddress output-only while its description says to supply it on create, so the copy accepts it and the generated tool does not; Slack's admin.conversations.lookup page example says channels while the Java SDK's recorded answer says channel_ids, and the copy answers channel_ids; PayPal's GET /v1/shipping/trackers summarises a list but its 200 schema holds one tracker, so the copy answers the first match. The live and sandbox PayPal hosts differ in one recorded header — the order of access-control-allow-methods on OPTIONS — and the copy answers like live by default, or like the sandbox when started with PAYPAL_TWIN_HOST=sandbox.

What data the copy holds

The copies are seeded with one fake company and use it everywhere, so a test can span providers: Example Co at example.com, Maya Patel as the main sandbox user, Sam Okafor, and two outside organisations — Northwind (a customer) and Acme Hosting (a supplier). The rule in the build notes is that no real person, no real company's data and nothing from a user's real accounts is used. Stripe's copy is that account with its own test keys, customers Northwind and Globex and two products; Slack's is an Enterprise org with the Example Co workspace, a guest whose account expires, an app and a hosted workflow; Zendesk's is one help desk with subdomain exampleco; Google's is Maya's mailbox and calendar.

Above that sits a per-company layer. Every company created in the product gets its own copy in every fake API, chosen by the x-twin-tenant header for backend calls and a twin_tenant cookie for sign-in pages, so company data does not mix: two Stripe companies see only their own customers, and connections are stored per company.

The records themselves are written by a data agent, not pulled from anywhere. It is given a domain and a company, researches how businesses of that domain work, reads each chosen app's fixed record kinds and formats, writes a single canonical world, then writes every app's records from it, in each kind's exact format, until a check_data pass accepts every kind. The point of one world is that the same fact holds in every app: the order with its payment and amounts, the customer's email, the staff with their mailboxes and handles, the thread that refers to the order.

What conformance testing checks, and what the copies report

Four kinds of suite run against a copy, and all of them run with the copy up. The replay suite calls the copy and compares status, body and every recorded header against the answers recorded from the real API. The conformance suite calls every operation with valid input and checks the answer against the pinned spec — every field the spec does not define, every wrong type, every value outside an enum. The behaviour suite exercises the documented side effects. The SDK suite drives the copy with the provider's own client library.

Two details keep those numbers honest. In the Stripe every-operation run, the 110 error answers are requests built from the spec's required parameters only — for example POST /v1/charges without source — or ids of objects the run deleted earlier, and each is a status the spec lists with a body matching the error schema. And the replay suites only ever saw unauthenticated or invalid-token traffic, because the recording rules forbid using an account or a key.

What a copy does not claim

This is the part a comparison page usually leaves out, so here it is plainly. Because no request with a valid key has been sent to real Stripe, PayPal, Google or Zendesk from these copies, the messages for errors raised with a valid key are documented but not verified: Stripe lists parameter errors, No such <type>, state errors and its card-error wording as documented-but-unverified; PayPal lists validation errors, token errors and several fee and expiry rules; Google lists authenticated error texts, default settings values and some messages.insert behaviours as needing one real test account to record and compare. Where the docs are silent, the copy's own values are listed too — ID formats, key order, some webhook and dispute details. Calendar's format checks are an exception the READMEs disclose: they use real Calendar answers published by other projects on GitHub, not recordings from Redline's own account.

Coverage has edges: Gmail's HTTP batch endpoint and resumable uploads are not implemented and users.watch cannot publish to Pub/Sub; Google's official Node auth library hard-codes oauth2.googleapis.com for token requests, so an app using the SDK directly needs a per-call rootUrl or network-level routing to the copy; the Stripe copy does not enforce rate limits and does not retry webhook deliveries, and its /v2 routes answer as recorded without a key but are unknown URLs with one. Read a copy's own "Not verified" and "Known gaps" sections before you treat a green suite as full coverage.

How a mock server is different

Mock servers are the older and cheaper answer, and they are good at things a copy is not.

The differences that matter for an agent run come down to three.

Who wrote the answer. A mock returns what you wrote or what a generator inferred from a schema. A copy returns what the provider's recorded API and documentation say, and where it cannot, it says so. That is why a copy can answer a wrong URL with the provider's own page, and why its errors carry the provider's status, body and headers.

Where the data comes from. A mock has whatever you stubbed; a Mockoon bucket is a small database you invent. A copy starts from a seeded fake company, and each company's copy holds that company's records, the same facts in every app.

What failure means. Injecting faults is a feature of WireMock and Mockoon — useful for testing retries and timeouts, and a good reason to choose them. A copy mostly tries to answer like the real API and gives you named controls for the failure paths the provider documents: Google's copy can expire Gmail history so the next sync gets a 404 and does a full sync; Slack's can make the next calls of a method answer 429 with Retry-After; Stripe's has a page where a 3D Secure challenge passes or fails and a Checkout page a POST can complete; PayPal's lets a payer approve an order, open a dispute or approve a vault setup token without a browser.

How a provider's sandbox is different

A sandbox is the vendor's own environment, and for anything the vendor runs it is the authority.

Stripe describes a sandbox as an isolated test environment inside your account where payments are not processed by card networks or payment providers, and states that the API key decides whether a request runs live or in a sandbox. Test cards simulate successes by brand and country, declines with the error and decline code the card returns, Radar blocks, disputes with winning or losing evidence, 3D Secure and refunds that fail asynchronously. Simulations with test clocks advance time to see how subscriptions, trials and renewals behave. Limits are documented as well: you cannot test IC+ pricing in a sandbox, you cannot create connections between a Connect platform's sandbox and connected-account sandboxes, and a card refund can be cancelled only in the Dashboard, within a 30-minute window that stands in for live mode's short but unspecified one.

PayPal describes its sandbox as a self-contained virtual environment that simulates the live PayPal production environment, mirrors it feature for feature, and gives you fictitious sandbox accounts whose credentials you use with the sandbox endpoints. It also lists what does not apply there, such as closing an account, issuing monthly statements and storing shipping preferences.

For an agent test, that means you get the vendor's real behaviour, including anything your copy has not modelled, and you do none of the maintenance: the vendor keeps the sandbox current, whereas a copy is kept current by a watcher and a builder. What you do not get is your own records or a run that spans tools. The sandbox holds the test objects you created in that vendor's account, against the vendor's host, with the scenarios its published test values allow. If your agent has to handle a refund on an order that exists in your shop, a ticket that exists in your help desk and a customer record that exists in your CRM, no single sandbox has that.

When a mock server or a sandbox is the better choice

Choose a mock server when the API you depend on does not exist yet, or when you only need a few endpoints and want them exactly as you imagined them. Choose one when you are testing your own client code with assertions, since a stub is simpler to change than a copy of a whole API. Choose WireMock when you need latency distribution or corrupted responses — a lognormal delay or a dribbled body is trivial there and not something a provider's API will produce on demand. Choose Prism when you have a spec and want a runnable mock in the same minute, with dynamic values so nobody codes against one example. Choose Mockoon when you want a small stateful JSON database behind REST routes without writing a server.

Choose a sandbox when you must use the vendor's real engine: fee and ledger arithmetic, asynchronous settlement, disputes and evidence, webhook signing and retries, versioning and account state. Choose it when there is no copy of that provider, when you have no capacity to keep a copy exact, or when the vendor's own sandbox tooling is the thing you need, as with Stripe's test clocks. Choose it as the last check before release, because it is the only environment in the comparison that the provider itself is accountable for.

Choose a replica when the question is what your agent does with your company's records across several apps, when you want to run the same scenario many times without moving money or mail, when you need a reset between runs, when you want to force a documented failure path deterministically, and when you want the run — model calls, tool calls, cost, failures — recorded against a copy rather than against production. Teams working through those scenarios per industry are collected in the support-agent and security-testing pages.

Common questions

Is an API replica just a mock server?

No. A mock server answers with stubs or generated examples; a replica answers with the provider's recorded and documented behaviour, routes every operation in the provider's published spec, keeps state, serves the provider's OAuth or token flow, and is seeded with one company's records. Both run locally; only the second claims to stand in for the API's contract, and it is checked against that contract by replay and conformance suites.

What does conformance-tested mean for a copy?

It means two suites. The replay suite compares status, body and recorded headers against answers recorded from the real API with no credentials and with an invalid credential — Stripe's copy reports 1242/1242 answers identical, Zendesk's identical: 2460, different: 0, Slack's identical: 1043, different: 0. The conformance suite calls every operation and checks the answer against the pinned spec, reporting counts such as Stripe 612/612 and Zendesk 673/673 operations answer as the spec says. Behaviour suites and official-SDK suites cover side effects and client compatibility.

Can I put my company's real data into a replica?

No. The build rule is made-up people and companies on example.com and .example domains, and never anything from a user's real accounts. What you get instead is a company in the product whose copy holds records written for it by the data agent, in each app's exact record formats. Existing articles on this site came from Redline's older security product; the replica product's accounts, companies and per-company copies are what is described here.

Which option costs the least to keep accurate?

A provider's sandbox, because the provider maintains it; Stripe's and PayPal's documents are the reference for what their environments support and where they stop. A mock server is next: it only has to stay as accurate as your tests need, and unmatched or unmodelled behaviour is your own gap. A replica costs the most to keep accurate, which is why it has a watcher that reports spec changes as breaking, not breaking or documentation-only, and why each copy's README carries its own "Not verified" list.

Does a replica replace testing against the real sandbox?

No. Use the copy to shape and repeat the scenario, then keep a final pass against the provider's sandbox — and against a live-mode check where the vendor requires one — because no valid-key request has been made to these providers from the copies, so authenticated error text and some defaults are documented rather than verified. If your team is running a payment or refund flow, the payment-agent walkthrough puts those steps in order.

Sources

• WireMock overview — response stubbing matchable on URL, headers and body, record/playback, delays and fault injection, stateful simulation, 5 million downloads a month, library or standalone.
• WireMock request matching — the request attributes a stub can match on.
• WireMock record and playback — recording and snapshotting stubs from a real API through proxying, served immediately afterwards.
• WireMock simulating faults — fixed, random and chunked-dribble delays, corrupted responses.
• WireMock Cloud cleanroom testing for AI agents — simulated stateful APIs for agents, with failure scenarios under your control.
• Prism (Stoplight), open-source HTTP mock and proxy server — mock servers from OpenAPI documents, dynamic examples, validation.
• Prism README (GitHub) — mock and validation-proxy commands, OpenAPI and Postman support, data persistence on the roadmap.
• Prism HTTP mocking guide — route-not-found 404 problem+json, static fallback values, dynamic generation.
• Mockoon CRUD routes — stateless HTTP routes, CRUD routes over a data bucket, bucket state and resets.
• Mockoon documentation: about — desktop app, CLI and hosted editions.
• Stripe sandboxes — isolated test environments, payments not processed by card networks, limits such as IC+ pricing and Connect sandbox connections.
• Stripe testing — test cards by brand and country, declines with error and decline codes, Radar cards, disputes and evidence, asynchronous refund cards, the 30-minute refund-cancellation window, swapping test keys for live ones.
• Stripe test clocks — simulations that advance time for Billing objects.
• Stripe API reference — sandbox versus live by API key, sk_test_ keys, HTTPS, error types and status codes.
• PayPal sandbox testing guide — the sandbox environment, sandbox accounts and endpoints, and the features that do not apply.