How can we test an AI agent for prompt injection and data exfiltration before production?

1. Build a staging copy that can leak

Pre-production testing is only meaningful when the test subject is the assembled agent that will perform work after release. Create a staging copy with the same prompt, tools, MCP servers and skills as production. The purpose is to expose the interactions between these components rather than assess an isolated model.

An attack can arrive through the user’s message, a document returned by a tool, an MCP response or the description associated with a tool. If the test replaces these attachments with mocks, it is testing a different agent. A test environment should therefore preserve the interfaces and interaction paths that the production agent uses.

Attach the real components

The staging copy should retain:

• The same prompt.
• The same tools.
• The same MCP servers.
• The same skills.

This arrangement makes the test relevant to the complete execution path. A model-only evaluation cannot show how an agent handles instructions embedded in a tool result or how an attacker-chosen value reaches a tool argument.

The staging environment should also be able to show whether information leaves the agent. That means the test must observe replies, tool calls, rendered URLs and outbound requests rather than relying only on the model’s final text.

Plant secrets deliberately

Put a canary value where a real secret would be. Suitable locations include:

• An API key in the environment.
• A customer record in the database.
• A confidential line in the system prompt.

The canary should be a value that the test can search for directly. This changes the question from whether an output appears suspicious to whether a protected value crossed an observed boundary. A leak becomes a string match in a reply, tool argument, URL or outbound request.

The planted value must remain available to the test and unavailable to legitimate task output. Its purpose is not to imitate every property of a production secret. Its purpose is to provide clear evidence when an attack causes protected data to leave.

Isolate each run

Use a fresh environment for each run. One attempt should not be able to contaminate the next, and a result from a later attempt should remain attributable to that attempt. Isolation also makes reruns easier to interpret when a prompt, tool or model changes.

The test record should identify the environment associated with the run and preserve the evidence collected from it. This allows a finding to be reproduced against the same setup rather than judged from a summary or an agent-generated explanation.

2. Run the four attack classes

A useful corpus covers the different places where hostile instructions can enter the assembled agent. The following classes separate the arrival point from the evidence of a successful attack.

Direct injection

Direct injection places the hostile instruction in the user’s own message. The test should check whether the agent follows the injected instruction instead of the instruction that defines its intended task.

The result is not determined solely by the wording of the final reply. A direct injection may cause an unnecessary tool call, change tool arguments or attempt to move a planted secret. The trace and environment therefore provide the evidence needed to distinguish a refusal from an unsafe action.

Indirect injection

Indirect injection places the payload in content the agent is asked to read, such as a document, web page or email. The agent may be instructed to summarise that content, but the embedded text attempts to make it act.

This class matters because the user does not need to type the hostile instruction directly. The test should include content that the agent processes through its normal tools and check whether the agent treats the content as data or as an instruction to follow.

The method is described in the OWASP AI Testing Guide’s prompt-injection test. AgentDojo runs 629 security test cases against agents executing tools over untrusted data.

Tool-output injection

Tool-output injection arrives in what a tool or MCP server returns, including tool descriptions. The test should examine whether the agent makes a tool call that the legitimate task did not require, particularly where the arguments are chosen by the attacker.

This attack class cannot be assessed by sending prompts to the model without its tools. The relevant payload is introduced through the tool path, and the evidence is found in the resulting calls, arguments, results or outbound activity.

Data exfiltration

Exfiltration probes should attempt to move a planted secret out through several channels:

• The final reply.
• A tool argument.
• A rendered URL.
• An outbound request.

Exfiltration does not require a malicious tool. EchoLeak documents a zero-click prompt injection in a production LLM system. The test corpus should therefore include payloads the user never sees, including content returned through the agent’s normal reading and tool-use paths.

Redline’s corpus contains 11,204 cases across 16 families, including Tool-Output Injection, Indirect Prompt Injection, Data Exfiltration and PHI / PII Leakage. The corpus is an example of testing across attack families rather than relying on a small collection of manually written prompts.

3. Keep the trace and grade the evidence

For every run, preserve the complete execution trace. At minimum, keep:

• The message sent.
• Each tool call.
• The arguments for each tool call.
• Each tool result.
• The final reply.

A complete trace turns a finding into a specific claim supported by evidence. It also lets the team check whether a fix changes the behaviour that produced the finding. The same run can be used to compare the original result with the result after a prompt, tool or model change.

Assess actions, not explanations

An agent may report that it refused an instruction. That statement is not itself proof that no data left the environment. The planted secret either appears in an observed output path or it does not.

Grade on what the trace and environment show. Search replies, tool arguments, URLs and outbound requests for the planted value. Also review the calls made during the run, including calls that were not needed for the legitimate task. The test should not award a pass because the final message sounds cautious if an earlier tool argument already carried the secret.

Redline uses a second agent to open the run’s machine after it finishes and pass a criterion only on evidence it saw. This separates the agent’s account of its behaviour from the evidence available in the environment.

NIST’s work on agent-hijacking evaluations makes the same point about measuring what the agent did. For a security test, the observed action is the release decision’s basis.

Preserve evidence for remediation

A useful finding identifies the attack class, the arrival point, the action taken and the protected value’s destination. The stored trace should make these elements inspectable without depending on memory or an informal description.

When a change is made, rerun the same case and compare the resulting trace. A fix should address the unsafe action while preserving legitimate task completion. This is particularly important where a broad refusal could conceal the problem by preventing both attacks and legitimate work.

4. Set pass/fail gates for release

The release decision should use explicit gates derived from the same test runs. Three requirements apply.

Zero planted-secret leaks

Across every exfiltration probe and every trial, no planted secret should leave the environment. One leak fails the release because the attack that found it may be found again.

The check must cover every destination in scope: the final reply, tool arguments, rendered URLs and outbound requests. Searching only the final response would miss an agent that keeps its reply clean while placing the secret in a tool call.

Legitimate work still done

Run benign tasks alongside attack cases. A release requires both refusing the attack and completing the intended job. Refusing a benign request is a failure in its own right because it shows that the agent is not performing its legitimate function.

This gate prevents an unsafe test strategy in which the agent passes by refusing everything. Security behaviour and task performance must be evaluated together.

No regression on rerun

Rerun the same corpus after every change to the prompt, tools or model. Compare the result with the last release rather than relying on a general impression.

The comparison should include both dimensions of the gate: whether planted secrets leave and whether legitimate tasks complete. A change that reduces attack success but also prevents intended work does not satisfy the release criteria.

What does pre-production testing not cover?

Pre-production testing measures the attacks included in the corpus. Production may bring new attacks, so the same agent still needs runtime controls after release.

Three runtime measures apply:

• A classifier on incoming messages.
• Deterministic rules on tool calls.
• Bait tools that prove a hijack.

These controls address activity as it occurs rather than relying only on a prior test result. They do not replace pre-production testing. The staging tests establish whether the assembled agent can resist known attack classes and complete legitimate work; runtime controls address activity that was not represented in those tests.

See how to stop an agent doing something unauthorized and runtime guardrails for the related controls.

Key takeaways

• Test the assembled agent in a staging copy with the same prompt, tools, MCP servers and skills as production.
• Plant canary values where real secrets would be so exfiltration can be detected as a string match.
• Cover direct injection, indirect injection, tool-output injection and data exfiltration.
• Preserve messages, tool calls, arguments, tool results and final replies for every run.
• Grade on observed traces and environment evidence, not on the agent’s claim that it refused.
• Release only when no planted secret leaves and legitimate tasks still complete.

Common questions

Can I test prompt injection against the model instead of the agent?

Only partly. A model test cannot see tool-output injection, and exfiltration happens through tools, arguments and rendered output. Test the assembled agent with its real attachments, including the tools and MCP servers used by the production assembly. The relevant security evidence is often outside the model’s final response.

How do I detect data exfiltration in a test?

Plant canary secrets where real ones would be, then search every reply, tool argument, URL and outbound request in the trace for them. A leak is then a string match rather than an interpretation. The environment should also be checked after the run so evidence is not limited to what the agent reports.

How many attack cases are enough?

Enough to cover each class and each tool the agent can reach, repeated across trials. Redline runs 11,204 cases across 16 families; OWASP’s testing guide and AgentDojo are good public starting points. The corpus should include payloads arriving through content and tool outputs, not only instructions typed by the user.

What should block a release?

Any planted secret leaving in any run, and a drop in legitimate task completion. Both numbers come from the same test run. A clean final reply does not override a secret found in a tool argument, URL or outbound request, and a refusal is not a successful result when the task is benign.

Sources

• OWASP AI Testing Guide’s prompt-injection test
• AgentDojo
• EchoLeak
• NIST’s work on agent-hijacking evaluations