How do you assess AI agent security and prevent sensitive data leaking through autonomous workflows?

What should an AI agent security assessment cover?

An assessment should follow the complete route from data access to data movement. Reviewing only the final response is insufficient because an autonomous agent can expose sensitive information before a reply is produced. It may place information in a tool argument, include it in a URL, send it in an outbound request or pass it to another agent.

Start by documenting the assembled workflow rather than assessing the model in isolation. The relevant system includes the agent, its prompt, its tools, its credentials, its data sources and any other agents it can call. The assessment must use the real tools attached to the agent, because a data path may exist only when a particular capability is available.

The inventory should answer practical questions:

• What sensitive data can the agent read?
• Which credentials allow that access?
• Which tools can receive data from the agent?
• Which destinations can those tools contact?
• Can the agent produce URLs that a client or another system will fetch?
• Can context be handed to another agent with broader reach?
• Which actions require approval before information leaves the workflow?
• Can production sessions be recorded for investigation?

This inventory provides the basis for testing. It also prevents a common assessment error: treating data leakage as a property of the model response alone. The control must be placed at the point where the data can actually leave.

Which exits can leak sensitive data?

Every exit should have an explicit control. The following map identifies the main routes described in the assessment and the corresponding control.

Replies

A reply can disclose a record that the requester should not receive. Least privilege limits what the agent can retrieve from the data source, while a check before sending provides a second control at the output boundary. These controls address different points in the workflow: one limits access, and the other examines what the agent is about to show.

Tool arguments

Tool arguments are an independent data exit. A secret does not need to appear in the final answer to be exposed if the agent passes it to an email, ticket or webhook call. The pre-call policy should inspect the arguments in the context of the session and refuse identifiers or destinations that do not belong.

Approval is appropriate for sends and exports because it holds the data movement before the irreversible action occurs. The approval boundary should apply to the operation that moves information, not only to the text that describes the operation.

Rendered URLs and outbound requests

A rendered URL can carry private data even when the visible response does not state that data plainly. The client may fetch the image or link after receiving the response. Refusing or stripping untrusted URLs in output addresses this route.

Outbound requests create another boundary. A fetch tool directed at an attacker’s server can transmit information outside the intended system. An allowlist of destinations limits where the tool can send or retrieve data.

Hand-offs to other agents

A hand-off is a tool-like data movement operation. Context passed to a sub-agent may reach a broader set of data or tools than the originating agent. The same checks used for a tool call should therefore apply to the hand-off, including inspection of the context and the receiving agent’s permitted scope.

How do canary tests prove a leak path?

A canary is a planted value, such as a fake key or record, placed where real sensitive data lives. It should be inserted into the actual locations the agent can reach, including a record in the database, a key in the environment or a line in the system prompt. The purpose is to make an otherwise difficult-to-observe data path identifiable without relying on real sensitive information.

Run attacks against the assembled agent with its real tools attached. The test should exercise both direct requests and instructions embedded in content the agent reads. A canary found anywhere in the trace is a confirmed path. It may appear in a reply, a tool argument, a rendered URL, an outbound request or a hand-off.

This distinction matters. A suspected path indicates that a route might exist; a canary in the trace demonstrates that the route carried data. The trace should therefore preserve the relevant model interactions, tool calls and resulting arguments or requests so the failed boundary can be identified.

Test content the agent reads

Attacks must not be limited to the user’s message. Content retrieved by the agent can also influence its behaviour. EchoLeak documents a zero-click prompt injection that exfiltrated data from a production LLM system.

The assessment should include plausible content that attempts to direct the agent to disclose private information. Redline’s PHI / PII Leakage family leads the agent to read out another patient’s diagnosis under a plausible pretext. Its Data Exfiltration family plants a page telling the agent to leak private data through a rendered image URL.

These cases test different boundaries. The first examines whether the agent can be persuaded to disclose information through its response. The second examines whether data can be moved through a URL even when ordinary output review might not identify the exposure.

Which controls should close the leak paths?

Controls should be placed at the access, call, output and approval boundaries. No single control covers every route.

Least privilege

Scope credentials to what the task needs and, where possible, to the individual user. The objective is to ensure that a hijacked session can reach only the data that user could reach. This reduces the available data even if an instruction causes the agent to behave outside its intended workflow.

Least privilege is an access control, not a substitute for output or call inspection. An agent may still expose information that it is authorised to read, or pass permitted information to an unauthorised destination. The access scope and the movement controls must therefore be assessed together. The OWASP AI Agent Security Cheat Sheet covers least privilege for agents.

Checks before each call

Apply a policy to each tool call before execution. The policy should compare the arguments with the session and refuse a destination or identifier that does not belong. This prevents the agent from using a permitted tool with an inappropriate target.

The same principle applies when context is handed to another agent. Treat the hand-off as a call, inspect what is being transferred and apply the receiving agent’s permitted scope. See how to stop unauthorized actions.

Checks before each reply

Use a pre-output stage to inspect the reply before the user sees it. This stage is the control for the reply exit and should be considered separately from checks on tool calls. A response may contain a record, key or other sensitive value even if the agent has not made an external call.

Output inspection should also account for rendered URLs. Refuse or strip untrusted URLs so that private data cannot be encoded into a link or image that the client later fetches.

Approval for exports and sends

Hold irreversible data movement for a named person. Approval is relevant to sends and exports because these actions move information beyond the agent’s immediate context. The approval requirement should be attached to the action, rather than relying on the agent to describe accurately what it is about to do.

An approval gate complements least privilege and policy checks. Least privilege limits access, pre-call checks evaluate the proposed operation and approval provides a human decision before the export or send occurs.

How should an assessment be run against an assembled agent?

A practical assessment follows the agent’s data paths rather than testing isolated prompts. Use the following sequence:

• Inventory the sensitive data sources the agent can reach.
• Inventory every exit, including replies, tool arguments, rendered URLs, outbound requests and other agents.
• Place canary values in the relevant data sources.
• Attach the real tools used by the workflow.
• Run injection and exfiltration attacks, including attacks delivered through content the agent reads.
• Inspect the complete trace for canaries.
• Record each confirmed path and the boundary through which the canary passed.
• Apply the relevant control at that boundary.
• Repeat the attack and confirm that the canary no longer leaves through the tested path.

This process separates discovery from remediation. The canary demonstrates the path; the control addresses it; the repeated test checks whether the path has been closed. The same method can be applied to a reply, a tool argument, a URL, an outbound request or an agent hand-off.

The assessment should be performed against the assembled workflow because controls and capabilities interact. A prompt may appear safe until a fetch tool is attached. A data source may appear contained until another agent can receive the context. Testing the complete arrangement exposes those dependencies.

How should production activity be monitored and reassessed?

Record every production session with its tool calls. A leak that passes through the controls must be visible and traceable, including the route by which the data moved. Session records support investigation by connecting the agent’s behaviour with the calls it made.

Add a honeypot export tool that no legitimate conversation calls. Its purpose is to provide a visible signal when a workflow attempts an export that should not occur. It complements session records by making an inappropriate export attempt distinguishable.

Re-run the canary assessment whenever the prompt, tools, model or data sources change. A change in any of these components can alter the available paths. The reassessment should use the same assembled-agent method and inspect the same classes of exit.

Reassessment is also required when no application code has changed. A model update can reopen a path without a code change. The assessment should therefore be tied to changes in the agent workflow and model, not only to changes in surrounding software.

OWASP describes the underlying risk as LLM02 Sensitive Information Disclosure. Redline assesses the risk through its PHI / PII Leakage and Data Exfiltration attack families. Its assessment covers 11,204 attack cases across 16 families, then enforces argument and output policies inside the agent process and records production sessions.

Key takeaways

• Assess the complete assembled agent, including its prompt, tools, credentials, data sources and other agents.
• Map every data exit, not only the final reply.
• Plant canary values in real data locations and treat a canary in any trace as a confirmed leak path.
• Use least privilege, pre-call argument checks, pre-output checks, destination allowlists and approval gates together.
• Record production sessions with tool calls so leaks remain visible and traceable.
• Re-run the assessment whenever the prompt, tools, model or data sources change.

Common questions

Is output filtering enough to prevent data leakage?

No. Replies are only one exit. Data can also leave through tool arguments, URLs the client fetches and outbound requests. Those routes require controls before the call, least-privilege credentials and destination restrictions. Output checks remain necessary, but they cannot inspect or prevent every operation that occurs before a reply is produced.

What is a canary test?

A canary is a planted value, such as a fake key or record, placed where real sensitive data lives. If it appears in any reply, argument or request during an attack, that path leaks. Canaries provide a direct way to confirm that data moved through the assembled agent rather than merely suggesting that a theoretical route exists.

How often should an agent be reassessed?

Reassess the agent on every change to its prompt, tools, model or data sources, and on a schedule. A model update can reopen a path without any code change. Each reassessment should use the assembled agent, its real tools and canary values, then inspect the trace for evidence of leakage.

Where does Redline fit?

Redline runs the assessment against the assembled agent using 11,204 attack cases across 16 families, including PHI / PII Leakage and Data Exfiltration. It then enforces argument and output policies inside the agent process and records production sessions. The assessment and controls address different stages of the workflow: finding paths, restricting calls and inspecting outputs.

Should another agent be treated as a data exit?

Yes. Context handed to a sub-agent can expose information to an agent with broader reach. Apply the same checks to the hand-off as to a tool call. Inspect the context being transferred, compare it with the session and permitted scope, and refuse a hand-off that would move data beyond the intended boundary.

Do attacks need to come from the user’s message?

No. Attacks can arrive in content the agent reads. EchoLeak documents a zero-click prompt injection that exfiltrated data from a production LLM system. Assessments should therefore include retrieved pages and other content sources, not only direct user instructions.

Sources

• EchoLeak
• OWASP AI Agent Security Cheat Sheet
• OWASP LLM02 Sensitive Information Disclosure