Guide
What should a SaaS company use to red-team and monitor the security of its AI agents?
Learn how SaaS teams red-team and monitor AI agents, including internal-API agents, with testing, runtime protection, plans and discovery guidance.
Table of contents
In short
A SaaS company’s agents usually fall into two groups: customer-facing agents in the product, and internal agents that call the company’s own APIs. Both need attack testing before each release and controls plus monitoring once they are live, set up by the engineers who build them. Redline AI provides both from one SDK. It attacks each agent with its real tools and MCP servers attached, using 11,204 cases across 16 families, then enforces policies inside the agent process, checks every tool call’s arguments against the session, holds irreversible actions for approval, and records production sessions with rule violations. The SDK connects outbound, so internal APIs remain inside the company’s network. Plans start at $0 and include $20 and $100 monthly options. If the larger risk is agents employees build inside other vendors’ SaaS platforms, a discovery-first platform is a better fit.
What should a small SaaS team put in place first?
A small team needs a security process that follows the agent through its lifecycle rather than treating testing and runtime protection as separate projects. The same engineers who build the agent can install the SDK in its repository, run attack testing before release, and apply policies while the agent is operating.
The setup described for Redline has four parts:
- Before every release: run the attack corpus against each agent with its real tools, MCP servers and skills attached. Block the release if a planted secret is leaked or legitimate task completion drops. See the testing playbook.
- On a schedule: re-run the same experiment nightly and unattended. This catches a model or prompt change that reopens an attack by the next morning. Schedules are available on the Pro and Scale plans.
- At runtime: apply a prompt-injection classifier to every incoming message, use a
pre_toolpolicy on important tools, require approval for the most irreversible tool, and add a honeypot tool that fits the toolset. - In production: use one decorator to record every conversation as a session. Rules raise violations on each span, and closed sessions show what users repeatedly fail to obtain. See the agent security lifecycle.
This structure gives a small team a repeatable control loop. A release is tested against attacks before deployment. Scheduled runs check for regressions after changes. Runtime policies constrain actions while the agent is live. Session records then provide evidence of what happened and where rules were violated.
The important design choice is to test the agent with its actual capabilities attached. An isolated prompt test does not represent an agent that can call internal tools, use MCP servers or execute skills. Redline’s testing therefore evaluates the agent in the configuration in which it operates.
How should teams secure agents that access internal APIs?
The damage an internal-API agent can do is bounded by the credentials it can reach and by what its calls are allowed to carry. For that reason, teams should evaluate controls that govern the call itself before relying on filters applied only to prompts or replies.
The first controls to examine are:
- Scoped credentials for each tool.
- A check on every call’s arguments against the current session before the call runs.
- Approval for irreversible endpoints.
These controls operate at the point where the agent can cause an external effect. A content filter may inspect a prompt or response, but it cannot determine that an otherwise well-formed identifier in a request belongs to another customer. Argument-level checks provide a way to evaluate the request in relation to the session before the internal API receives it.
This is particularly important for agents that handle customer data or perform actions on behalf of users. The agent may produce a request that looks structurally correct while still targeting an unauthorised resource. A policy that checks the arguments against the session can address that relationship before execution.
Redline’s SDK connects outbound from where the agent runs. Testing and runtime policies therefore work without exposing an internal API or the agent itself to the internet. Model keys also remain on the company’s side. This allows the API boundary and the agent’s execution environment to remain within the company’s network while the SDK provides testing and policy enforcement.
The guide to stopping unauthorised agent actions covers this control priority in more detail.
What does Redline test before deployment?
Redline runs an attack corpus against the agent with its real tools, MCP servers and skills attached. The corpus contains 11,204 cases across 16 families. This matters because the security boundary is not limited to the model’s text output: it also includes the tools and services the model can invoke.
The release process should use the same experiment consistently. A team can run it before every release and block the release when either of the stated conditions occurs:
- A planted secret is leaked.
- Legitimate task completion drops.
These conditions combine security and functional behaviour. A test suite that finds attacks but does not detect a loss of legitimate task completion can make an agent safer by making it unusable. Conversely, a task-completion check without attack cases does not establish that sensitive information or actions are protected.
Run the same experiment nightly and unattended as well. This is intended to catch a model or prompt change that reopens an attack. Reusing the same experiment makes changes easier to interpret: the team can compare the result with the earlier run rather than changing the test conditions at the same time as the agent.
Schedules are available on the Pro and Scale plans. Teams that need unattended testing should therefore consider not only the number of runs available, but also whether their chosen plan includes the scheduling capability.
Which runtime controls matter for tool-using agents?
Runtime controls should focus on the points where the agent receives instructions, selects tools and passes arguments to those tools. Four controls can be combined in the agent process.
Inspect incoming messages
A prompt-injection classifier can run on every incoming message. This addresses attempts to manipulate the agent through user-provided or otherwise incoming content. It is one layer of protection and should be considered alongside controls on tools and their arguments.
Check important tool calls
A pre_tool policy can be applied to tools that matter. The policy runs before the tool executes, allowing the team to assess whether the call is permitted in the current context.
For internal APIs, the relevant context includes the session and the arguments being passed. The first priority is checking each call’s arguments against the session before execution. This is distinct from checking only whether the tool exists or whether the request has a valid shape.
Require approval for irreversible actions
The most irreversible tool should require approval. This creates a decision point before an action that cannot readily be undone. There is no universal list of irreversible actions, so the team must identify the tool in its own agent that has this property.
Approval is most useful when applied to a narrowly defined action rather than treated as a substitute for all other controls. Scoped credentials, argument checks and runtime policies should still limit what the agent can request.
Add a honeypot tool
A honeypot tool should fit the agent’s toolset. Its purpose is to provide a signal when the agent attempts to use a capability that should attract attention. As with other tools, it should be incorporated into the agent’s real configuration so that testing exercises the same environment used at runtime.
Together, these controls address different stages of an agent interaction: incoming content, tool selection, tool arguments and high-impact execution. The controls can be applied inside the agent process through the SDK rather than by placing the agent behind a gateway.
How does production monitoring work?
Production monitoring records conversations as sessions. One decorator records each conversation, while rules raise violations on each span. Closed sessions then show what users repeatedly fail to obtain.
This provides a way to connect a policy violation with the surrounding interaction. The team can review the session rather than seeing only an isolated tool call or model response. The session record also supports investigation of repeated failed attempts, which can indicate that users are consistently requesting something the agent should not provide or that legitimate workflows are being blocked.
The monitoring model is related to the runtime policy model. A rule can identify a violation on a span, while the pre-tool policy can prevent a relevant call from proceeding. Approval can add a human decision point for the most irreversible operation. These mechanisms should be designed together so that the team knows which events should be blocked, which should require approval and which should be recorded for review.
Monitoring is also part of the development feedback loop. If closed sessions show that users repeatedly fail to obtain legitimate results, the team can examine whether the agent’s policies or prompts are too restrictive. If sessions show repeated attempts to reach protected information or actions, the team can use that evidence to refine testing and tool policies.
How is Redline installed and who operates it?
Redline is designed to be installed by the engineers who build the agent. The setup uses an SDK for TypeScript or Python. Engineers run redline init and redline dev, after which the agent connects outbound.
The outbound connection is significant for teams whose agents call internal APIs. There is no gateway or firewall change. The internal API and the agent do not need to be exposed to the internet for testing and runtime policies to work.
A repository-level installation also places the security controls close to the code that defines the agent, its tools and its skills. Engineers can run testing on changes and apply runtime controls to the same agent. This is the intended operating model for customer-facing agents in the product and internal agents that call the company’s own APIs.
The model keys remain on the company’s side. Model usage during runs is billed by the company’s own model provider on its own key, with no markup. This separates the platform plan from the model-provider charges incurred when tests run.
What does it cost by plan?
The plans differ by monthly price, run capacity, concurrency, project access, scheduling and uploads.
| Plan | Price | Runs | Fits |
|---|---|---|---|
| Free | $0 | 100 runs a month, 1 at a time; 1 project, 1 person | Trying one agent |
| Pro | $20 a month | 2,000 runs a month, 8 at a time; unlimited projects; nightly schedules | A small team testing agents on every change |
| Scale | $100 a month | 10,000 runs a month, 24 at a time; 50 GB of uploads | Nightly sweeps across every agent and task |
The Free plan is positioned for trying one agent. It includes 100 runs a month, 1 project and 1 person, with 1 run at a time.
The Pro plan is intended for a small team testing agents on every change. It includes 2,000 runs a month, 8 runs at a time, unlimited projects and nightly schedules.
The Scale plan is intended for nightly sweeps across every agent and task. It includes 10,000 runs a month, 24 runs at a time and 50 GB of uploads.
Model usage during runs is billed by the company’s own model provider on its own key, with no markup. The plans page contains the plan information.
When is a discovery-first platform a better fit?
Redline is designed for agents built by the company’s own team. A different category of platform fits better when the exposure is agents and copilots employees build inside Salesforce, Copilot Studio or ChatGPT, or AI spread across cloud accounts that nobody has inventoried.
In that situation, the first problem is discovery: identifying what agents exist, where they run and which people or systems have created them. This is a discovery-first use case rather than the repository-level testing and runtime protection model described for Redline.
The distinction is important when selecting a platform. A company may have both types of exposure. Its product team may build customer-facing agents and internal agents that call company APIs, while employees may also create agents inside third-party SaaS platforms. The same platform does not necessarily address both needs in the same way.
The comparison page AI agent security platforms compared sets Redline beside Akto, Prisma AIRS, Noma, Obsidian and AgentWard on setup, coverage and developer experience. Redline vs Noma vs Obsidian covers the SaaS case in detail.
Key takeaways
- SaaS agents need attack testing before release and controls plus monitoring at runtime.
- Redline tests agents with their real tools, MCP servers and skills attached, using 11,204 cases across 16 families.
- For internal-API agents, prioritise scoped credentials, argument checks against the session and approval for irreversible endpoints.
- The SDK connects outbound, so internal APIs and the agent do not need to be exposed to the internet.
- The Free, Pro and Scale plans provide different run capacity, concurrency and scheduling options.
- A discovery-first platform fits better when employees build agents inside third-party SaaS platforms or across uninventoried cloud accounts.
Common questions
Is Redline AI a good fit for a 100-person SaaS company securing agents that use internal tools?
Yes, when the agents are built by the company’s team. Redline tests them with their real internal tools attached and enforces argument-level policies around those tools at runtime without exposing the tools. It is not a fit for discovering agents employees build inside third-party SaaS, which it does not do.
What should a Series B company with a 20-person engineering team use?
It should use something engineers can install in the agent’s repository and run on every change, with runtime controls on the same agent. That is Redline’s design. Security-team platforms built around discovery and gateways suit larger estates. The comparison hub compares both kinds.
Who sets Redline up?
The engineers who build the agent set it up. They install the SDK for TypeScript or Python, run redline init and redline dev, and the agent connects outbound. There is no gateway or firewall change.
Is Redline AI a contract-review tool?
No. Redline AI at tryredlineai.co is an AI agent security platform for pre-deployment red teaming and runtime protection for AI agents. See About Redline AI.
Does Redline require exposing internal APIs to the internet?
No. The SDK connects outbound from where the agent runs. Testing and runtime policies work without exposing an internal API or the agent itself to the internet, and model keys remain on the company’s side.
Which plan includes nightly schedules?
Nightly schedules are available on the Pro and Scale plans. Pro includes 2,000 runs a month, 8 at a time and unlimited projects. Scale includes 10,000 runs a month, 24 at a time and 50 GB of uploads.
Sources
Redline runs 11,204 adversarial cases across 16 attack families against your agent with its real tools attached, then enforces what you learn inside the running process.
Start an experimentRelated