How accurate are prompt-injection classifiers?

The table, wins and losses

The comparison was measured on 5 September 2026 against ProtectAI's deberta-v3-base-prompt-injection-v2 and PIGuard. All three models were scored on identical rows in a single run. That detail matters: the comparison is intended to hold the evaluated examples constant while exposing how the models behave across different datasets.

The seven external sets represent different evaluation conditions. They can be checked at their respective dataset pages: BIPIA, Qualifire, StrongREJECT, SORRY-Bench, Gandalf, NeuralChemy and Mindgard.

Redline Guard v5 leads on four sets: BIPIA, Qualifire, StrongREJECT and SORRY-Bench. ProtectAI v2 leads on NeuralChemy and Mindgard, while also reaching the highest recall on Gandalf. PIGuard does not lead on any of the seven sets in this comparison.

The table does not support a universal ranking independent of the data. It supports a conditional conclusion: performance changes with the set, the metric and the operating point. A model that leads on one dataset may trail on another, and the magnitude of that difference can be substantial.

Qualifire is the only row in the table using AUROC rather than recall. The result is 0.9703 for Redline Guard v5, compared with 0.8305 for ProtectAI v2 and 0.8384 for PIGuard. That difference should be read alongside the false-positive discussion rather than treated as a substitute for an operating-point result.

What does false-positive rate add to recall?

Recall without its false-alarm rate is not a complete result. A detector can increase recall by flagging more traffic, but those flags also apply to legitimate messages. In an operational system, the cost is paid where legitimate traffic is inspected, delayed or rejected.

On BIPIA, Redline Guard v5 reached 0.9484 recall at a 1.4% false-positive rate. PIGuard reached 0.8320 recall at 10%. These figures describe more than which detector caught more examples. They show the recall achieved at the stated false-positive operating point.

The same distinction appears on Qualifire. This set contains both classes and therefore measures false positives directly. Redline Guard v5 recorded a 2.2% false-positive rate, compared with 23% for ProtectAI v2. Its AUROC was 0.9703, while ProtectAI v2 recorded 0.8305 and PIGuard recorded 0.8384.

How to read the operating point

For an engineering team, the useful question is not simply which model has the highest recall. It is which model provides an acceptable balance between missed injections and false alarms for the traffic being protected.

A benchmark result should therefore be recorded as a pair wherever both figures are available:

• Recall or AUROC, depending on the set.
• The false-positive rate associated with the selected operating point.
• The dataset and rows used for the comparison.
• The model version and evaluation date.
• Whether the rows were evaluated identically across models.

This prevents a high recall number from being presented without the condition that produced it. It also makes later threshold changes easier to interpret. A change in recall may reflect a changed operating point rather than a change in the underlying ordering of examples.

The BIPIA comparison illustrates the distinction directly. Redline Guard v5 has the higher recall and the lower stated false-positive rate. The Qualifire comparison provides a second example in which the false-positive rate changes the practical reading of the recall result.

Where does the classifier lose?

Three sets go the other way. On Mindgard, Redline Guard v5 records 0.2921 recall against ProtectAI v2 at 0.7210. On NeuralChemy, Redline Guard v5 records 0.5165 against ProtectAI v2 at 0.8813. On Gandalf, Redline Guard v5 records 0.9193 against ProtectAI v2 at 1.0000.

These losses are material and should remain visible in any evaluation summary. A comparison that reports only the four winning sets would give an incomplete account of the measured behaviour. Publishing the complete table allows security teams to assess whether the evaluated pattern matches their own traffic and threat model.

The NeuralChemy result also shows why recall alone can obscure the nature of a difference. Redline Guard v5 has an AUROC of 0.8978 on that set against 0.9750 for ProtectAI v2. The ordering is closer than the recall figures might suggest, while the chosen threshold produces the lower recall shown in the table.

That is a trade rather than an unexplained contradiction. A threshold determines which scores become flags. A detector can have a useful ordering of examples while operating at a point that favours a lower false-positive rate. The relevant production decision is whether that trade is suitable for the input boundary being protected.

The StrongREJECT row requires a separate interpretation. ProtectAI v2 records 0.0000 recall, while Redline Guard v5 records 0.7891 and PIGuard records 0.0863. ProtectAI does not catch StrongREJECT's harmful requests because refusing harmful requests is not what it was built for. At the same time, it pays 23% false positives on clean traffic for the injection recall it does have.

This is why a benchmark row should not be interpreted outside its task. A classifier can be designed for one decision boundary while a dataset tests another. The score remains a useful observation, but the engineering conclusion must account for what the set contains and what the detector is intended to identify.

Why can random splits inflate detector results?

Public injection corpora contain near-duplicates: the same attack template can appear with a different noun. If rows are split randomly, a test row can sit beside its own twin from training. The resulting score can measure memorisation of the shared pattern rather than generalisation to an unseen attack structure.

The measured effect is substantial. A random split placed 29.4% of test rows in the same group as a training row, and 98% of one source's rows were in the same situation. It moved the headline from 0.9544 to 0.9760.

The difference is not evidence of a general improvement in detection. It is evidence that the split changed what the evaluation permitted the model to see during training and testing. When related examples cross the split boundary, the test set is less independent of the training set.

What split should an evaluation disclose?

An evaluation should state how related rows were separated. The BIPIA figure in this comparison uses the quarter held back by hash rather than a random slice. That design is intended to prevent near-duplicate groups from being distributed freely between training and test data.

The split method belongs beside the metric, not in a footnote that can be omitted when the result is quoted. Without it, a detector number should be read as an upper bound. The same metric can describe materially different evidence depending on whether the rows are independent, grouped or related through a shared template.

For security engineers reviewing a published result, the following questions are practical:

• Were all models evaluated on identical rows?
• Was the evaluation performed in a single run?
• Were near-duplicates grouped before splitting?
• Does the reported figure use a random slice or a hash-held-out quarter?
• Is the metric recall, AUROC or a false-positive rate?
• Is the operating point stated alongside recall?

These questions do not require a new benchmark. They determine whether the existing number can support the decision being considered.

What is a classifier for in an agent security model?

A prompt-injection classifier is one gate at the input boundary, not a complete security model. It runs before the model reads the message. Its role is to make straightforward attacks inexpensive to stop so that the remaining controls can handle cases that require more context.

Those controls include scoped tools, approval on irreversible calls and bait that proves a hijack. The classifier should therefore be treated as one layer in a sequence of decisions rather than as permission to allow every message that it does not flag. See runtime guardrails.

This layered design matters because the classifier is probabilistic and sees only the input boundary. It may miss an injection, misclassify legitimate content or encounter an attack expressed in a form that differs from the evaluated sets. The controls that do not depend on the classifier being right provide protection when that happens.

How should teams use the benchmark table?

Use the table to identify the conditions under which a detector is strong or weak. Start with the datasets most closely related to the traffic under review, then examine the metric and false-positive rate. Do not collapse all rows into a single accuracy claim.

Next, treat the losses as design input. The Mindgard and NeuralChemy results indicate cases in which ProtectAI v2 scores higher. The Gandalf result shows a case in which ProtectAI v2 reaches 1.0000 recall. Those rows should influence fallback controls, threshold selection and test coverage.

Finally, validate the chosen operating point against the surrounding agent controls. A detector can stop an input before the model reads it, but it does not replace argument scoping, approval for irreversible tools or a mechanism that can establish that a hijack occurred. The classifier reduces the amount of work passed to those controls; it does not remove the need for them.

Key takeaways

• Prompt-injection classifier accuracy varies by dataset, metric and operating point.
• Redline Guard v5 beat ProtectAI v2 and PIGuard on four of seven evaluated sets and lost on three.
• On BIPIA, Redline Guard v5 recorded 0.9484 recall at a 1.4% false-positive rate, compared with PIGuard at 0.8320 recall and 10%.
• Recall must be read with its false-positive rate; otherwise the operating cost is hidden.
• Random splits can place near-duplicate rows in both training and test data, moving a headline from 0.9544 to 0.9760.
• A classifier is an input gate and should be combined with scoped tools, approval on irreversible calls and bait that proves a hijack.

Common questions

What is BIPIA?

BIPIA is a benchmark for indirect prompt injection. The payload rides inside a document that the agent reads through a tool rather than being typed directly at it. It is the closest public proxy in this comparison for the attack that matters to an agent with tools, because the injection enters through content the agent is processing.

Is a higher recall always better?

No. Recall without a false-positive rate is unbounded: flag everything and recall is 1.0. The pair is the result. A detector at 0.83 recall and 10% false alarms is worse in production than one at 0.95 recall and 1.4%, because the latter achieves the higher recall with fewer stated false alarms.

Why publish the losses?

A table showing only wins is a marketing claim and cannot be checked properly. Three of the seven sets go against Redline Guard v5, and those figures are shown beside the four sets where it leads. Publishing both outcomes lets a buyer or engineer judge whether the measured strengths and weaknesses are relevant to their own evaluation.

Can a classifier be the whole defence?

No. A classifier is probabilistic and only sees the input boundary. Controls that do not depend on the classifier being right are needed when it misses or misclassifies an input. Argument scoping, approval on irreversible tools and honeypots are the controls identified here as holding when the classifier is wrong.

Why is false-positive rate necessary when recall is available?

False-positive rate shows what happens to legitimate traffic at the chosen operating point. Recall alone can be increased by flagging more inputs, but that also increases flags on clean traffic. The BIPIA and Qualifire comparisons demonstrate why recall and false-positive rate should be reported together.

Can results from one dataset predict results on another?

No. The table shows different leaders across the evaluated sets. Redline Guard v5 leads on BIPIA, Qualifire, StrongREJECT and SORRY-Bench, while ProtectAI v2 leads on NeuralChemy and Mindgard and reaches 1.0000 recall on Gandalf. Dataset composition and task alignment affect how each result should be interpreted.

Sources

• ProtectAI's deberta-v3-base-prompt-injection-v2
• PIGuard
• BIPIA
• Qualifire
• StrongREJECT
• SORRY-Bench
• Gandalf
• NeuralChemy
• Mindgard