BENCHMARK RESULTS
Public benchmark results for 768 ATR rules
Every benchmark below includes the raw data source, reproducible methodology, and ATR version that ran it. No cherry-picking.
All precision / FP figures are per-corpus Layer 1 measurements, not engine-wide claims. Benign-gate false positives are lane-based: ~0.24% enforce / ~9% hunt (default) on 65,000 benign samples. There is no single engine-wide FP number.
Garak (NVIDIA jailbreak corpus)
NVIDIA garak is the leading open-source LLM red-teaming framework. We ran ATR v2.1.2 against the full garak corpus to measure adversarial-prompt detection.
Recall
97.2%
Sample size
650 samples
Layer
Regex only (no LLM second opinion)
ATR version
v2.1.2
Reproduce
pnpm bench:garak (in agent-threat-rules repo)SKILL.md (PanGuard wild corpus)
Manually labeled corpus of 498 AI agent skills from ClawHub, OpenClaw, and Skills.sh. Half malicious, half benign. Used to validate that ATR catches threats without false-positive bloat.
Recall
100%
Precision
97%
False positive rate
0.2%
Sample size
498 samples
How to read these numbers
Precision / FP here are Layer 1 (deterministic-rule) measurements on this specific corpus, not an engine-wide figure. Real-world false positives are lane-based: ~0.24% enforce / ~9% hunt (default) on 65,000 benign samples.
Reproduce
pnpm bench:skill (in agent-threat-rules repo)PINT (self-built PINT-format corpus)
A self-built 850-sample corpus in PINT format, assembled from the deepset prompt-injections dataset plus Lakera Gandalf. This is NOT Lakera official PINT benchmark and is not attributable to any third party. Lower recall than Garak/SKILL.md reflects the corpus mixing in SIEM-style detection patterns — Sigma migration via PanGuard Migrator closes the gap.
Recall
63.64%
Precision (PINT-format corpus, self-built)
99.65%
Sample size
850 samples
Layer
Layer 1 — regex only
How to read these numbers
Precision / FP here are Layer 1 (deterministic-rule) measurements on this specific corpus, not an engine-wide figure. Real-world false positives are lane-based: ~0.24% enforce / ~9% hunt (default) on 65,000 benign samples.
Reproduce
pnpm bench:pint (in agent-threat-rules repo)Wild Scan (full ecosystem audit)
Live audit of every AI agent skill we could crawl across ClawHub, OpenClaw, Skills.sh. Not a curated benchmark — actual production skills shipped by real authors. Result: 1,302 flagged out of 96,096 scanned, 552 confirmed malware after manual review.
Skills scanned
96,096
Flagged
1,302
Confirmed malware
552
Triple-threat packages
249
Postinstall scripts
122
Reproduce
scripts/wild-scan.ts (in panguard-ai monorepo)HackAPrompt cluster mining
ATR run against the HackAPrompt EMNLP 2023 competition corpus (4,780 deterministic samples). Result: 69.6% recall, 0 new false positives. The number is honest and below closed-source ML detector claims. Methodology and rule additions documented in the public engineering blog.
HackAPrompt recall
69.6%
Baseline recall
28.6%
Sample size
4,780 deterministic
New FPs introduced
0
Reproduce
pnpm bench:hackapromptWant to run ATR on your corpus and publish the results? Open a PR at Agent-Threat-Rule/agent-threat-rules. We add your benchmark to this page with full attribution.
Reviewed by Adam Lin · Last reviewed 2026-05-12