Prompt-injection defense for agents

# Prompt-injection defense for agents

Reach 0 high-severity findings in a 200-probe red-team run (indirect-prompt-injection, data-exfil, excessive-agency, tool-discovery) against an agent that reads untrusted content, while losing no more than 5 points on its normal task eval.

## Recommended stack

- **promptfoo 0.124 red team** - Attack generation and scoring in CI. Ships agent-specific plugins (indirect-prompt-injection, data-exfil, excessive-agency, tool-discovery, mcp) and runs from the CLI.
- **AI SDK 7** - Agent loop controls. toolApproval pauses side-effecting tools for a human and stopWhen: isStepCount(n) caps loops.
- **Network egress allowlist (proxy or platform firewall)** - Blocks data leaving to unknown hosts. A network rule holds even when the model is fooled; prompt filters do not.
- **OWASP Top 10 for Agentic Applications 2026** - Threat checklist. Names the risks to test: ASI01 goal hijack, ASI02 tool misuse, ASI03 identity and privilege abuse, ASI06 memory poisoning.

## Build steps

1. Map each agent against the lethal trifecta: access to private data, exposure to untrusted content (web pages, email, PDFs, tool output), and a way to send data out (HTTP, email, rendered links or images). Any agent with all three needs a structural fix.
2. Choose a design pattern per agent and write down why: plan-then-execute (fix the tool plan before reading untrusted content), dual LLM (a quarantined model reads untrusted text and returns only typed values), action-selector, or context minimization.
3. Shrink capabilities: per-task tool allowlists, read-only database roles, short-lived credentials scoped to the requesting user, and no tool that fetches arbitrary URLs.
4. Gate side effects with `toolApproval: { sendEmail: "user-approval" }` and show full arguments to the approver; cap loops with `stopWhen: isStepCount(8)` and a per-run token budget.
5. In the UI, do not render remote images or auto-link URLs from model output unless the domain is allowlisted; enforce the same allowlist for outbound requests at the network layer.
6. Write a redteam block in promptfooconfig.yaml (or run `npx promptfoo@0.124 redteam init --no-gui` locally once): target the agent endpoint, set purpose, numTests so plugins x numTests is about 200, and enable indirect-prompt-injection (with config.indirectInjectionVar naming the prompt variable that carries untrusted content), data-exfil, excessive-agency, tool-discovery, and mcp. Then run `npx promptfoo@0.124 redteam run` and triage every finding. Set PROMPTFOO_DISABLE_REMOTE_GENERATION=true if the agent's purpose or data must not leave your network.
7. Schedule the red-team run nightly in CI, fail on any high-severity finding, (parse the results JSON for failed high-severity plugins; there is no built-in severity gate), and turn each fixed exploit into a regression case.

## Watch out for

- Relying on a classifier or an "ignore instructions in documents" system prompt: both are probabilistic and bypassable. Only removing a capability or a data path gives a guarantee.
- Markdown image exfiltration: a rendered image whose URL carries data in its query string leaks it with no tool call at all; strip or proxy remote images in model output.
- Approval fatigue: if every call needs approval, people click yes. Approve only side effects and show a diff of what will change.
- Shared service credentials let a hijacked agent act beyond the user's own permissions (ASI03). Act with the user's scoped identity.

## Definition of done

- A threat table exists per agent showing trifecta status and the chosen pattern.
- The promptfoo red-team report shows 0 high-severity findings.
- A test document containing a planted instruction to send data to an external address produces no outbound request in the egress log.
- The normal task eval scores within 5 points of the pre-hardening baseline.