The Filter AI / Resources

Check it yourself.

Everything in this course rests on work published by other people, and this page collects it. Each entry is graded by the kind of evidence it represents, and each one states what it does not establish as well as what it does. Where anything here contradicts something written elsewhere on the site, this page takes precedence.

Step 5 of 5 20 sources 3 grades A reference, not a read-through Every link resolved 31 Aug 2026
  1. 01  Build
  2. 02  Attacks
  3. 03  Solutions
  4. 04  Labs
  5. 05  Resources
How grading works

Three grades, and what they mean.

By the end of this section you will know how much weight each source can carry on its own.

This page is not designed to be read from beginning to end. It exists so that you can verify a claim made elsewhere in the course, or read further on a topic that interested you.

A grade describes the kind of evidence a source represents rather than the quality of the work itself.

Some of the most useful writing in this field carries a grade of C, and several grade A papers have since been superseded. The grade indicates only how much weight a claim can carry on its own.

GradeMeansUse it for
APeer-reviewed research, a published standard, or a measurement that can be reproduced from a public method.Claims that carry weight. This is what to cite in a design review.
BDocumentation written by the people who built the thing. Accurate about how it behaves, and not a neutral party on how well it performs.Understanding how something works, rather than how well it works.
CCommentary, classification schemes and framing written by practitioners. Frequently the clearest explanation available, and not a measurement.Building vocabulary and intuition, but never as proof of a claim.
Standards

Standards and taxonomies.

These give you shared words. Their value is less in what they say than in the fact that other people have read the same document.

A

OWASP Top 10 for LLM Applications, 2026

OWASP GenAI Security Project · 2026 edition

The reference list of risks in LLM applications, and the naming scheme used throughout this site. Prompt injection has held the top position across every edition. The 2026 revision renumbered eight of the ten entries, moved excessive agency up to third, dropped improper output handling to tenth, and renamed system prompt leakage to hidden context exposure to cover everything an application holds and never shows you. Where a number changed, this site prints the old one beside it.

Does not prove: that these are the ten most common risks in the wild. It is expert consensus weighted with incident data, not measurement. And it is a list of risks, not a set of fixes. Also note: the 2026 ordering used here was taken from the published edition summary and cross-checked against two independent write-ups that agree exactly. OWASP’s own per-risk pages under genai.owasp.org/llm-top-10 still carried 2025 numbering when this was checked on 31 August 2026, so deep links to individual risks point at the 2025 slugs.

A

NIST AI 100-2e2025 · Adversarial Machine Learning

NIST · 2025

The most rigorous taxonomy of attacks and mitigations available, spanning predictive and generative systems, with a literature base of several hundred references. The 2025 edition adds explicit treatment of AI agents and of misuse violations. PDF.

Does not prove: that any listed mitigation works in your system. It is a taxonomy and a survey. It describes on purpose. It does not prescribe.

A

MITRE ATLAS

MITRE · continuously updated

ATT&CK-style tactics and techniques for attacks on AI systems, with case studies drawn from real incidents. Useful when you need to describe an attack chain to people who already think in ATT&CK.

Does not prove: prevalence. Case study coverage reflects what was published and disclosed, which is a small and unrepresentative slice.

C

The Arcanum Prompt Injection Taxonomy

Jason Haddix, Arcanum Information Security · v1.6

The most granular open classification of prompt injection specifically: intents, techniques and evasions, each with a citable code and cross-references to how OWASP, ATLAS and NIST name the same thing. Repository, with the data as plain JSON.

Does not prove: effectiveness or frequency. It is a naming system built from practitioner experience, which is exactly what it claims to be. It is useful when the OWASP list of ten is too broad for what you are looking at.

Foundations

Foundational papers.

A

Universal and transferable adversarial attacks on aligned language models

Zou et al. · 2023

Demonstrated automatically generated suffixes that transfer across models, including ones the attack was never optimised against. The reason "we fixed the jailbreaks we knew about" is not a security posture.

Does not prove: that current models are equally vulnerable. Specific published strings get trained against; the search procedure that produced them does not go away.

Defences

Defences with published methods.

A

Defending against indirect prompt injection attacks with spotlighting

Hines et al., Microsoft · 2024

Formalises marking untrusted content so the model can distinguish it from instructions: delimiting, datamarking and encoding. The basis of DP-05, and the cheapest structural improvement most teams can make.

Does not prove: robustness against an adaptive attacker. It raises the cost of an attack. Do not make it load-bearing.

A

StruQ: defending against prompt injection with structured queries

Chen et al. · 2024

Separates instructions from data at the model level by training on a structured query format, rather than asking a general model to respect a convention at inference time.

Does not prove: applicability to hosted models. The approach requires influence over training, which most application teams do not have.

A

Defeating prompt injections by design (CaMeL)

Debenedetti et al. · 2025

Extracts control flow from the trusted query and runs it in an interpreter, with capabilities attached to data so untrusted values cannot redirect execution. The strongest containment story published so far, and honest about the capability it costs. Basis of DP-12. Code.

Does not prove: practicality at product scale. The evaluation is on a benchmark, the engineering cost is substantial, and the paper documents residual channels rather than claiming a total solution.

C

The dual LLM pattern

Simon Willison · 2023

The clearest short description of splitting a system into a privileged model that holds tools and a quarantined model that reads untrusted content. Basis of DP-11.

Does not prove: anything empirically. It is a design proposal, and the author says so; its value is that it makes the architecture obvious once you have read it.

Evaluation

Evaluation and skepticism.

The papers that stop you over-trusting the section above. Every one of these is a reason to prefer containment over detection.

Tooling

Tooling you can actually run.

Grade B throughout: first-party documentation from the people who built it. Accurate about behaviour, interested in the outcome.

B

LLM Guard

Protect AI · open source

Scanners you can stack on input and output. Injection detection, secret detection, personal data, topic limits. The code is readable enough to learn from even if you do not deploy it.

Does not prove: its own detection quality. Measure on your traffic, and log the allows as well as the blocks or you cannot compute precision at all.

B

PurpleLlama (Llama Guard, Prompt Guard, CyberSecEval)

Meta · open weights and tools

Open classifier models for input and output moderation, plus a security evaluation suite. Useful as a self-hosted baseline when sending traffic to a third-party scoring API is not acceptable.

Does not prove: coverage outside the data it was trained on. That matters most in less-resourced languages, where a detection gap turns into a fairness problem.

B

NeMo Guardrails

NVIDIA · open source

Programmable rails for conversational flow, topic boundaries and tool use, expressed as configuration rather than as prompt text. Sits closer to the constrain shape than the filter shape.

Does not prove: containment. Rails constrain a cooperative model; they are not an authorisation boundary and should not be mistaken for one.

B

Azure AI Content Safety · Prompt Shields

Microsoft · product documentation

A managed classifier for direct and indirect injection. The documentation is worth reading even on another platform, because it distinguishes the two delivery routes clearly at the API surface.

Does not prove: published detection figures on your data. Vendor evaluation sets are not your traffic distribution.

C

Gandalf

Lakera · free browser game

The best known attacker-seat game: extract a password through escalating defences. Genuinely instructive about how brittle prompt-level protection is, and it is where most people's intuition about this comes from.

Does not prove: defensive skill. It trains the attacker seat. That is precisely the gap this site exists to fill.

Commentary

Ongoing commentary worth following.

C

Simon Willison on prompt injection

Ongoing since 2022

The longest-running practitioner record of the problem, including the coining of the term. If you want to understand why the field has not solved this, reading it in order is the fastest route.

Does not prove: anything on its own. It is commentary, consistently well-sourced, and it links to the primary work.

C

The lethal trifecta

Simon Willison · 2025

The framing that an agent is dangerous when it combines access to private data, exposure to untrusted content, and a way to communicate externally. The most useful triage question available, and the reasoning behind DP-04.

Does not prove: completeness. Destructive actions need no outbound channel at all, so removing the third leg reduces exfiltration rather than risk in general.

Method

What is missing.

Every link on this page was opened and verified on 31 August 2026.

A source is included when it is the original work rather than a description of it: the standard itself, the paper itself, or the maintainer's own documentation.

Where the course makes a claim that rests on a single source, that source is linked at the point of the claim as well as appearing here.

The gaps below matter as much as the entries above, because a reading list presented as complete is misleading by omission.

  • Real incident numbers are not published. There is no public record of actual attacks with their frequencies and outcomes. Every statement about how common this is, including any on this site, is an estimate.
  • Detection quality outside English is barely measured. The gap is widely admitted and thinly quantified. That is why switching language is treated here as a fairness problem, not just a coverage one.
  • False positive rates are almost never published. Vendors report what they catch. The second number is left to you. That absence is the entire reason the base rate calculator exists.
  • Agent defences are young. AgentDojo and CaMeL are recent. The work on attacking them is younger still. Expect this section to age fastest.

Found something wrong?

Out of date, mis-stated, or missing entirely. Get in touch. Corrections to this page are more welcome than compliments about the others.

You have finished the course

Across five steps you have learned to name the parts of an AI application, identify where it goes wrong, recognise the attacks, decide what to build instead, and build a working example yourself.

Returning to step 1 now is worth the twenty minutes, because it reads very differently once the rest is in place.

The sources explain it. The exercises make it stick.

The Filter AI turns this material into 35 scored decisions across seven units, every one anchored to a named risk. Every answer is scored on both numbers. Unit one is free.

Get the full training $39.90 Start again at step 1