How an AI application works, and where it breaks.

Why this comes first.

By the end of this section you will understand what this page is for and why the chat window is not the whole system.

Most people first encounter artificial intelligence through a chat window. You type a question, an answer appears, and it is reasonable to assume that the chat window is the system.

In practice, the chat window is only the entrance. The working parts of the application sit behind it. There is somewhere that documents are stored, there are functions that can send an email or change a record, and there are often connections to services built by other companies.

You cannot judge whether an application is safe by examining its entrance alone. This page introduces the parts that sit behind it, and then examines each one in turn.

Understanding these parts will change your work in three specific ways.

• You will ask a more useful question. Asking whether a system can be tricked always produces the same answer, because with enough attempts it can be. The more useful question is what a successful attempt would actually reach, and that question has a definite answer.
• You will know where a control belongs. Every defence described later in this course attaches to one particular part of the system. Being able to point at that part is what turns a general concern into a specific, actionable request.
• You will be able to follow a technical discussion. Engineers describe their systems using the vocabulary introduced below. Sharing that vocabulary is the difference between being consulted while a design is being decided and being informed once it has shipped.

Three principles run through this page and the four steps that follow it.

• You are the defender. Every technique is described by the signal that reveals it and the cost of missing it, rather than as a set of instructions. You will not find a working attack anywhere in this course.
• Two numbers, in every case. A defence is never described only by what it stops. You are also told what it fails to stop, and who it blocks in error. Section 10 explains why that second figure matters so much.
• Published names. Where something corresponds to a recognised industry risk, it is labelled as such, so that what you learn here is something you can raise by name in a meeting.

How a model is made.

By the end of this section you will be able to describe the six stages that produce a model, and identify which one you control.

A model is not written in the way ordinary software is written. It is trained, and that training happens in six stages.

In almost every case, another organisation carries out the first five stages and you inherit the result of their work. Read each stage below with a single question in mind: what does this hand to me?

The six parts of an AI application.

By the end of this section you will be able to name all six parts of any AI product that is described to you.

Different teams choose different frameworks, and they frequently use different words for the same component. Underneath those differences, every AI application is assembled from the same six parts.

Once you can name these six, you will be able to read almost any architecture diagram in this field, including one drawn by someone using an entirely different vocabulary.

One request, traced from beginning to end.

By the end of this section you will be able to follow a single request through a system and say where each piece of it lives.

A user types a short question, such as summarise my tickets. A great deal happens between that moment and the answer appearing on screen.

The diagram below follows that request through all seven steps. Each box names the part, says who produced the text it carries, and gives the place you would actually find it on a working machine.

Colour indicates where text came from. Green marks text you wrote yourself. Red marks text that arrived from outside your organisation. Amber marks text that is a mixture of the two, which means somebody has to decide how far to trust it.

If you are wondering where these parts would be on your own computer, the table below gives the concrete answer for each step. You will build a working version of exactly this in step 4 of the course.

Five points in that sequence deserve particular attention.

• Step 2 is where trust is decided, and it is code that you wrote. Nobody else can record where a piece of text came from on your behalf, and no later step can recover a label that was never attached.
• Step 3 discards those labels. Four sources carrying four different levels of trust arrive as a single block of text. The colours in the diagram exist inside your application; they do not exist inside the model.
• Step 5 cannot be undone. Everything before it is an opinion held in memory. This is the step that justifies a confirmation dialogue (DP-07) and a narrowly scoped credential (DP-01).
• Step 6 returns to the beginning. Whatever the tool produced re-enters as fresh outside text. An agent is therefore not a single request with a filter in front of it, but a cycle in which the door reopens on every pass.
• Step 7 is ordinary web security. It is also the point at which a successful attempt turns into data actually leaving your organisation (DP-02, DP-04).

One flaw, and everything that follows from it.

By the end of this section you will understand why this problem has no clean fix, and what that changes about your work.

Section 04 showed the moment this happens. Four sources of text, each with a different claim to your trust, joined into one block at step 3 of the trace. This section explains why that moment cannot simply be engineered away.

A model receives one stream of text. Your instructions, the user's message, a document your application retrieved, and whatever a tool returned all arrive as a single flat sequence.

There is no separate channel reserved for your rules, and there is no marker on any word that identifies it as an instruction rather than as information.

This means that when a retrieved page contains a sentence shaped like an instruction, the model is not being deceived in any meaningful sense. It is doing exactly what it was built to do, which is to continue the most plausible text. An instruction inside a document looks like an instruction.

That single conclusion reshapes the whole task.

The question is no longer how to detect every hostile message. It becomes what this model is able to do, whose permissions it is using, and what it can reach on the way out.

Detection still has a place. It is inexpensive and it removes the obvious majority. But it is the outer surface rather than the structure. The structure is how the application is built.

Four shapes, and how to recognise them.

By the end of this section you will be able to place any AI product into one of four shapes and describe how much damage a successful attack could cause.

Nearly every AI product takes one of four shapes, or combines several of them. Each shape below is described in the same way: what it looks like, where outside text enters it, how far an attack could reach, and the smallest set of defences worth arguing for before it is released.

Seven places where things go wrong.

By the end of this section you will have seven categories to sort any problem into, and one question to ask at each of them.

Most security advice about this subject arrives as a long flat list, which makes it almost impossible to tell whether you have covered anything.

Layers work better, because a control belongs to a layer, and at every layer you ask the same plain question: what crosses this boundary, and do I trust it?

The published list of risks.

By the end of this section you will recognise the ten named risks and know which layer each one belongs to.

An organisation called OWASP publishes a list of the ten most significant risks in AI applications. It is the closest thing this field has to a shared vocabulary.

You do not need to memorise the list. What matters is recognising the names when an engineer uses one, and knowing where to look when they do.

It is a list of risks rather than a list of fixes, so the useful approach is to place each risk on a layer and then ask what genuinely helps at that layer. The 2026 edition renumbered eight of the ten entries, and where a number changed the previous one is shown alongside it.

Four kinds of defence.

By the end of this section you will be able to identify which kind of defence you are reaching for and predict how it will fail.

Almost every control you will ever build takes one of four forms. They are not ranked against one another, and they are not alternatives. A working system uses all four at different layers.

What matters is knowing which one you have chosen, because each fails in its own characteristic way.

Each of these forms is documented properly, with its components and its costs, in step 3.

The two numbers, and the calculation behind them.

By the end of this section you will be able to perform the one calculation that determines whether a filter is worth deploying. It requires only multiplication.

Every guardrail fails in two directions at once. It misses attacks, and it blocks real people.

Improving either figure on its own is straightforward and useless. Allow everything and you never inconvenience a customer. Block everything and you never suffer a breach. Neither arrangement is a filter.

You cannot improve one number without paying for it in the other, and managing that exchange is the whole of the work.

Consider what happens when those percentages meet real traffic.

Twenty-one out of every twenty-two people you blocked had done nothing wrong. The model is not at fault, and neither is the threshold you chose.

The rarity of attacks is what produces this result. The same effect appears in fraud detection, in spam filtering and in medical screening. When the thing you are searching for is rare, a small error rate overwhelms a high detection rate every time.

Three conclusions follow from that.

• A false positive rate means nothing on its own. It acquires meaning only alongside your traffic volume, because that percentage is a fact about you rather than about the attacker.
• Blocking has a cost, and somebody receives the bill. Usually it is the security professional asking a reasonable question, the customer writing in a language your training data barely covered, or the developer whose ordinary work now appears suspicious.
• This is the argument for isolation. A defence that reduces what a successful attack is worth has no false positive rate at all, which is why section 09 recommends building it first.

How real systems fail.

By the end of this section you will have a checklist you can work through in front of any real system.

The items below are not theoretical weaknesses. They are the patterns that recur, and any one of them that you cannot rule out is worth writing down as a finding.

• The security policy is written into the prompt. A sentence such as "never reveal data belonging to another customer" is written as instruction rather than enforced in the query. Instructions are advice, whereas code is not.
• Retrieved text is not treated as an input. The user's message receives careful scrutiny, while the forty thousand characters of retrieved document sitting beside it receive none.
• The agent runs with administrative permissions. One account with full access, because narrowing it for each task was recorded as a follow-up ticket.
• The answer reaches somewhere consequential. Model output is rendered as raw markup, passed to a command shell, or joined into a database query. This is an old category of bug arriving from a new source.
• The outbound route is open. The model can be told what to reveal and also has a means of transmitting it, whether through an image address it constructs, an outbound request, or a link it renders. Reading, acting and sending within one session is the combination that turns an attempt into an incident.
• Confirmation without comprehension. A dialogue asking whether to run a tool, without showing what it will do, to what, and on whose behalf.
• Memory preserves the attack. Text written into long-term notes continues to take effect in sessions that never touched the original document.
• The loop has no ceiling. A single request continues until either the bill or the rate limiter brings it to a halt.
• Guardrail decisions are not recorded. Neither of your two numbers can be calculated afterwards if you stored only the blocks and not the approvals.
• There is no route of appeal. Nobody wrongly blocked has any way to say so, which makes your false positive rate invisible and therefore, on paper, zero.

What is actually within your control.

By the end of this section you will know which problems you can address yourself, beginning this week.

You have now walked through the whole system. The division below is an honest account of what you can and cannot change. It is shorter than most people expect on one side, and considerably longer on the other.

Every item in the third column is engineering work you can begin this quarter, and none of it requires cooperation from a model vendor.

More importantly, every item in that column continues to protect you on the day an attack succeeds. That is the argument this page has been building towards, and it is why the solutions in step 3 are ordered as they are.

The wider toolchain.

By the end of this section you will know what each layer of the toolchain does and whether adding it helps or hinders you.

The table below lists the layers a team assembles and explains what each one contributes. Product names in this field change quickly, so the final column is the one worth reading closely. It describes what adding each layer does to the number of places an attacker can reach, and that does not change when the names do.

What is still being argued about.

By the end of this section you will know which questions remain open, so that disagreement between experts does not catch you unprepared.

Not all of this is settled. The four questions below are ones on which competent people still disagree, presented with the current state of the evidence.

Can this be solved within the model itself?

One position holds that improved training will eventually make models reliably prefer the instructions their operator supplied.

The opposing position holds that a system built on statistical estimates can always be argued with, given sufficient attempts, and that containment is therefore the only durable answer.

Where matters stand: models have measurably improved at ignoring obvious attacks, while new attacks continue to find the gaps (The Attacker Moves Second, 2025). Build as though the problem remains unsolved.

Should you buy a guardrail or change the architecture?

Vendors selling detection report strong benchmark scores. Architectural approaches such as CaMeL argue for containment that can be reasoned about formally, at the cost of capability and engineering time.

Where matters stand: detector scores tend to fall once somebody attacks them deliberately (Are Firewalls All You Need?). Containment is genuine but slower to build. Most production systems will use both, and should be honest internally about which one is carrying the weight.

Is "simply do not give it dangerous tools" a real answer?

It is the most effective advice available, and it is also the advice product teams reject, because the tools are frequently the product.

Where matters stand: the practical version is not removing capability but binding it. Credentials for one task, a tool list for each step, and a person approving anything final. A capability that cannot survive those constraints is probably not ready to ship.

Does publishing a catalogue of attacks help attackers?

This is the familiar disclosure argument, applied to a new subject.

Where matters stand: these attacks already circulate widely and most are straightforward to attempt. Defenders are the group lacking shared vocabulary, which is why step 2 describes what reveals each technique and what it costs, rather than supplying working examples.

Common questions.

Where to go next.

You can now name the parts of an AI application, recognise the shape it takes, and say where it goes wrong. Step 2 shows what the attacks themselves look like.