Caching: three kinds, three risks

The syllabus names three kinds of caching, and they sit in a very clear order of
risk. Two are close to free and one can quietly break your system.

The three

Prompt caching. Reuses the unchanged prefix of a prompt — the system message,
the tool definitions, the instructions that do not vary. Nearly free, no correctness
risk, and it works precisely because the reused part is identical rather than
similar. Do this one first, always.

Semantic caching. Returns a stored answer for a similar question. The saving
is the largest of the three. So is the risk.

Response caching. Exact-match reuse of a full response. Safe where the question
is genuinely identical, which is rarer in conversational systems than it looks.

What semantic caching gets wrong

The whole mechanism turns on a similarity threshold, and similarity is not
equivalence.

Consider: "What is the policy for contractors?" and "What is the policy for
contractors in France?" Those are semantically very close. They have different
answers. A semantic cache tuned loosely enough to be useful is tuned loosely enough
to confuse them.

Two more failure modes worth naming:

Caching across users leaks. An answer built from one user's permitted data,
served to another user because their question was similar, is an entitlement bypass
with a performance benefit. If the cache key does not include identity, this is not
a risk — it is a certainty waiting for the right pair of questions.

Staleness is invisible. A cached answer does not look old. It looks confident,
and it looks exactly like a fresh one. There is no visual or structural cue that
anything is wrong, which is why cache invalidation on a fast-changing source has to
be designed rather than assumed.

So the rules are

Include identity in the cache key wherever the answer depends on permissions.

Include a version or a time bound wherever the source changes.

Set the similarity threshold conservatively, and accept a lower hit rate as the
price of correctness.

And measure the hit rate honestly, because a semantic cache that hits ten per cent
of the time and is wrong one per cent of the time is not a saving.

Where caching sits on the cost ladder

Fifth of seven — after trimming tool returns, trimming the prompt, fixing handoffs,
and right-sizing the model. It is worth knowing it comes that late, because caching
feels like the sophisticated optimisation and the four things above it are usually
larger and always safer.

The trap

Semantic caching turns up in cost-optimisation scenarios as the impressive-sounding
option, and it is frequently the wrong answer.

The tell is in the setup rather than the question: if the scenario mentions per-user
data, permissions, or a source that changes often, semantic caching is the
distractor. The right answer is either a cheaper lever further up the ladder, or
semantic caching with the cache key qualified — and if the options do not offer the
qualified version, they are not offering the correct one.