Prompt caching is a cost-saving mechanism in large-model APIs: when the same opening block keeps appearing across turns or requests, the provider computes it once, stores the result, and reuses it whenever the next request starts with a byte-identical prefix. What you save is both input-token money and the wait for the first token.
It caches a half-finished computation, not an answer
Before answering, a model must process everything you sent, layer by layer; this step is called prefill, and the longer the text, the slower and pricier it gets. Prompt caching stores the intermediate state from prefill (often called the KV state). On a hit, the model skips the repeated work, processes only what you newly added, and then generates the answer token by token as usual. It never replays an old answer: the same cached document paired with a different question still produces a fresh reply.
Three conditions must hold at the same time
First, the prefix must match exactly, from the very first token. System prompts, tool definitions and fixed documents sit at the front and hit most easily; anything dynamic placed early blocks every hit behind it. Second, the block must be long enough: providers set a minimum, roughly around a thousand tokens, so a few short sentences never trigger it. Third, it must be reused inside its lifetime, usually a few minutes, extended on each hit. Details differ: some platforms ask developers to mark cache breakpoints explicitly, charging a small premium on writes and a steep discount on reads, while others match prefixes automatically.
Not the same as the other two caches
Semantic caching stores whole question-and-answer pairs and returns the old answer when a new question is similar in meaning. Prompt caching returns no old answers at all; it only skips repeated computation. The context window is different again: it describes how much text the model can see at once, a capacity question, while caching is about whether that text gets recomputed. A huge window does not make repeated content free.
Three common misconceptions
One: changing a single word at the start is harmless. Matching is exact, so everything from the first difference onward is recomputed; putting a date or a user name at the very front is the classic way to kill your hit rate. Two: caching makes calls free. The matched input becomes much cheaper, but new content and every generated token still bill at full price. Three: longer chats always save more. Once tool results or retrieved chunks that change every turn enter the middle, the later half rarely hits. The practical layout is stable content first, volatile content last.
Prompt caching pays off most for agents with long system prompts and tool definitions, knowledge bases questioned repeatedly over the same long document, and batch jobs sharing one instruction set. If you ask one-off questions with short, ever-changing prompts, you will barely notice it exists.
