The KV cache holds intermediate results a model keeps while generating text. Each time the model produces a token, it stores the related Key and Value vectors; on the next step, work that is already done is reused instead of recomputed, and only the new token is processed. The name comes from those two stored sets of numbers, and the KV cache is one of the most common ways to speed up long conversations and large-model inference.
What exactly does it save?
Text generation runs step by step. Without a KV cache, generating each new token means recomputing everything that came before, so longer conversations get slower with every step. With the cache, earlier results are reused and each step only handles the new token, so response time does not climb steeply as the conversation grows. What is saved is repeated computation, not a gain in intelligence: answer quality does not improve just because caching is used.
It can also run out of room
The KV cache uses GPU memory, and it grows with conversation length and model size. In a very long conversation it can fill the available memory, forcing the system to drop early parts of the conversation or move some cache entries to slower memory. Replies may then slow down or lose details mentioned at the start. Many products limit the length of a single conversation for exactly this reason: it is not that the model refuses to remember, but that storing and computing everything becomes too costly.
Three common misunderstandings
First, the KV cache is not the model's memory. Close the page or switch devices and the cache is gone; the model has not "learned" anything about you from it. Second, it is not the same as a long-term memory feature. The memory offered by a product is usually a separately stored summary or note, which is different from the cache used during inference. Third, a high cache hit rate does not mean the answer is correct. Caching only guarantees faster computation; whether the model answers correctly depends on its training and input.
Understanding the KV cache helps explain everyday behaviour: a conversation that slows down as it grows is usually paying more in cache and computation, and a fresh conversation feels fast again largely because the cache has been cleared and restarted. When that happens, starting a new conversation and restating the key information is usually quicker than repeatedly re-asking in the old one.
