Explainer
Why KV caching turns context into a capacity question
Understand the memory side of token generation, then ask better questions about prompt length, concurrency and serving limits.
THE SHORT ANSWER
A cache can avoid repeated work while still becoming a memory and scheduling constraint.
The useful mechanism
Avi Chawla’s archived tutorial introduces KV caching as reuse of attention keys and values during generation. The PagedAttention paper examines how those stored states consume memory and how allocation affects serving capacity. Its paged design addresses fragmentation and sharing. Neither source justifies applying a single speedup figure to every model, sequence length or machine.
Ask two different questions
For an interactive assistant, ask how long a user waits for the first useful token. For a busy service, ask how many requests can remain active without exhausting memory or causing unacceptable delay. Keep prompt length, generated length and concurrent requests visible. A short demo with one user cannot answer the second question, even when the streaming output looks fast.
Plan a bounded comparison
If you later benchmark a server, hold model revision, hardware and generation settings constant. Compare a short and long input at a quiet load, then repeat at the concurrency your application expects. Record failures and timeouts as outcomes. This is a proposed measurement plan, not a serving result. Use it to locate the next bottleneck before selecting an optimization or making a capacity promise.
Put it to work.
- Separate first-token delay and throughput.
- Record input and output lengths.
- Include concurrent load.
- Retain memory failures.
Go to the source.
Original material, with its own scope and publication date. This note is an explanation, not a CooScope performance test.
Source published · Editorial review
Archived English post text; primary sources reviewed separately. Generated annotations, media and missing replies are outside the evidence scope.
- @_avichawla — archived X post (2026-03-20)
- Kwon et al. — PagedAttention paper, version 1Link / source review: 2026-10-15