KV Cache in LLMs Explained: How It Works and Memory Usage

KV Cache in LLMs is a memory mechanism that stores previously calculated Key and Value tensors during text generation. Instead of recalculating attention information for every previous token, the model reuses stored values and computes only what is needed for the next token. This reduces redundant computation and makes autoregressive text generation faster.

The tradeoff is GPU memory. As conversations grow longer, the cache requires more storage, limiting how many requests a model can process simultaneously. Understanding this relationship is essential for LLM Inference Optimization, especially when working with long contexts, limited GPU resources, or production applications serving multiple users.

How KV Cache Works in LLMs

Large language models generate text one token at a time. Each new token depends on the information that came before it.

Without caching, the model processes the previous context again when generating each subsequent token. This repeats calculations that have already produced valid results.

KV caching removes much of that repeated work.

To understand the mechanism, consider the three components of the Attention mechanism:

Query (Q): Represents the information the current token seeks from the available context.

Key (K): Represents information used to determine how strongly tokens should attend to one another.

Value (V): Contains information combined according to the calculated attention weights.

During inference, the model calculates Key and Value tensors for the tokens in the input prompt and stores them in the cache.

When generating the next token, it retrieves the stored Key and Value tensors and computes the new information required for the current position.

The new Key and Value tensors are added to the cache. The process continues as the model generates additional tokens.

KV Cache Example

Consider an LLM processing this prompt:

"The company's revenue increased because"

The model processes the input and generates the next token, such as "sales."

At the next generation step, it needs to consider both the original prompt and the newly generated token.

Without KV caching, the model repeats the relevant calculations for the entire sequence.

With KV caching, the Key and Value tensors for the original prompt are already available. The model processes the new token and combines its information with the stored tensors.

As the response grows, the model continues extending the cache rather than rebuilding the previous context.

This is the central benefit of KV caching: previously calculated attention information remains available throughout generation.

Why Query Is Not Cached

A common point of confusion is why the mechanism stores Key and Value tensors but not Query tensors.

The reason comes from how causal attention works during generation.

The current token's Query is used to calculate attention against the Key tensors of the tokens available at that step. Those attention weights are then applied to the corresponding Value tensors.

Once the current token has been processed, its Query is not needed to calculate attention for a future token.

The next generation step creates its own Query.

Previous Key and Value tensors remain useful because every subsequent token can attend to earlier positions.

This makes storing K and V valuable while retaining historical Query tensors would add memory consumption without serving the same computational purpose.

KV Cache vs No Cache

The difference becomes clearer when comparing the computation required during consecutive generation steps.

Imagine a prompt containing 1,000 tokens.

After the model begins generating its response, the context grows with every new token.

Without a KV cache, calculating the next token involves processing the preceding sequence again. The model repeatedly produces Key and Value representations for tokens it has already processed.

With a KV cache, previously calculated representations remain stored. The model computes the new token's representations and uses them alongside the existing cache.

For a conventional causal Transformer, this changes the amount of repeated computation required during incremental decoding.

Without caching, each new step includes computation over the preceding token representations. With caching, the newly computed attention still needs to consider the previous positions, but the historical Key and Value projections do not need to be recreated.

This distinction is important. KV caching does not eliminate attention computation over the context. It eliminates the need to recalculate the stored representations of previous tokens.

The speed benefit becomes particularly valuable as the generated sequence grows.

However, the first processing of a completely new prompt still requires computation. The cache created during that operation primarily benefits subsequent generation steps.

Reusing the same prompt across separate requests requires an additional mechanism, such as prefix caching, rather than relying solely on the cache created within one request.

KV Cache Memory Usage

The performance benefit of KV caching comes with a memory requirement that increases as more tokens are retained.

Every cached token contributes Key and Value tensors across the model's attention layers.

For standard full attention, the theoretical cache size grows linearly with the number of stored tokens.

A longer context therefore requires more KV memory.

The number of concurrent requests matters as well. Each active sequence needs access to the attention information associated with its context.

Consider a server processing four separate conversations. If each conversation contains 4,096 retained tokens, the system needs sufficient cache capacity for those sequences, subject to any supported sharing or memory reuse.

This requirement exists in addition to the memory occupied by model weights, temporary tensors, and runtime operations.

KV Cache and GPU Memory

GPU memory is a limited resource shared by model parameters and execution data.

If model weights occupy most of the available capacity, less memory remains for KV caching.

Longer prompts, longer generated responses, and additional active sequences can push total memory usage beyond the available limit.

The result can be an allocation failure, reduced concurrency, or additional latency when the serving system must move data or reschedule requests.

When evaluating a model for production, we need to consider more than whether its weights fit on the GPU.

The relevant question is whether the weights, expected KV cache, and execution overhead fit under the intended workload.

Grouped Query Attention and Memory Efficiency

Not every Transformer architecture stores the same number of Key and Value heads.

Traditional Multi Head Attention uses a corresponding set of Key and Value heads for its attention heads.

Grouped Query Attention, or GQA, allows multiple Query heads to share fewer Key and Value heads.

Multi Query Attention, or MQA, extends that idea by using a single Key head and Value head shared across Query heads within an attention layer.

This reduces the number of stored Key and Value elements.

For example, a model with 32 Query heads and eight KV heads needs one quarter of the KV tensor storage of an otherwise equivalent architecture using 32 KV heads.

The total cache requirement still depends on the number of layers, head dimensions, context length, and storage precision.

This explains why parameter count alone cannot determine a model's KV cache memory consumption.

Calculating KV Cache Size

KV Cache Memory Calculation is one of the most useful steps when estimating hardware requirements for LLM inference.

For a conventional decoder Transformer with equal attention dimensions across its layers, the theoretical storage requirement is:

KV Cache Bytes = 2 × Layers × Sequence Length × KV Heads × Head Dimension × Bytes per Element × Active Sequences

Each component has a specific meaning.

The factor of two represents the Key and Value tensors.

Layers refers to the number of attention layers storing those tensors. Sequence Length is the number of retained tokens per sequence, including prompt tokens and the generated context represented in the cache.

KV Heads is the number of Key and Value heads, not necessarily the number of Query heads. Head Dimension describes the size of each head.

Bytes per Element depends on the cache's numerical precision. FP16 and BF16 each require two bytes per stored element.

The Active Sequences factor applies when calculating memory for multiple sequences of equal retained length. When sequence lengths differ, their individual token counts must be added instead.

Practical Memory Calculation

Consider a hypothetical LLM configuration with these specifications:

Attention layers: 32

KV heads: 8

Head dimension: 128

Context length: 4,096 tokens

Cache precision: BF16

Active sequences: 1

The calculation is:

2 × 32 × 4,096 × 8 × 128 × 2 = 536,870,912 bytes

That equals 512 MiB of theoretical KV cache storage.

If the retained context increases to 8,192 tokens, the requirement becomes 1 GiB.

Four independent sequences with 4,096 retained tokens each require approximately 2 GiB of KV tensor storage under the same assumptions.

These figures describe the stored tensors, not the total GPU memory allocated by the serving system.

Actual memory usage also includes allocation overhead, model weights, temporary buffers, and memory reserved by the runtime.

Architectures using sliding window attention, compressed latent attention, or different cache layouts require calculations adapted to their internal design.

KV Cache Types and Storage Strategies

KV Cache implementations differ in how they allocate memory, store values, and manage access to earlier tokens.

The distinction matters because the fastest allocation method is not necessarily the one that uses the least memory.

Dynamic KV Cache

A Dynamic KV Cache expands as additional tokens are processed.

The serving system adds new Key and Value tensors as the sequence grows.

This avoids reserving the entire maximum context capacity for every request in advance.

Dynamic allocation is practical when requests have different lengths, but changing tensor shapes can limit compatibility with execution techniques that depend on fixed memory shapes.

Static KV Cache

A Static KV Cache reserves storage for a specified maximum sequence length.

Its fixed dimensions support compilation and execution optimizations that benefit from predictable tensor shapes.

The tradeoff is reserved capacity.

If a request uses only a small portion of the allocated context, the remaining storage can represent unused memory capacity.

Static caching is worth considering when fixed tensor shapes provide a measurable execution benefit and the expected sequence lengths justify the reservation.

Quantized KV Cache

Quantized caching stores Key and Value tensors using reduced numerical precision.

For example, storing suitable cache tensors in eight bits instead of sixteen reduces their raw storage requirement by approximately half.

Actual savings depend on quantization metadata, residual storage, and implementation details.

The additional conversion operations can also affect execution speed. For short contexts that fit comfortably in GPU memory, quantization overhead can outweigh the memory benefit.

KV cache quantization is distinct from quantizing model weights. Both reduce memory requirements, but they operate on different data.

Offloaded KV Cache

Cache offloading moves selected KV tensors from GPU memory to another storage tier, such as system RAM.

This frees GPU capacity for model execution and other active data.

The cost is additional data movement.

If the model repeatedly transfers large amounts of cached information between CPU and GPU memory, transfer latency can reduce generation performance.

Offloading is useful when memory capacity is the primary limitation and the application can accommodate the additional transfer overhead.

KV Cache Memory Management

The most effective cache management strategy depends on the application's context length, available GPU memory, and performance requirements.

Rather than treating every implementation as interchangeable, it helps to match each approach to the problem it solves.

KV Cache Management Comparison

The following table provides a quick decision guide for choosing a cache strategy.

These approaches solve different problems and are not mutually exclusive.

For example, PagedAttention changes how cached data is allocated and accessed, while quantization changes its numerical representation.

Prefix caching addresses repeated computation across compatible requests rather than reducing the size of every stored tensor.

The correct choice depends on whether the bottleneck is memory capacity, allocation efficiency, repeated prompt processing, or execution latency.

PagedAttention

PagedAttention addresses the difficulty of storing caches for requests with unpredictable sequence lengths.

Instead of requiring a large contiguous memory allocation for every sequence, it organizes Key and Value tensors into blocks.

The blocks can be allocated as needed and accessed through a mapping maintained by the serving system.

This reduces memory fragmentation and unused reservations, improving the amount of available GPU memory that can be used productively.

PagedAttention does not change the theoretical number of Key and Value elements required by a conventional full attention model.

Its main contribution is more efficient storage management.

Prefix Caching

Prefix caching allows a serving system to reuse KV tensors from a previously processed prompt when a new request shares the same valid prefix.

Consider an application that sends the same lengthy system instructions and reference document with multiple user questions.

Without prefix reuse, the shared content must be processed again for each request.

With a compatible prefix cache, previously calculated tensors can be reused, reducing the repeated work required to process that shared context.

This improves prompt processing efficiency.

It does not directly accelerate the computation of new output tokens after the shared prefix has been processed.

A valid cache match must also account for the exact token sequence, model configuration, positional information, and other execution conditions required by the serving engine.

KV Cache in Production

KV caching becomes an operational concern when a model serves multiple users, supports long conversations, or operates under strict GPU memory limits.

A production system needs to manage cache allocation, growth, reuse, and release as requests enter and leave the execution queue.

Estimating Concurrent Request Capacity

The first calculation should separate memory reserved for model weights and runtime operations from memory available for KV storage.

For example, a GPU with 24 GiB of memory cannot dedicate all 24 GiB to the cache.

If model weights and execution overhead consume 16 GiB, the remaining theoretical capacity is 8 GiB before additional allocation requirements and safety margins.

Using the earlier example of 512 MiB per 4,096 token sequence, that capacity corresponds to 16 such sequences in pure KV tensor storage.

This is a theoretical memory calculation, not a guaranteed concurrency limit. The runtime requires additional capacity, and real requests do not all have identical lengths.

The correct operational limit must be established through measurements.

Cache Efficiency Under Long Conversations

Long conversations present two separate challenges.

First, the retained context grows as the user and model exchange messages.

Second, a serving application must ensure that its available cache capacity supports the context required by each active conversation.

For standard full attention, longer retained contexts create larger caches.

Models using sliding window attention can restrict cache growth in the relevant layers by retaining a limited attention window.

That behavior depends on the model architecture. Removing older cached positions from a full attention model changes the available context and can change its predictions.

Cache eviction must therefore respect the model's attention behavior rather than simply discarding the oldest tensors.

Evaluating Cache Performance

We recommend evaluating cache behavior using representative prompt lengths and output lengths rather than a single short test.

The measurements should include GPU memory usage, token generation speed, request completion time, and the number of active sequences supported before capacity becomes constrained.

For prefix caching, the cache hit rate and reduction in prompt processing time provide additional evidence.

Memory efficiency also needs to be considered alongside latency. An implementation that reduces GPU usage but introduces substantial data transfer time can make the application slower.

The useful outcome is a cache configuration that supports the required context and concurrency while meeting the application's performance limits.

KV Cache and Efficient Text Generation

KV Cache is a fundamental part of efficient autoregressive inference because it allows attention information from earlier tokens to be reused.

Its benefit is straightforward: less repeated computation during generation.

Its limitation is equally important: every retained token requires storage, and that storage competes with model weights and other execution resources.

Understanding the number of KV heads, the retained context length, and cache precision makes memory requirements easier to estimate. Choosing between dynamic allocation, static allocation, quantization, offloading, and block based management then becomes a technical decision grounded in the workload.

The most useful principle is to measure cache memory and token generation performance together. Faster generation has limited value if the resulting memory demand prevents the application from serving its required number of users.

FAQ

Can I save a KV cache and use it again tomorrow?

Yes, if the serving framework supports persistent cache storage and restoration. The saved tensors must remain compatible with the exact model configuration, token sequence, positional encoding, and execution requirements. Changes to the model or tokenization can invalidate the stored cache. Saved tensors also require appropriate access controls because they represent information derived from processed inputs.

Does using KV Cache change the model's answers?

A correctly implemented KV cache is designed to preserve the underlying attention calculation. With equivalent numerical operations and generation settings, cached and uncached inference should produce equivalent results. Different kernel implementations and floating point operation orders can introduce small numerical differences that affect token selection near a decision boundary.

Can I use KV Cache when training or fine tuning an LLM?

KV caching is intended for autoregressive inference, where previously processed token representations can be reused during generation. Standard training and fine tuning require gradient computation across the relevant sequence. Reusing detached historical KV tensors in that process does not provide the required gradient relationships and can produce incorrect training behavior.

What happens to the cache if I edit an earlier message?

Changing an earlier part of the prompt invalidates cached representations that depend on the changed content. A compatible serving engine can retain the unchanged prefix and recompute the affected portion. Reusing cache entries derived from the old prompt without checking their validity can produce incorrect results.

Why is my GPU memory still occupied after a response finishes?

The inference framework or GPU memory allocator can retain allocated memory for reuse after a request completes. A serving engine can also preserve reusable prefix cache blocks according to its retention policy. Reserved memory is not necessarily memory occupied by active sequence tensors. GPU memory statistics should distinguish active allocations from reserved capacity before diagnosing a leak.