Aryan PuttaDev JournalThe KV Cache Is the Memory Wall
The weights are not what run you out of GPU memory first. The KV cache is, and it scales with your traffic, not your model size. Here is how I think about that bottleneck and how EigenKache compresses it: 60.35% less memory, 19.87% less latency.
Why I Started Here
When I look at any system, I ask the same question every time: where is the bottleneck, what is the constraint, and how does this behave under real conditions. For LLM inference the answer surprised me. Everyone talks about model size, but the thing that actually runs a serving box out of memory first is usually not the weights. It is the KV cache. So I built a project, EigenKache, to understand that wall by attacking it.
What The KV Cache Actually Is
A transformer generates one token at a time, and each new token attends to every token before it. If you recomputed all of that attention at every step, decoding would be quadratic and unusably slow. So the model saves the key and value vectors for every past token, at every layer, for every attention head, and reuses them. That saved state is the KV cache. It is the price you pay to make decoding fast.
The Number That Made It Click
cache memory = layers x heads x head dimension x sequence length x batch size x 4 bytes
The Realization
Read that with serving in mind. Your model size sets a constant. Your traffic sets the sequence length and the batch size. So the resource that decides how many users you can serve, and how long their context can be, scales with demand, not with the model you picked. At long context the cache can grow larger than the model itself. This is the pattern I keep seeing in systems: the thing that breaks you is the resource that scales with load, not the one that scales with the artifact. Find that resource first.
The Bet Behind EigenKache
If the cache is the wall, the real question is whether you need all of it at full fidelity. I bet you do not. Attention is concentrated. For most queries a small number of past tokens carry the weight and the rest contribute almost nothing. If that holds, you can keep a compressed summary of the cache instead of every entry, and reconstruct attention against the summary. The risk is obvious: compress too much and the model forgets something it needed, and quality drops. So the object of study is not one setting. It is the tradeoff curve between memory saved and quality kept.
The Design Decisions
Two decisions mattered. First, I select landmark vectors that summarize the cache instead of dropping tokens at random, so the summary keeps the structure attention cares about. Second, I run selection per head, not once for the whole layer. Heads attend to different structure, and a single shared summary throws that difference away, which is exactly where quality leaks out. Per-head selection costs more bookkeeping, but it is the difference between a compression that holds and one that quietly degrades.
What I Measured
Against a full-cache baseline, EigenKache cut KV-cache memory by 60.35% and decode latency by 19.87%, with per-head support and 14 passing tests for reproducibility. The latency win is worth saying plainly: when you store and move less cache per step, every decode step reads less memory, so it finishes sooner. Less data is its own kind of speed.
What It Does Not Solve
This is one compression policy, not a general answer, and I want to be honest about that. Landmark selection has its own cost. At short context the overhead is not worth it, because there was never enough cache to compress. The quality side needs broader evaluation across tasks before any of these numbers earns the word free. The curve has a knee, not a cliff, and the interesting engineering lives right at that knee.
What I Took From It
The lesson generalizes past this project. When you want to make a system faster or cheaper, do not start with the biggest or most famous component. Start with the resource that grows with the workload, measure it under real conditions, and spend effort where the curve is steep. For inference right now, that resource is the KV cache. Next I want to test landmark selection against eviction and quantization on the same benchmark, so the tradeoffs are comparable across methods instead of told in isolation.