Keep the most recent budget token positions in the KV cache and forget the
rest. It is the obvious thing to do, and it is here as the honest baseline every
other policy in this domain has to beat.
It is a much stronger baseline than it sounds. Attention is dominated by recency: the token being generated attends most heavily to the handful just before it. So keeping only recent tokens still retains most of the attention mass. Anyone proposing something cleverer has to beat this first.
When to use it
- As the reference line. Knowing what the naive answer scores is what makes every other policy’s number mean something.
- When the cache is large relative to the context, so little is ever evicted and the choice barely matters. At a budget of a quarter of the sequence, every policy in this domain looks alike.
- When you cannot see attention weights at all. Some serving stacks do not expose them per step, and this policy needs nothing but the position.
- When absolute simplicity is worth more than quality. This is a ring buffer. There is no scoring pass, no tuning, and nothing to get wrong.
When not to use it
- Almost anywhere you would actually deploy it, because streaming-llm is the same ring buffer plus four pinned positions and is strictly better. On the canonical trace at a budget of 256 it retains 0.7367 of the attention mass against this policy’s 0.5966, the entire gap being the attention sinks this policy throws away first.
- When the model has attention sinks, which trained transformers generally do. This policy evicts them before anything else, purely because they are the oldest thing it holds, and a model deprived of them degrades sharply rather than gracefully.
- When old tokens carry real information: a long document, a system prompt, a retrieved passage. Everything outside the window is gone regardless of how much attention it was receiving.
- When you can see attention weights and can afford to score them. h2o and its neighbours exist precisely because the important old tokens are findable, and this policy does not look.
How it works
on_decode_step(pos): window.push_back(pos)
evict(budget): while len(window) > budget: yield window.pop_front()A ring buffer of positions with exactly budget + 1 slots, one more than the
budget, because that is the most that can be held at the moment the harness asks
for an eviction. O(1) per step, and nothing allocated after construction.
The cache starts holding position 0, whose token exists before the first decode step (see the domain interface). It is therefore also the first thing evicted.
Tie-breaking. Victims come back oldest-first. Positions only ever increase, so arrival order and position order coincide here and the domain’s “lower position first” rule is satisfied automatically.
Parameters
A budget of 1 is legitimate, keeping only the newest token, and a vector
covers it. Zero is refused: a cache that holds nothing is not a cache.
Complexity
O(1) time per decode step and per eviction. O(budget) space: budget + 1
positions of uint32_t in C, so 2,052 bytes at the default plus the struct.
Attention is never read, so the per-step cost does not grow with the number of kept positions, which is not true of any attention-aware policy here.
Source
Folklore. Windowed attention long predates the KV-cache eviction literature and appears in some form in most long-context work. This is the degenerate version with no extras at all.
Its nearest neighbour is streaming-llm, which differs by exactly four pinned positions. The distinguishing vector in both policies runs the same steps at the same budget: this one evicts positions 0, 1 and 2 in turn, and that one keeps them and evicts 4 instead.
Notes
No patents known.
The Rng is accepted at construction and never used: this policy is entirely
deterministic. The attention argument is not even a parameter of the TypeScript
onDecodeStep, which is the clearest way to state that it cannot be read.