Which KV-cache eviction policy?

Policies for deciding which tokens to forget while a language model generates. A transformer's KV cache grows by one entry per token, per layer, per head. At a long context that is gigabytes, and the cost is linear in the sequence while the value of any individual token is not.

Pick up to four and run them on the same trace. Any difference you see is the policies. The URL holds the whole setup, so you can share a comparison as a link.

The runner uses the same code as the benchmark tables on the page. It replays a shorter prefix of each trace, so use the tables for the final numbers.