policybook

Domains

• cache Policies for deciding which key to drop when a cache is full. A cache holds at most `capacity` keys, so when a new one arrives something has to go, and which one is the whole question. The difference between a good and a bad eviction rule on the same workload is routinely ten points of hit rate, for identical memory and near-identical cost per operation.
• rate-limiter Policies for deciding whether a request may proceed. They differ far less in *how much* they let through than most comparisons suggest, and far more in what they cost to run and how they behave at the edges.
• retry Policies for deciding how long to wait before trying again. The delay curve is the part everyone already knows. The part that decides outcomes is where the randomness goes.
• kv-cache Policies for deciding which tokens to forget while a language model generates. A transformer's KV cache grows by one entry per token, per layer, per head. At a long context that is gigabytes, and the cost is linear in the sequence while the value of any individual token is not.

Use it in your repo

1. npx policybook list shows what there is
2. npx policybook add cache/sieve --out src/cache copies the one you want (--lang python or --lang c for the other ports)
3. import Sieve from "./cache/sieve", like any other file in your project.

Prefer a dependency? pip install policybook gives you every policy in Python (from policybook.cache import Sieve). C is a single header on the releases page. The npm package has the CLI and all the source.

New here? Check out the tutorial.