Token counting
The engine reports token reduction, not byte reduction. It counts locally with an embedded BPE tokenizer, so the same payload gets the same count without a network call.
A ratio is only useful if every surface counts the same way. The engine owns the counter, and CLI, proxy compression, MCP, and context packing use that basis instead of inventing their own.
The default counter
The default is a real BPE tokenizer using OpenAI's o200k_base encoding (the GPT-4o-family encoding). Its vocabulary is compiled into the binary. Counting never reaches the network and never varies between runs: the same bytes always yield the same count.
// The shared default counter, used across every content type.
ctr := tokens.Default() // o200k_base BPE, offline
n := ctr.Count([]byte(payload)) // the inferred token count
name := ctr.Name() // "o200k_base"Two properties matter and they come for free from embedding the vocab:
- Deterministic — counting is a pure function of the bytes. There is no sampling, no model call, no clock. A ratio you compute locally is the same ratio CI computes.
- Offline — no token count ever leaves the machine, and no network is required to produce one. The engine works the same on an air-gapped box as on a laptop.
The Counter interface
Every compressor and the context packer count through one small interface, so a per-provider tokenizer can drop in later without touching any compressor:
type Counter interface {
// Count returns the inferred token count of b.
Count(b []byte) int
// Name identifies the counting basis (the encoding name) for
// display and cross-surface parity.
Name() string
}Count must be deterministic — the same bytes always produce the same number. If the embedded codec ever fails to load (a build-time invariant that should not fail at runtime), the engine degrades to a deterministic chars/4 approximation rather than crash. It never guesses high.
Why counts are always inferred
The engine's counter is a local estimate. The only verified token count is the one a provider returns in its usage field, and the engine never sees it — the proxy reads that downstream. So every ratio the engine prints carries basis: "inferred".
Honesty rule
inferred is a per-run, local estimate from an offline counter. The engine never says verified and never re-projects a ratio into a monthly figure.
This counter is the source of every percentage you see anywhere in the engine — the per-type targets on the compressors page, the ratio in a compress report, and the budget accounting the context packer does. One counter, one basis, one honest number.