DocsCompression engineOverview

Compression engine

The engine detects payload type, routes to a matching compressor, counts tokens with an offline counter, and stores originals for recovery when output is lossy.

Fig. — the compression path

Agents often send repetitive JSON, long logs, diffs, search results, and source files where only part of the text matters. The engine uses different compressors for those shapes and labels every token reduction inferred.

The four-call API

Every front door onto the engine — the CLI, the proxy, the MCP server, the SDKs, the WASM build — shares the same small surface:

text

Compress(input, opts) → { output, content_type, tokens_before, tokens_after, ratio, basis: "inferred", recovery_handle? }
Retrieve(handle)      → the byte-exact original
Detect(input)         → "json" | "diff" | "code" | "log" | "search-result" | "text"
Stats()               → { totals, by_content_type }

record mode is pass-through. compress mode routes to a compressor. Parse problem, not-smaller output, unknown safety class, or missing recovery store means original bytes come back.

Detect, then route

Detect is a content router. It reads the payload and classifies it; low-confidence input falls back to text, the gentlest compressor. Each class has a compressor tuned to it, with an honest token-reduction target range:

Content	Keeps	Target
`json`	keys, structure, error/message subtrees	70–90%
`log`	errors, stack traces, first/last lines	85–95%
`code`	imports, signatures, types (syntax stays valid)	40–70%
`diff`	file/hunk headers and changed lines	60–80%
`search-result`	top/bottom hits, diagnostic/security hits	80–95%
`text`	headings, opening/closing context, key sections	50–80%

These are targets, not guarantees — real payloads vary. Full compressor reference →

Lossy, at S4 — and recoverable

Every compressor sits at the top of the safety ladder.

Fig. — the safety ladder

S4lossy · recoverable

S4 means lossy: bytes are dropped. What makes it honest is the rule attached to the class — S4 RequiresCCR. A lossy result is only ever emitted if the original was first stored for byte-exact recovery. No store, no compression; the engine falls back to pass-through. So the model sees a smaller payload, and the full original is always one Retrieve away.

Honesty rule

The engine reports inferred ratios from an offline token counter — never verified, never re-projected into a monthly figure. It also fails closed: an unknown mode becomes record, an unknown content type becomes text. The conservative path is always the default.

What's inside

Compressors

The six detected compressors, plus the forced-only tool-schema and TOON re-encoders.

BSL

Recoverable compression

The content-addressed store that makes lossy honest.

BSL

Token counting

The offline BPE counter behind every ratio.

BSL

Context-window packing

Deterministic BM25 packing into a token budget.

BSL

Engine CLI

compress · detect · retrieve · stats · evals run.

BSL

Build it

The engine is a Go binary. The pure-Go build compresses Go source; build with cgo to add tree-sitter for Python and JS/TS. JSON, logs, diffs, search results, and prose compress the same in either build.

terminal

go build -o ./bin/caveman-engine ./public/engine/cmd/caveman-engine
echo '{"items":[1,1,1,1,1]}' | ./bin/caveman-engine compress