Compressors

The engine routes a payload to a compressor by content type. Detect classifies the bytes, the engine hands them to the matching compressor, the offline counter measures the token reduction, and — because these compressors are lossy — the original is stored for byte-exact recovery before any compressed result is emitted. S4lossy · recoverable is the class for every detected compressor: lossy to the bytes, but recoverable.

Each compressor is structural and deterministic. The engine wraps it with token counting, CCR recovery, and pass-through rules.

The six detected compressors

Detect routes to one of these six. Low confidence falls through to text. The percentages are honest token-reduction targets by content type, not measured guarantees — actual results depend on the payload.

The principle is the same across all six: keep the part a downstream model would need to answer correctly, drop the part that only adds tokens. A log compressor that throws away a stack trace would be useless; one that keeps every INFO line defeats the point.

The code compressor and tree-sitter

The code compressor parses source so it can keep declarations and elide bodies without producing invalid syntax. How it parses depends on the build:

• Built with cgo, it uses tree-sitter and understands Go, Python, and JavaScript/TypeScript.
• Built cgo-free (or for WASM), it falls back to a pure-Go go/ast parser that compresses Go only.

The fallback is deliberate and honest: rather than mangle a Python file it can't parse, the cgo-free build leaves non-Go code as a pass-through. The embedded eval fixtures cover all three languages under cgo.

The two forced-only compressors

Two compressors exist but are never auto-detected. Detect must never route to them — they only run when a caller explicitly forces the type via Options.Type (the CLI's --type flag, or cave compress --toon).

• toolschema — compresses MCP/OpenAI tool catalogs: drops annotation bloat (examples, titles, defaults, $schema) and truncates descriptions while keeping every selection token (tool and parameter names, types, enums, required) byte-for-byte, so the model still picks the same tool. This is the compressor caveman-shrink wraps.
• toon — a TOON re-encode of tabular JSON. Unlike the detected compressors it is lossless to the model; you ask for it on purpose, never by detection.

Compressing JSON

The detected path needs no flags. Pipe a payload into caveman-engine compress; the clean compressed bytes land on stdout and the inferred report (ratio, basis, recovery handle) goes to stderr.

# stdout = compressed payload, stderr = the inferred JSON report
caveman-engine compress < tool-output.json > compressed.txt
# {"content_type":"json","tokens_before":16098,"tokens_after":1091,
#  "ratio":0.93,"basis":"inferred","recovery_handle":"..."}

That recovery handle is how you get the original back, byte-for-byte:

caveman-engine retrieve <handle> > original.json

On the committed sample, this run reproduces the headline demo:

See also