Engine CLI

This is the binary the cave CLI, and other tools, shell out to. It builds from source — there's no published package yet.

Build

go build -o ./bin/caveman-engine ./public/engine/cmd/caveman-engine

That uses the public engine source in this repo. For Python and JS/TS code compression, build with cgo enabled so tree-sitter is available. A cgo-free build still compresses Go plus non-code content types.

Verbs

compress

Reads stdin, writes the compressed payload to stdout, and writes the JSON report (ratio, basis, recovery handle) to stderr — so stdout stays a clean payload you can pipe.

caveman-engine compress < tool-output.json > compressed.txt
# stderr:
# {"content_type":"json","tokens_before":1366,"tokens_after":246,
#  "ratio":0.82,"basis":"inferred","recovery_handle":"..."}

Detection is automatic. To force a content type — including the forced-only toolschema and toon compressors that detection never routes to — pass --type:

caveman-engine compress --type toon < table.json > compressed.txt
caveman-engine compress --type auto < payload          # explicit auto-detect

detect

Prints the detected content type of stdin and nothing else. It needs no recovery store. Low-confidence input falls through to text.

caveman-engine detect < payload
# json

The result is one of json, diff, code, log, search-result, or text. detect never returns toolschema or toon — those are forced-only.

retrieve <handle> [query]

Prints the byte-exact original for a recovery handle from a prior compress. This is the recoverable half of the lossy compressors: nothing is destroyed. An optional second argument narrows the result to the sections most relevant to that query (BM25) instead of the whole original — the same path the MCP caveman_retrieve tool uses.

caveman-engine retrieve 7f3c... > original.json
caveman-engine retrieve 7f3c... "the part about the deploy key"   # narrowed, still byte-exact per section

An unknown handle is an error, not a silent empty result.

stats

Prints aggregate compression stats as JSON — totals and a per-content-type breakdown, all inferred.

caveman-engine stats
# { "totals": { ... }, "by_content_type": { "json": { ... } } }

toon encode / toon decode

A stateless JSON⇄TOON converter — no CCR, no recovery store. encode takes JSON on stdin and writes the TOON re-encoding to stdout; decode reverses it. Both fail closed: input that isn't valid JSON (for encode) or valid TOON (for decode), or a JSON shape outside the proven round-trip subset, is an error rather than a lossy guess.

caveman-engine toon encode < table.json > table.toon
caveman-engine toon decode < table.toon > table.json

pixel render / pixel simulate

Dev tooling for the pixel text→PNG renderer, the S4-lossy transform ported from pxpipe that the proxy's pixel mode uses. render reads a text file, writes one or more <file>.pxN.png pages next to it on disk, and prints a JSONL report (per-page dimensions, chars rendered/dropped, estimated image tokens, plus a summary line) to stdout. simulate sniffs a request body (OpenAI Responses input, Gemini contents, or Anthropic Messages messages) and prints the transform's TransformInfo to stdout — no PNGs written, no network call.

caveman-engine pixel render --dense large-log.txt > report.jsonl
caveman-engine pixel simulate --model claude-fable-5 request.json

Neither CLI verb touches CCR — this is an inspection tool, not the recoverable compress path. Plain (non-dense) rendering is rarely token-profitable once you account for image tokens; --dense is the configuration that actually wins in practice. In the proxy, pixel only ever applies to an allowlisted model (CAVE_PIXEL_MODELS, default claude-fable-5,gpt-5.6) and only when a gate confirms the rendering is cheaper than sending the text.

evals run

Runs the local eval harness against embedded fixtures and prints the report as JSON. It is the quality gate: any fixture or grader failure is a non-zero exit, so it drops straight into CI.

caveman-engine evals run || echo "quality gate failed"

At a glance

See also