Token efficiency you can recompute.

CaveBench measures how much an optimizer cuts cost while holding quality, then signs the run. Re-run the open harness on the open data and the signature recomputes, byte for byte.

77.2%

fewer tokens

10,577 → 2,416

100%

quality kept

floor ≥ 99%

100%

byte-recoverable

exact via CCR

71.8%

cost saved

at held quality

signed23f03a34c8…ef083ckey cavebench-localcatalog 2026-08-13basis inferred

Where the savings come from.

Reported per content type, because that is where the tradeoff lives: Caveman's S4 transforms elide model-visible content behind a task-quality gate, while CCR recovers each exact original. The measured prose fixture passed through unchanged.

content typetoken reductioncost savedqualitybyte-recov.

Repetitive output

S4 lossy + recoverable

91%

86%100%100%

Build & run logs

S4 lossy + recoverable

91%

89%100%100%

Terminal sessions

S4 lossy + recoverable

91%

86%100%100%

Tool & JSON output

S4 lossy + recoverable

82%

80%100%100%

Configuration

S4 lossy + recoverable

73%

68%100%100%

HTML

S4 lossy + recoverable

69%

59%100%100%

Tabular data

S4 lossy + recoverable

64%

61%100%100%

Source code

S4 lossy + recoverable

57%

52%100%100%

Tool schemas

S4 lossy + recoverable

55%

45%100%100%

Tabular JSON (TOON)

lossless to model

21%

14%100%100%

Free prose & chat

S4 lossy + recoverable

none

—100%100%

Byte-recoverability is 100% on every compressed class because Caveman stores the original out of band in CCR before emitting a lossy transform. Recovery proves durability, not that compressed model input was lossless.

Cost per correct task

At identical 100% task success, Caveman costs 3.2× less per correct task — the savings come from the input tokens it never sends.

raw passthrough

389µ$ / 500 tok

caveman

123µ$ / 146 tok

How it's measured.

The construct is inferred token reduction at held quality on representative tasks — not raw byte savings, provider billing, or verified live-traffic savings.

Sweep the curve, never one number

Compression versus quality is non-monotonic, so the harness sweeps a grid of targets and reports the curve, stratified by context length — never a single context-free percentage.

Two ratios, for two reasons

A word-ratio that compares across providers, and an engine-token ratio with its tokenizer disclosed. Provider-reported usage is separate; neither benchmark ratio is invoice truth.

Decontaminate, then test significance

A 13-gram overlap filter removes leaked items, and Versus deltas are reported as a mean with a 95% interval — a delta that straddles zero is not a win.

This run grades quality offline against deterministic answer keys (exact_match, json_schema, tool_sequence) and estimates cost from the dated catalog rather than a billed invoice — 17/17 tasks passed, both ways. Savings stay inferred and non-billable; a benchmark or active rollout never enters the verified ledger. Single-number compression claims — say, 20× at near-zero loss — do not survive this harness, which is why it sweeps the curve and reports intervals.

The receipt you can reproduce.

Every run is wrapped in an Ed25519-signed, hash-chained receipt carrying the token counts, the cost, the eval-gate result, and the exact formula and catalog versions. Public leaderboards publish a number; CaveBench publishes the number and the signature that recomputes it.

caveman.receipt.v4signed

day: 2026-06-21
seq: #1
requests: 17
tokens before: 11,065
tokens after: 2,906
benchmark cost: 2405µ$
eval gate: 17 pass · 0 fail
verified savings: $0; honest zero
formula: cavebench.self.v1

report hash

receipt hash

prev receipt∅ genesis

Ed25519 signaturekey cavebench-local

Reproduce it

One deterministic command regenerates the golden and its signature. Same inputs, same toolchain, same catalog snapshot → an identical receipt hash.

$ make bench-golden

verifies this run

trust anchor · production

Trust model

What this claims is reproducibility, not neutral attestation: the signing key is founder-held and repo-pinned. Receipts carry their key_id, so historic ones stay verifiable through rotation. Custody & neutrality policy: docs/cavebench/GOVERNANCE.md.

What's live, and what's next.

CaveBench-Self is live and signed — every number above is from it. Two stages remain, blocked on the work below, not on the result.

CaveBench-Selflive · signed
Caveman grades and signs its own savings on a fixed fixture suite — every number on this page.
CaveBench-Versusnot yet published
The same harness head-to-head against rival compressors. Opens once the rival adapters and the canonical open suite are configured.
CaveBench-Publicnot yet published
A standing leaderboard ranking any agent or model by cost-per-correct-task. Opens after the first signed full run on a decontaminated open suite.

Caveman runs the benchmark it sells, and judges its own entry under the same rules — same suite, same contamination and significance checks, no privileged conditions. Key custody and the neutrality stance: docs/cavebench/GOVERNANCE.md.

Leaderboard status

The harness is open. The receipt is reproducible.

Clone the repo, run make bench-golden on the open data, and compare your signed receipt against the one above.

Browse the harness Leaderboard status