Evals & graders
@caveman-cloud/evals exports one grader entry point: grade(grader, value, deps?). It dispatches over 15 canonical grader types and fails closed: an unknown grader never returns passed: true.
The package is MIT-licensed and has no runtime dependencies. Network graders use injected deps in tests, so they can be checked without hitting real endpoints.
The contract
import { grade } from "@caveman-cloud/evals";
import type { Grader, GradeDeps } from "@caveman-cloud/evals";
const result = grade(grader, value, deps);
// result: { passed: boolean; reason: string }grade() dispatches on the grader's discriminant and returns a GradeResult of { passed, reason }. deps is an injectable GradeDeps carrying fetch and an ssrfCheck — tests inject both so a grader is never hitting the real network.
The 15 grader types
The Grader union is exactly these — no more, no fewer:
| Grader | Checks |
|---|---|
exact_match | Output equals the expected value, normalized (case-insensitive, key-order-insensitive) |
contains | Output contains a substring |
regex | Output matches a regular expression |
json_schema | Output validates against a JSON Schema subset |
json_path_assertion | A JSONPath into the output equals an expected value |
tool_called | A named tool was called |
tool_not_called | A named tool was not called |
tool_sequence | Named tools appear as an ordered subsequence of the calls |
tool_argument_assertion | A specific argument of a tool call equals an expected value |
http_status | The response carried an expected HTTP status |
latency_threshold | Latency was under a threshold |
cost_threshold | Cost was under a threshold |
token_threshold | Token count was under a threshold |
custom_webhook | A webhook (SSRF-guarded) returns a pass verdict |
llm_judge | A model, prompted through the gateway, returns PASS |
A few of these carry contracts worth stating plainly:
exact_matchis normalized — case-insensitive and key-order-insensitive — specifically so its verdict matches the Python optimizer's. A rawJSON.stringifycomparison would diverge; this package does not use one.tool_sequencechecks an ordered subsequence, not an exact, contiguous sequence. The named tools must appear in order, but other calls may sit between them.llm_judgePOSTs to<gateway_url>/openai/v1/responsesand parses PASS or FAIL from the model's text. Ifgateway_urlis missing it errors clearly — it never silently passes.
Network graders are guarded
Two graders reach the network: custom_webhook and llm_judge. Both run defaultSsrfCheck before any fetch. IP literals and bare hostnames are blocked unless you inject a resolver-backed ssrfCheck through GradeDeps. The guard runs first; the request runs only if it passes.
Fail closed, always
The default branch returns fail(), never pass()
This is the no-placeholder rule made executable. The default branch of grade() returns fail(...) — an unrecognized or unsupported grader is treated as not passed, never as passed. The Python optimizer's legacy semantic and custom graders are Python-only; ask for one here and it falls through to the default and fails closed as an unknown grader. A grader you cannot evaluate is a grader that did not pass.
This is the same instinct that runs through the rest of the stack: when in doubt, prefer the honest negative. A gate that defaults to "passed" is a gate that lets bad rollouts through.
Keep it in sync
The TypeScript types and the Python grader names must stay aligned. There are two intentional asymmetries, by design, not drift:
- The legacy
semantic/customgraders exist only in Python; this package is the 15 canonical types. - The TypeScript
GradeResultis{ passed, reason }; the Python result is{ passed, grader, score, reason }.
Adding a grader means extending the Grader union, adding a case in grade(), and adding a runtime test — and mirroring it in the Python optimizer.
pnpm build # tsc
pnpm test # tsc + node --test against dist/