Source of truth: , , Last updated: 2026-05-13 — v3.8.0

) that registers in-memory built-in suites, evaluates outputs against expected criteria, and aggregates scorecards.
• ) for custom (user-defined) suites and historical runs in SQLite.
• ) that executes each case by dispatching real calls to , captures latency and outputs, and persists the run.
• (management-auth only).
• ().

and one or more cases. Suites come from two sources:

	at boot
	+

):

— 10 baseline cases across greeting/math/translation/safety
• — Python/JS/SQL/TS/bug detection
• — syllogisms, word problems, pattern recognition
• — translation and language detection
• — PII, jailbreak, refusal, bias awareness
• — JSON-only, numbered lists, language constraints
• — head-to-head coding tasks intended for compare mode

	targeting
	— sent to
	— scoring rubric (see below)
	, , )

in :

		field
		)

and , the field is required (enforced by Zod ). When is provided, both targets must differ — the runner persists both runs under the same for A/B comparison.

(evalRunner.ts):

	is truthy
	returns truthy (built-in only)

Note: Custom-function scoring is reserved for code-defined (built-in) suites because functions cannot be serialized through the API. The only accepts for user-created suites.

.

and ):

	, , )
	, ,
	, , , , , , ,

not stored in the DB. They live in memory and are re-registered every time is imported.

) — they are not part of the public proxy surface.

curl -X POST http://localhost:20128/api/evals \
  -H "Cookie: auth_token=..." \
  -H "Content-Type: application/json" \
  -d '{
    "suiteId": "golden-set",
    "target": { "type": "combo", "id": "my-combo" },
    "apiKeyId": "optional-api-key-uuid"
  }'

— of pre-computed outputs. When provided, the runner skips dispatch and only scores the cached outputs (useful for offline evaluation).
• — second target to run in parallel; both runs share a generated for head-to-head viewing.
• — internal API key used to authenticate the dispatched calls. Required when is enabled.

curl -X POST http://localhost:20128/api/evals/suites \
  -H "Cookie: auth_token=..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Production smoke",
    "description": "Quick sanity check before deploy",
    "cases": [
      {
        "name": "JSON shape",
        "model": "gpt-4o",
        "input": { "messages": [{ "role": "user", "content": "Reply with {\"ok\": true}" }] },
        "expected": { "strategy": "regex", "value": "\"ok\"\\s*:\\s*true" }
      }
    ]
  }'

():

1. to with the case's , the resolved , , and (or the case override).
3. or the Responses-API payload.
4. , then persists via .

sequentially. There is no concurrency flag today.

(). From there you can:

• , optionally an API key, then run on demand.
• scope.

(see also AUTO-COMBO.md for the live scoring engine). That subsystem targets the Auto Combo engine — automatically scoring providers and models so combos can self-heal when upstreams fail. It uses its own runner, its own categorizer, and its own scoring logic.

broader, general-purpose testing surface. Prefer it for arbitrary regression suites, A/B comparisons, and per-release smoke tests. Use the Auto-Assessment subsystem when you need real-time provider health to influence routing decisions.

npm script today. Two paths if you want to gate releases on eval results:

HTTP path: stand up the server, hit with a known + , and assert in the response.
• In-process path: import from from a script, run against a test DB, and check the returned .

and .

New scoring strategy — extend the block in () and widen in plus in .
• New built-in suite — define a suite object and call at the bottom of . It will be auto-discovered by .
• Run with concurrency — change the sequential loop in to a bounded (no concurrency control exists today).
• Stream/tool-call cases — currently the runner forces . Streaming or tool-aware evaluation would require changes in (capture and aggregate SSE chunks before scoring).

USER_GUIDE.md — overall product walkthrough
• ARCHITECTURE.md — request pipeline reference
• AUTO-COMBO.md — Auto Combo scoring engine (live runtime)
• , ,