Evaluation

Coverage

The public core suite has cases for instruction following, coding, mathematics, reasoning, factuality, summarization, structured JSON, context retrieval, tool calls, safety, and consistency. Collection records input/output tokens, wall time, tokens per second, and process peak RSS.

tidelm benchmark \
  --adapter models/candidate \
  --label candidate \
  --output-dir evaluation/results/local/candidate

Grade an already collected bundle without model dependencies:

tidelm evaluate \
  --outputs evaluation/results/local/candidate/outputs.jsonl \
  --report evaluation/results/local/candidate/report.json

What graders can establish

• exact format and JSON schema adherence;
• arithmetic final answers;
• required/forbidden factual strings;
• Python syntax plus small tests after AST filtering;
• summary length and required facts;
• lexical refusal and safe-alternative markers;
• cross-prompt response similarity; and
• measured local speed.

They cannot establish general reasoning, truthfulness, safety, or code quality. The suite is small and project-authored, so each release reports sample counts and raw case details.

Blind AI critic

The harness assigns A/B per case with a secret deterministic key. The fresh critic receives only prompt, context, outputs, and rubric—not checkpoint names, chronology, losses, or automated scores. It rates correctness, usefulness, coding quality, reasoning, instruction following, hallucination, clarity, and consistency. The key is revealed only after its signed report is saved.

An AI critic is not an independent human study. Its provider/model, fresh session, prompt hash, raw response, parse status, and ties are recorded.

Regression gate

A candidate is not promoted merely because training completed. The gate checks:

1. all repository/data/artifact tests pass;
2. candidate aggregate diagnostic score does not decline;
3. no critical safety, structured-output, or coding case declines;
4. validation loss is finite and reported (not used alone);
5. blind critic prefers the candidate or reports a defensible tie without a

critical regression;

1. inference remains usable on measured hardware; and
2. security/license checks pass.

Any exception must be explicit in the experiment decision. Rejected candidates remain recorded and never replace stable.