Specification 01 Eval infrastructure assurance
Sieve challenges benchmark tasks, answer keys, and graders before their scores influence a model decision. It finds tests that cannot be solved, cannot be failed, or cannot grade reliably.
The problem
Most eval systems measure the model. Sieve measures the test: whether the task is valid, the oracle is credible, and the grader rejects the wrong things while accepting the right ones.
The oracle cannot pass, a null response passes, or the scenario is degenerate.
Constructed wrong answers are accepted or valid variants are rejected.
The reference answer disagrees with trusted evidence, hiding a false negative.
Assurance method
Sieve runs cheap red flags first, spends a declared budget on probes, and refuses to turn missing evidence into false certainty.
Run a declared or bundled oracle. A rejected oracle is a task or label defect candidate.
ORACLE → GRADERTry empty, refusal, and prompt-echo agents. Passing null behavior exposes an unfailable task.
NULL × REFUSE × ECHOFeed plausible-but-wrong answers to estimate a grader false-positive lower bound.
WRONG → FP [WILSON 95%]Use legitimate variants to measure false negatives and surface answer-key errors.
CORRECT → FN [WILSON 95%]Proof, not promise
FlawedBench contains 20 deterministic local tasks and five seeded defects. Sieve finds all five in the canonical audit.
Rates are based on constructed mutations and correct variants. The application shows counts and Wilson 95% intervals for every task.
Validity, FP, and FN findings turn a single reported score into an explicit sensitivity range.
Correct variants and wrong mutations stay separate, so aggregate accuracy cannot hide the error direction.
Change the budget, withhold oracles, vary mutation coverage, step through probes, filter findings, copy reproducers, and export the evidence package.
Architecture
The shipped probes and graders are local, scripted, and network-free. The demo costs $0.
A normalized task contract separates suite ingestion from audit logic.
Every grader call spends one unit. Skipped work and abstention are always reported.
Declared and bundled oracles are labeled. Missing trusted evidence becomes undetermined.
About the thesis
Evaluation infrastructure is the decision layer beneath model launches, benchmark claims, and research roadmaps. Sieve treats that layer like production code: adversarially test it, retain exact evidence, and state uncertainty before anyone optimizes against a misleading number.
View the implementation on GitHubThe deterministic FlawedBench regression fixture: 20 tasks, five known defect classes, 165 of 200 probes used, and a 65–90% sensitivity band from a reported 80% score.
Sieve cannot prove semantic correctness without trusted oracle evidence. Unknown grader exploits are outside the mutation denominator, so FP is a lower bound.
LLM-generated probes, simulation fleet mode, automatic changed-task discovery, and a disclosed real public-benchmark audit remain future work.