Eval infrastructure audit · Journey 0

FlawedBench under evidence.

20 tasks were challenged for solvability, failability, label quality, and grader behavior. Every finding below retains its evidence tier and exact reproducer.

5findings

165/200probe budget

0skipped probes

0.0%abstention rate

$0keyless run

Defect register

Findings

Primary verdict · evidence · reproducer

task-03 high

GRADER_FP

accepted 1/3 constructed wrong answers

secondary — evidence declared-oracle FP lower bound yes

sieve audit flawedbench --task task-03

task-07 high

TASK_UNSOLVABLE

declared oracle solution rejected by the grader

secondary — evidence declared-oracle FP lower bound no

sieve audit flawedbench --task task-07

task-12 critical

TASK_UNFAILABLE

3/3 null/refuse/echo probes accepted

secondary — evidence declared-oracle FP lower bound no

sieve audit flawedbench --task task-12

task-15 high

GRADER_FN

trusted oracle answer rejected; keyed answer is a label-error candidate

secondary LABEL_ERROR evidence bundled-oracle FP lower bound no

sieve audit flawedbench --task task-15

task-19 critical

WEAK_GRADER

accepted 4/6 constructed wrong answers

secondary — evidence declared-oracle FP lower bound yes

sieve audit flawedbench --task task-19

Grader calibration

Measured FP/FN · Wilson 95% CI

FP is a lower bound over constructed mutations.

Per-task grader behavior

counts before rates

task	FP count	FP rate [CI]	FN count	FN rate [CI]
task-01	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]
task-02	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]
task-03	1/3	33.3% [6.1%, 79.2%]	0/2	0.0% [0.0%, 65.8%]
task-04	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]
task-05	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]
task-06	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]
task-07	0/2	0.0% [0.0%, 65.8%]	1/1	100.0% [20.7%, 100.0%]
task-08	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]
task-09	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]
task-10	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]
task-11	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]
task-12	4/4	100.0% [51.0%, 100.0%]	0/1	0.0% [0.0%, 79.3%]
task-13	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]
task-14	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]
task-15	0/2	0.0% [0.0%, 65.8%]	2/2	100.0% [34.2%, 100.0%]
task-16	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]
task-17	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]
task-18	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]
task-19	4/6	66.7% [30.0%, 90.3%]	0/2	0.0% [0.0%, 65.8%]
task-20	0/2	0.0% [0.0%, 65.8%]	0/2	0.0% [0.0%, 65.8%]