Human-in-the-loop evaluation · signal detection

The Catch

A model-output judgment instrument for the harder question than “did you notice something weird?”: can you distinguish real errors from clean output that merely looks suspicious, and can you identify the mechanism that actually failed?

Not every miss means the same thing

Each reference item carries explicit solvability—cold, primed, or insider—plus separate stimulus and label provenance. That prevents missing an insider item with an adjudicated label from being silently treated as equivalent to missing a cold, measured engineering defect.

Partly right is its own failure mode

The instrument does not flatten everything into binary right/wrong. It includes cases where the final result is true but the reasoning is false or overstated, clean output that attracts false alarms, and actual errors whose detection matters. Scoring includes raw and cost-weighted d-prime, criterion, ROC points, hit/false-alarm counts, latency, and breakdowns by register, solvability, and provenance.

Referent vs. echo

Items can expose three evidence probes: define it, show the number, and where’s it from. A probe response is typed as measurement, source, restatement, or absent. The design point is simple: asking a model for evidence does not mean the response contains evidence. A confident restatement of the claim is still an echo.

Public demo status

A separate public test surface is still being designed. The reference architecture already defines a cold-start slice, but a useful public version needs a replaceable item set and an answer-handling model that does not turn the live bank into training data. Until that exists, this page publishes the method and boundaries—not the questions.