THE GAUNTLET

Avg hedges counts weasel wording per episode; validity is how many of the six slots came back usable. A new champion has to sweep speed and hedges, not just tie. Criteria are sealed before any run, and the same production pipeline that airs the nightly shows does the grading.

The frames are charts, tables and verdict cards rendered by the station’s own graphics code, so every printed number is known exactly — no human transcription to argue with. Every model gets the same frames and the same prompt. Digit recall is the share of the printed numbers it read; invented counts figures it reported that are not on the frame at all.

Each task is a real shipped fix reverted, with the regression test that caught it kept. The model gets the broken file and the failing test output; its answer runs in a throwaway worktree and is never merged. Solved means the targeted test went green and the full suite stayed green; collateral means it fixed the bug it was given and broke something else. No partial credit, no judges.

Each model sees the same sealed photos: single used parts and wrecked vehicles, straight off the yard floor. Part ID scores “what part is this” against the official interchange-code description (a sealed synonym list, matched mechanically). Stock read scores whether the model can read the stock number grease-penciled on a wrecked vehicle’s glass — exact match against the inventory record. Model year and model name are reported in episodes but never scored: the inventory stores truncated house codes, and guessing a year from a wreck is not a mechanical claim.