Behaviour, not pixels
Checks open a fresh page, use trusted mouse and keyboard input, and inspect state transitions, rules, edge cases, keyboard access and faithful data encoding.
Method & evidence
Judge what a user can do.
Each task declares a product brief and a small test interface. The judge drives the artifact through trusted user input and checks observable behaviour. Reference solutions and faulty variants test the checks themselves.
A declared test interface
Prompts name required data-testid hooks and, where applicable, a read-only window.__vab.state(). They leave layout and implementation choices to the model.
Deterministic validation
HTML and SVG checks use a paused virtual clock, seeded randomness, offline Chromium and fixed viewports. Full-stack checks use an isolated server and real time. Task admission requires three identical judgements, a reference score of 1.0 and every seeded mutant killed.
Score components
Missing measurements remain pending.
Specified behaviour · S
The share of executable task checks passed. Critical checks and gates also determine the strict pass result.
Implicit quality · I
Accessibility and robustness beyond the brief: accessible names, contrast, keyboard reach, visible focus, target size, phone reflow and errors.
Design quality · D
Where measured, a vision-model panel compares rendered artifacts pairwise in both orders. Interpretation depends on the panel's calibration and compatible comparison pools.
Quality and consistency
With both I and D measured, Q = S × (0.6 + 0.2·I + 0.2·D). With one applicable component measured, Q = S × (0.75 + 0.25·component). With neither, Q = S. For each task, reliability is mean Q across runs − 1 sample standard deviation; the reported R averages those task values. With one run per task, R equals mean Q.
Only components supported by comparable evidence should enter a reported comparison. A score from a shared subset cannot be presented as a score over the full roster.
Generation and judgement are separate evidence
Available artifacts
Generated means an artifact exists; qualified means it meets generation validity requirements. Native completed means the agent ended normally. A run that reaches its time or step limit may still have an inspectable artifact, and its limit remains visible.
Published measurements
Judged means executable checks have evidence. Audited means failed checks were reviewed against the artifact. A judge failure that is not an artifact defect requires correction and a new judgement before publication.
One 77-case roster
The Results page gives every listed model configuration the same 77 cases. Coverage makes unavailable cases explicit. A full-roster aggregate stays pending until actual full-roster judgement evidence is available; historical per-case measurements remain inspectable on Tasks.
Configuration identity
Model API IDs, harness versions and seeds identify the execution configuration. Combining disjoint batches of the same configuration does not create two models. Different harness identities remain separate.
Inspecting a case
On Tasks, select a case and add the models you want to compare. Available HTML and SVG artifacts are interactive in isolated previews. Switch the viewport or open a focus view to inspect responsive behaviour.
Full-stack cases need their backend to run. The static showcase uses available screenshots and source for those cases and labels that limitation. The brief, exact prompt, checks and reported scores sit alongside the artifact.
The site refreshes current data every 30 seconds. Your selected case and comparison models stay in place, and an unchanged artifact preview keeps its current interaction state.