Silicon Subjects · Lab
Simulated populations are now a standard instrument. The check that usually gets run is mean-only, and a mean-only check cannot see the two failures that matter most. This lab scores a population on three separate axes against 17 classic paradigms, on human anchors taken from the published literature.
Orange cells mark the weakest protocol for that population. That row is what the suite-level 0.3 × worst term keys on.
One JSON object per line, or CSV with a protocol_id,condition_id,value header.
Scoring runs entirely in this page, nothing is uploaded.
| axis | anchor | rule |
|---|---|---|
| location | human mean per condition | SMD = (mean_sim − mean_human) / sd_human, score = max(0, 1 − |SMD| / 1.5) |
| dispersion | human sd per condition | ratio = sd_sim / sd_human, score = max(0, 1 − |ln ratio| / ln 2.5) |
| effect | direction and size of the contrast | wrong sign scores 0, otherwise the log-ratio rule above |
Per protocol 0.4 × worst axis + 0.6 × mean of axes, so one collapsed axis cannot be
averaged away. Suite 0.3 × worst protocol + 0.7 × mean over protocols.
Verdict at ≥ 75 usable as a pilot signal, 50–75 directional only,
< 50 not usable.
With 500 simulated subjects a 0.1-unit difference is "significant" while being practically meaningless, and with 20 nothing is significant while everything is wrong. The axes are equivalence formulations: how far off the population is, in human standard deviations or in log-ratio, on scales you can interpret without a sample size.
In the reference benchmark, the population "right mean, low variance" lands almost exactly on the human means (location near the ceiling) and is still not usable, because its spread is a third of the human spread. Every downstream subgroup analysis on it produces confident noise. That is the case this suite exists to catch.
A static Space. The reference benchmark was computed once by
build_static.py from the engine in silab/ and frozen into
data/results.json. The "score your data" tab runs a JavaScript port of the same
rules, checked against the Python engine by scripts/check_scoring_parity.py
(7 populations × 17 protocols, agreement to 1e-15).
The Gradio version is kept as app.py and adds a run a model tab that sends
the protocol prompts to any OpenAI-compatible endpoint, so you can generate a population from
a real model and score it. That needs a server, so it does not run here.
confidence field and a citation; check the source before leaning on one number.