Dataset and Adjudication
Recruit domain annotators, resolve contested labels and add independently held-out procurement families. Repeated calls do not create independent domains.
Policy Interpretation
Policy Interpretation
Eighteen model configurations translate source rules and verify statements from three short fictional procurement texts. Results retain three repetitions, missing requests and disputed provisional labels.
- Input
- 3 short fictional procurement texts
- Task
- Source-linked translation and statement verification
- Models
- 18 serving configurations
- Repetitions
- 3 per assigned task
- Execution
- OpenRouter API; serving provider in receipts
Evidence status: provisional labels and maintainer-led observations. Independent human validation is pending; no numerical quality ranks are assigned. Historical adapter excerpts are a separate tab.
…distinct model IDs
… configurations
… configurations
…assigned call slots
(cohort)
(cohort)
…procurement
responses complete
responses complete
…failed / incomplete
attempted calls
attempted calls
…not run
… pre-call blocked
… pre-call blocked
…provider calls dispatched3 cases · 3 repetitions · 2 tasks
Loading recorded model observations.
Results workspace
Model results
Loading archived evidence…
Data Unavailable
Results will not be replaced with placeholder scores. Read the report or try loading the data again.
Read reportAlphabetical by default. Sorting explores recorded values without assigning ranks. Developer-country metadata is available in model details and extended columns.
Within-cohort comparison
Selected configurations
Reviewer diagnostics
Model-assisted countertests and collection-channel checks are documented separately from model results. Inspect reviewer evidence and limitations ↗
Scope and next steps
Method and limitations
These records support internal engineering and exploratory comparisons. Qualification requires a separate, independently governed study.
Task and Configuration
Historical adapter diagnostics and procurement interpretation answer different questions. Column sorting is an exploration tool, not a qualified model ranking. Missing calls remain in assigned denominators.
Metric Validation and Data Custody
Appoint an independent custodian, validate detection and false-alarm rates, publish uncertainty by family and obtain an independent reproduction before assigning an index.
Configurations and evidence scope
Model roster
Alphabetical configurations, frozen primary labels and all three cases. Agreement uses received classifications; missing classifications retain their assigned opportunities.
18 configurations in the Policy Interpretation; one authored source group; three repetitions. Execution uses the OpenRouter API, with the serving provider recorded separately. Each request sees one case. The 10 historical adapter configurations remain a separate cohort. The 35 benchmark references are methodological comparisons, not suites run on these models.
| Model | Serving provider | Matches / assigned | Agreement / received | Missing | Reasoning | Observed input tokens per call | Complete calls |
|---|---|---|---|---|---|---|---|
| Reflection: Beam 1.0 501B A23B | nebius/fp4 | 324/324 | 100.0% | 0 | high | 378 to 1327 | 18/18 |
| Anthropic: Claude Opus 5.5 | anthropic | 322/324 | 99.4% | 0 | high | 521 to 2045 | 18/18 |
| Cohere: Command A+ | cohere/fp8 | 324/324 | 100.0% | 0 | enabled | 305 to 1154 | 17/18 |
| DeepSeek: DeepSeek V4.1 Flash 1017 | relace/fp4 | 288/324 | 100.0% | 36 | high | 339 to 1210 | 18/18 |
| Baidu: ERNIE-5.5-Flash-Preview | baidu | 321/324 | 99.1% | 0 | high | 335 to 1197 | 17/18 |
| Sakana: Fugu Max | sakana | 0/324 | N/A | 324 | high | N/A to N/A | 0/18 |
| Google: Gemini 4 Argon | google-ai-studio | 288/324 | 100.0% | 36 | high | 375 to 1521 | 17/18 |
| Z.ai: GLM 5.3 | wafer/us | 324/324 | 100.0% | 0 | high | 321 to 1207 | 18/18 |
| OpenAI: GPT-6.1 Sol | openai | 324/324 | 100.0% | 0 | high | 319 to 1231 | 18/18 |
| Tencent: Hy4 | tencent/fp4 | 321/324 | 99.1% | 0 | high | 330 to 1201 | 18/18 |
| Mistral: Mistral Large 4 | mistral/zdr | 315/324 | 97.2% | 0 | high | 329 to 1303 | 18/18 |
| Nex AGI: Nex-N2.6 | nex-agi/fp8 | 323/324 | 99.7% | 0 | high | 321 to 1207 | 18/18 |
| Qwen: Qwen3.8 Max Prime | alibaba | 322/324 | 99.4% | 0 | high | 389 to 1329 | 18/18 |
| Inference.net: Schematron V2 Small | inference-net/us | 62/324 | 31.6% | 128 | not configured | 343 to 1197 | 17/18 |
| Inference.net: Schematron V2 Turbo | inference-net/us | 47/324 | 31.5% | 175 | not configured | 335 to 1180 | 18/18 |
| Upstage: Solar Mini 4 | upstage/zdr | 324/324 | 100.0% | 0 | high | 382 to 1320 | 18/18 |
| PrismML: Ternary Bonsai 2 27B | darkbloom/int4 | 216/324 | 100.0% | 108 | xhigh | 389 to 1455 | 15/18 |
| PrismML: Ternary Bonsai 2 9B | darkbloom/int4 | 324/324 | 100.0% | 0 | enabled | 351 to 1417 | 18/18 |
Observed input tokens include instructions and payload. They are not the advertised context window or evidence of 100 MB processing. Exact configurations, output caps, catalogue ceilings, campaign identities and raw-record links are in the JSON export.
Inspect · reproduce · challenge
Evidence and reproduction
The interface is a way into the record. It does not confer independent validation, certify Workbench or turn a small internal study into a general claim about a model.
01 / REPORTResults & limitationsDesign, costs, contested label and post-hoc sensitivity ↗02 / ARCHIVEOriginal baseline archiveThree-model freeze, outputs, review bank and replay ↓03 / DATAEvery displayed observationSeparate cohorts, settings and recorded calls ↓
Results Interpretation
Configurations, not interchangeable models. Provider, reasoning, prompt, adapter, output ceiling and revision are part of the observation. A catalogue context window is not the amount of document text processed. High reasoning does not represent equal compute across providers.
Known, unknown and missing remain distinct. No response is not a semantic error. A charge without evidence is not zero. Primary labels remain frozen, even when an internal challenge makes a label questionable. The sensitivity lens is explicitly post-hoc.
Qualification remains open. There is no independent custodian, independently adjudicated domain gold or validated measure here. Repeated calls within one authored family do not establish general performance. No overall score, confidence interval or quality rank is assigned.
Other tests
Local advisory agent diagnostics · Process Interpretation · Specification Generation · Independent-validation requirements