Development-exposed synthetic material · Observation revision 2026-11-11.4
Process Interpretation
Three models interpret six fictional process descriptions and generate source-linked workflow drafts.
This test examines workflow generation and interpretation of provisional labels. It does not establish independent semantic accuracy, model superiority or statistical sufficiency. The Policy Interpretation remains a separate dataset.
- Input
- 6 fictional process descriptions
- Task
- 48 propositions, 6 workflows and 7 paired variations
- Models
- 3 configurations: OpenAI, Anthropic and Google
- Repetitions
- 1 per assigned task
- Execution
- 60 requests through OpenRouter API
Execution trace
Evaluation Protocol
Each stage has its own evidence boundary.
- 01Freeze inputsSource and prompt hashes
- 02Record responseRaw bytes and request receipt
- 03Check contractIdentifiers, coverage and parsing
- 04Compare draftsProvisional labels and event decisions
- 05Human reviewPending domain adjudication
Fixed assignments · descriptive observations
Request explorer
No observations loaded.
Inspect the authored material
Source Documents
The draft answer key is provisional. These fictional policies are not procurement law or verified business practice.
Controlled source variations
Metamorphic observations
Document order should preserve judgments. A declared threshold change should affect only specified targets. The proposed relations still need expert review.
P002 · P013 · P036
Advisory label review
Automated opinions are recorded separately. They do not fill the human annotation forms or alter frozen historical labels. P036 combines a request obligation and an unspecified submission channel; the review packet asks for separate propositions before a final human decision.
Evidence boundaries
Method and next steps
Model rationale punctuation is normalised for display. Original responses and analysis remain unchanged in the evidence archive and Git links.
- The public study bank is exposed development material. It cannot become an independently held-out set by renaming it.
- Coverage of 48 propositions is not proof of statistical sufficiency.
- Draft blueprints require business approval. Exporting a file does not establish a deployable application.
Released evidence
6 fictional process descriptions · 48 provisional propositions · 3 serving configurations · 60/60 completed requests.
17/18 workflow drafts pass structural checks. Provider-reported test charge: USD 2.5539187. These counts do not establish semantic accuracy.
Versioned evidence release on GitHub ↗ · Presentation 2026-11-11.8 · Frozen observation revision 2026-11-11.4
Qualification Requirements
Independent acceptance gates remain open. Specification generation study is closed incomplete: 3 documents, 200 supplied pages, 27 assigned calls across three serving configurations, including stopped and incomplete requests. Current completion and accounting appear on the specification generation page.
Qualification Criteria
- Real procurement document families
Build licensed, source-hashed procurement packs spanning tens of pages, with amendments, precedence, exceptions and cross-document conflicts. Use 60–85% only as a development difficulty diagnostic after the metric is valid; never tune or select the sealed test to force that range.
Status: 3 public documents, 200 supplied pages; development-exposed - Document-to-specification generation
Make open generation from documents the primary task. Assess atomic rules, actors, guards, exceptions, data constraints, provenance and omissions against expert gold. Seeded-statement classification remains a secondary diagnostic.
Status: Closed incomplete: 27 assigned, 14 completed, 5 incomplete, 8 not dispatched - Blind domain annotation and adjudication
Freeze the contradiction versus not-established rubric first. Two procurement experts annotate independently without model outputs or identities; a third adjudicates. Retain pre-adjudication labels and publish per-category agreement, disagreement and unresolved cases.
Status: Two annotators and a third adjudicator needed - External selection and sealed custody
An external custodian selects new source families and controls sealed gold and access. Everything already public or inspected during development remains development material. Publish verified role and conflict disclosures, dated commitments and access receipts.
Status: No external custodian appointed - Seeded-defect measurement validation
Run a preregistered study with human-adjudicated faithful controls and semantic defects. Report detection TP/(TP+FN), false alarms FP/(FP+TN), abstentions, parsing failures and missingness separately, with uncertainty by source family and error category.
Status: Independent metric-validation study not executed - Replicated comparisons and uncertainty bands
Freeze comparable resource conditions, model/provider versions, full serving settings and retry rules before calls; collect at least three repetitions. Aggregate by independent document family, estimate paired uncertainty and keep missing responses separate. Assign an ordering only after preregistered practical-effect and multiplicity-controlled uncertainty criteria are met; otherwise show unresolved comparison bands, not forced ranks.
Status: No qualified bands or ranks
Protocol, rubric and acceptance evidence ↗ · Validation status · Measurement protocol (draft)