Three separate input sets, with different tasks and evidence status.
Synthetic means written for the experiment. Public documents come from an external issuing body. Neither category is automatically independent test material or a validated reference answer.
Policy Interpretation
Input
3 short fictional procurement texts
Task
Source-linked translation and statement verification
Models
18 configurations across retained collection groups
Repetitions
3 per assigned task
Execution
OpenRouter API; serving provider in receipts
The texts describe invoice matching, purchase approval and vendor onboarding. Together they contain 36 source rules and 108 candidate statements. All were authored within the development process. This is one authoring provenance group, not three independently sourced organisations.
48 propositions, 6 workflow tasks and 7 paired variations
Models
3 configurations: OpenAI, Anthropic and Google
Repetitions
1 per assigned task
Execution
60 requests through OpenRouter API
The process descriptions introduce exceptions, scope conditions, dependencies and conflicting interpretations. They share one automated authoring provenance group. The source bank is exposed development material; its labels await human adjudication.
Emergency spare-parts purchasing8 propositions
Framework call-offs and aggregation8 propositions
Supplier bank change and emergency payment8 propositions
Effective dates and unresolved policy conflict8 propositions
Currency conversion and approval ownership8 propositions
Milestone acceptance, partial completion and clocks8 propositions
3 public procurement documents; 200 supplied pages
Task
Generate a structured specification from source documents
Models
3 configurations; see the study page
Repetitions
3 planned repetitions per input and configuration
Execution
OpenRouter API through MCP; status on the study page
This study uses externally issued documents rather than fictional source text. The document register records editions, URLs, retained bytes, extraction coverage and reuse conditions. Its generated outputs require source-based human review; public availability does not make a reference answer independent or complete.
FAR Part 15: Contracting by Negotiation
FAC 2026-01; effective 2026-03-13 · 56 supplied pages.
PyMuPDF 1.27.2.3 get_text(sort=True), UTF-8, no OCR, no retrieval or summarisation
Supplied physical PDF pages: 429 to 484 inclusive. Parent document: 2026 pages. Extracted source text: 240,988 characters.
Source PDF SHA-256: f22a17e30ff99682edcba9f6827e349cac0ecbe8908888c08616a81397e85bca
Public source inspected during task development; not held out
US federal government authored text; 17 USC 105. Federal text is reproduced; source PDF images, agency marks and third-party material are not relicensed or included. Code MIT does not relicense sources.
FTA Circular 4220.1G: Third Party Contracting Guidance
2025-01-17 document issue; interpretation is bounded to supplied edition · 97 supplied pages.
PyMuPDF 1.27.2.3 get_text(sort=True), UTF-8, no OCR, no retrieval or summarisation
Supplied physical PDF pages: 1 to 97 inclusive. Parent document: 97 pages. Extracted source text: 293,023 characters.
Source PDF SHA-256: 67306963820781a23f2dfceddf217a0e42c95fead209130a4f481890c022eb5d
Public source inspected during task development; not held out
US federal government authored text; 17 USC 105. Federal text is reproduced; source PDF images, agency marks and third-party material are not relicensed or included. Code MIT does not relicense sources.
FEMA Contract Provisions Guide
Version 2, effective 2024-10-01 · 47 supplied pages.
PyMuPDF 1.27.2.3 get_text(sort=True), UTF-8, no OCR, no retrieval or summarisation
Supplied physical PDF pages: 1 to 47 inclusive. Parent document: 47 pages. Extracted source text: 112,220 characters.
Source PDF SHA-256: 79d9515150a290e82fdb7f144018433d8b349df669ad196a32fe2afd55705795
Public source inspected during task development; not held out
US federal government authored text; 17 USC 105. Federal text is reproduced; source PDF images, agency marks and third-party material are not relicensed or included. Code MIT does not relicense sources.
3 short excerpts from public documents, 4,994 UTF-8 bytes
Task
Historical PALO extraction-contract checks
Models
10 historical configurations
Repetitions
1 assigned call per excerpt
Execution
Historical API records; incomplete and differing configurations
These excerpts concern planning documents and aircraft weighing. They are not the three synthetic procurement texts described above. Their structural acceptance counts diagnose adapter behaviour; they do not measure semantic quality or demonstrate processing of the larger source repositories.
Current synthetic labels are provisional, including the disputed P002, P013 and P036 items. Automated advisory responses are retained separately from human annotation templates. Changes made after observing a response do not silently redefine the original result.
