Document-to-specification evaluation
Specification Generation
Generated specifications from published procurement documents.
Models read the supplied document text and generate atomic requirements, process steps and executable decision rules. The comparison separates completed requests, accepted output contracts and agreement with pre-output reference decisions.
- Input
- 3 published documents · 200 supplied pages
- Task
- Generate requirements, workflow and decision rules
- Models
- 3 model IDs · 3 serving configurations
- Repetitions
- 3 per document and configuration · 27 assigned calls
- Execution
- OpenRouter MCP · serial pinned cloud providers
Closed incomplete · 14 complete, 5 incomplete and 8 not dispatched · Reference decisions are assistant drafts · Assessment scope and limitations
Execution of generated rules
Counts refer to a fixed set of draft reference decisions. A citation match or a matching test outcome does not establish full semantic correctness. Missing output and rejected contracts remain in the assigned denominator.
FEMA draft-key review, first repetition.
6 disagreements in the reviewed first-repetition outputs share an ambiguous other contract category. This is one candidate input/key ambiguity, not an established model error. Frozen counts are retained; human adjudication is pending. Review scope and competing interpretations
Recorded executions
| Model / execution | Document / repetition | Contract | Matches / assigned | Diverges | Unassessed / missing | Known USD | Duration | Native provider tokens input / output | Evidence |
|---|
Documents and generation tasks
Serving configurations
All calls use cloud inference through OpenRouter. Configuration cards distinguish original serial MCP access from supplementary batch access. No local inference, browser chat or model-assisted reviewer output is included in this comparison. Each model and pinned provider is a distinct configuration. Full source text is supplied directly; no RAG, embeddings or retrieval are used. Observed tokens are recorded per request and differ from the advertised context limit.
Method and limitations
Study closure and assigned calls · Workflow connectivity diagnostic · Draft annotation guidelines
Quote-check correction v1.1
Quote checker v1.1 corrects PDF hyphen-linewrap matching while retaining the hyphen. Source documents, model answers and draft decision labels are unchanged. Correction record · Original v1.0 analysis · v1.0 replay on the same receipts
Assessment boundaries and reproducibility
There are three document inputs and three assigned repetitions per configuration, not additional independent document families. A transport supplement is a separate configuration, not a replacement for original failed assignments. The 36 private-to-prompt test situations contain 132 component decisions per model repetition. Labels were prepared by an assistant before collection and require independent human adjudication. No confidence interval or quality ordering is inferred from these observations.
Tasks cover a defined process within each source. Full semantic coverage of every requirement in the 200 pages has not been measured. Provider defaults for temperature and seed are not controlled by the MCP interface. Exact tool arguments and returned tool envelopes are retained; original HTTP response bytes are unavailable.
Costs and authorised budget
Reported charges include whatever input, output, reasoning and cache usage the serving provider billed. Native provider counts and router-reported counts can use different tokenisers. Reasoning and cache counters may overlap the reported totals; they must not be added again. A reported cache adjustment can be negative and is not interpreted as measured savings. Token subcategories are displayed when returned; missing accounting is not replaced by zero. Source acquisition and deterministic local processing have no billed model calls. The USD 40 ceiling applies cumulatively, including preceding tests and unresolved-cost reserves.
Research method · Download reproducibility archive · Source code and complete records