Document-to-specification evaluation
Assessing how AI systems preserve business requirements when they turn documents into structured specifications.
Business documents contain permissions, obligations, exceptions, responsibilities and conditions. An apparently plausible specification can change their meaning. BlueprintBench examines these transformations, records the supporting source text and makes disagreements available for review.
Internal tests with provisional reference answers. Review status and limitations.
These are separate studies with different inputs and tasks. Their results are not combined into one score.
Three short fictional procurement texts. Eighteen model configurations perform source-linked translation and statement verification, with three repetitions.
Useful for inspecting compatibility, label disagreements and incomplete requests.
Explore model resultsSix fictional process descriptions, 48 propositions and six workflow tasks. Three models are tested once, with document-order and threshold variations.
Useful for examining difficult interpretation choices and draft process decisions.
Explore the six-process testA separate study of specification generation from three public document inputs. Three model IDs and three repetitions per input. A separate Google serving-provider supplement brings the total to four configurations, with original failed assignments retained.
Consult the study page for source coverage and recorded execution status.
Explore document generationWhether an output preserves the source's scope, modality, exceptions and responsible actors. Provisional label agreement is reported separately from independently adjudicated correctness.
Whether a system can generate a structured, source-linked process description. Format checks establish whether an output can be inspected, not whether its meaning is correct.
Completion, missing outputs, observed token use, provider-reported charges and request duration. Each observation retains the model, serving settings and collection conditions.
Inspect the documents and task before comparing outputs. Identify disagreements that matter for your process, review the evidence behind them, and examine whether the tested provider configuration fits your operational constraints. The current findings support a focused evaluation plan; they do not identify a universally best model or certify a production system.
The corpus register distinguishes fictional text, fictional process descriptions, public documents and earlier adapter diagnostics.
Inspect the corpusTask definitions, scoring boundaries, reviewer roles and the requirements for independent validation.
Read the method · Related benchmarksExplore source-fidelity review observations for local model configurations, including hardware, prompts, incomplete calls and evidence. These development tests use provisional model references and are not independently validated.
Explore local agent diagnostics