Statement verification
A source text and candidate statement are supplied together. The system classifies whether the statement is supported, contradicted, not established, or affected by an unresolved conflict where the test rubric permits it.
Research method
Method
The unit of evaluation is a configured system performing a defined document task.
Model identity alone does not describe an experiment. The request, source material, serving provider, reasoning setting, output limit, parsing contract and review procedure affect the observed result.
Tasks and evidence
Specification generation
The system proposes entities, rules and process decisions from the source. Outputs retain references so reviewers can assess meaning, missing requirements and unsupported additions.
Controlled variations
Document order or a stated condition is varied. Relations between paired outputs are examined. A variation that also changes the question batch cannot isolate a single causal factor.
Result Interpretation
Fixed assignments and missingness
Assigned requests remain in the record when they fail or are not run. Missing output, rejected structure and disagreement with a provisional label are different outcomes. No missing charge is converted to zero.
Structure and meaning
A valid JSON object or source reference can still express the wrong rule. Structural acceptance is a contract check. Label agreement is descriptive until the labels and rubric have been adjudicated independently.
Costs, time and context
Reported token usage and charges describe the retained requests. Maximum configured output and advertised context capacity are separate from observed prompt length. Different tokenisers, settings and scheduling prevent uncontrolled efficiency rankings.
Disputes and changes
Frozen primary labels remain visible. Predeclared exclusions, postrun ambiguity notes and sensitivity analyses are identified separately. Model agreement is advisory evidence, not majority-vote gold.
Execution and provenance
The model-result studies use the OpenRouter API, with the serving provider recorded separately where the receipt supplies it. A provider pin and the returned provider are not the same field. Missing identity details remain unavailable. These API measurements do not establish local hardware performance.
Reviewer diagnostics use separate collection channels, including APIs, a browser, desktop applications and a local model server. Their prompts and observation conditions differ from those used to produce the model results. Review those diagnostic records separately.
In this method, a test identifies the input set and task. A run is one recorded execution. A configuration identifies the model and serving settings. Archived files may retain older terms such as campaign or case; those identifiers do not create additional independent datasets.
Requirements for independent validation
- Domain experts construct and adjudicate reference propositions, including alternatives and prohibited inferences.
- An external custodian retains source sets outside the system-development process.
- The measure is tested against known semantic defects and false alarms.
- Repeated runs use an appropriate sampling design and uncertainty analysis.
- An independent party reproduces the evaluation and signs the review record.
These requirements are pending. Automated reviews can identify problems, but cannot substitute for independent domain expertise or custody. Inspect the qualification roadmap.
Governance and limitations
PALO maintains both Workbench and BlueprintBench. That relationship is a declared conflict of interest. The current observations do not certify Workbench, demonstrate organisation-wide document processing, or establish a production governance control. Source reuse conditions and provider-specific configuration limits remain part of each evidence record.
The evaluator release and the presentation revision have separate identities. Historical evidence retains its original bytes. The earlier evaluator 0.2 package is not the corrected evaluator. Inspect versioned releases.
Research context
Related benchmarks
These references compare documented tasks, evidence and evaluation practices, not model scores. PALO differences and transfer priorities are maintainer assessments, not claims of superiority or exclusive novelty. External suites were not executed. Sources checked 2026-11-11.
Reference catalogue
The original 20 references and 15 additional benchmark families selected for documented visibility.
Visibility profiles
Model-launch references, public leaderboards, competitions and dated repository signals. No hype or quality score is inferred.
Transfer priorities
Expert rubrics, workflow reliability, separate structural and semantic evidence, and unseen-case custody. Proposed work for PALO.
Purposive high-visibility selection using official model-launch references, dedicated public leaderboards/competitions and dated GitHub repository participation. At least one source-backed visibility signal is required per added family. Not an exhaustive ranking of global hype, social reach or benchmark quality. No comparable social/media time series was collected. Repository stars are cumulative attention proxies with age and project-scope bias; the Gorilla count is parent-project-wide. Different benchmark versions remain separate. No external suite was executed.
Priorities for PALO
Expert gold and atomic rubrics
Recruit procurement experts, freeze allowed alternatives and forbidden inferences, adjudicate disagreements and report human agreement before qualification.
References: GDPval, DeepResearch Bench / II, GPQA / Diamond. Proposed work; not completed qualification.
End-to-end workflow reliability
Test document import, interpretation, human correction and blueprint export with repeated runs, environment identity, failure states and cost/time accounting.
References: tau2-bench, OSWorld, Terminal-Bench. Proposed work; not completed qualification.
Structural and Semantic Assessment
Report schema/tool validity, source support, semantic violations and downstream executable checks as separate outcomes.
References: Berkeley Function Calling Leaderboard, SWE-Bench Pro, BrowseComp. Proposed work; not completed qualification.
Freshness, custody and disclosure
Use externally selected unseen families, date windows, protected answer custody, sponsor/data-access disclosures and versioned judge/adapter policies.
References: LiveCodeBench, Humanity's Last Exam, ARC-AGI, FrontierMath. Proposed work; not completed qualification.
Additional Benchmark Families
| Reference and version | Target and evidence | Visibility signal | PALO difference and caution | Practice to adopt |
|---|---|---|---|---|
| Humanity's Last ExamExpert academic benchmark Public and private sets; tool-enabled and tool-free runs must remain distinct. | Expert-level questions spanning text and images. Public evaluation and a separate private set intended to monitor overfitting. Primary source reviewed | Featured in Google's annual research recap and maintained on a dedicated Scale leaderboard. Source 1 / Source 2 | PALO targets faithful interpretation of supplied business documents, rather than broad expert knowledge. Exam performance is not business-process validity; public exposure and tool access affect interpretation. | Keep public diagnostics separate from unseen evaluation; record tool access and calibration alongside correctness.Transfer: Method |
| ARC-AGIReasoning and interactive benchmark family ARC-AGI-1, ARC-AGI-2 and ARC-AGI-3 are separate evaluations. | Generalisation in visual abstraction tasks and, in ARC-AGI-3, interactive environments. Separate benchmark editions and verified/community leaderboards, with cost information. Primary source reviewed | Public prize competitions and distinct verified/community leaderboards provide visible participation signals. Source 1 / Source 2 | PALO tests preservation of explicit source meaning, not abstract task generalisation. Changes in task format, compute budget and evaluation split prevent direct score comparisons across editions. | Display verification status, task edition and cost per task separately; never join incompatible editions into one progress line.Transfer: Method |
| GPQA / DiamondExpert science question answering GPQA main and Diamond subsets must be named; prompt and tool configurations are separate. | Difficult expert-written multiple-choice science questions. Domain-expert and non-expert assessment; public data and baseline code. Primary source reviewed | Explicitly used in the GPT-5 launch and Google research recap. Source 1 / Source 2548 GitHub starsRepository snapshot, not benchmark usage; 2026-11-11 | PALO requires evidence-bound interpretations and permissible alternatives, not a single exam option. Expert authorship does not make every answer immutable; subset choice and reasoning budget must be frozen. | Use domain-qualified annotators, disagreement adjudication and an explicit correction channel for flawed gold.Transfer: Method |
| MMLU-ProMulti-domain knowledge and reasoning MMLU-Pro is distinct from MMLU and multilingual MMMLU. | Harder multiple-choice questions across subject domains, with ten answer options. The paper evaluates robustness across 24 prompt styles; public dataset and leaderboard. Primary source reviewed | Dedicated Hugging Face benchmark leaderboard and an active public evaluation repository. Source 1 / Source 2432 GitHub starsRepository snapshot, not benchmark usage; 2026-11-11 | PALO needs source-specific rules and exceptions; broad subject accuracy cannot replace that measure. Aggregate subject accuracy can hide weak domains; broad popularity is not evidence of procurement fidelity. | Add a preregistered prompt-robustness check while keeping the primary prompt fixed.Transfer: Context |
| AIME as an LLM evaluationCompetition-derived mathematics evaluation AIME 2025 launch results are a specific historical example, not a universal AIME protocol. | Invitational mathematics problems with objectively checkable answers. An established human competition reused in model evaluations; provider protocols define the AI test. Primary source reviewed | Prominent in GPT-5 and Claude Sonnet 4.5 launch evaluations. Source 1 / Source 2 | PALO has ambiguous business interpretation and human review needs that exact-answer mathematics does not represent. Small annual sets, repeated sampling and tool-assisted solutions can materially change results. MAA does not endorse PALO or define one universal LLM leaderboard. | Publish edition, problem IDs, attempts, tool access and aggregation rules for every reported result.Transfer: Context |
| FrontierMathAdvanced mathematics programme Tiers 1-4, Open Problems and Erdos tracks must not be pooled. | Mathematician-authored difficult problems and separately defined research-mathematics tracks. Tiers 1-4, Open Problems and formalised Erdos problems have different evaluation contracts. Primary source reviewed | Epoch documents its inclusion in the o3 announcement and the resulting sponsorship/data-access clarification. Source 1 / Source 2 | PALO assesses business-rule meaning and correction effort, not mathematical research capability. Sponsorship and data access require explicit disclosure; a hard task set alone does not establish evaluator independence. | Disclose funding, advance data access and custodian roles before collecting submissions.Transfer: Method |
| LiveCodeBenchTime-windowed coding evaluation Release windows and coding scenarios are part of the benchmark identity. | Coding tasks collected from programming contests with recorded release dates. Public harness, task dates, leaderboard and several coding scenarios. Primary source reviewed | Public multi-model leaderboard and community evaluation repository. Source 1 / Source 2974 GitHub starsRepository snapshot, not benchmark usage; 2026-11-11 | PALO uses business source families rather than programming-contest tasks. Date filtering mitigates exposure risk but cannot prove absence from undisclosed model training data. | Assign held-out source families by date and publish the evaluation window and contamination checks.Transfer: Method |
| Terminal-BenchTerminal-agent benchmark family Current homepage shows 4.1. The published 2.0 methodology and older launch results refer to separate releases. | Agents complete tasks in executable terminal environments. The current page separates model and agent and displays resolution, cost, tokens and uncertainty. Primary source reviewed | Featured in Claude launch evaluation notes; a dedicated public leaderboard reports agent outcomes. Source 1 / Source 2 | PALO's document interpretation track precedes execution; Workbench end-to-end tests can reuse the environment discipline. Agent scaffolding, tools and infrastructure affect results. Resolution rate is not a pure model capability score. | Freeze environment, agent and model identities; retain traces, resource costs and uncertainty over repeated tasks.Transfer: Direct |
| SWE-Bench ProSoftware-engineering agent evaluation SWE-Bench Pro is distinct from SWE-bench Verified; public and private results are separate. | Long-horizon changes to substantial software repositories. Reproducible environments, fail-to-pass and regression tests, public and held-out partitions. Primary source reviewed | Dedicated Scale leaderboard and public release with inspectable task infrastructure. Source 1 / Source 2559 GitHub starsRepository snapshot, not benchmark usage; 2026-11-11 | PALO evaluates the specification produced before coding, while this suite evaluates repository changes. Passing tests does not establish faithful business intent; repository licensing alone does not prove contamination avoidance. | Use executable downstream checks to complement semantic review and preserve environment failures in the ledger.Transfer: Method |
| BrowseCompWeb-browsing agent evaluation Original BrowseComp task set; tool setup and test-time computation are configuration fields. | Finding difficult, distributed information on the web. Human-created questions favour short verifiable answers; the release documents search and compute effects. Primary source reviewed | OpenAI benchmark launch explicitly compares browsing models and compute strategies. Source 1 | PALO already has a supplied document corpus and needs to preserve its meaning, not primarily discover obscure web facts. The authors explicitly note that short-answer performance may not transfer to open-ended user work. | Measure retrieval success separately from interpretation and retain source URLs, retrieval times and tool budgets.Transfer: Method |
| DeepResearch Bench / IIResearch-report evaluation family Original and II are separate sets. The original repository records evaluator and cleaning-model changes in 2026. | Research report quality, evidence use and citation reliability. Original RACE/FACT methods and a separate Bench II with fine-grained rubrics derived from expert reports. Primary source reviewed | Public multi-agent leaderboard, released outputs and an active follow-on expert-rubric benchmark. Source 1 / Source 2849 GitHub starsRepository snapshot, not benchmark usage; 2026-11-11 | PALO seeks source-to-specification fidelity; report-level quality and evidence checks are adjacent rather than interchangeable. LLM judges can change scores or share model-family biases. Independent human calibration remains necessary. | Build expert-reviewed atomic rubrics and version every judge, cleaning step and citation checker; test agreement against human labels.Transfer: Direct |
| OSWorldComputer-use benchmark family OSWorld-Verified and OSWorld 2.0 remain separate; steps and repetitions must be recorded. | Computer agents execute workflows in real application environments. Verified repairs earlier tasks; 2.0 extends long-horizon evaluation and distinguishes binary and partial reward. Primary source reviewed | Featured in Claude Sonnet launch evaluations, with a public project and subsequent 2.0 leaderboard. Source 1 / Source 2 / Source 33,258 GitHub starsRepository snapshot, not benchmark usage; 2026-11-11 | PALO business usability also depends on completing import, review and export, beyond the quality of extracted text. A GUI success score depends on action budget and environment; it is not a direct semantic-fidelity score. | Test complete user journeys with environment resets, final-artifact checks, failure traces and explicit partial completion.Transfer: Direct |
| tau2-benchPolicy and tool-agent-user interaction tau2 is distinct from tau-bench; domain, simulator, policy revision and modality must be pinned. | Agents work with users and tools under explicit domain policies. Domain policies, environments, agent/user simulators and repeatable execution are provided. Primary source reviewed | Explicitly reported in Claude Sonnet 4.5 launch methodology, with substantial repository participation. Source 1 / Source 22,434 GitHub starsRepository snapshot, not benchmark usage; 2026-11-11 | PALO extracts policies into a specification; this suite tests following policies during interactions. User simulators and policy prompt amendments can change outcomes; they must be part of the tested system identity. | Link each source obligation to a testable state transition and measure repeated compliance, including exceptions and unauthorised actions.Transfer: Direct |
| Berkeley Function Calling LeaderboardTool-use and agentic evaluation BFCL V4; do not mix scores from earlier category compositions or native/prompt interfaces. | Function selection, arguments and multi-turn or agentic tool use. V4 exposes category scores, model interfaces, costs, latency, model outputs and a pinned evaluator checkpoint. Primary source reviewed | Public Berkeley leaderboard with sortable metrics, comparison visualisation and a widely followed parent repository. Source 1 / Source 213,069 GitHub starsWhole Gorilla repository, not BFCL alone; 2026-11-11 | PALO needs semantic business validation in addition to syntactically valid tool calls. Gorilla stars cover the parent project, not BFCL alone. Valid tool syntax is not proof of a correct business decision. | Separate contract conformity from task correctness, publish adapter versions and category-level errors, and distinguish native tool APIs from text prompting.Transfer: Direct |
| GDPvalProfessional work-product evaluation Published GDPval release and public subset; human grading and automated grading are distinct. | Economically valuable work products across professional occupations. Domain professionals create and review tasks and compare model deliverables with human work. Primary source reviewed | Dedicated OpenAI release compares several major providers on professional work products. Source 1 | PALO's proposed procurement study targets document-to-specification fidelity, a narrower unit than a finished professional deliverable. The authors do not treat experimental automated grading as a replacement for expert assessment. Occupation coverage does not establish all enterprise use cases. | Use actual procurement experts for blind rubric design and review, and measure repair effort alongside deliverable acceptance.Transfer: Direct |
Original Reference Set
| Reference and version | Target and evidence | Visibility signal | PALO difference and caution | Practice to adopt |
|---|---|---|---|---|
| HELMEvaluation framework | Broad scenario and metric coverage for foundation models. Public framework, prompts/responses inspection and leaderboards. README reports maintenance mode since June 2026. | Original reference set; visibility not reassessed in this extension. | PALO proposes a narrower procurement specification-fidelity study. Multi-dimensional evaluation is established prior art. | Retain scenario, adapter, model and metric versions.Transfer: Not assessed in the visibility extension |
| BIG-benchTask suite | Collaborative tasks probing diverse language-model capabilities. Public tasks, evaluation code and published experiments. Repository archived on 17 April 2026; retained as historical prior art. | Original reference set; visibility not reassessed in this extension. | PALO targets business-rule preservation in a reviewed specification, rather than broad capability coverage. | Use explicit task definitions and diverse independently selected cases.Transfer: Not assessed in the visibility extension |
| BigBIODataset library | Harmonised biomedical datasets and task schemas. Dataloaders, provenance, licensing metadata and dataset documentation. Primary source reviewed | Original reference set; visibility not reassessed in this extension. | A dataset library is a reference for corpus governance, not a competing procurement-specification score. | Document source rights, schema mappings and provenance.Transfer: Not assessed in the visibility extension |
| BigScience EvaluationEvaluation framework | Task-based language-model evaluation, including translation and question answering. Code, task abstractions and evaluation instructions. README retains 2021 milestones; no current deployment qualification inferred. | Original reference set; visibility not reassessed in this extension. | PALO adds a scoped evidence contract for annotated business propositions; it has not demonstrated broader capability coverage. | Keep task-specific evaluation separate from model execution.Transfer: Not assessed in the visibility extension |
| LM Evaluation HarnessEvaluation framework | Configurable execution of academic language-model tasks across backends. Task configurations, model adapters, metrics and sample logging. Primary source reviewed | Original reference set; visibility not reassessed in this extension. | This is reusable execution infrastructure. PALO supplies a proposed domain-specific task and adjudication contract; no integration is claimed. | Reuse configuration discipline and record prompt, backend and generation settings.Transfer: Not assessed in the visibility extension |
| Scholar EvalsUnverified directory entry | Scope could not be verified from the referenced repository. The referenced repository returned HTTP 404 during this review. Unavailable at the referenced URL; reason unknown. | Original reference set; visibility not reassessed in this extension. | No capability comparison or judgement of benchmark quality is made from an unavailable source. | Preserve the discovery reference and mark unknown evidence explicitly.Transfer: Not assessed in the visibility extension |
| BigCode Evaluation HarnessCoding evaluation framework | Code generation and related coding tasks, including unit-test-based evaluation. Task implementations, generation/evaluation tooling and container instructions. Primary source reviewed | Original reference set; visibility not reassessed in this extension. | Correct code against supplied tests does not establish that the originating business requirement was interpreted faithfully. | Keep code-test outcomes separate from source-rule fidelity.Transfer: Not assessed in the visibility extension |
| Chatbot Arena / ArenaPreference evaluation platform | Model comparison with uncertainty-aware score and rank presentation. Published ranking methodology and confidence/rank intervals. Current methodology checked; directory points to historical FastChat infrastructure. | Original reference set; visibility not reassessed in this extension. | PALO proposes source-based adjudication of obligations and exceptions; preference and semantic fidelity answer different questions. | Show uncertainty and avoid treating a small numerical ordering as a stable ranking.Transfer: Not assessed in the visibility extension |
| GLUELanguage-understanding suite | General natural-language understanding across multiple tasks. Task suite, analysis platform and published methodology. The directory links jiant, an implementation toolkit whose README says it is no longer actively maintained. | Original reference set; visibility not reassessed in this extension. | PALO examines source-to-specification transformations and the correction burden within a business domain. | Use task-specific gold and publish per-task results.Transfer: Not assessed in the visibility extension |
| SuperGLUELanguage-understanding suite | More challenging general language-understanding tasks following GLUE. Benchmark tasks, evaluation methodology and analysis resources. Distinguish the benchmark from the jiant implementation linked by the directory. | Original reference set; visibility not reassessed in this extension. | PALO is narrower and currently unqualified; neither general NLU performance nor PALO diagnostics imply success on the other. | Define acceptance and task difficulty before comparing systems.Transfer: Not assessed in the visibility extension |
| CLUELanguage-understanding suite | Chinese-language understanding through representative classification and reading tasks. Datasets, baselines, task metrics and leaderboards. Primary source reviewed | Original reference set; visibility not reassessed in this extension. | PALO has not qualified multilingual specification fidelity. Country of model origin is not a language-quality measurement. | Treat language and localisation as explicit evaluation conditions.Transfer: Not assessed in the visibility extension |
| CodeXGLUECode-intelligence suite | Code understanding and generation, including code/text transformations. Task datasets, baselines and evaluation resources. Primary source reviewed | Original reference set; visibility not reassessed in this extension. | PALO proposes checking the business meaning before code is produced; code/text overlap is acknowledged. | Separate transformation types and use outcome-appropriate metrics.Transfer: Not assessed in the visibility extension |
| Vals AIApplied evaluation platform | Industry-oriented tasks and model comparisons across professional domains. Published benchmark pages and methodologies; some benchmarks are proprietary. Primary source reviewed | Original reference set; visibility not reassessed in this extension. | Enterprise evaluation is already established. PALO proposes a specific document-to-specification fidelity study, without claiming broader coverage or superior validation. | Use realistic tasks and make public/private evidence boundaries explicit.Transfer: Not assessed in the visibility extension |
| Artificial AnalysisEvaluation platform | Quality, intelligence, performance and price of models, endpoints and systems. Published measurement definitions and comparative results. Primary source reviewed | Original reference set; visibility not reassessed in this extension. | PALO records configuration-specific evidence and proposes workflow correction effort; current cost records are not full enterprise cost measurements. | Distinguish model, serving endpoint and complete system.Transfer: Not assessed in the visibility extension |
| OfficeQADocument-reasoning benchmark | Complex questions over real-world financial documents. Evaluation code and corpus tooling; source documents, parsed data and gated answer sets are documented. Primary source reviewed | Original reference set; visibility not reassessed in this extension. | A close comparator. PALO proposes evaluating the resulting specification and altered rules, rather than only answers to questions. | Separate parsing, retrieval and reasoning; keep unseen document families.Transfer: Not assessed in the visibility extension |
| DABstepData-agent benchmark | Multi-step business data analysis with factoid answers. Tasks, answer-format requirements and automated answer checks. Primary source reviewed | Original reference set; visibility not reassessed in this extension. | PALO propositions can admit multiple legitimate formulations and require semantic adjudication; answer normalisation alone is insufficient. | Use deterministic checks where valid, without confusing formatting with correctness.Transfer: Not assessed in the visibility extension |
| OmniDocBenchDocument-parsing benchmark | Document parsing, including text, tables, formulas and layout. Annotated document data and evaluation tooling. Primary source reviewed | Original reference set; visibility not reassessed in this extension. | Parsing is an upstream component; PALO proposes checking whether extracted content retains its business meaning in a specification. | Measure extraction independently from interpretation.Transfer: Not assessed in the visibility extension |
| SWE-bench VerifiedSoftware-agent benchmark | Resolution of repository issues using a human-validated task subset. Task definitions, evaluation resources and published agent results. Primary source reviewed | Original reference set; visibility not reassessed in this extension. | PALO focuses on interpreting the source requirement before implementation; passing software tests addresses a different evaluation stage. | Freeze the harness and distinguish model comparisons from whole-agent comparisons.Transfer: Not assessed in the visibility extension |
| FACTS GroundingGrounded-generation benchmark | Factual grounding of generated responses in supplied documents. Published benchmark and multi-judge evaluation methodology. Primary source reviewed | Original reference set; visibility not reassessed in this extension. | A close semantic comparator. PALO proposes domain-specific critical-rule inversions and human repair effort; that additional value still needs validation. | Calibrate semantic judging against independent reference decisions.Transfer: Not assessed in the visibility extension |
| Andon Labs Blueprint-Bench 2Spatial-reasoning benchmark | Reconstruction of apartment floor plans from photographs. Published task description, connectivity metric and leaderboard. Primary source reviewed | Original reference set; visibility not reassessed in this extension. | Distinct project and meaning of blueprint: spatial floor plans versus business development specifications. No affiliation or shared score. | Disambiguate the name and compare only commensurate tasks.Transfer: Not assessed in the visibility extension |
Direct: transferable to business workflow, evidence or operational checks. Method: useful evaluation design. Context: broad capability context. These are maintainer assessments, not external validation. Discovery: terryyz/llm-benchmark. All 12 directory entries remain covered. References imply no endorsement or integration.
Read the full comparison on GitHub / Download CSV / Inspect source records / Repository snapshots / Earlier methodological review
Contributions
The next stage needs procurement-domain annotators, an external custodian and an independent methods reviewer. Reviewers should state their role and conflicts, assess source material before seeing model identities, and retain pre-adjudication judgments.
Read the reviewer kit · Contact the project. Do not send confidential business documents or personal datasets.