id,name,kind,source_url,focus,evidence,palo_difference,lesson,source_status,in_requested_directory,checked_on,review_group,version_scope,visibility_basis,visibility_sources,repo_stars,repo_star_scope,caution,palo_transfer,repo_url
helm,HELM,Evaluation framework,https://github.com/stanford-crfm/helm,Broad scenario and metric coverage for foundation models.,"Public framework, prompts/responses inspection and leaderboards.",PALO proposes a narrower procurement specification-fidelity study. Multi-dimensional evaluation is established prior art.,"Retain scenario, adapter, model and metric versions.",README reports maintenance mode since June 2026.,True,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
bigbench,BIG-bench,Task suite,https://github.com/google/BIG-bench,Collaborative tasks probing diverse language-model capabilities.,"Public tasks, evaluation code and published experiments.","PALO targets business-rule preservation in a reviewed specification, rather than broad capability coverage.",Use explicit task definitions and diverse independently selected cases.,Repository archived on 17 April 2026; retained as historical prior art.,True,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
bigbio,BigBIO,Dataset library,https://github.com/bigscience-workshop/biomedical,Harmonised biomedical datasets and task schemas.,"Dataloaders, provenance, licensing metadata and dataset documentation.","A dataset library is a reference for corpus governance, not a competing procurement-specification score.","Document source rights, schema mappings and provenance.",Primary source reviewed,True,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
bigscience,BigScience Evaluation,Evaluation framework,https://github.com/bigscience-workshop/evaluation,"Task-based language-model evaluation, including translation and question answering.","Code, task abstractions and evaluation instructions.",PALO adds a scoped evidence contract for annotated business propositions; it has not demonstrated broader capability coverage.,Keep task-specific evaluation separate from model execution.,README retains 2021 milestones; no current deployment qualification inferred.,True,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
lm-eval,LM Evaluation Harness,Evaluation framework,https://github.com/EleutherAI/lm-evaluation-harness,Configurable execution of academic language-model tasks across backends.,"Task configurations, model adapters, metrics and sample logging.",This is reusable execution infrastructure. PALO supplies a proposed domain-specific task and adjudication contract; no integration is claimed.,"Reuse configuration discipline and record prompt, backend and generation settings.",Primary source reviewed,True,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
scholar,Scholar Evals,Unverified directory entry,https://github.com/scholar-org/scholar-evals,Scope could not be verified from the referenced repository.,The referenced repository returned HTTP 404 during this review.,No capability comparison or judgement of benchmark quality is made from an unavailable source.,Preserve the discovery reference and mark unknown evidence explicitly.,Unavailable at the referenced URL; reason unknown.,True,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
bigcode,BigCode Evaluation Harness,Coding evaluation framework,https://github.com/bigcode-project/bigcode-evaluation-harness,"Code generation and related coding tasks, including unit-test-based evaluation.","Task implementations, generation/evaluation tooling and container instructions.",Correct code against supplied tests does not establish that the originating business requirement was interpreted faithfully.,Keep code-test outcomes separate from source-rule fidelity.,Primary source reviewed,True,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
arena,Chatbot Arena / Arena,Preference evaluation platform,https://arena.ai/blog/ranking-method,Model comparison with uncertainty-aware score and rank presentation.,Published ranking methodology and confidence/rank intervals.,PALO proposes source-based adjudication of obligations and exceptions; preference and semantic fidelity answer different questions.,Show uncertainty and avoid treating a small numerical ordering as a stable ranking.,Current methodology checked; directory points to historical FastChat infrastructure.,True,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
glue,GLUE,Language-understanding suite,https://arxiv.org/abs/1804.07461,General natural-language understanding across multiple tasks.,"Task suite, analysis platform and published methodology.",PALO examines source-to-specification transformations and the correction burden within a business domain.,Use task-specific gold and publish per-task results.,"The directory links jiant, an implementation toolkit whose README says it is no longer actively maintained.",True,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
superglue,SuperGLUE,Language-understanding suite,https://arxiv.org/abs/1905.00537,More challenging general language-understanding tasks following GLUE.,"Benchmark tasks, evaluation methodology and analysis resources.",PALO is narrower and currently unqualified; neither general NLU performance nor PALO diagnostics imply success on the other.,Define acceptance and task difficulty before comparing systems.,Distinguish the benchmark from the jiant implementation linked by the directory.,True,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
clue,CLUE,Language-understanding suite,https://github.com/CLUEbenchmark/CLUE,Chinese-language understanding through representative classification and reading tasks.,"Datasets, baselines, task metrics and leaderboards.",PALO has not qualified multilingual specification fidelity. Country of model origin is not a language-quality measurement.,Treat language and localisation as explicit evaluation conditions.,Primary source reviewed,True,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
codexglue,CodeXGLUE,Code-intelligence suite,https://github.com/microsoft/CodeXGLUE,"Code understanding and generation, including code/text transformations.","Task datasets, baselines and evaluation resources.",PALO proposes checking the business meaning before code is produced; code/text overlap is acknowledged.,Separate transformation types and use outcome-appropriate metrics.,Primary source reviewed,True,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
vals,Vals AI,Applied evaluation platform,https://www.vals.ai/benchmarks,Industry-oriented tasks and model comparisons across professional domains.,Published benchmark pages and methodologies; some benchmarks are proprietary.,"Enterprise evaluation is already established. PALO proposes a specific document-to-specification fidelity study, without claiming broader coverage or superior validation.",Use realistic tasks and make public/private evidence boundaries explicit.,Primary source reviewed,False,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
artificial-analysis,Artificial Analysis,Evaluation platform,https://artificialanalysis.ai/methodology,"Quality, intelligence, performance and price of models, endpoints and systems.",Published measurement definitions and comparative results.,PALO records configuration-specific evidence and proposes workflow correction effort; current cost records are not full enterprise cost measurements.,"Distinguish model, serving endpoint and complete system.",Primary source reviewed,False,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
officeqa,OfficeQA,Document-reasoning benchmark,https://github.com/databricks/officeqa,Complex questions over real-world financial documents.,"Evaluation code and corpus tooling; source documents, parsed data and gated answer sets are documented.","A close comparator. PALO proposes evaluating the resulting specification and altered rules, rather than only answers to questions.","Separate parsing, retrieval and reasoning; keep unseen document families.",Primary source reviewed,False,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
dabstep,DABstep,Data-agent benchmark,https://huggingface.co/blog/dabstep,Multi-step business data analysis with factoid answers.,"Tasks, answer-format requirements and automated answer checks.",PALO propositions can admit multiple legitimate formulations and require semantic adjudication; answer normalisation alone is insufficient.,"Use deterministic checks where valid, without confusing formatting with correctness.",Primary source reviewed,False,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
omnidocbench,OmniDocBench,Document-parsing benchmark,https://github.com/opendatalab/OmniDocBench,"Document parsing, including text, tables, formulas and layout.",Annotated document data and evaluation tooling.,Parsing is an upstream component; PALO proposes checking whether extracted content retains its business meaning in a specification.,Measure extraction independently from interpretation.,Primary source reviewed,False,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
swebench,SWE-bench Verified,Software-agent benchmark,https://www.swebench.com/verified,Resolution of repository issues using a human-validated task subset.,"Task definitions, evaluation resources and published agent results.",PALO focuses on interpreting the source requirement before implementation; passing software tests addresses a different evaluation stage.,Freeze the harness and distinguish model comparisons from whole-agent comparisons.,Primary source reviewed,False,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
facts,FACTS Grounding,Grounded-generation benchmark,https://arxiv.org/abs/2501.03200,Factual grounding of generated responses in supplied documents.,Published benchmark and multi-judge evaluation methodology.,A close semantic comparator. PALO proposes domain-specific critical-rule inversions and human repair effort; that additional value still needs validation.,Calibrate semantic judging against independent reference decisions.,Primary source reviewed,False,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
andon,Andon Labs Blueprint-Bench 2,Spatial-reasoning benchmark,https://andonlabs.com/evals/blueprint-bench-2,Reconstruction of apartment floor plans from photographs.,"Published task description, connectivity metric and leaderboard.",Distinct project and meaning of blueprint: spatial floor plans versus business development specifications. No affiliation or shared score.,Disambiguate the name and compare only commensurate tasks.,Primary source reviewed,False,2026-11-11,original-review,,Not assessed in the visibility extension,,,,,Not assessed in the visibility extension,
hle,Humanity's Last Exam,Expert academic benchmark,https://labs.scale.com/leaderboard/humanitys_last_exam,Expert-level questions spanning text and images.,Public evaluation and a separate private set intended to monitor overfitting.,"PALO targets faithful interpretation of supplied business documents, rather than broad expert knowledge.",Keep public diagnostics separate from unseen evaluation; record tool access and calibration alongside correctness.,Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,Public and private sets; tool-enabled and tool-free runs must remain distinct.,Featured in Google's annual research recap and maintained on a dedicated Scale leaderboard.,https://blog.google/innovation-and-ai/products/2025-research-breakthroughs/ ; https://labs.scale.com/leaderboard/humanitys_last_exam,,Not measured,Exam performance is not business-process validity; public exposure and tool access affect interpretation.,Method,
arc-agi,ARC-AGI,Reasoning and interactive benchmark family,https://arcprize.org/leaderboard,"Generalisation in visual abstraction tasks and, in ARC-AGI-3, interactive environments.","Separate benchmark editions and verified/community leaderboards, with cost information.","PALO tests preservation of explicit source meaning, not abstract task generalisation.","Display verification status, task edition and cost per task separately; never join incompatible editions into one progress line.",Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,"ARC-AGI-1, ARC-AGI-2 and ARC-AGI-3 are separate evaluations.",Public prize competitions and distinct verified/community leaderboards provide visible participation signals.,https://arcprize.org/leaderboard ; https://arcprize.org/results,,Not measured,"Changes in task format, compute budget and evaluation split prevent direct score comparisons across editions.",Method,
gpqa,GPQA / Diamond,Expert science question answering,https://arxiv.org/abs/2311.12022,Difficult expert-written multiple-choice science questions.,Domain-expert and non-expert assessment; public data and baseline code.,"PALO requires evidence-bound interpretations and permissible alternatives, not a single exam option.","Use domain-qualified annotators, disagreement adjudication and an explicit correction channel for flawed gold.",Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,GPQA main and Diamond subsets must be named; prompt and tool configurations are separate.,Explicitly used in the GPT-5 launch and Google research recap.,https://openai.com/index/introducing-gpt-5/ ; https://blog.google/innovation-and-ai/products/2025-research-breakthroughs/,548,"Repository snapshot, not benchmark usage",Expert authorship does not make every answer immutable; subset choice and reasoning budget must be frozen.,Method,https://github.com/idavidrein/gpqa
mmlu-pro,MMLU-Pro,Multi-domain knowledge and reasoning,https://arxiv.org/abs/2406.01574,"Harder multiple-choice questions across subject domains, with ten answer options.",The paper evaluates robustness across 24 prompt styles; public dataset and leaderboard.,PALO needs source-specific rules and exceptions; broad subject accuracy cannot replace that measure.,Add a preregistered prompt-robustness check while keeping the primary prompt fixed.,Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,MMLU-Pro is distinct from MMLU and multilingual MMMLU.,Dedicated Hugging Face benchmark leaderboard and an active public evaluation repository.,https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro ; https://github.com/TIGER-AI-Lab/MMLU-Pro,432,"Repository snapshot, not benchmark usage",Aggregate subject accuracy can hide weak domains; broad popularity is not evidence of procurement fidelity.,Context,https://github.com/TIGER-AI-Lab/MMLU-Pro
aime,AIME as an LLM evaluation,Competition-derived mathematics evaluation,https://maa.org/maa-invitational-competitions/,Invitational mathematics problems with objectively checkable answers.,An established human competition reused in model evaluations; provider protocols define the AI test.,PALO has ambiguous business interpretation and human review needs that exact-answer mathematics does not represent.,"Publish edition, problem IDs, attempts, tool access and aggregation rules for every reported result.",Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,"AIME 2025 launch results are a specific historical example, not a universal AIME protocol.",Prominent in GPT-5 and Claude Sonnet 4.5 launch evaluations.,https://openai.com/index/introducing-gpt-5/ ; https://www.anthropic.com/news/claude-sonnet-4-5,,Not measured,"Small annual sets, repeated sampling and tool-assisted solutions can materially change results. MAA does not endorse PALO or define one universal LLM leaderboard.",Context,
frontiermath,FrontierMath,Advanced mathematics programme,https://epoch.ai/frontiermath,Mathematician-authored difficult problems and separately defined research-mathematics tracks.,"Tiers 1-4, Open Problems and formalised Erdos problems have different evaluation contracts.","PALO assesses business-rule meaning and correction effort, not mathematical research capability.","Disclose funding, advance data access and custodian roles before collecting submissions.",Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,"Tiers 1-4, Open Problems and Erdos tracks must not be pooled.",Epoch documents its inclusion in the o3 announcement and the resulting sponsorship/data-access clarification.,https://epoch.ai/latest/epoch-impact-report-2024 ; https://epoch.ai/latest/openai-and-frontiermath,,Not measured,Sponsorship and data access require explicit disclosure; a hard task set alone does not establish evaluator independence.,Method,
livecodebench,LiveCodeBench,Time-windowed coding evaluation,https://livecodebench.github.io/,Coding tasks collected from programming contests with recorded release dates.,"Public harness, task dates, leaderboard and several coding scenarios.",PALO uses business source families rather than programming-contest tasks.,Assign held-out source families by date and publish the evaluation window and contamination checks.,Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,Release windows and coding scenarios are part of the benchmark identity.,Public multi-model leaderboard and community evaluation repository.,https://livecodebench.github.io/ ; https://github.com/LiveCodeBench/LiveCodeBench,974,"Repository snapshot, not benchmark usage",Date filtering mitigates exposure risk but cannot prove absence from undisclosed model training data.,Method,https://github.com/LiveCodeBench/LiveCodeBench
terminal-bench,Terminal-Bench,Terminal-agent benchmark family,https://www.tbench.ai/,Agents complete tasks in executable terminal environments.,"The current page separates model and agent and displays resolution, cost, tokens and uncertainty.",PALO's document interpretation track precedes execution; Workbench end-to-end tests can reuse the environment discipline.,"Freeze environment, agent and model identities; retain traces, resource costs and uncertainty over repeated tasks.",Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,Current homepage shows 4.1. The published 2.0 methodology and older launch results refer to separate releases.,Featured in Claude launch evaluation notes; a dedicated public leaderboard reports agent outcomes.,https://www.anthropic.com/news/claude-sonnet-4-5 ; https://www.tbench.ai/,,Not measured,"Agent scaffolding, tools and infrastructure affect results. Resolution rate is not a pure model capability score.",Direct,
swebench-pro,SWE-Bench Pro,Software-engineering agent evaluation,https://labs.scale.com/leaderboard/swe_bench_pro,Long-horizon changes to substantial software repositories.,"Reproducible environments, fail-to-pass and regression tests, public and held-out partitions.","PALO evaluates the specification produced before coding, while this suite evaluates repository changes.",Use executable downstream checks to complement semantic review and preserve environment failures in the ledger.,Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,SWE-Bench Pro is distinct from SWE-bench Verified; public and private results are separate.,Dedicated Scale leaderboard and public release with inspectable task infrastructure.,https://labs.scale.com/leaderboard/swe_bench_pro ; https://github.com/scaleapi/SWE-bench_Pro-os,559,"Repository snapshot, not benchmark usage",Passing tests does not establish faithful business intent; repository licensing alone does not prove contamination avoidance.,Method,https://github.com/scaleapi/SWE-bench_Pro-os
browsecomp,BrowseComp,Web-browsing agent evaluation,https://openai.com/index/browsecomp/,"Finding difficult, distributed information on the web.",Human-created questions favour short verifiable answers; the release documents search and compute effects.,"PALO already has a supplied document corpus and needs to preserve its meaning, not primarily discover obscure web facts.","Measure retrieval success separately from interpretation and retain source URLs, retrieval times and tool budgets.",Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,Original BrowseComp task set; tool setup and test-time computation are configuration fields.,OpenAI benchmark launch explicitly compares browsing models and compute strategies.,https://openai.com/index/browsecomp/,,Not measured,The authors explicitly note that short-answer performance may not transfer to open-ended user work.,Method,
deepresearch-bench,DeepResearch Bench / II,Research-report evaluation family,https://deepresearch-bench.github.io/,"Research report quality, evidence use and citation reliability.",Original RACE/FACT methods and a separate Bench II with fine-grained rubrics derived from expert reports.,PALO seeks source-to-specification fidelity; report-level quality and evidence checks are adjacent rather than interchangeable.,"Build expert-reviewed atomic rubrics and version every judge, cleaning step and citation checker; test agreement against human labels.",Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,Original and II are separate sets. The original repository records evaluator and cleaning-model changes in 2026.,"Public multi-agent leaderboard, released outputs and an active follow-on expert-rubric benchmark.",https://github.com/Ayanami0730/deep_research_bench ; https://arxiv.org/abs/2601.08536,849,"Repository snapshot, not benchmark usage",LLM judges can change scores or share model-family biases. Independent human calibration remains necessary.,Direct,https://github.com/Ayanami0730/deep_research_bench
osworld,OSWorld,Computer-use benchmark family,https://osworld-v2.xlang.ai/,Computer agents execute workflows in real application environments.,Verified repairs earlier tasks; 2.0 extends long-horizon evaluation and distinguishes binary and partial reward.,"PALO business usability also depends on completing import, review and export, beyond the quality of extracted text.","Test complete user journeys with environment resets, final-artifact checks, failure traces and explicit partial completion.",Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,OSWorld-Verified and OSWorld 2.0 remain separate; steps and repetitions must be recorded.,"Featured in Claude Sonnet launch evaluations, with a public project and subsequent 2.0 leaderboard.",https://www.anthropic.com/news/claude-sonnet-4-5 ; https://osworld-v2.xlang.ai/ ; https://xlang.ai/blog/osworld-verified,3258,"Repository snapshot, not benchmark usage",A GUI success score depends on action budget and environment; it is not a direct semantic-fidelity score.,Direct,https://github.com/xlang-ai/OSWorld
tau2,tau2-bench,Policy and tool-agent-user interaction,https://github.com/sierra-research/tau2-bench,Agents work with users and tools under explicit domain policies.,"Domain policies, environments, agent/user simulators and repeatable execution are provided.",PALO extracts policies into a specification; this suite tests following policies during interactions.,"Link each source obligation to a testable state transition and measure repeated compliance, including exceptions and unauthorised actions.",Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,"tau2 is distinct from tau-bench; domain, simulator, policy revision and modality must be pinned.","Explicitly reported in Claude Sonnet 4.5 launch methodology, with substantial repository participation.",https://www.anthropic.com/news/claude-sonnet-4-5 ; https://github.com/sierra-research/tau2-bench,2434,"Repository snapshot, not benchmark usage",User simulators and policy prompt amendments can change outcomes; they must be part of the tested system identity.,Direct,https://github.com/sierra-research/tau2-bench
bfcl,Berkeley Function Calling Leaderboard,Tool-use and agentic evaluation,https://gorilla.cs.berkeley.edu/leaderboard.html,"Function selection, arguments and multi-turn or agentic tool use.","V4 exposes category scores, model interfaces, costs, latency, model outputs and a pinned evaluator checkpoint.",PALO needs semantic business validation in addition to syntactically valid tool calls.,"Separate contract conformity from task correctness, publish adapter versions and category-level errors, and distinguish native tool APIs from text prompting.",Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,BFCL V4; do not mix scores from earlier category compositions or native/prompt interfaces.,"Public Berkeley leaderboard with sortable metrics, comparison visualisation and a widely followed parent repository.",https://gorilla.cs.berkeley.edu/leaderboard.html ; https://github.com/ShishirPatil/gorilla,13069,"Whole Gorilla repository, not BFCL alone","Gorilla stars cover the parent project, not BFCL alone. Valid tool syntax is not proof of a correct business decision.",Direct,https://github.com/ShishirPatil/gorilla
gdpval,GDPval,Professional work-product evaluation,https://openai.com/index/gdpval/,Economically valuable work products across professional occupations.,Domain professionals create and review tasks and compare model deliverables with human work.,"PALO's proposed procurement study targets document-to-specification fidelity, a narrower unit than a finished professional deliverable.","Use actual procurement experts for blind rubric design and review, and measure repair effort alongside deliverable acceptance.",Primary source reviewed,False,2026-11-11,visibility-expansion-20261111,Published GDPval release and public subset; human grading and automated grading are distinct.,Dedicated OpenAI release compares several major providers on professional work products.,https://openai.com/index/gdpval/,,Not measured,The authors do not treat experimental automated grading as a replacement for expert assessment. Occupation coverage does not establish all enterprise use cases.,Direct,