Advisory review records · Presentation 2026-11-11.6
Reviewer diagnostics
Inspect how advisory reviews were collected and where the process remains unresolved.
Fable was captured through an API, Gemini through a web interface, and GLM through a desktop app with an added instruction. The channels, prompts and attempts differ. These records cannot establish which reviewer is more accurate. A malformed capture has no semantic score; it is not evidence of incorrect reasoning.
Model-family overlap and exposed synthetic material prevent independent qualification. Historical observations are retained without repairing the captured response bytes.
Targeted review
Retained reviewer observations
Read the source, the output and each reviewer’s decision. Targeted model countertests leave the frozen draft labels and original test counts unchanged.
Internal review only. Cases were selected for known issues; these findings do not estimate general error rates or establish independent validation.
Loading retained countertest evidence.
Review design and limitations
The sample contains 18 verification probes, selected as two per original draft-label class in each case, plus 18 source-bound draft translations. It is deliberately targeted and cannot represent the full cohort.
Author model identities and prior labels were hidden in reviewer prompts. The reviewer families overlap the tested cohort. Reviewers can share errors, and model agreement is not independent human adjudication.
GLM, Gemini and the local Qwen model provide additional internal evidence. Missing or incomplete reviews stay visible. No PALO Blueprint Index or revised ranking is assigned from this exercise.
Internal method checks
Reviewer capture diagnostics
Six seeded alterations, six unchanged rules and six paraphrases. One authored source family, three procurement cases. This checks an elementary reviewer workflow, not general judging ability or independent accuracy.
- Freeze source and draft rubric
- Review without expected labels
- Check response structure
- Compare semantic judgments
- Export source-bound drafts
Fable 5.2 High
Capture channel: API
Envelope accepted
Retained judgments available; no comparative accuracy claim.
Historical capture counts and advisory text
Detected alterations: 6 / 6. False alarms: 0 / 12. Unresolved: 0 / 18.
The plan is sound as internal engineering hygiene and worth building, but it does not by itself support any qualified, independent or validated claim. It makes results traceable; it does not validate the proposition bank, the judge or the calibration set. The findings below block a qualified claim, not the implementation.
GLM 5.3 Max
Capture channel: ZCode desktop
Captured envelope invalid
Semantic assessment unavailable; capture failure is not a reasoning score.
Historical capture counts and advisory text
Detected alterations: N/A / 6. False alarms: N/A / 12. Unresolved: 18 / 18.
Raw advisory text is preserved on GitHub; no semantic score is assigned.
Gemini 4 Argon High
Capture channel: Gemini web
Captured envelope invalid
Semantic assessment unavailable; capture failure is not a reasoning score.
Historical capture counts and advisory text
Detected alterations: N/A / 6. False alarms: N/A / 12. Unresolved: 18 / 18.
Raw advisory text is preserved on GitHub; no semantic score is assigned.
Interpretation: Missing, invalid and uncertain answers remain assigned. A zero false-alarm count is not interpretable without complete control responses. Fable flags C10 as a potentially ambiguous paraphrase; the original draft label is preserved. These easy examples do not validate the judge or change the recorded model results. Channels, prompts and attempt histories differ; the counts cannot rank reviewers.
Post-hoc lexical baseline: exact source/candidate equality detects 6/6 alterations but raises 6/12 false alarms on faithful controls. This exposes the weakness of copy matching; it does not establish model generalisation. Gemini and GLM advisory text is preserved even when its captured JSON is invalid. Their copied interface responses are not provider API bytes; the source of the formatting defect cannot be isolated.
Review agreements, disagreements and implementation decisions ↗
Delivered checks and further development
- Versioned source-bound proposition bank and explicit draft rubric. Independent expert gold remains pending.
- Replayable review-to-export harness, strict identifiers and fixed denominators. No new Workbench document-ingestion or 100 MB test is claimed here.
- Structural failures and semantic outcomes are separate. A schema-valid judgment can still be wrong.
- Exposed sources remain ineligible for independent qualification. Local hashes establish byte identity, not external custody.
- Next study: subtle alterations, instruction-bearing candidates, order-shuffled repetitions, unanswerable items, new independent source families and blind human adjudication.
Review protocol
Use identical frozen packets, compatible capture contracts and predeclared retry rules for automated diagnostics. Independent domain reviewers must adjudicate semantic labels separately.
Case coverage and human adjudication planIndependent review requirements
These advisory reviewer captures remain separate from model results. The next qualification study requires human-adjudicated controls, independent custody and source-family uncertainty.
Acceptance gates and current status ↗
