PALO FRAMEWORKAI governance indicators are useful when they connect a defined objective or risk to evidence, an accountable reviewer and an action. A dashboard with many metrics is not a substitute for that connection.
PALO's KPI Generator suggests technical, business and ethical measures based on model type, objective and sector. The examples below explain how to turn a selection into a reviewable indicator register. They are illustrative definitions, not measured PALO deployment results or universal thresholds.
What distinguishes a KPI from a KRI?#
A key performance indicator tracks an intended outcome, such as the quality of reviewed answers. A key risk indicator signals exposure, such as consequential actions attempted outside a declared authority boundary. Whether a measure is useful depends on its scope, denominator, observation quality and the response it triggers.
Three worked indicator definitions#
| Indicator | Illustrative calculation | Evidence and owner | Review action |
|---|---|---|---|
| Supported-answer rate | Reviewed answers meeting the documented support rubric / all answers in the review sample | Versioned rubric, sampled answers and source checks; knowledge-service owner | Investigate failure categories and evaluate a revised system before expanding use |
| Authority-boundary attempt rate | Observed action requests outside the allowed scope / all observed action requests | Policy decisions with reason codes and a stated logging boundary; agent-service owner | Examine prompting, tool permissions, delegation and blocked-attempt patterns |
| Escalation completion rate | Escalations receiving a recorded accountable disposition within the agreed window / escalations due for review | Queue timestamps and decision records; operational review owner | Investigate overdue decisions, workload and whether escalation reaches a person able to act |
A blocked unauthorized request and a completed unauthorized action are different events. Track them separately. A low observed rate can also reflect missing logs rather than effective controls.
Numerical example with a stated denominator#
Suppose a reviewer examines 100 synthetic support answers using a documented rubric. If 86 satisfy every required support criterion, the supported-answer rate in that sample is 86%.
That figure describes the selected sample only. It does not establish the population rate, fairness across groups, safety of consequential actions or performance after a model change. Record how the sample was selected, whether the cases are representative, the review procedure and unresolved disagreements. If some records cannot be evaluated, report that missingness rather than silently dropping it from the denominator.
Choose thresholds from the decision context#
Set targets and escalation conditions with the accountable owner. Consider the consequence of failure, exposure, user needs, baseline evidence and the ability to detect or reverse an unwanted effect. Do not adopt a suggested numerical threshold solely because it appears in a template.
Pair efficiency measures with quality and risk measures. Lower handling time could accompany worse answers, reduced oversight or a larger review backlog. Investigate the trade-off before describing a change as an improvement.
Minimum fields for an indicator register#
- Name, purpose and relevant decision or control.
- Formula, unit, numerator, denominator and exclusions.
- Data source, observation boundary and known missingness.
- System version, population or sample, and measurement period.
- Owner, review cadence and agreed escalation action.
- Supporting evidence, limitations and next review date.
Use the versioned PALO KPI/KRI registry for the framework's indicator definitions. A project-specific register should preserve the relationship between a selected measure and the evidence that supports it.
Keep measurement connected to review#
After generating indicators, carry them into the Assessment Path. Review them alongside the use-case context and controls, rather than treating the export as an independent assurance conclusion.
The invoice-agent worked example shows how an expected effect becomes a verification question. The board review template helps record the accountable decision and follow-up actions.