How we measure, and where we’re honest about the limits.

A hand-labelled set of 118 financial promotions: 71 containing genuine breaches (spread across all 10 rules, guarantees, missing warnings, unbalanced presentation, past-performance misuse, hidden costs, disguised promotions, targeting, s21, social/standalone, and a handful of prompt-injection attempts embedded in the promotion text), 34 compliant promotions that must pass (savings, ISAs, properly-warned investments, credit, insurance, pensions), and 13 deliberately thin inputs that should be refused. Each case is labelled with the rules that should fire and the rules that must not. Rule set applied: fca-cobs4-v1-2026-07.

The 4 missed expected-rule citations (of 114 across the set) are all secondary-rule completeness: on a promotion the tool had already correctly flagged as ISSUES_FOUND, it cited the specific breach but not every overlapping rule the label expected too. Zero breaching promotions were wrongly cleared to PASS — the worst failure mode did not occur in this run. Two assessment misses went the other way: a compliant promotion flagged as issues, and a compliant promotion refused for insufficient grounding. Both are the cautious direction of error; we still count them against ourselves rather than hide them behind the headline numbers.

A finding is only shown if it names a real rule — that’s checked in code for every finding, no exceptions. Where the finding is that something is present in the promotion, it must also quote text that is verbatim-present there; a quote that isn’t actually in your text gets dropped in code before you see it. Where the finding is that something is missing (an omission — a required risk warning that isn’t there, for example), there’s no text to quote by definition, so it’s marked as the model’s judgement instead of a verbatim-checked claim. An “issues” verdict left with no valid findings is corrected to a pass. The 100% grounding-integrity number above is specifically about the quotes that exist: every one of them is verbatim in the input — an enforced invariant, not a hope. It is not a claim that every finding carries a quote.

The evaluation harness ships with the product. npm run eval runs the labelled set through the live engine and regenerates the results file. We welcome an independent compliance professional labelling a held-out set, that is what turns this internal signal into an external benchmark, and we’ll publish the result whichever way it goes.