Accuracy & methodology
How we measure, and where we’re honest about the limits.
Most AI compliance tools publish no accuracy number at all. We publish ours, the method behind it, and the failure modes. It’s a first-pass aid your team reviews, not a replacement for a qualified reviewer, and we don’t pretend otherwise.
94%
Breach recall
of expected breach-rule findings caught (94 of 100)
0%
Over-flag rate
false-alarm rule flags on compliant copy (0 of 82 must-not-flag checks)
96%
Assessment accuracy
correct pass / issues / refusal (96/100)
100%
Grounding integrity
every quote is verbatim in the input (306/306)
94% breach recall, with a 0% false-alarm rate. It caught the breaching promotions in all but one case (a borderline costs teaser where charges were disclosed but the headline was arguably misleading), and no must-not-flag rule fired on the compliant set. Where it errs, it errs cautiously, refusing or flagging rather than clearing. We report the miss rather than hide it.
The labelled set
A hand-labelled set of 100 financial promotions: 62 containing genuine breaches (spread across all 10 rules, guarantees, missing warnings, unbalanced presentation, past-performance misuse, hidden costs, disguised promotions, targeting, s21, social/standalone), 27 compliant promotions that must pass (savings, ISAs, properly-warned investments, credit, insurance, pensions), and 11 deliberately thin inputs that should be refused. Each case is labelled with the rules that should fire and the rules that must not. Rule set applied:
fca-cobs4-v1-2026-07.What we measure, and why
- Missed-breach rate (recall), the metric a compliance buyer cares about most: a breach the tool failed to flag.
- Over-flag rate, false positives on compliant promotions; over-flagging is what makes teams distrust a tool.
- Grounding integrity, whether every quoted excerpt is genuinely present in the promotion. Enforced in code, so it should be 100%.
- Wrongly-cleared breaches, the worst failure mode: a real breach returned as a pass.
Reading the 6% honestly
The remaining ~6% of “missed” expected rules is mostly secondary-rule completeness: the tool cited the specific breach (e.g. a misleading guarantee) but not also an overlapping general rule, on promotions it had already flagged. Only one breaching promotion was wrongly cleared (a borderline costs teaser); we count that against ourselves rather than hide it behind the recall figure.
How grounding is enforced (not just prompted)
A finding is only shown if it names a real rule and quotes text that is verbatim-present in the promotion. Anything that fails either test is dropped in code before you see it, and an “issues” verdict left with no valid findings is corrected to a pass. That is why grounding integrity is 100%, it’s an enforced invariant, not a hope.
Reproduce it
The evaluation harness ships with the product.
npm run eval runs the labelled set through the live engine and regenerates the results file. We welcome an independent compliance professional labelling a held-out set, that is what turns this internal signal into an external benchmark, and we’ll publish the result whichever way it goes.Limitations, stated plainly
- • This is a self-constructed set (n=100), not an independent benchmark. Small n; may not generalise to your product lines.
- • PromoCheck is a first-pass aid, human-reviewed. It does not replace an s21 approver’s sign-off, and it will miss some breaches, never rely on it to “catch everything.”
- • It reviews pre-publication copy. It does not do post-publication monitoring of live pages or social.
- • Numbers are measured on the currently-configured model; a different provider will move them.
See it on your own promotions.
The honest way to judge it is on your real copy, with the evidence pack to review.
Open the checker →