Updated each quarter as marking volume grows
A score is only worth having if it isn’t one person’s opinion — and that applies to us before it applies to anyone we mark. Marking is done by a calibrated panel; one call in five is independently marked a second time by another assessor, working blind. This page reports how closely those pairs have agreed across all our marking to date. A firm that sells measurement should be willing to be measured.
Each double-marked call is scored by two assessors independently; the second never sees the first's marks, the AI's score, the source report, or the participant's real name. Two figures are reported, and both are published rather than the flattering one alone:
| Double-marked call | Assessor pair | Criteria | Exact | Within one band |
|---|---|---|---|---|
| Call 1 | two of three | 20 | 65% | 100% |
| Call 2 | two of three | 25 | 80% | 96% |
| Combined | panel of 3 | 45 | 73% | 98% |
Calls are identified by number and participants by code in all reliability reporting; assessors by an anonymous reference. Marks are 0–4 with an explicit not-applicable option. No client, programme or rubric is identified here, and none will be: client statements are issued privately and are the client’s to share.
Where a selected second-marking pass yields no comparable score — for example a
not-applicable pass, or a recording too compromised to score fairly — it is left out of the figures
above rather than counted as disagreement, and noted here, so that a data-quality finding is never
misreported as an assessor one.
To date: two further participants were selected for second marking, but the second assessor recorded not-applicable on every criterion, so those passes are excluded and reported here rather than counted as disagreement.
These figures are drawn from 45 double-marked criteria across two calls — a baseline, and a small one. The limitation we would rather state than have you find: in this sample the lead assessor was one half of every pair, so what is measured is the panel calibrating against a common reference point rather than fully independent inter-rater reliability. The figure is re-established as marking volume grows and more assessor pairings are drawn. Movement away from the baseline, in either direction, is the signal worth watching: a falling figure asks whether the rubric or the calibration needs attention; a rising one confirms the panel is converging.
