You Don't Need to Re-Mark Every Call. You Need to Re-Mark the Right Ones.

Volume is not validity. A team that runs ten thousand calls through an AI scorer and points to that number as proof the system works has confused throughput with accuracy. If the model has a systematic error, scoring ten thousand calls at scale does not correct the error. It industrialises it. The people being scored, and the managers making decisions on the back of those scores, deserve better than that. So does anyone who has to stand in front of a sceptical stakeholder and answer the question: "How do you know the score is right?"

The honest answer, for most teams right now, is that they do not know. They have volume. Volume is not an answer.

Why Full Re-Review Is Not the Solution Either

The instinct, once you accept that volume alone proves nothing, is to swing the other way: re-mark everything. Pull every call back through a human reviewer and compare. That sounds thorough. It is also impractical, statistically naive, and often counterproductive.

Human marking at scale introduces its own drift. Reviewer fatigue is real. Calibration degrades across a long session. If you send a reviewer through two hundred calls in a week, the marks they assign on call one hundred and fifty are not comparable to the marks on call twelve. You have traded one unreliable signal for another, slower one.

More importantly, you do not need full coverage to find systematic error. You need the right coverage.

What "Well-Constructed" Actually Means

A well-constructed sample for AI scorer audit is not random, and it is not cherry-picked. It is stratified, blinded, and sized to surface the failure mode you are most likely to have.

Stratified means the sample deliberately includes calls from across the score distribution, not just the edges. A common mistake is to pull only low-scoring calls for human review because those feel most at risk of error. They are also the calls where the human and AI are most likely to agree, because poor calls are easier to score. The interesting disagreement almost always lives in the middle band, where nuance matters and rubric language starts to strain.

A workable starting distribution for a team of, say, thirty reps running forty scored calls each per month:

Forty calls. That is it. Reviewed blind by two calibrated human markers working independently, without access to the AI score.

Blinded means the reviewers do not see the AI score before they mark. This is non-negotiable. If reviewers see the AI score first, they anchor to it. The exercise stops being an audit and becomes a ratification.

Sized to surface the failure mode means thinking before you sample about what you are actually testing for. Are you worried the model misreads objection handling? Over-rewards rapport talk? Misclassifies discovery questions as pitch? Build the sample to stress those moments specifically. Pull calls where those moments definitely occur, not just any calls.

What the Re-Mark Is Testing For

Human reviewers re-marking a blinded sample should be testing three things, in this order.

First, rubric fidelity: does the AI score reflect the written rubric criteria, or is it scoring something adjacent to what the rubric says? A common failure mode is an AI that scores "active listening" by measuring talk-to-listen ratio, when the rubric actually defines active listening as verbal acknowledgement and targeted follow-up questions. The numbers look correlated. The constructs are not the same.

Second, consistency across rep archetypes: does the model score a high-energy closer the same way it scores a quieter consultative rep, for the same underlying behaviour? AI scorers trained on majority-archetype call data often have a systematic bias here that is invisible until you look for it.

Third, boundary case accuracy: the calls where a skilled human reviewer would say "this is a marginal pass" or "this is a marginal fail." Those are the calls where the rubric is doing its real work, and they are the calls most likely to generate decisions, development plans, certifications, or performance conversations. Getting boundary cases wrong is where AI scoring causes the most damage.

What Disagreement Rate Tells You

Once you have the sample re-marked, you compare human and AI scores at the framework level, not the overall call level. An overall score match can mask significant disagreement underneath.

A one-band disagreement (AI scores a competency at 3, human scores it at 2 or 4 on a 5-point scale) is expected and acceptable up to a point. Raters disagree. The question is the rate. Above roughly 15% one-band disagreement on a well-calibrated sample, you have a rubric problem, a training data problem, or both.

A two-band disagreement (AI scores a 4, human scores a 2, or vice versa) on any call in your sample is a serious signal. Two-band disagreements do not usually indicate random noise. They indicate the AI and the human are measuring different things. If you see more than two or three of these in a forty-call sample, the validity of that competency dimension needs to be suspended until you understand why.

The Human vs AI Scoring Agreement Checker is built specifically to run this calculation across your sample without requiring a statistician to interpret it. It surfaces disagreement rates by band and by competency dimension, which is where the useful signal lives.

Building This Into a Rhythm, Not a Project

An audit run once, after a vendor challenge or a complaint, is better than nothing. An audit run quarterly as standard practice is what actually gives you defensible scoring.

The cadence I recommend to teams running AI assessment at any meaningful scale: forty-call stratified sample, blind dual-mark, disagreement analysis at competency level, quarterly. Document the results. If a competency shows persistent elevated disagreement, revisit the rubric language before you revisit the model. In my experience, the rubric is wrong more often than the model is.

If you are certifying people using scored judgements, the audit trail matters beyond accuracy. It matters for the conversation you will eventually have with HR, with a line manager, or with a rep who disputes a score. "We ran a forty-call blinded sample last quarter, saw 9% one-band disagreement and zero two-band disagreements on this competency" is a defensible answer. "We scored ten thousand calls" is not.

Rigour is not a function of volume. It is a function of whether you have constructed the right test and looked honestly at what it tells you. Most teams have not done that yet. Most teams could do it in a week.