# The Score Looks Precise. The Precision Is the Problem.

> AI assessment outputs a number. That number has consequences. Almost nobody has checked whether the number is right.

By Greg McCallum, founder of PeerLab. Published 2026-10-09 in AI Scoring & Assurance. Source: https://www.peerlab.ai/blog/what-a-two-band-disagreement-costs-the-person-being-scored/

A percentage looks like a fact. It lands in a spreadsheet, it sits inside a dashboard, it gets screenshotted into a calibration deck, and from that moment forward it does persuasive work that nobody voted for it to do. The problem with AI call scoring is not that the models are always wrong. Some of the time they are right. The problem is that the format of the output, a band, a tier, a digit, suppresses the scrutiny that would catch the times they are wrong. And the times they are wrong have names attached to them.

## What a two-band disagreement actually costs

Let's be specific. A rep scores 61% on a discovery call assessment. Your system bands that as Developing. The human marker who listens to the same call scores it 74%, which bands as Proficient. Two bands apart. In most frameworks that is the difference between a performance flag and a pass.

Here is what flows from the 61%:

- A coaching plan built around a deficit that may not exist. The manager pulls the call, focuses on the diagnosed weakness, and spends four weeks reinforcing something that was not actually broken.
- A promotion case that stalls. The rep's quarterly roll-up shows a Developing band. The panel defers. The rep, who does not know why, starts looking elsewhere.
- A performance flag that follows them into Q3. In some systems, two consecutive Developing scores triggers a formal PIP conversation. One of those scores was wrong.

None of this is hypothetical. It is the operational consequence of treating a disputed score as ground truth, which is exactly what happens when the output is a number rather than a considered opinion.

## Why numbers suppress scrutiny

If I tell you "I think this rep is developing but close to the boundary", you will probably ask what I saw and why I landed there. If a system tells you the rep scored 61%, you will probably write 61% in the box. The number feels like it has already been examined. It implies a method, a calibration, a standard. The work appears to have been done.

Written qualitative outputs get challenged. Numeric outputs get filed. This is not a hypothesis about human psychology, it is the daily reality of anyone who has sat in a calibration session where a manager says "well, the system gave them 58, so..." and the conversation ends.

The mechanism matters because it explains why errors propagate so much further in AI scoring than in human scoring. A human marker who calls a rep Developing when they are Proficient will probably get pushed back in the next calibration. The AI scores three hundred calls a month and nobody re-listens to the ones that look plausible.

## The validation gap nobody is advertising

Here is the question you should be asking your vendor, and almost certainly are not: what is your model's inter-rater agreement rate with expert human markers, measured on your actual use case, not a benchmark dataset?

Inter-rater agreement, specifically Cohen's Kappa or a weighted equivalent, tells you whether two independent scorers looking at the same material land in the same band at a rate better than chance. A Kappa above 0.6 is generally considered substantial agreement. Below 0.4 is marginal. Most vendors do not publish this number. Some do not calculate it. A few will tell you their model achieves 87% accuracy, which sounds reassuring until you ask what it was measured against and discover the answer is "agreement with the training labels", which is circular.

The precision of the display is often inversely related to the rigour of the validation behind it. A system that shows you a score to one decimal place has not necessarily earned the right to report integers. The decimal is a design choice, not a measurement claim. It just reads like one.

## What good looks like

If you are buying, operating or white-labelling an AI scoring system and acting on the output, here is the minimum you should be demanding before you trust a digit.

| What to demand | Why it matters | What a weak answer sounds like |
|---|---|---|
| Published inter-rater agreement (Kappa or equivalent) | Tells you how often the model and a qualified human marker agree at band level | "Our accuracy is 90%" with no reference to what it was tested against |
| Validation on your call type and ICP | Generic training data does not transfer reliably to your context | "The model was trained on over a million sales calls" |
| Defined band boundaries with behavioural anchors | Without these, the numeric output cannot be interpreted consistently | "Bands are defined by percentile within your cohort" |
| A human sign-off protocol for consequential scores | Any score that triggers a coaching plan, a flag or a promotion decision needs a second marker | "Our model doesn't need sign-off, that's the point" |
| Drift monitoring | Models degrade as conversation norms shift; you need evidence the vendor is tracking this | "We update the model regularly" |

That last column is what you will actually hear. I have heard most of those answers. They are not malicious; they are the result of building a scoring product for speed and not for consequence.

[Human vs AI Scoring Agreement Checker](https://www.peerlab.ai/library/tools/human-vs-ai-scoring-agreement-checker/) is a starting point if you want to run a quick comparison between your current AI output and a calibrated human assessment. It does not replace a full validation study, but it will tell you immediately whether the gap is worth investigating.

## The specific population this hurts most

It is not the obviously high performers and it is not the obviously struggling ones. The model tends to agree with humans at the extremes. It is the middle band, the 55 to 75% range, the Developing-to-Proficient borderline, where disagreement is most frequent and where the consequences of a wrong score are most serious. These are the people on the edge of a flag, the edge of a promotion, the edge of a PIP. They are also the people who most benefit from accurate diagnosis because they are actually coachable in a direction that exists.

If your system is scoring at scale and nobody has checked its agreement rate in that band specifically, you do not know what you are doing to those people. That is a strong sentence and I mean it plainly.

## What operators owe the people being scored

There is a governance question buried in here that does not get asked enough. When a company deploys AI scoring and acts on it, they are making consequential decisions about individuals based on a model's output. The rep does not know the model's Kappa score. They do not know that the system was trained on a call population that does not resemble their deals. They see a band, their manager sees a band, and decisions get made.

The minimum ethical floor is a human sign-off protocol for consequential scores. Not a human reviewing every call, that is not operationally realistic. But any score that will be used in a coaching plan, a performance review or a promotion conversation should have a qualified second marker before it is treated as fact. [Human vs AI Scoring Agreement Checker](https://www.peerlab.ai/library/tools/human-vs-ai-scoring-agreement-checker/) gives you a lightweight mechanism to build that into your workflow without adding two weeks to every cycle.

The vendors whose product scores people have a version of this obligation too. Shipping a precision display without a published validation is not a neutral act. It is a claim, dressed as a format.

The number on the screen is not more accurate because it has a decimal point. It is more persuasive. Those are not the same thing, and the difference belongs to the person being scored.