# The Criteria in Your AI Scoring Rubric That No Model Can Actually See

> Some of what your rubric asks an AI to score cannot be inferred from a transcript, and most teams have never mapped which criteria those are.

By Greg McCallum, founder of PeerLab. Published 2026-10-30 in AI Scoring & Assurance. Source: https://www.peerlab.ai/blog/ai-judge-rubric-criteria-no-model-can-observe/

Most AI scoring rubrics were written by people who had never thought carefully about what a transcript actually contains. They took a human call scorecard, fed it to a vendor, and the vendor said "yes, we can score that." The vendor was not lying exactly. The model does produce a number. What nobody checked is whether that number means what the rubric says it means, or whether the model quietly swapped in something it could see in place of something it could not.

This matters because rubric scores are being used to make decisions: who gets coaching, who passes certification, who is flagged for a performance review. If a meaningful slice of your rubric is being estimated by proxy, those decisions have an unknown margin of error baked into them. Unknown because nobody mapped the rubric before they shipped it.

## What a transcript can and cannot contain

A transcript is a record of words and, depending on your tooling, timing data. Audio adds prosody: pace, volume, some tonal cues. That is the complete set of raw material available to any model scoring a sales call.

From that material, a model can legitimately evaluate structure (did the rep open with X before Y), vocabulary (were specific phrases used), talk-to-listen ratio, whether certain questions were asked, response latency in aggregate, and rough sentiment in spoken language. Those are observable criteria. A well-calibrated rubric scores them and the numbers are defensible.

Everything else is inference from incomplete data, or it is a proxy substituted without disclosure.

## The ranked list: unobservable criteria by rubric prevalence

The following eight criteria appear regularly in AI-scored rubrics. They are ranked by how often I encounter them in rubric reviews, starting with the most common. For each one I have named what the model is most likely scoring instead, and what that substitution costs you.

**1. Active listening**

The most overrepresented criterion in any scorecard. Listening, by definition, produces silence and subsequent response. A transcript cannot show whether silence was attentive or distracted. What the model typically scores: whether the rep's next turn acknowledged vocabulary from the buyer's previous turn, plus talk-to-listen ratio. That is a linguistic echo check, not active listening. A rep who parrots keywords while mentally composing their next pitch scores well. A rep who listens carefully and responds with a reframe that shifts vocabulary scores less well.

**2. Deliberate use of silence**

This appears in discovery rubrics constantly. "Rep demonstrates comfort with silence to allow the buyer to expand." Silence is literally invisible in most transcripts and ambiguous in timing data. What the model scores: whether there is a measurable pause. It cannot distinguish a tactical pause from a rep fumbling for a thought. If your rubric rewards long pauses and your rep has a poor broadband connection, you have a confound nobody will notice.

**3. Rapport and relationship quality**

Rapport built over time, through shared context and physical presence, does not survive transcription in any useful form. What the model scores: warmth of language, use of the buyer's name, presence of small-talk preamble. Those are rapport-adjacent behaviours. They do not capture whether the buyer trusts the rep, whether the rep remembered something personal from last quarter, or whether the tone in the room was relaxed or strained. A model has no access to the room.

**4. Whether a concession was pre-planned or reactive**

Negotiation rubrics often include something like "rep concedes strategically rather than under pressure." A transcript shows that a concession happened. It cannot show what was in the rep's head when it happened, whether the concession was within a planned range, or whether the buyer's tone shifted in a way that triggered a capitulation the rep later rationalised. What the model scores: the timing and size of the concession relative to buyer pushback. That proxy might correlate with strategic concession behaviour sometimes. Sometimes it does not.

**5. Buyer affect versus buyer words**

Many rubrics ask: "did the rep respond appropriately to the buyer's emotional state?" The buyer's emotional state is partly conveyed by words, but significantly by tone, body language, and what is left unsaid. A buyer who says "that's fine" when it is not fine is giving the transcript one signal and the room a different one. What the model scores: sentiment in the buyer's language. If the buyer is politely dismissive, the model often reads it as neutral or positive. The rep who reads the room correctly and adjusts gets no credit.

**6. Credibility conveyed through presence and physicality**

Executive-level selling rubrics sometimes include criteria around gravitas or executive presence. These are partly conveyed by how someone sits, how they pause, how they own a room. Audio captures some of this via pace and volume. A transcript captures none of it. What the model scores: vocabulary sophistication, use of data, avoidance of filler phrases. A rep who speaks with genuine authority in person but uses simple language scores flat. A rep who uses impressive-sounding words at the wrong moment scores well.

**7. Pre-call preparation quality**

Some rubrics credit preparation: "rep demonstrates knowledge of the buyer's business." A rep who prepared extensively and wove that knowledge naturally into conversation may surface fewer explicit research references than a rep who name-drops the buyer's latest press release and moves on. What the model scores: presence of specific named references to the buyer's company, role, or industry. Depth of preparation is not detectable; surface display of preparation is.

**8. Whether objections were genuine or buying signals**

Late-stage rubrics often ask reps to distinguish real objections from test objections, which experienced buyers use to see how a rep responds. This distinction lives almost entirely in the buyer's intent, which is not in the transcript. What the model scores: whether the rep used a recognised objection-handling framework. That tells you about framework compliance, not whether the rep correctly read what the buyer was actually doing.

## What to do with this

The immediate step is to go through your rubric criterion by criterion and sort each one into two columns: observable from transcript or audio, and requires information not in the recording. If you cannot finish that exercise in an hour, the rubric is either too long or too vaguely written.

For the unobservable column, you have three options: remove the criterion, replace it with a proxy that you explicitly label as a proxy, or retain it and route it to human review. What you cannot responsibly do is leave it in the AI-scored column and report its output at the same confidence level as criteria the model can actually evaluate.

The [Human vs AI Scoring Agreement Checker](https://www.peerlab.ai/library/tools/human-vs-ai-scoring-agreement-checker/) is a useful starting point if you want to quantify where your model and your human reviewers are currently diverging. Divergence on observable criteria usually means a calibration problem. Divergence on unobservable criteria usually means the model has been scoring something else all along, and you are only now finding out.

A rubric that mixes observable and unobservable criteria without labelling which is which is not a measurement instrument. It is a blended opinion with a confidence interval nobody has calculated. The fact that it produces a number does not make it measurement. It makes it a number.