# What Your AI Scorer Does When the Rubric Has a Gap

> Every rubric has undefined territory. The question is whether your AI scorer flags it or fills it in quietly.

By Greg McCallum, founder of PeerLab. Published 2026-10-23 in AI Scoring & Assurance. Source: https://www.peerlab.ai/blog/what-your-ai-scorer-does-when-the-rubric-has-a-gap/

Most teams running AI scoring at scale have checked for hallucination. Fewer have checked for something quieter and, in practice, more damaging: the model filling a rubric gap with a confident, plausible-sounding score that was never grounded in anything you wrote. The output looks normal. The number lands in the spreadsheet. Nobody flags it. That is the problem.

This post is about that specific failure mode, how it happens mechanically, what it costs when you miss it, and how to find it before it compounds.

## What a rubric gap actually is

A rubric is a set of defined criteria: behaviours you want to see, anchored to score levels, with enough specificity that two human raters would agree more than they disagree. A gap is any input the model receives that the rubric does not cover. Not a grey area within a criterion, a situation the criterion simply does not address.

Gaps are inevitable. You write the rubric for the calls you imagined. Reps do the calls that actually happen. Common gap types in sales assessment:

- **Behaviour present, criterion absent.** The rep does something notable (a creative reframe, an aggressive price anchor, a disclosure that changes the dynamic) and none of your criteria speak to it.
- **Criterion present, context absent.** Your rubric scores "objection handling" but the call had no objections. The criterion exists; the trigger condition does not.
- **Ambiguous polarity.** The rubric says "asks open questions" but does not specify whether a closed question used strategically to confirm understanding counts for or against the rep.
- **Scope mismatch.** The rubric was written for discovery calls. Someone ran it over a renewal call, a legal qualification call, or a cold voicemail.

In a human QA process, a trained rater would note the gap, possibly mark N/A, and escalate. They would not invent a score. An LLM-based judge will almost always invent a score, because that is what language models do: they complete the sequence.

## The mechanics of silent interpolation

When an LLM receives a prompt asking it to score behaviour X against criterion Y, and the call contains no clear instance of X, the model does not return null. It searches for the nearest plausible proxy, scores the proxy as if it were X, and presents the result with the same formatting and confidence language as a grounded score. This is not a bug in the conventional sense. It is the model doing exactly what it was trained to do: produce coherent, contextually appropriate output. The failure is architectural. You gave it a rubric but you did not give it an abstention protocol.

The score produced by gap-filling tends to have three properties that make it hard to catch in normal review:

1. It is numerically plausible. Not 0 or 5 on a 1-5 scale. Typically 2, 3, or 4, the range where human scores cluster.
2. The rationale is coherent. The model writes something that sounds like a real observation. "Rep demonstrated partial awareness of customer concern" is not meaningless; it just refers to something the rubric never asked about.
3. It is consistent across similar calls. If the model interpolates the same way every time it hits the same gap, your inter-rater reliability numbers look healthy. You have consistent scoring of a fictitious dimension.

That third point deserves attention. Consistency is normally a sign of a working system. In this case, high consistency across a gap is evidence that the model has settled on a stable interpolation strategy, not that the rubric is functioning correctly.

## What it costs

The direct cost is misevaluation: reps getting scored on dimensions the rubric never defined, which means coaching conversations referencing evidence that was constructed rather than observed. A rep coached to "improve their use of open questions" based on a gap-filled score may change behaviour that was never actually measured.

The indirect cost is calibration drift. If gap-filled scores go undetected, they enter your benchmark data. Future human reviewers anchor to them. Rubric revision decisions get made on the basis of them. Within two or three review cycles, the original rubric intent can be substantially diluted without anyone having deliberately changed it.

There is also a fairness exposure. If the model's interpolation strategy correlates with anything other than the rubric criteria, which it may, because the training data underlying the LLM is not neutral, then you are systematically scoring some reps on a different basis than others. That is not a theoretical risk. It is the kind of thing that surfaces in an employment tribunal.

## How to find the gaps before they compound

You do not need a full audit. A targeted sampling protocol can expose most structural gaps in a working day.

**Step 1: Pull a stratified call sample.** Take 20-30 calls from the last 30 days that span different deal stages, rep tenures, and call outcomes. Include some calls that went unusually, calls with no objections, calls that ended early, calls involving pricing discussions the rubric may not have anticipated.

**Step 2: Run the scorer and capture full rationale.** Do not just look at scores. Pull the model's written justification for every criterion on every call. This is where gap-filling shows up: rationale that references behaviour not in the criterion definition, or that explains a score without citing observable evidence.

**Step 3: Flag rationale that does not map to the rubric text.** For each criterion, take the rationale and ask: does this cite a behaviour that my rubric criterion actually specifies? If the rationale mentions something the criterion does not define, you have found a gap. Log the gap type from the taxonomy above.

**Step 4: Compare flagged scores to human re-scores.** Have a calibrated human rater score the same calls blind, on the same criteria. Where the human returns N/A or scores differently with a gap-based explanation, that is confirmation. The [Human vs AI Scoring Agreement Checker](https://www.peerlab.ai/library/tools/human-vs-ai-scoring-agreement-checker/) is useful here: it structures the comparison and surfaces the criteria where agreement breaks down systematically.

**Step 5: Revise, don't just annotate.** The fix is rubric surgery, not a note in the margin. Each gap type requires a different intervention:

| Gap type | Fix |
|---|---|
| Behaviour present, criterion absent | Add the criterion or explicitly exclude the behaviour |
| Criterion present, context absent | Add a trigger condition: "score only if objection is raised" |
| Ambiguous polarity | Add polarity guidance with an example of positive and negative instances |
| Scope mismatch | Separate rubrics by call type, full stop |

## The abstention question

The deeper fix, and the one most vendor products currently do not offer, is building an abstention protocol into the scoring prompt. The model should be instructed to return a specific output, "insufficient evidence to score," when the call does not contain a clear instance of the behaviour the criterion defines. This requires you to define, per criterion, what constitutes sufficient evidence. That is hard work. It is also exactly the work that separates a rubric from a vague wish-list.

If your current AI scoring vendor cannot show you what their system does when a criterion's trigger condition is absent, that is a diagnostic in itself. The answer should not be "it scores it anyway."

[Human vs AI Scoring Agreement Checker](https://www.peerlab.ai/library/tools/human-vs-ai-scoring-agreement-checker/)

Rubrics are not self-maintaining. The model will not tell you when it has run out of road. It will keep scoring, confidently, into terrain you never mapped.