# AI Scoring Inter-Rater Reliability Audit: Consistency Without Stats > Before you trust what your AI scorer says about your reps, find out whether it agrees with itself. This guide gives you a step-by-step consistency audit you can run in a morning, without a data team or a statistics background. Free guide for sales managers. Topic: [AI Assessment, Scoring & Assurance](https://www.peerlab.ai/library/topic/ai-assessment-scoring-assurance/). Published by PeerLab in the [Sales Enablement Resource Library](https://www.peerlab.ai/library/). Source: https://www.peerlab.ai/library/ai-scoring-inter-rater-reliability-audit/ ## How to use it Block two to three hours, pull the call sample using the selection criteria in Step 1, and run the comparison using the variance table provided. When you find a failure mode, go straight to the fix section for that specific problem rather than reading the whole guide end to end. ## What's inside - A plain-English definition of inter-rater reliability and why it matters before any other AI scoring question - A call selection protocol for building a matched sample set across different reps and contexts - A variance threshold table with named bands (acceptable, investigate, halt) requiring no statistical knowledge - Instructions for separating genuine performance differences from model noise - Named failure modes: prompt drift, criteria overlap, and anchor creep, with definitions and diagnostic questions - A fix protocol for each of the three failure modes - A re-audit trigger list telling you exactly when to repeat the process - A one-page audit log template for recording findings and actions - A "when NOT to trust this audit" section with honest caveats ## Why This Comes First Most AI scoring audits ask: does the model agree with a human? That is the wrong first question. If the model is internally inconsistent, comparing it to human judgement is noise comparing with noise. Internal consistency means the model scores the same behaviour the same way regardless of who performed it, which call it appeared in, or when the call was scored. This property is called inter-rater reliability. You do not need a statistician to test it. You need a structured sample and a clear threshold for what counts as a problem. Run this audit before you use AI scores in coaching decisions, performance reviews, or any calibration process. --- ## Step 1: Build a Matched Sample Set The goal is a set of calls where the underlying behaviour is as similar as possible, so that score differences reflect the model, not the rep. **Target sample size:** 20-30 calls minimum. Fewer than 20 and any pattern you find is too easy to explain away. More than 30 and you are doing more work than this stage needs. **Selection criteria (apply all four):** | Criterion | What to control for | |---|---| | Call type | Same stage of your sales cycle (e.g. all discovery calls) | | Duration | Within a 5-minute band (e.g. 28-33 minutes) | | Deal size | Same tier or bracket | | Outcome | Mix of progressed and stalled, not all wins or all losses | **Across the sample, vary:** - At least 6 different reps - At least 3 different time periods (e.g. calls from three separate months) - Where possible, include two calls per rep to allow within-rep comparison Pull the calls before you look at any scores. Selection bias, choosing calls you already sense were scored inconsistently, will corrupt your findings. --- ## Step 2: Score Everything Fresh Before comparing anything, rescore all calls in the sample through the AI system from scratch. Do not use archived scores. Models update, prompts drift, and an archived score may have been generated under different conditions. Log the rescore date. You will need this for the re-audit trigger list in Step 6. --- ## Step 3: Compare Scores Without Statistics For each scoring criterion your system uses, record the scores across all calls in the sample. You are looking at spread, not average. **The variance table:** | Spread across matched calls | What it means | Action | |---|---|---| | 0-8 points on a 100-point scale | Acceptable. Normal sensitivity to genuine small differences. | No action needed. | | 9-15 points | Investigate. Could be real performance variance or model noise. | Dig into the specific criterion. See Step 4. | | 16+ points | Halt. The criterion is not scoring consistently enough to use. | Do not use this criterion in any decision until fixed. | If your system uses a different scale (e.g. 1-5 or 1-10), convert: acceptable is roughly 8% of the scale, investigate is 9-15%, halt is 16%+. **Do this for every scored criterion separately.** A system can be consistent on one criterion (e.g. "questioning technique") and completely unreliable on another (e.g. "rapport building") at the same time. An overall score that looks stable is often masking criterion-level chaos. --- ## Step 4: Separate Model Noise from Real Performance Differences When a criterion lands in the "investigate" band, you need to rule out the possibility that the spread reflects genuine rep-to-rep differences before you blame the model. Pull the two highest-scoring and two lowest-scoring calls on that criterion. Listen to the relevant segment of each call (typically 3-5 minutes). Ask: - Could a neutral observer clearly explain why Call A scored higher than Call B using the written criterion definition? - If you described the behaviour in Call A to the model in plain language, would it score it differently to Call B? If yes to both: the spread is real. The reps are performing differently and the model is picking it up. No action needed on the model. If you cannot explain the difference, or if two clearly similar behaviours produced a 12-point gap: you have model noise. Move to Step 5. --- ## Step 5: The Three Failure Modes and What To Do About Each ### Failure Mode 1: Prompt Drift **What it is:** The scoring prompt has been edited since the system was first configured, often by a vendor applying a platform update, or by a well-meaning admin who "clarified" the language. The model is now applying a subtly different definition to the same criterion than it was six months ago. Calls scored at different times therefore get different scores for the same behaviour. **Diagnostic question:** Are the inconsistencies concentrated in calls from different time periods rather than different reps? **Fix:** Retrieve the current live prompt for the affected criterion. Compare it to any previous version you have. If you do not have a previous version, treat the current prompt as suspect and rewrite it from your original criterion definition. Lock the prompt in writing and note the date. Rescore the full sample under the new prompt to confirm stability. ### Failure Mode 2: Criteria Overlap **What it is:** Two or more criteria in your scorecard are measuring the same underlying behaviour. The model splits its attention inconsistently between them, producing phantom variance on both. You often see this as one criterion scoring high and a related criterion scoring low on the same call, with no clear reason. **Diagnostic question:** Take any call in the "investigate" or "halt" band. Does the score on the problem criterion correlate oddly with a different criterion on the same call, either both high when you'd expect one low, or inversely when they should move together? **Fix:** Map every criterion definition against every other one. Highlight language that describes the same moment in a call. Merge overlapping criteria or rewrite them so each covers a distinct, non-overlapping behaviour. Less is more here. Eight clean criteria score more consistently than fourteen muddy ones. ### Failure Mode 3: Anchor Creep **What it is:** The example calls or benchmark descriptions used to calibrate the model's scoring have drifted away from actual current performance norms. The model has a mental benchmark of what a "7 out of 10" looks like, but that benchmark was set when your best rep was performing at a level your average rep now matches. Scores compress or spread in ways that do not reflect real differences. **Diagnostic question:** Are high performers and average performers receiving almost identical scores on criteria where you know, from listening, they are clearly different? **Fix:** Replace the calibration examples. Pull two or three calls that clearly represent the bottom, middle, and top of actual current performance on the criterion in question. Submit these to the vendor or update your configuration to use them as the new anchors. Rescore the sample and check that the spread now reflects what you hear when you listen. --- ## Step 6: Re-Audit Triggers Run this audit again when any of the following happen: - The vendor releases a platform update - Any scoring prompt is edited for any reason - A new criterion is added to the scorecard - A new rep cohort joins and is scored for the first time - More than 90 days pass with no audit The 90-day rule is not bureaucracy. Models behave differently as the call corpus grows. What was consistent in month one may not be consistent in month four. --- ## Audit Log Template | Date | Criterion audited | Sample size | Spread found | Band (accept/investigate/halt) | Failure mode identified | Action taken | Re-audit date set | |---|---|---|---|---|---|---|---| | | | | | | | | | Keep this log. If you ever need to defend a coaching or performance decision that was informed by AI scores, the log shows due diligence. --- ## When NOT to Trust This Audit - If your sample is smaller than 20 calls, your findings are directional at best. Do not act on "halt" findings from a 10-call sample. Expand the sample first. - If your organisation has only one or two reps, you cannot separate model noise from genuine performance differences with confidence. You need cross-rep variance to isolate the model. - If the AI system has been updated between the time you pulled calls and the time you rescored them, your comparison is invalid. Pull and score on the same day. - This audit tests consistency, not accuracy. A model can be perfectly consistent and consistently wrong. Run a human-vs-AI calibration separately once consistency is confirmed.