A frequency tabulation of outcome labels over the platform assessment corpus (n=15,878 atomic behavioural observations, as of 2026-08-19) returns three labels that are not mastery grades. Absence accounts for 1,949 observations, 12.3% of the corpus. Not applicable accounts for 1,555 observations, 9.8%. Insufficient signal accounts for 708 observations, 4.5%. The remaining observations carry one of four ordered mastery grades: foundation 3,984 (25.1%), proficient 3,981 (25.1%), advanced 2,383 (15.0%), expert 1,318 (8.3%). The three non-grade labels correspond to different objects of measurement. Absence is a statement about a person's behaviour on an occasion where the competency was in frame. Non-applicability is a statement about the occasion rather than about the person. Insufficient signal is a statement about the evidentiary record. Only the third is abstention in the sense used in the selective-classification literature, and it is the smallest of the three. The evidence available for this report does not state how any of the labels were defined, who or what assigned them, or what decision rule separated them, and no agreement statistics are reported.
Method
The measurement is a single cross-sectional tabulation. Counts for seven mutually exclusive outcome labels were read from the assessment corpus at one as-of date, 2026-08-19, against a corpus total of 15,878 observations. Every share reported below is a label count divided by that same total at that same date.
The unit of analysis is the observation, defined in the evidence as an atomic behavioural observation. It is not the person, the competency, or the conversation. No person-level or organisation-level aggregation is reported, so nothing here describes how non-grade outcomes distribute across individuals or across cohorts of organisations.
Several method facts are absent from the evidence and are therefore not asserted. The definitions attached to each of the seven labels are not reported. Whether labels were assigned by human assessors, by a model, or by a mixed procedure is not reported. Whether the label set was fixed before the corpus was collected is not reported. No inter-rater or inter-run agreement figure is reported, and no confidence score, uncertainty score or threshold is reported for any observation, which bounds what can be said about the abstention behaviour of the instrument. No time series is available, so no claim about drift or about change following any change of procedure is made. No per-competency or per-framework breakdown of the outcome distribution is reported.
Subtotals across labels are not reported in the evidence and are not computed here. Statements about the effect of choosing one denominator rather than another are therefore directional and rest on the label counts as given.
Corpus
The assessment corpus contains 15,878 atomic behavioural observations (n=15,878, as of 2026-08-19). All outcome shares in this report are computed against that total, and no figure in this report is drawn from any other denominator.
The evidence carries further counts, each measured at its own sample size and none sharing the observation denominator: commercial conversations assessed, competencies in the ontology, frameworks represented, behavioural primitives held in a distinct cross-tenant licensable store, and the subset of those primitives cleared as licensable. Those counts are not restated here and are not combined with the outcome distribution, because doing so would let a figure measured on one population borrow the authority of a figure measured on another. Their existence still bears on interpretation in a way that requires no arithmetic.
No mapping between conversation counts and observation counts appears in the evidence. No observations-per-conversation ratio is therefore computed, and no statement is made about how observations are distributed within conversations or about how many competencies were placed in frame by any single conversation.
No crosswalk between competencies and frameworks is reported, so nothing is said about how many frameworks a given competency belongs to or about how framework membership relates to the outcome distribution.
The observations discussed here belong to the platform assessment corpus and are not held in the cross-tenant licensable store. The relationship between behavioural primitives and observations is not reported, and the licensable store plays no part in the distribution below.
Results
| Outcome label | Observations | Share of corpus |
|---|---|---|
| Foundation | 3,984 | 25.1% |
| Proficient | 3,981 | 25.1% |
| Advanced | 2,383 | 15.0% |
| Expert | 1,318 | 8.3% |
| Absence | 1,949 | 12.3% |
| Not applicable | 1,555 | 9.8% |
| Insufficient signal | 708 | 4.5% |
| Corpus | 15,878 |
Within the graded labels the counts decline from the second level upward. Foundation (3,984) and proficient (3,981) are close enough that no ordering between them can be defended, since no agreement figure is reported and the labelling error rate is unknown. Advanced (2,383) sits below both, and expert (1,318) below that. The distribution of graded observations is therefore concentrated in the lower half of the ordered scale, with the two lowest levels together larger than the two highest.
Among the three non-grade labels, absence (1,949) is the largest, not applicable (1,555) the second, insufficient signal (708) the smallest. Absence alone exceeds expert (1,318). Absence and not applicable each exceed insufficient signal.
The choice of denominator changes the reportable coverage of the instrument, and the three labels push in different directions. Treating all three as failures to produce an assessment removes the largest number of observations from the numerator and yields the lowest coverage figure. Treating only insufficient signal (708) as a failure yields the highest, since it is the smallest of the three. Removing not applicable (1,555) from the denominator on the ground that those observations were never eligible for a grade raises coverage further without changing the count of unresolved records. Because no subtotals are reported in the evidence, the specific values of these alternatives are not stated; the ordering follows from the counts.
Category structure of the non-grade outcomes
The three labels differ in what is being asserted and in what would falsify the assertion.
Absence asserts something about the person's behaviour on an occasion where the competency was in frame: the behaviour was available to be exhibited and was not exhibited. Under that reading it is a substantive measurement result rather than a missing value, and it belongs at or below the floor of the ordered scale rather than outside it. The corpus places it outside the scale, and the evidence does not report the reasoning for that placement. The consequence is arithmetic: 1,949 observations that may carry information about mastery are excluded from any statistic computed over graded observations only. Whether those observations should re-enter such statistics depends on a definition that is not published.
Not applicable asserts something about the occasion rather than about the person. The competency was not in frame, so no behaviour was owed and none can be inferred from its absence. Structurally this is the assessment analogue of a survey skip pattern, in which an item is not administered because the respondent's path through the instrument makes it irrelevant. Item non-response, by contrast, occurs when an administered item goes unanswered; the missing-data literature treats the two as different problems with different remedies, and imputation methods such as chained equations and the Amelia family are aimed at the latter, as documented in the bibliographic record for one recent treatment of item non-response. Non-applicability at 1,555 observations, 9.8% of the corpus, is a denominator fact. It records that the frame and the ontology were mismatched for those observations, which is a property of the pairing of conversation and competency.
Insufficient signal asserts something about the record: the evidence did not license a grade. This is the only one of the three that corresponds to abstention. Its magnitude, 708 observations or 4.5% of the corpus, is the quantity a selective-classification analysis would treat as the complement of coverage. Pooling it with the other two reports as abstention a quantity substantially larger than the record-quality problem actually observed, since absence (1,949) and non-applicability (1,555) each exceed insufficient signal (708) on their own.
A caution from the missing-data literature applies directly to the pooled treatment. Guidance on longitudinal non-response lists the creation of an extra "missing" category for a categorical variable as a practice that introduces bias, and lists dropping all incomplete records as merely inefficient, in a summary of standard approaches to attrition and item non-response. The same source notes that weighting for attrition is inefficient because it can use only complete data for each combination of variables analysed. A single non-grade bucket spanning the three labels here would be such an extra category, carrying three incompatible meanings under one name.
Related work
Selective classification supplies the formal vocabulary for the insufficient-signal case. Coverage is defined as the proportion of instances on which a model makes a prediction rather than rejecting it, and selective risk at a given coverage is the standard working-point metric; a recent critique of evaluation practice sets out both definitions and proposes the area under the generalised risk-coverage curve as an alternative summary interpretable as the average risk of undetected failures (Overcoming Common Flaws in the Evaluation of Selective Classification). That work reports an empirical benchmark spanning 6 datasets. It compares 13 confidence scoring functions, and reports that the proposed metric substantially changes metric rankings on 5 of the 6 datasets. The optimal-strategy result for reject-option classifiers shows that cost-based, bounded-improvement and bounded-coverage formulations lead to the same optimum, a Bayes classifier combined with a randomised Bayes selection function, and traces the cost-based formulation to Chow's 1970 model in which the classifier abstains when conditional expected risk exceeds a reject cost (Optimal strategies for reject option classifiers). The same line of work introduces a proper uncertainty score, a scalar summary of prediction uncertainty sufficient to construct the selection function, and gives two learning algorithms for it with Fisher consistency proofs (expanded version). A hypothesis-testing treatment characterises the optimal rejection rule as a likelihood-ratio test and implements abstention by thresholding a real-valued confidence score (Know When to Abstain).
Two features of that literature bear on the present corpus. Every construction cited above presumes a scalar uncertainty or confidence quantity on which a threshold operates. No such quantity is reported for the 708 insufficient-signal observations, so no coverage-risk curve can be drawn and no claim can be made about whether the abstention rate sits at an optimum for any cost. Abstention is also not distributionally neutral: selective classification has been shown to magnify existing accuracy disparities between groups, with increasing abstention decreasing accuracy for some groups, and the same work proves that for symmetric margin distributions the direction of the effect is determined by full-coverage accuracy and log-concavity of the margin distribution, and that models trained to equalise full-coverage accuracy across groups permit uniform improvement (Selective Classification Can Magnify Disparities Across Groups). That paper reports the disparity-magnifying behaviour consistently on five vision and NLP datasets. Because the evidence here reports no group-level breakdown of insufficient-signal observations, the risk cannot be evaluated in either direction.
Applied assessment schemes show both the conflation and the separation. A published document-assessment specification defines three statuses, covered, partial and gap, with gap applying when no relevant text is found in the document after thorough review, and requires a numeric confidence value on every assessment along with evidence quotes, severity and reasoning (Control Assessment skill specification). That design merges the distinction at issue here: a gap status covers the case where the control is genuinely absent and the case where the document does not speak to it, with the confidence field carrying whatever separation exists. The same specification distinguishes strong evidence, characterised by specific procedures, named roles, quantified timelines and technical specifications, from weak evidence such as aspirational language and undefined terms, which is a record-quality criterion operating alongside the status label rather than inside it.
A national curriculum assessment framework instead judges performance at the construct level using analytic scoring rubrics with the descriptors Exceptional, Outstanding, Satisfactory, Basic and Elementary, mapping to final subject grades A to E (NCDC A-Level assessment guidelines summary). That framework covers 29 subjects, each with subject-specific guidelines. Every descriptor listed there is a performance level, and the documented scheme carries no outcome for an unassessable construct, which is defensible when the assessment occasion is constructed rather than observed.
Interpretation
Transcript-based competency assessment differs from both of those cases in that the assessment occasion is not constructed. A conversation happens for its own reasons, and the competencies it puts in frame are a consequence of the conversation rather than of the assessment design. That is the structural source of the 1,555 non-applicable observations, and it means non-applicability rates carry information about the fit between the ontology and the observed occasions rather than about the people assessed.
The three-way separation matters for any statistic built on top of the corpus. A per-person mastery profile computed over graded observations only would discard 1,949 absence observations that plausibly carry negative evidence about mastery, and would treat unresolved records and out-of-frame competencies identically to observations that never occurred. A profile computed by scoring all non-grade outcomes at the floor would penalise people for the frame of their conversations and for the quality of the records. The two errors run in opposite directions and their magnitudes are unequal: absence is 12.3% of the corpus, non-applicability 9.8%, insufficient signal 4.5%.
For the instrument itself, the reportable abstention rate is 4.5% of observations at the corpus level, with the caveats that no confidence quantity underlies it and no selective risk can be computed against it. Whether that rate is well calibrated cannot be determined from a frequency table. Establishing it would require ground truth on a sample of the 708 insufficient-signal observations and on a comparison sample of graded observations, which the evidence does not report.
For a career-level reading, the distinction between the labels determines what a mastery profile is evidence of. An absence label, if its published definition matches the reading given above, is evidence about what a person did on an occasion where the competency was in frame, and repeated absence at a given competency would be the observable trace of an unmastered competency. A non-applicable label is evidence about which competencies the person's work put in frame, which is closer to a description of role scope than of capability. An insufficient-signal label is evidence about the recording and processing pipeline. Reporting the three separately keeps those three readings separable for whoever consumes a profile; pooling them makes the profile uninterpretable in a specific way, because a low count of graded observations could then arise from behaviour, from role scope, or from record quality with no way to tell which.
Limitations
The tabulation is a single snapshot at one date and supports no statement about stability, drift, or the effect of any change in procedure. Label definitions are not reported, so the boundary between absence and insufficient signal, which is the boundary determining whether a non-grade outcome is evidence about a person or evidence about a record, cannot be inspected. The assignment procedure is not reported, and no agreement statistics appear in the evidence, so the reliability of that boundary is unknown. No confidence or uncertainty score is reported, which forecloses coverage-risk analysis of the kind the selective-classification literature treats as standard.
The corpus is not broken out by competency, by framework, by person, or by organisation. Without a per-competency breakdown, the 1,555 non-applicable observations cannot be traced to particular competencies that are systematically out of frame, which is the analysis that would turn a denominator fact into an ontology revision. Without a group-level breakdown, the disparity-magnification result cited above can be neither confirmed nor excluded for this instrument. No linkage between observations and assessed conversations is reported, so clustering within conversations cannot be modelled and no interval is reported for any share above. Subtotals across labels are not reported in the evidence, so coverage under alternative denominators is described only by direction.
The other counts present in the evidence sit at their own sample sizes and are not comparable to the observation counts without a stated mapping, and no such mapping is reported. Nothing here should be read as relating the size of the ontology, the number of conversations assessed, or the size of the licensable primitive store to the outcome distribution.
Further work
Publication of the operative definitions and the assignment procedure for the seven labels would remove the largest constraint, since it determines whether absence and insufficient signal are separable at all. Publication of any scalar confidence quantity attached to observations, with a coverage-risk curve computed against adjudicated labels on a sample, would make the abstention rate assessable against the standard metrics. A per-competency non-applicability rate would locate frame mismatch and identify candidates for ontology revision. Re-adjudication of a sample of insufficient-signal observations against longer or richer records would estimate what fraction of that label is a property of the record rather than of the instrument. A stated convention for whether absence enters the ordered scale at floor or remains outside it would fix the treatment of 1,949 observations that currently sit outside every mastery statistic. A reported linkage between observations and the conversations they were drawn from would permit interval estimates that account for clustering.
Sources
- Optimal strategies for reject option classifiers
- Optimal strategies for reject option classifiers (expanded version)
- Know When to Abstain: Optimal Selective Classification
- Overcoming Common Flaws in the Evaluation of Selective Classification
- Selective Classification Can Magnify Disparities Across Groups
- Predicting Missing Values in Survey Data Using Prompt Engineering for Addressing Item Non-Response (bibliographic record)
- Addressing Non-response and Missing Data in Longitudinal Surveys
- Control Assessment skill specification
- NCDC A-Level assessment guidelines summary