Balancing the Equation: Why HITL Is a Variable, Not a Constant

A common thread running through my recent work is the concept of the expanded validation boundary for probabilistic models. When reproducibility is bounded, not absolute, and the validated state is a moving target, validating the probabilistic model in isolation is insufficient. Instead, we validate the system as a whole, and monitor its behavior over time, providing assured evidence that it is operating within its pre-specified validated context. Runtime drift outside of this boundary triggers a change control and re-validation event.

The validated system boundary includes the context-of-use, the model (and its inputs and outputs), the harness, and the human-in-the-loop (HITL). This piece explores the concept of the HITL as a variable, not a constant.

The Kenya Study's Hidden Lesson

When it comes to Large Language Models (LLMs), and other probabilistic models, context is king. That’s the lesson behind the recent stress-test of a fine-tuned clinical LLM that demonstrated very different results than the benchmarks had originally suggested when implemented into a real-world workflow.

The promise of LLMs in regulated science, including clinical workflows and GxP settings, is increased efficiency, along with potential time and cost savings. Structured benchmarks against human evaluators support this hypothesis: several large language models (LLMs) have matched or outperformed both structured dataset comparisons and standardized medical examinations, including the USMLE. For instance, a study by Mwaniki et al. (2025) in Nature found that on average, clinician mean ratings were lower than the LLM’s mean ratings (five models were benchmarked, including GPT-4.1, Gemini-2.5-Flash, DeepSeek-R1, MedGemma, and o3) when both groups were asked to rate a randomly sampled set of patient case vignettes.

But when these same LLMs enter the real world, the performance doesn’t quite hold up as expected. A recent Nature study from June 2026 (Agweyu et. al) found that when clinical medical professionals in Kenya utilizing the electronic medical record were separated into two subgroups, one with LLM assistance and one without, and evaluated on the expert-judged composite of treatment failure events occurring within 14 days of enrollment, the overall ratio of patients experiencing treatment failure did not differ significantly between the subgroups.

While directional accuracy can be confirmed, the study did include several caveats: the experts could choose whether or not to accept the LLM’s suggestion, and the study was designed around a larger effect size than the one in the paper.

This discrepancy raises the question of where and how LLMs can provide a significant value-add in real-world clinical workflows, and what factor(s) explain the discrepancy seen between LLMs’ performance on structured workflows and real-world scenarios. In particular, it draws attention to the human variable. The study did not control for the model-human interaction.

The Unmeasured Variable

Every human whose judgment carries regulatory weight in a pharmaceutical operation is qualified before they're trusted with it: except one.

An aseptic operator qualifies through gowning qualification and media fills, with acceptance criteria and genuine disqualification on failure. A visual inspection operator is qualified against seeded defect kits and must detect known defects at a specified rate before they're permitted to inspect parenterals. A QC analyst undergoes proficiency testing. In each case the logic is the same: the person is a component of the process, human performance is a variable, and a variable that carries risk gets qualified, monitored, and requalified on a cycle.

The AI reviewer is the exception. Under draft Annex 22, documented human oversight at each decision point is the condition on which generative AI is permitted in non-critical functions at all. That reviewer's judgment is the load-bearing control: and it is the only human in the plant carrying regulatory load without a qualification regime for the specific competence the role now demands.

HITL and Model Performance

The reason this competence resists the existing toolkit is that reviewer failure is not independent of model failure. And worse, these failures are correlated with the model's, because what the model is good at (fluent, confident output) is exactly what switches human scrutiny off. Defense-in-depth assumes independence. Two barriers each catching most errors compound toward near-certainty only if their failures are uncorrelated. The human-in-the-loop is often placed in the architecture on exactly this assumption: the model may err, and the reviewer is the second, independent line that catches it.

But the mechanisms by which a probabilistic model produces a wrong output — fluency, confidence, internal coherence — are the same mechanisms that suppresses the reviewer's scrutiny of it. A confident, well-formed, plausible answer is precisely the one a reviewer is least likely to challenge. So the reviewer does not fail independently of the model; they may fail in step with it. Several cognitive biases well-established in literature, including automation bias and the fluency heuristic, are relevant here. It is cognitively taxing to identify errors in fluently-written text, and we are biased towards agreeing with computerized output.

Staying critical when evaluating fluent output is the specific thing that has to be qualified, and qualifying it is harder than it looks. It degrades exactly when it's needed most, and it degrades more as the model improves: a better model produces more fluent, more confident output and a lower base rate of real errors, which shifts the reviewer toward acceptance. The control weakens as the model’s performance improves.

The Measurement Problem

The measurement problem: the quantity that matters is detection conditional on the model being confidently wrong, and it cannot be estimated retrospectively, because retrospective sampling only surfaces errors that were caught. The censoring is on the exact variable of interest. Georgi Tancev (2026) describes two forms of failure modes in his paper Governing Large Language Models in Pharmaceutical Environments: A Risk Engineering Framework: remediable (the occurrence probability is reducible by engineering effort) and irreducible (the occurrence cannot be reduced to zero through known engineering approaches). He notes himself that “hallucination cannot be eliminated by better prompting, retrieval augmentation, or human review protocols that assume the model is probably correct.”

The measurement tool required here is a seeded challenge: known-bad artifacts injected live, ground truth known by construction, no human adjudicator in the loop. This is the only unbiased instrument. Within HITL, the validated operating envelope must consider that qualified human performance holds only under specified conditions: queue depth, session length, error base rate. Session length and queue configuration become qualified parameters, analogous to a biology assay’s validated range. The unified capture layer can mark a non-production artifact, exclude it from the GxP record, and retain the reviewer's disposition as qualification evidence.

Ultimately, you can only control what you can bind, measure, and capture. In GxP workflows, the human-in-the-loop is no exception.