The Last Deterministic Thing: The Unified Capture Layer
The Unified Capture Layer as the Deterministic Instrument
Four sequential pieces in my blog series have explored the concept of the expanded validation boundary for probabilistic models: I dissected each component of the equation (context-of-use x model (inputs and outputs) x validation harness x human performance). In a fifth piece, I also explored five harness layers as the continuous monitoring surface (step 7 of the Probabilistic Validation Lifecycle).
For clarification, as the location of HITL/HOTL within the validation boundary has shifted throughout the series: the human performance sits inside the harness itself as one of the five monitored surfaces, while the HITL sits within the validation boundary in the layer surrounding the harness.
This piece explores the thread tying together each component of the validated system boundary: the unified capture layer. The UCL provides the substrate for continuous monitoring via recording runtime evidence post-deployment.
The “Deterministic” Term: Resolving the Ambiguity
It is critical to define what precisely is meant by the term “determinism.” A frozen model with locked weights may be deterministic at its own boundary, but the system containing it is deterministic only if the components within the harness around the model are also controlled.
Determinism is often assumed to refer to reproducibility of output within the scope of the model: same input into the model leads to the same output from the model, every time (static/frozen model). For the purposes of this piece, we will refer to this as Definition A. This is a correct definition of determinism. It must be clarified, however, that a model may produce identical outputs given identical inputs, but still employ stochastic methods to arrive at that output. Inversely, in some contexts, a model may employ no stochastic method and yet return a different output given the same input. The computational serving layer executes batches of concurrent requests, and the size of these requests determines the downstream pathway order - which is not mathematically associative.
Determinism can also refer to reproducibility of the control path (Definition B), or the ability to replay a recorded execution (Definition C). Definition B is checkable only through Definition C.
A deterministic model does not automatically imply a deterministic system. If the model is embedded in a harness with drifting or configurable components (i.e. retrieval, thresholds, action, dependencies) or an untraceable control path it may be deterministic under Definition A, but it is not necessarily deterministic under Definition B or Definition C. Similarly, the control path reproduces within a given configuration state. A configurable harness may change over time. This is what makes B checkable only through C.
A probabilistic model cannot be deterministic under Definition A. This is why probabilistic model validation requires bounded reproducibility testing. However, a probabilistic model may be part of a deterministic system under Definition B or Definition C. The harness carries the determinism under Definition B, and the replay of recorded execution via the UCL carries the determinism under Definition C.
The Evidence Capture Instrument
The unified capture layer is what converts the inputs, a trace, and permission check into evidence: the assurance evidence required for demonstrating acceptance criteria. Without it, we are left assembling evidence retrospectively, by comparing evidence from multiple engineering harnesses.
Five harness layers need to be monitored after go-live: the retrieval layer, the instruction layer, the action layer, the control layer, and the human performance layer. When the action layer checks "is this agent permitted to do X in this context" and the answer is no, that verdict has to land in the immutable store exactly like a guardrail verdict in the control layer.
The UCL isn't just evidence storage. It's the measurement instrument for the one variable that cannot be measured retrospectively: the human performance layer. Precedent for non-AI-specific unified capture in regulated pharma exists; EBR/MES audit trails and data historians such as OSIsoft PI serve analogous functions for real-time manufacturing process recording.
The OpenAI breach, first detected by OpenAI five full days after the incident occurred (https://openai.com/index/hugging-face-model-evaluation-security-incident/), is a prototypical example for why a unified capture layer is necessary infrastructure. After OpenAI identified the breach and notified the affected party, Hugging Face, its CEO Clem Delangue asked for execution traces from the models involved. Independent researchers were able to use the execution traces to study the attack chain.
Jake Williams of IANS Research (https://techcrunch.com/2026/07/22/how-an-openais-human-mistake-led-to-the-ai-powered-hack-on-hugging-face/) called it a massive control failure, observing that “one man’s ‘the model escaped the sandbox’ paradigm is another man’s admission that you built the sandbox wrong.” And as University of Amsterdam social scientist Hannes Cools phrased it (https://www.wypr.org/2026-07-23/openai-blamed-a-hacking-event-on-its-ai-models-gone-rogue-here-is-what-to-know), “It is a human decision to turn off certain safeguards.” Knowing that capable agentic models may utilize paths unaligned with human values to accomplish their goals, we must design infrastructure with these critical risks in mind.
Step-level monitoring was already in place, but this was insufficient to detect the attack. To do a post-mortem audit, the evaluating organization (in this example, Hugging Face), needs to trace the model’s actions. A unified capture layer allows for a full, granular reconstruction of the model’s commit, in order to study the attack chain. Like looking through a microscope to view a specimen, a failure can only be located if the resolution is powerful enough.
Enabling the Replay
The primary function of the UCL is producing objective, deterministic replay evidence to serve as acceptance criteria (Step 4 of the validation lifecycle). The replay can be validated under binary pass/fail acceptance criteria and shown to an inspector. If the harness and human performance are components of the regulated unit, like the model itself, they both require assurance evidence to ensure they remain inside of the validated boundary. The UCL is the source of this assurance evidence:
Regress
The harness-as-determinism view poses the question: if the harness carries the regulatory load and the harness drifts, what validates the monitor?
The regress terminates because the unified capture layer is deterministic. Same inputs, same replay, every time. Which means it's testable under classic IQ/OQ/PQ CSV methodology, and more importantly, deterministic replay is a binary pass/fail acceptance criterion an inspector can execute without trusting the sponsor. Sample N decisions from the record, attempt full replay, score pass/fail.
The UCL can trace the full path of an agent:
Input
Pre-processing and PII handling
Retrieval
Model inference
Guardrail and ontology evaluations/Confidence scoring
Routing decision
Tool call to write the case
Human disposition
We can take the following use case: a pharmacovigilance (PV) generative AI intake processing agent processes the hypothetical unstructured adverse event (AE) submission below. Six months later, an inspector asks why this case was coded the way it was.
From: J. Alvarez, NP — Riverside Rheumatology
Pt 67F on [Product X] 40mg daily since March for RA. Called in today, severe abd pain x3 days, went to ED last night, kept overnight. Query GI bleed. Drug held. She also mentioned some SOB but that's likely her baseline COPD. Hgb 8.2 (was 11.4 in May).
Input: The generative AI intake processing agent receives the unstructured AE submission above as input.
Pre-processing and PII handling: The raw data enters the pre-processing and personal health information filtering pipeline. “Read” tools are used: MedDRA lookup and auto-encoder to retrieve the effective MedDRA code.
Retrieval: Using Retrieval Augmented Generation grounded with an ontology, the model pulls the correct CCDS and MedDRA versions.
Model inference: The model constructs agentic pathways to convert the raw input to structured data.
{
"case_id": "SYN-2026-00417",
"patient": { "age": 67, "sex": "F", "span": [14, 17] },
"reporter": { "name": "J. Alvarez", "qualification": "NP",
"org": "Riverside Rheumatology", "span": [6, 40] },
"suspect_product": { "verbatim": "Product X 40mg daily",
"product_id": "PRD-00219", "span": [22, 42],
"resolved_by": "product_master v2026.03" },
"events": [
{ "verbatim": "severe abd pain x3 days", "span": [98, 121],
"meddra_pt": "Abdominal pain", "meddra_version": "27.1",
"confidence": 0.94 },
{ "verbatim": "Query GI bleed", "span": [166, 180],
"meddra_pt": null, "coding_status": "unconfirmed_suspicion",
"confidence": 0.61 },
{ "verbatim": "SOB", "span": [201, 204],
"meddra_pt": "Dyspnoea", "confounder_noted": true,
"confidence": 0.72 }
],
"seriousness": { "criterion": "hospitalization",
"basis_span": [140, 154], "confidence": 0.88 },
"expectedness": "requires_retrieval"
}Guardrail and confidence scoring: The outputs are filtered through the guardrail layers, and the output is assigned a confidence score (79%) based on how certain the model is of its correctness. Validation and monitoring capabilities vary depending on the outsourcing of the guardrail.
Routing decision: The filtered output is either routed directly for translation into the final generated text, if the confidence score is above the human threshold, or routed to a qualified human reviewer if it’s below the threshold. As the threshold is set at 90%, the decision is made to escalate the output to the reviewer.
Tool call: The agent calls the following tools to generate the case text to be stored in the database (write tools): case creation in the safety database, field population on an existing case, regulatory timeline calculation, narrative generation, and queue assignment or escalation to a reviewer.
Human disposition: Because the confidence score is flagged as being below-threshold, the qualified reviewer’s disposition is recorded, and this final decision is recorded in the database.
2. Outsourced Platforms and Validation
There are several types of guardrails:
In-house: For an in-house model, evidence capture is straightforward. The data exists internally and the validation evidence can be captured.
Hosted model, sponsor-built harness (guardrails are hosted internally)
Outsourced:
Vendor agent platform, configurable guardrails
Guardrail-as-a-service, hosted evaluation layer, content-filter API, managed grounding/retrieval service
Full AI-as-a-service
For an outsourced model, evidence capture becomes more challenging. It is critical to capture across the trust boundary. When the guardrail fires inside the vendor’s infrastructure, the refusal lands in the vendor’s logs, not your internal log. A unified capture layer becomes essential in these instances to record evidence across the boundary.
3. Independent Executability
The UCL must be capable of logging, at minimum:
Inputs
Retrieval context (effective documents, version numbers, entities)
Model identity and version number
Raw output
Guardrail and ontology evaluations/Confidence scoring
Routing decision
Human disposition
4. Draft Annex 22
Draft Annex 22 explicitly permits deterministic models only, prohibiting “models with a probabilistic output which, when given identical inputs, might not provide identical outputs . . . in critical GMP applications.” While this prevents inadequate probabilistic validation, for which robust validation methodology standards have yet to be established, it still has its own failure mode. Deterministic models lacking a harness with retrieval context, an ontology, and guardrails can still fail, and when they do, they fail silently.
This ties back to the deterministic definition ambiguity problem. A deterministic model under Definition A may be permitted to be deployed in a critical GMP context under draft Annex 22, but still be non-deterministic under Definition B or Definition C. A truly deterministic system must be deterministic under all three definitions. This requires a unified capture layer.
Draft Annex 22 also permits probabilistic models in non-critical GMP applications, under the condition that there is “personnel with adequate qualification and training” responsible for ensuring that the outputs from such models are suitable for the intended use (i.e. a human-in-the-loop (HITL)). This poses the question: how do we operationalize the qualification component in an objective manner?
The Seeded Challenge: Human Performance Qualification
PIC/S PI 041-1, a harmonized inspection guidance adopted by PIC/S participating authorities, interpreting existing GMP/GDP data integrity requirements, contains a recommendation in the self-inspection of data governance controls section to use a sample of previously verified data to check the efficacy of the routine process. Section 5.6.2 recommends that the review of reported data against raw entries “may review data not included during the routine data verification checks... and/or a sample of previously verified data to ensure the continued effectiveness of the routine process.”
This is conceptually analogous to the human performance qualification seeded challenge, outlined in detail in my recent piece “Balancing the Equation: Why HITL is a Variable, Not a Constant.” When the failure mode is variable, efficacy checks on the effectiveness of the process ensure the process continues to function as intended. In the case of human performance qualification, where the human-in-the-loop functions as a control, inserting ground-truth material periodically to test the reviewer’s competence at the task ensures the integrity of the control.
The unified capture layer (UCL) is the instrument that operationalizes human performance qualification via the seeded challenge in a GxP environment. This layer has three primary functions: it determines what data is presented to the human; it records the human interface interaction; and it enforces the validated boundary. To ensure data integrity, the capture layer must exist as an intermediary between the human and the probabilistic model. It identifies non-production artifacts (known-false tasks), records the reviewer’s interaction, and excludes it from the GxP record.
The workflow ensures the integrity of the seeded challenge, records the reviewer-artifact interaction, and measures the interaction against the ground truth:
1. The Orchestration Router (The Blinder)
This is the intake valve. It receives draft outputs from the LLM and holds them. It also interfaces with a database of seeded challenges: pre-written outputs with known, expertly crafted errors designed specifically to exploit automation bias (e.g., highly fluent text with a subtle factual inversion).
The router's job is to seamlessly interleave these seeded challenges into the reviewer's queue alongside live production tasks.
The UI constraint: There can be zero metadata, latency, or visual cues differentiating a real model output from a seeded challenge.
The frequency variable: The injection rate is dynamic. If the queue depth is high, or the session length extends beyond 45 minutes, the router increases the frequency of seeded challenges to probe for reviewer fatigue.
2. The Capture Subsystem (The Measurement)
When the human reviewer interacts with an artifact (approving, rejecting, or editing), the capture subsystem records the disposition. This is where the workflow diverges:
If the artifact is a Production Task: The output, along with the human's documented oversight (audit trail, timestamp, explicit sign-off), is passed to the system of record.
If the artifact is a Seeded Challenge: The artifact is immediately intercepted. It never enters the GxP record. Instead, the human's decision is routed to the Qualification Monitor.
3. The Qualification Monitor (The Adjudicator)
This component evaluates the human's performance against the ground truth of the seeded challenge. Because the system injected the error, it possesses absolute certainty about what the human should have done.
It calculates specific metrics:
Catch Rate: Did they reject the confidently wrong output?
Dwell Time: How long did they spend evaluating the task before approving? (Rapid approvals of fluent text strongly indicate automation bias).
False Rejection Rate: Are they rejecting perfectly valid outputs just to appear thorough?
By building the unified capture layer, you transform the unmeasured human reviewer into a bounded, controlled, and continuously monitored parameter. The validation no longer rests on the hope that the human is paying attention; it rests on hard, empirical evidence generated by the system itself.
The Harness as an Error Mapping Surface
Four of the six origin categories in my published error taxonomy (”Bounding the Uncertainty: An Error Taxonomy for Non-Deterministic AI in Regulated Life Sciences*”,* indexed on Zenodo and under consideration via Therapeutic Innovation & Regulatory Science under the title "Establishing AI State of Control: An Error Taxonomy and Assurance Framework for Transformer-Based Probabilistic Systems in Regulated Life Sciences”) map to harness drift surfaces. The UCL also allows the harness to be the source for root cause analysis evidence. For instance, a record of retrieved and reference data can detect irregularities and inconsistencies, making root cause analysis traceable, and ensuring runtime data remains within its validated boundary.
My published supplier qualification methodology, VALID Trust, includes a step that explicitly requires locating the validation-inheritance boundary, and identifying for each pillar which evidence the sponsor can inherit and which must be generated by the sponsor themselves. The boundary extends beyond the model to the harness layers, meaning the UCL allows the sponsor to generate the necessary evidence when the boundary extends outside the sponsor’s walls.
Tying It All Together: The Deterministic Solution
The unified capture layer resolves several of the core components of AI/ML validation, both deterministic and probabilistic. For deterministic models (under Definition A), the UCL extends the determinism beyond the model itself to the entire system, satisfying draft Annex 22’s determinism requirement.
For probabilistic models, the UCL replay makes system-level validation possible by:
Operationalizing human performance qualification, satisfying draft Annex 22’s requirement for “personnel with adequate qualification and training” ensuring the integrity of the model’s output.
Providing validation evidence across the inheritance boundary.
Functioning as the continuous monitoring instrument for the validated system.
Identifying the root cause for probabilistic errors.
Determinism doesn’t have to live in the model if it lives in the record of what the model did, when it was done, and who dispositioned it. The append-only UCL is the answer to the probabilistic problem. While methodology appropriate to the validation of the probabilistic model itself is still necessary, the UCL enables validation of the system surrounding the model, resolving in a deterministic record.
