My March piece on cybersecurity, “Your Frozen Architecture May Have a Backdoor,” opens with a concerning but resolvable scenario: an attacker modifies efficacy endpoint data in a validated electronic data capture system. IT notices an anomaly in the audit trail and resolves it. The next paragraph introduces a different scenario: a clinical drug development team has hundreds of millions of pathology slide tiles that funnel through a multimodal classifier, with federated data sharing between academic medical centers and community health systems.
Instead of modifying data in one organization’s system, the attacker injects mislabeled samples into a community site’s fine-tuning data: just a fraction of early-stage malignancies labeled as benign. The model doesn’t flag an error; it just recalibrates slightly, and its classification threshold drifts just enough to reduce sensitivity. The data contamination propagates downstream, and no one notices that anything has changed. Misclassification that hits both arms equally biases the treatment effect toward the null, so the drug appears less effective than it is.
Two features of the second scenario stand out: the quantity of samples needed to shift the signal is very small. And the length of time between the attack and the signal shift is so large that no one can link the attack with the signal shift. This points to a need for upstream governance measures, as organizations cannot rely on corrective measures alone with probabilistic AI systems. OWASP listed data and model poisoning as #5 on its Top 10 for LLM Applications 2026, and #4 on its 2025 list.
What does this mean for validation and quality professionals in GxP environments?
• • •
The Last Thread in the House
Every layer and thread of the House of AI Trust has, at minimum, one article written about what its architecture contributes to probabilistic system AI governance and validation as a whole, with the exception of security. That’s where this piece picks up.
In this piece, I describe why probabilistic AI systems have unique security risk surfaces, and outline a high-level framework for security-specific probabilistic AI system assurance, covering three primary scopes:
Security-informed probabilistic acceptance criteria with re-verification triggers
A fourth-party evidence model for foundation-model providers
Why attack and intrinsic failure are hard to distinguish
• • •
Security Throughout the Probabilistic Lifecycle: The Certification Gap
SOC 2 and ISO 27001 are both necessary certifications. However, they cover infrastructure, not model behavior within a probabilistic AI system. Similarly, OWASP and MITRE ATLAS describe attacks, not qualification evidence that the system is in an assured state. Annex 11 and Annex 22 outline the principles that sponsors may need to implement, but do not describe the “how.”
STAR for AI Level 2 is one of the few recent certifications that explicitly covers AI-specific risks, including model poisoning, adversarial attacks, training data contamination, or the unique shared responsibility models in AI-as-a-Service deployments. This is a certification sponsors may want to look for when evaluating suppliers. The submission window opened on November 20, 2025. The Cloud Security Alliance recommends several prerequisite actions for suppliers seeking to submit to STAR for AI Level 2. Among them:
Initiate ISO 42001 certification.
Review the AI Controls Matrix and AI-CAIQ documentation.
Compile supporting evidence.
Establish STAR submission account infrastructure.
Document shared responsibility boundaries.
Once #1-#4 are completed, #5 (document shared responsibility boundaries) is frequently a challenge for suppliers, and it’s the document sponsors most need from them. Generative AI is the first supplier category in regulated pharma where the gap between evidence and accountability for model behavior can’t be closed by the usual tools (audit, quality agreement, traditional supplier test packages).
• • •
The Fourth-Party Problem: An Evidence Model for Foundation Model Providers
There is a distinction to be drawn between inheriting supplier evidence and delegating accountability. In a March 2026 warning letter (Warning Letter 320-26-57), the FDA cited a company for a software vendor change to electronic batch records outside the audit trail. The eBR didn’t capture the vendor’s changes. The supplier made the change, per the manufacturer’s instructions, and the manufacturer was cited.
Evidence may be inherited, and under GAMP 5 (2nd Edition), leveraging supplier qualification is encouraged. However, ensuring that the validation evidence demonstrates that the chosen model is fit for its intended purpose within the chosen context of use is solely the accountability of the sponsor. Inheriting evidence means reusing a supplier’s artifact to reduce your own verification effort, after you have assessed that artifact’s scope, currency and relevance.
The gaps listed above include model-behavior attestation, audit rights, GxP/Part 11 mapping, notice periods, silent serving-layer changes, and certificates that don’t attach to hosted routes. Model and supply chain updates are both qualification events and potential security risk surfaces. Supply chain was listed as #3 on the 2025 OWASP Top 10 for LLM Applications and as #4 on the 2026 list. The question then arises: how can we create an evidence model for foundation model providers?
Version pinning: Change control is a prominent gap. On 14 October 2026, OpenAI gave only 92 days’ notice, compared with its six-month policy. The default is toward updates; the assigned system owner should own and document version pinning under change control. The route should be pinned along with the model, as retirement dates may differ by route.
Pre-determined change control (PCCP for AI-enabled device software, ICH Q12): Given the gap between GxP revalidation timelines and foundation model notice periods, pre-determined change control envelopes tied to system validation metrics and appropriate benchmark testing may be beneficial. Robust system testing upfront provides a thorough baseline to measure against when the system goes into production.
Re-verification/gap assessment triggers: Items including regulatory events (e.g., Annex 11/22 finalization), expiry of SOC 2 or ISO certificates, a model provider security incident, route change, model provider disclosure of modification to provider or host-side safety filters, control configuration or system-prompt policies, and challenge-set metric drift beyond pre-defined validated envelope.
Sponsor-side security testing: Provider results are typically gated. Therefore, adversarial testing should be a component of both validation testing and determining valid acceptance criteria, and sits with the sponsor.
Model configuration as a CoU component: Model configuration (model training, weights, safety tuning, etc.) must be demonstrated by the sponsor to be suitable for the chosen context-of-use. The model/system card for the pinned version may be leveraged during the risk assessment.
Vendor contract: The vendor is the one auditable layer, and contractual obligations around model and version disclosure, pinning, notice periods, and log export can be enforced here.
On-site audit: As this is effectively not an option for the vast majority of foundation model providers, this should be considered during the risk assessment under ICH Q9(R1) and documented as a residual risk.
• • •
Security as a Data Integrity Issue: Attack vs. Intrinsic Failure
A security failure is possible at any point in the supply chain boundary. To understand why security is synonymous with data integrity in GxP settings, we must understand how transformer-based architectures process input. These systems do not distinguish between user prompts, multimodal inputs, retrieved data, or hidden context. All types of data are transformed into “tokens,” which are the symbols that the system uses to compute the probabilities that determine the output. Their behavior is stochastic, so receiving a different output from the same input is expected.
For deterministic GxP systems, as in the first opening scenario, a security failure usually leaves a forensic trail: a changed record, an anomalous login, an audit trail gap. For probabilistic AI GxP systems, as in the second scenario, a security failure is frequently silent, and may propagate downstream.
Draft Annex 11 is the strongest regulatory anchor around these types of concerns; Section 2.7 requires regulated users to “keep updated about new security threats to GMP systems” and Section 4.5 requires risk assessment of “the vulnerability of data to deliberate or indeliberate alteration, deletion or loss, and the likelihood of detection.” While AI is not called out explicitly, probabilistic AI systems have unique properties that increase risk:
Outputs vary by design: outputs are stochastic, so changed outputs cannot be attributed to a security failure by default.
Attacks act on distributions, not records: each individual record looks plausible.
The decisive evidence lives upstream: the supply chain boundary means that not all evidence sits with the sponsor. The model version, serving configuration, guardrail verdicts, retrieved chunks and tool definitions typically sit with the foundation model, hyperscaler/cloud host, or vendor layer.
This means that defense must be both preventative and detective, because reactive alone fails. Appropriate governance and design controls must be put in place upstream to bound the risk. Analogous to the “Swiss cheese” model, multiple types of controls are necessary to ensure a robust defense. Potential controls to restore detectability include:
Sponsor-side transaction capture at the point of creation
Fixed challenge sets run on a schedule and after every notified change (called out above as foundation-model evidence assurance)
Version pinning and change notification (called out above as foundation-model evidence assurance)
Corpus and tool integrity
Log export rights
Independent verification
The starting point in the Probabilistic Validation Lifecycle is the context-of-use: what is the system’s intended purpose? From there, risk tiering using a risk assessment via ICH Q9(R1) may be utilized to determine the potential impact from a security standpoint. Based on the documented risk tier, appropriate testing should be added into the validation plan, with type and depth scaled appropriately. Unscripted, exploratory testing is well suited for this purpose.
• • •
The Human and the Next Piece
Most validation professionals are not trained to evaluate systems from a security standpoint. Under draft Annex 11 section 3.1, security incidents feed periodic review, and potentially re-validation as well, if sponsor data is affected. Looping back to the opening scenario, case-by-case human review won’t catch subtle drift. Historically, security was owned by IT, and rarely if ever became a direct validation question. When errors propagate downstream while evidence remains where the incident occurred, security is a data integrity concern, and therefore a validation one as well. Distinguishing between an attack and intrinsic failure is the subject of the next piece in this series.
A six-question “security addendum” ships next week on Zenodo.
