The Version Bump Nobody Change-Controlled

Your supplier updates the foundation model from 5.1 to 5.2. Nobody tells you. There is no clause in your agreement requiring them to, no field in your supplier file to record it, and no trigger in your change control that fires.

The sponsor liability

You don't find out until six months later, when the SOP dictates periodic review is required. By this point, the model that shifted post-validation has touched hundreds of GxP records. Your quality team has to open a CAPA and trace through the model's decisions. Ultimately, the performance of the model pre- and post-update must be compared to demonstrate that the outputs of the updated model are within the same bound.

That comparison is only executable if the pre-update baseline was captured while the prior model was still live. If it wasn't, there is nothing to compare against. The CAPA cannot close on data; it closes on a written justification, and the burden of demonstrating state of control shifts entirely onto the sponsor's argument.

Three things that make this comparison routine for deterministic software are not available here:

The decision path cannot be rebuilt after the fact. Unlike with deterministic software, the reasoning used to generate a probabilistic output cannot be reconstructed retrospectively unless it was captured contemporaneously.

Release notes exist, but they are not scoped to your context of use. They report aggregate benchmark performance and describe the changes the supplier chose to version. They will not tell you whether your acceptance criteria still hold, and they cannot be used to predetermine affected scope.

Downgrade and re-execution require a pin. Calling the prior version to reproduce a decision is only possible if you pinned that version in advance and the supplier is still serving it. Absent a pin, the prior model is gone, and your comparison is limited to whatever you captured while it was live.

What could have prevented this?

Sponsor-side strategies

The sponsor has several mitigating strategies: prospective, architectural, and organizational. These are meant to be layered, and controls in one category are not substitutive for controls in another.

Prospective

• Supplier Contractual Agreement: Before the supplier contractual agreement is signed, the sponsor has the opportunity to negotiate the inclusion of clauses requiring the vendor to delay and/or disclose model updates. This is expanded upon in the next section.

A note on leverage. This category assumes you have a negotiating position, and many sponsors do not. If you are consuming a frontier model through a standard API agreement, the terms are offered rather than negotiated, and there is no counterparty across the table to negotiate with. Where that is the case, the prospective layer is unavailable, and the architectural and organizational layers stop functioning as defense-in-depth. They become the entirety of your control set, and they have to be specified accordingly.

This also makes model sourcing a purchasing decision, not only a technical one. An enterprise agreement or an intermediary that carries notification and pinning terms may be worth more to a GxP deployment than a marginally better-performing model you cannot get terms on.

Architectural

• Automated Testing: Embedding periodic automated testing into the software during sponsor-side qualification testing allows the quality team to quickly identify results that fall out of the validated boundary. This limits the retrospective examination to the window between the current time and the most recent validated snapshot.

• Golden Set Rescoring: Golden set rescoring can be layered onto automated testing, or implemented as an independent process layer.

• Unified Capture Layer: The unified capture layer (UCL) is the architectural layer that requires the least manual intervention. The UCL enables reconstruction of a full decision path, as well as a visualization into the state of the harness layers around the model.

Organizational

• Performance-Based QMS Triggers: Pre-determined triggers that alert the quality team when outputs fall outside the validated boundary.

• Risk-Tiering: Under GAMP and ICH Q9(R1), the type and degree of controls required is determined by system criticality. For instance, high-criticality systems may require automated testing with golden set rescoring alongside the Unified Capture Layer (UCL), while a lower-criticality system may require only periodic automated testing.

What to ask your supplier

1. What do you define as a change? Ask them to list what triggers a notice and what doesn't: weights, system prompt, tokenizer, safety filters, load-based routing or fallback to a smaller model, context assembly, retrieval config. A clause keyed to "model version" only covers what the supplier chooses to version.

2. How much notice, and can we test before it reaches production? Concretely: notice period in days, and whether there's a staging endpoint or a parallel window where both versions are callable.

3. Can we pin a version, and what is the deprecation notice period? If you can pin, and if you get a defined window before the pinned version is retired, the pre/post comparison stays executable. Without it, you can only compare against whatever you captured before the change. Pinning buys a window, not a permanent solution.

Checking your own vendor agreement

Draft EU GMP Annex 11 includes a clause requiring that contracts with service providers cover that the supplier “Agrees on the process for release of new system versions and on the regulated user’s possibility to test these prior to release.” Based on regulatory direction, sponsors should not rely only on manually checking vendor release notes, or on assuming SOC 2 Type II and the vendor IOQ covers validation for the intended use.

Is a model-version notification clause present in your vendor agreement?

If the answer is no, next week: the question set your procurement team can utilize.