Critical Manufacturing: AI Accountability in Manufacturing—What Happens When AI Makes a Decision That Can’t Be Explained?

Can You Delegate a Decision to AI?

Since agent-based AI in manufacturing is increasingly not just advising but also taking action, the question is no longer whether it is intelligent enough. Rather, the question is whether anyone can explain what it just did.

Yes, I know that sounds scary to many people, but even if you’re still resistant to this idea today, you’ll soon be letting AI take action on your production line.

Take a moment to imagine what that might look like: releasing a batch, adjusting a setpoint, routing material for rework, or putting a batch on hold. The technology is ready, and the pressure to deploy it is undeniable.

The problem is: Some time after that decision, someone with decision-making authority will ask why the system acted that way. It could be a regulatory agency during an inspection. Or an auditor reconstructing an incident. It could be your own executive management following a committee incident. Or a customer’s quality team before they sign the contract.

And in that situation, “the model was very reliable” is not an answer. It’s the beginning of a very bad day.

Why isn’t a “human-in-the-loop” approach enough to hold AI accountable?

The question that determines whether you can delegate a decision to AI isn’t how intelligent the system is. It’s whether the system can explain itself to someone who holds you accountable.

And as it turns out, explainability isn’t something you add to AI after the fact. It’s a property of what the AI is built on.

If this seems hypothetical or far-fetched to you, think again. There was an initial warning letter from the FDA citing the use of AI in pharmaceutical manufacturing. Investigators found that the company had not performed process validation prior to distributing its drugs, as required by 21 CFR 211.100. The company’s response stated, verbatim, that it had been unaware of this requirement because the AI agent it used “never informed [them] that this was required.”

Note that the FDA’s position was not that AI had been used. Rather, the issue was that AI results must be reviewed and approved by a person responsible for CGMP compliance, and that the use of AI to assist in the creation of documents does not eliminate the obligation to verify their accuracy and compliance. The agent’s silence is no justification.

Why MES Traceability Is the Foundation for Responsible AI

There are two ways an AI system can answer the question “Why did you do that?”

The first is to generate a plausible justification after the fact. If the reasoning was not recorded anywhere, the model reconstructs a narrative that fits the result.

The second is to trace the decision back through the structure that produced it. The system took this action because a specific rule applied to this step in this version of the recipe, for this product, based on the state of the material and the quality result recorded at that time.

This is an explanation because the reasoning was captured before the decision was made. You’re not asking the AI to justify itself after the fact; you’re following a path that already exists.

To understand why this distinction is important, set the AI aside for a moment and consider a single number in manufacturing.

How MES Traceability Provides the Context AI Needs

A measurement value on its own means nothing. It is just noise until it can be placed in context: which product it describes, which piece of equipment generated it, which recipe version was running, which operator performed the work and whether their certification was up to date, which material batch was used, and which order linked all of this together.

When you add these connections, the number becomes a fact. Without them, it’s just a value without meaning.

This context must be compiled in one of two places:

– Late

Events end up in separate locations, and when a question is asked, the system immediately re-links everything. This can work, but you pay the price for this linking with every question, and a system that reconstructs relationships it did not witness firsthand can only guess.
Sometimes the guess is wrong, and no one can tell.

– Early

The moment the step is completed, the execution system already knows the product, the equipment, the recipe version, the operator, the batch, and the order. Not because it looked them up, but because it enforced them a moment earlier. It verified the certification before approving the step. It confirmed that the equipment was qualified and linked the batch to the order. The context was the process itself.

AI can only explain a decision if the reasoning was captured at the very moment it took place—and that is precisely what the MES does by enforcing rules (certifications, equipment qualification, batch linking)—rather than reconstructing them after the fact.

Interestingly, this is not a new standard invented specifically for AI. It is the standard by which more heavily regulated manufacturing environments already operate.

The principles of data integrity require that records be traceable and captured at the time of the event. Regulatory authorities also expect electronic records to reconstruct the event, not merely summarize it.

The reason for this is simple: if you cannot trace a decision back to the moment it was made, you can hardly defend it. AI does not change this requirement; it increases the pressure, because the entity seeking trust is no longer a person.

A well-designed MES is what makes traceability possible in the first place

The reason a decision can be traced at all is that a well-designed MES does more than just record what happened. Whether explicitly or not—every production step generates a graph of relationships. The material, the batch, the equipment, the operator, and the recipe step become nodes, while the processes between them form the edges.

Every production decision generates a traceable context chain

MES traceability captures these relationships in real time, thereby creating a coherent record of how production actually unfolded.

It is this graph that provides an explanation. When an auditor asks “why,” the answer is a traceable path through the graph: from the decision back to the rule that triggered it, the step it controlled, as well as the material, the quality status, and the person involved.

The explanation does not have to be reconstructed on demand. It already exists, since the system captured these relationships during factory operations.

To be fair to the alternative: A contextualization layer built on top of the execution can reconstruct much of this retrospectively. In many plant- and process-oriented environments, this works well enough to be useful.

However, the reconstruction is limited by what the execution system actually recorded. It can roughly trace the lineage, but it cannot restore meaning that was never recorded. This gap between “roughly” and “verifiable” is greatest where it matters most.

In highly variant, regulated, discrete manufacturing, where genealogy, deviations, and rework are not edge cases but are fundamental to operational processes.

Explainability vs. Traceability: These Are Not the Same Thing

That is why you should be cautious of vendors who claim to have solved the problem of “explainability.” Two different things are being sold under this term.

One is the understanding of how a model reasons internally—a problem that has not yet been solved by experts and for which there isn’t even a standardized method of evaluation.

The other is an audit trail that shows which inputs led to a decision—and that is exactly what most explainability tools actually offer.

That’s why the foundation is so important when it comes to agentic AI. Without the right foundation, AI cannot provide reliable answers.

Determinism, guardrails, and human oversight ensure AI accountability

Certain topics come up again and again when we discuss decisions made by agent-based AI. These are important, even if they aren’t necessarily related to the topic of explainability.

Determinism. This is very important, even if it merely prevents the use of generative AI in critical runtime execution functions. In regulated environments or risk-averse organizations, determinism—where identical input conditions always lead to the same output decisions—is a must.

In these cases, generative AI belongs in the design phase, not the execution phase. Whether through vibe coding or, more intelligently, through vibe metadating (AI-based configurations utilized by the platform)—companies can accelerate their business rules and processes; however, only deterministic behavior makes it into runtime execution.

Safety barriers. These are also crucial, but they are not enough. With safety barriers, you define the space in which the AI operates, and the boundaries themselves are rules that must be explainable. However, they do not explain the decisions made within those boundaries.

Human oversight. “Human-in-the-loop” is constantly cited as the answer to all of this. On its own, however, this is not enough. If a human approves a decision that they cannot question, that is not oversight. It is liability with a signature.

Nevertheless, the decision is yours

So, back to explainability and decisions. None of the solutions mentioned relieves you of the most difficult decision. Even a system that can explain everything perfectly won’t tell you how much trust is sufficient to act. This threshold—the line between support and autonomy in a given decision—is a judgment for which someone with accountability must take responsibility. The platform can make the threshold visible, verifiable, and customizable. But it cannot set it for you.

That is precisely why this is no longer an IT issue, but a matter of leadership. A leader can decide how much trust to place in the system. But no leader can defend a decision that the system cannot explain.

Regulation of Agent-Based AI in Manufacturing

In regulated environments, the direction is already clear: The European Commission’s draft Annex 22 explicitly draws the line: Generative AI and large language models are not covered and should not be used in critical GMP applications; and where AI is used in non-critical applications, qualified personnel remain responsible, within the framework of a “human-in-the-loop” approach, for ensuring that the results are appropriate.

The principle underlying the draft is that intelligence does not transfer responsibility. So before the debate about how autonomous your AI should be even begins, there is a preliminary question—one that is more architectural than strategic in nature. When your AI takes action and someone with decision-making authority asks for the reason, does your factory respond with an explanation or with a predefined procedure?

You cannot delegate a decision that your system cannot explain. And your system can only explain what it was originally designed to do.

– – – – –

Related Links

👉 www.criticalmanufacturing.com  

Photo: Critical Manufacturing