AI That Explains Why It Made a Decision — Not Just What

The AI identified the reactor’s ramping constraints—how quickly flow rates can change—as the most influential factors in deciding optimal control actions, even though traditional solvers don’t highlight them. For one test case, the system pinpointed the edge between the initial inlet flow rate $u_0$ and the mass balance equation at $t=1$ as the single most important connection, scoring it a perfect 1.0 on the explanation scale. This wasn’t just symbolic: when tested, the simplified “explanation graph” reproduced the full model’s predictions with a mean absolute fidelity error of just 0.34—less than half a liter per minute deviation across a time horizon where flows range up to 800 L/min. In high-stakes industrial processes, such precision paired with interpretability could be transformative.

This isn’t about replacing human engineers. It’s about giving them an intelligent collaborator that doesn’t just say what to do, but why. And it works fast: each explanation took about two seconds to generate.

The Science

Optimal control problems are everywhere in engineering—from guiding rockets to stabilizing power grids to managing chemical reactors. At their core, they involve solving complex optimization puzzles: given a system’s dynamics, constraints, and objectives, what sequence of actions minimizes error or energy use? Traditionally, these are solved numerically using algorithms like interior-point methods, which return a solution and maybe some Lagrange multipliers—but rarely any intuitive insight into why that solution is optimal.

Enter machine learning. In recent years, researchers have trained neural networks to predict optimal control actions directly, bypassing slow solvers. But these models are typically black boxes: feed in a set-point profile, get out a control trajectory, with no visibility into the reasoning.

Jiyong Lee and Ilias Mitrai from the University of Texas at Austin propose a new hybrid approach: use graph neural networks (GNNs) to predict solutions, then apply explainable AI (XAI) techniques to reveal which parts of the problem structure drove those predictions (Lee & Mitrai, 2026). Their framework has two stages.

First, they represent the entire optimal control problem as a variable–constraint bipartite graph. Each variable (like reactant concentration $x_t$ or inlet flow $u_t$ at time $t$) is a node. Each constraint (like a differential equation or a physical limit) is another node. An edge connects a variable to a constraint if that variable appears in the constraint. On top of this skeleton, they attach features: bounds, ramping rates, set-points, initial conditions, and time indices. This rich graph encodes both the topology and physics of the control problem.

Then, a graph convolutional network (GCN) learns to map this graph to the optimal manipulated variables. Using message-passing layers, the GNN aggregates information from neighboring nodes—so the prediction for $u_0$ incorporates not just its own features, but also those of connected constraints like the mass balance equation and ramping limits. After several layers, the model combines local node embeddings with a global summary of the whole graph before passing through a final MLP to produce scalar outputs.

Once trained, the real innovation begins: explaining the prediction.

They use GNNExplainer (Ying et al., 2019), an algorithm designed to find the smallest subgraph that preserves a GNN’s output. Mathematically, this means solving an optimization problem to identify a subset of edges and node features—represented by binary masks $M$ and $\hat{M}$—such that feeding the masked adjacency and feature matrices $(\hat{A} = A \odot M, \hat{\Phi} = \Phi \odot \hat{M})$ into the GNN yields nearly the same prediction as the full graph. To make it tractable, they relax the binary constraints and add entropy penalties so gradients can flow.

The result? A distilled version of the problem—the “explanation graph”—that captures only the most decision-relevant elements.

They applied this framework to an isothermal continuous stirred-tank reactor (CSTR), a canonical system in chemical engineering where a reactant $A$ turns into product $B$ via the reaction $A \rightarrow 3B$. The goal is to control the inlet flow rate $u(t)$ to keep the concentration $x(t)$ close to a desired set-point $x^{sp}(t)$, while respecting physical limits on how fast flows can ramp up or down.

What They Found

The GNN worked remarkably well. Trained on 10,000 simulated optimization instances—each with randomly sampled initial conditions, set-point profiles, and ramping rates—it achieved a normalized mean squared error of $9.96 \times 10^{-6}$ on held-out test data. For the actual flow values, that translated to a mean absolute error of 0.75 L/min and a mean relative error of 2.4% across the time horizon—well within acceptable ranges for industrial applications.

Training and Test Loss Over Iterations

Label	Value
0	0
500	0
1000	0
1500	0
2000	0

Normalized mean squared error converges to ~1e-5 after 2,000 iterations.

But accuracy was only half the story. The real breakthrough came from the explanations.

When analyzing the prediction for the first control action $u_0$, GNNExplainer revealed that certain edges dominated the decision-making process. The top-ranked was the connection between $u_0$ and the mass balance constraint at $t=1$, scoring a perfect 1.0. This makes physical sense: the value of $u_0$ directly affects how the concentration evolves from $x_0$ to $x_1$, so the model correctly identifies this causal link as critical.

Even more telling were the next two edges: $u_0$ linked to the ramping-up and ramping-down constraints at $t=0$, with scores of 0.896 and 0.848 respectively. These are inequality constraints limiting how much the flow can increase or decrease from the initial value $u^{init}$. Crucially, the solver output showed that the ramping-down constraint was active (i.e., binding) at the optimal solution for the first five time steps—meaning the controller was literally hitting the floor on how fast it could reduce flow.

Top-5 Variable-Constraint Edges by Importance Score

Label	Value
u₀ – Mass balance (t=1)	0
x₁₉ – Mass balance (t=19)	0
u₀ – Ramping up (t=0)	0
u₀ – Ramping down (t=0)	0
x₁₇ – Mass balance (t=17)	0

Scores derived from GNNExplainer; higher values indicate greater influence on prediction.

So the AI didn’t just notice these constraints—it spotlighted the ones that were actually constraining the solution.

Other high-ranking nodes included $x_{17}, x_{18}, x_{19}$—state variables near the end of the time horizon. This might seem counterintuitive: why would future states affect the first control move? But it reflects a known property of model predictive control (MPC): decisions are made with foresight. The controller knows that aggressive moves now could lead to violations later, so it plans conservatively. The GNN, having learned from thousands of such examples, internalized this long-horizon reasoning.

At the feature level,

Feature Importance for Predicting First Control Action

Label	Value
Node Type	0
Variable Bounds	0
Ramping Rates	0
Set-point	0
Initial Value	0
Time Index	0

Feature-level explanation shows structural and dynamic parameters dominate prediction logic.

shows that node type, ramping rates, and variable bounds were the most important inputs overall. Set-point and initial values mattered less, and time index least of all. This suggests the model relies more on structural knowledge—what kind of entity a node represents and what limits it faces—than on specific numerical values.

And critically, the explanations were faithful. When the pruned explanation graph was fed back into the GNN, the predicted control actions deviated by only 0.34 L/min on average from the original. At worst, the difference was 0.86 L/min—still negligible in context. This high fidelity means the explanation isn’t just a heuristic; it captures the actual logic the model uses.

Why This Changes Things

Today, many industrial control systems run on model predictive control (MPC), where an optimizer solves a constrained problem every few seconds to determine the next action. While powerful, this is computationally expensive and opaque. If the controller suddenly commands a sharp drop in flow, operators may hesitate—especially if it contradicts intuition or past experience.

Machine learning surrogates promise speed: once trained, a GNN can predict the entire control trajectory in milliseconds. But without trust, adoption stalls. Engineers won’t hand over control to a system that can’t justify its decisions.

This paper bridges that gap. It’s not just predicting faster—it’s explaining better than classical methods.

Consider Lagrange multipliers, the traditional tool for sensitivity analysis. They tell you how much the objective would change if you relaxed a constraint—but only locally, and only for active constraints. They’re silent on inactive ones, even if those constraints shaped the decision indirectly. And they require re-solving the problem to assess different scenarios.

In contrast, GNNExplainer highlights both active and inactive constraints that influence the decision—not just those that bind. It reveals the causal backbone of the solution: which equations and limits the AI deemed essential.

Moreover, because the explanation is derived from a fast surrogate model, it can be generated in real time—within two seconds per action in this study. That opens the door to interactive diagnostics: imagine a control room interface where an operator clicks on a suspicious control move and instantly sees a visual overlay of the most relevant constraints and variables.

Beyond usability, there’s a deeper implication: we may have been overlooking key drivers of optimal behavior. The fact that ramping constraints—often treated as secondary—emerged as top influencers suggests our mental models of these systems might be incomplete. By surfacing hidden patterns, explainable AI can feed back into theory, helping engineers refine their understanding of complex dynamics.

This matters far beyond chemical reactors. Similar frameworks could clarify decisions in energy grids (where transmission limits shape dispatch), autonomous vehicles (where safety constraints dictate maneuvers), or supply chains (where capacity bottlenecks alter routing). Anywhere optimization drives action, interpretability builds trust.

What's Next

The authors are careful to note a crucial caveat: “The proposed approach does not explain the solver output, rather it explains the prediction of the GNN model which is trained using the optimal solution obtained from the solver.” (Lee & Mitrai, 2026). In other words, the explanation reflects what the AI learned, not necessarily the mathematical necessity of the optimum.

This distinction is vital. If the GNN is slightly inaccurate—or biased due to training data limitations—the explanation will reflect those flaws. So validation against ground-truth solvers, as done here with Fig. 5, remains essential.

Several open questions remain. Can this approach scale to larger, multi-variable systems with dozens of states and controls? How robust are the explanations to distribution shifts—say, when the plant operates under new conditions not seen during training? And can we extend this to counterfactual reasoning—e.g., “What if we relaxed the ramping limit by 10%?”—to support decision-making under uncertainty?

One promising direction is combining this with symbolic regression or neural theorem proving to convert the highlighted subgraphs into human-readable rules. Instead of saying “these edges matter,” the system could say, “the controller avoids rapid decreases because it would violate the lower ramping bound within three steps.”

Another is integration with digital twins—virtual replicas of physical systems used in industry. Today’s digital twins simulate but don’t always explain. Adding explainable AI layers could turn them into true cognitive partners, capable of not just forecasting outcomes but teaching engineers why things happen.

Finally, there’s the question of actionability. An explanation is only useful if it leads to better decisions. Future work should test whether providing these visualizations improves operator performance—reducing hesitation, catching errors, or inspiring design improvements.

We’re moving from AI that acts to AI that collaborates. Not by mimicking human reasoning, but by developing its own—and then translating it into terms we can understand. In safety-critical domains like chemical processing, aviation, or medicine, that translation isn’t just nice to have. It’s the price of admission.