In 1968, David Kleinman published a discovery that would quietly reshape how engineers design controllers for everything from aircraft to factories. His insight was elegant: instead of solving a complex matrix equation from scratch every time, you could start with a rough guess of a good controller and iteratively refine it until the math told you to stop. Each step brought you closer to the optimal solution, guaranteed. The method worked beautifully—but it required knowing the exact physics of the system you were controlling. Nobody knew how to use it when those equations were unknown.
For fifty years, researchers tried to extend Kleinman's iteration to situations where the physics was uncertain. They developed reinforcement learning approaches that could estimate the needed quantities from data alone. But these data-driven methods carried a hidden tax: they required estimating something called a "critic"—an internal representation of how good each control policy was—at every step. The critic was computationally expensive to identify and imposed strict requirements on the data you collected. It was a necessary evil, everyone assumed.
A paper published in 2026 shows that the critic was never necessary at all. Jiacheng Wu, Yang Zhu, and Hongye Su at Zhejiang University have developed a method that eliminates the critic entirely from the policy-improvement step. Their algorithm achieves the same optimal controllers as the classic Kleinman iteration, with the same mathematical guarantees of stability and convergence, using only a fraction of the computational effort. The regression problem shrinks from solving for hundreds of unknowns simultaneously to solving for only the controller parameters themselves.
The implications extend beyond any single algorithm. The paper reveals a fundamental misconception that has shaped a decade of research on data-driven optimal control. If the critic was never essential, then the additional data requirements and computational costs imposed by critic identification may have been unnecessary obstacles all along. Engineers working with limited data, high-dimensional systems, or real-time constraints may find that the path forward was hiding in plain sight for half a century.
The Problem of Controlling Unknown Systems
Every engineered system that moves or changes—robots, aircraft, power grids, chemical plants—needs a controller. A controller is a set of rules that tells the system how to respond to whatever situation it encounters. The controller observes the current state, decides what to do, applies an action, and repeats. The goal is usually to achieve some objective: land an aircraft smoothly, maintain a temperature precisely, minimize fuel consumption.
In the simplest case, the physics of the system is known. Engineers can write down equations describing how the state evolves when they apply a control action. These equations take a specific form in many applications: a linear system, where the rate of change of the state depends linearly on the current state and the applied control. This is the setup the paper addresses, called continuous-time linear quadratic regulation, or LQR.
The "quadratic" refers to the cost function: a mathematical statement of what the controller is trying to minimize. Typically, you want the system states to stay near zero (or some desired trajectory) while using as little control effort as possible. The trade-off between these objectives is encoded in quadratic penalties, which have nice mathematical properties that make the optimal controller take a specific form.
When the system matrices—call them A and B—are known, solving for the optimal controller reduces to solving an algebraic Riccati equation, a matrix equation that captures the trade-off between state error and control effort. The solution is a feedback gain matrix K* that tells you, for any state, exactly what control to apply. This is the crown jewel of linear control theory: a complete, exact solution to an optimization problem with guarantees of stability and optimality.
But in the real world, the system matrices are often unknown. You might be controlling a new aircraft design whose aerodynamics aren't precisely characterized. You might be managing a power grid with components that change over time as they age. You might be tuning a controller for a robot in an environment where contact forces are difficult to model precisely. The elegant Riccati equation solution requires knowledge you don't have.
The practical response has been to learn from data. If you can run experiments on the system—applying different control signals, measuring how the states respond—you can estimate the unknown quantities. The field of data-driven optimal control, also called reinforcement learning for continuous systems, has developed a variety of approaches for doing exactly this.
The Standard Approach and Its Hidden Burden
The conventional data-driven method for LQR with unknown systems is off-policy policy iteration, developed over roughly two decades of research. The name contains two important ideas.
"Policy iteration" refers to a two-step procedure that repeats until convergence. First, evaluate the current policy: figure out how well it's performing by estimating its value—the total cost it accumulates over infinite time when applied to the system. Second, improve the policy: compute a new, better controller based on the evaluation. These two steps alternate, with each iteration producing a controller at least as good as the previous one, and the sequence converging to the optimal controller.
"Off-policy" refers to how the data is collected. Rather than evaluating the current policy by applying it to the system—which would require perfect knowledge of the system dynamics to compute the value—the method uses data collected by a different "behavior" policy. You can excite the system with arbitrary control signals, observe the resulting state trajectories, and use those observations to evaluate and improve whatever controller you're currently using. This flexibility is crucial for learning in real systems, where you often need exploration to gather informative data.
The two-step structure maps naturally onto two mathematical objects. The "critic" is the value matrix that represents how good the current policy is. The "actor" is the feedback gain that determines the control action. The evaluation step identifies the critic; the improvement step uses the critic to compute a better actor.
This is where the hidden burden appears. In the standard formulation, both the critic and the actor must be estimated simultaneously from the same batch of data. The joint regression problem requires solving for the entire value matrix—which contains n(n+1)/2 independent entries for an n-dimensional state—alongside the feedback gain, which contains m×n entries for an m-dimensional control input. The combined problem dimension is r + mn, where r = n(n+1)/2.
This matters for several reasons. First, computational cost: solving a larger regression problem requires more computation at each iteration. For real-time applications, where you need to update the controller quickly, this can be prohibitive. Second, data requirements: a larger regression problem typically requires more informative data to solve reliably. The matrix of collected data must have full rank, meaning it must contain enough independent information to uniquely determine all the unknowns.
For high-dimensional systems—say, controlling a robot with dozens of states—the critic alone might contain thousands of entries. The rank requirement for the data matrix becomes severe. And crucially, the policy improvement step only needs the new feedback gain, not the critic itself. The critic information is estimated in full, then immediately used to compute a single improved controller. It's as if you calculated every detail of a map to find the direction of a single path.
A Different Perspective: Working in Policy Space
The paper's key insight begins with a reframing. Rather than thinking of the Riccati equation as an equation to be solved for the value matrix, think of it as an equation to be solved for the controller itself.
This perspective is not entirely new. Previous researchers had noted that when the system matrices are known, the optimal controller can be characterized by a "policy-space Riccati equation"—an equation whose unknowns are the entries of the feedback gain rather than the value matrix. This reformulation shows that the optimal controller is the unique stabilizing solution to a particular equation in the space of possible controllers.
The challenge is that this reformulation still requires knowing the system matrices. The value matrix appears inside the equation, and without knowing the physics, you cannot compute it. Previous work had not resolved how to translate this policy-space perspective into a data-driven, off-policy setting.
The paper's first contribution is to show how to anchor this reformulation at a known stabilizing gain. The requirement is simple: you need at least one stabilizing controller F—some way of controlling the system that doesn't cause instability. This is often available in practice, either from rough engineering knowledge, from preliminary experiments, or from a conservative initial design. The anchor doesn't need to be good; it just needs to keep the system stable.
With this anchor in place, something remarkable happens. Every possible controller K can be lifted to a unique symmetric matrix P_F(K) by solving a stable Lyapunov equation. This matrix depends on the anchor controller and encodes how well K performs, but it can be computed without knowing the system matrices—only the anchor's properties matter.
Define a policy-space residual G_F(K) = RK - B^T P_F(K). When the system matrices are unknown, this quantity is not directly computable, but its structure captures exactly when K is optimal: G_F(K) = 0 if and only if K is the optimal controller. The residual tells you, for any candidate controller, how far you are from optimality—purely in terms of the controller itself, not the underlying value matrix.
This reformulation moves the problem into policy space. Instead of solving for a value matrix and then deriving a controller, you're solving directly for the controller. But the residual still contains the unknown term B^T P_F(K), which depends on the unknown system matrix B. The remaining challenge is to eliminate this term from the data equation.
The Null-Space Projection Trick
The paper's core technical contribution is a method for removing the value-matrix dependence from the integral data equation while preserving the information needed for policy improvement.
The starting point is an integral identity that holds for any controller K. If you run experiments on the system with a behavior policy, collecting trajectories of states and control inputs, you can write down an equation relating the controller to the collected data. This equation involves an endpoint term that contains P_F(K). The problem is that this endpoint term makes the equation depend on the unknown value matrix.
The insight is to project the equation onto a null space. Specifically, look at the weighted sum of the equations across different time intervals, where the weights are chosen so that the endpoint terms cancel out. This is possible when the weights lie in the kernel of the matrix of endpoint data. The weights don't need to be zero for each interval individually; they just need to cancel in the aggregate.
The mathematical details are careful but the intuition is clear: you're finding a linear combination of your measurements where the unknown quantity drops out. After projection, the resulting equation involves only terms that are directly computable from the data and the current controller estimate. The projected data matrix that multiplies the controller variables is the key object.
The paper shows that this projection is independent of the particular choice of weights in the null space—all valid choices give equivalent results, up to a rotation. The rank of the projected data matrix determines what you can learn from the data.
Under a verifiable rank condition on this projected matrix—which the paper calls the "projected actor rank condition"—the projected data equation is exactly equivalent to the policy-space residual equation. This means that solving the data equation for the controller is mathematically identical to finding where the residual vanishes.
In other words, you can find the optimal controller from data without ever estimating the critic. The projected equation contains exactly the right information to locate the solution, no more and no less.
The Connection to Kleinman Iteration
The equivalence is not just abstract. The paper proves that each iteration of the proposed algorithm coincides exactly with Kleinman iteration—the classical method that requires knowing the system matrices.
This is a striking result. Kleinman iteration has well-understood convergence properties: starting from any stabilizing controller, it produces a sequence of controllers that monotonically improve and converge to the optimal controller. The new algorithm reproduces this sequence using only input-state data from an arbitrary behavior policy, with no model knowledge.
The implication is that the stabilizing and convergence properties of Kleinman iteration transfer wholesale to the data-driven setting. The algorithm doesn't approximate Kleinman iteration; it computes the same update from different information. The guarantees hold because the data equation is mathematically equivalent to the residual equation, not because of approximation arguments.
This is unusual in the reinforcement learning literature, where convergence proofs often require additional assumptions or provide only approximate guarantees. The paper's result is exact: under the rank condition, the data-driven update produces exactly the same controller as model-based Kleinman iteration.
What Gets Removed and What Remains
Perhaps the most clarifying contribution of the paper is a precise accounting of what the conventional approach requires and what the new approach requires.
Standard off-policy policy iteration imposes a single full-rank condition on the data matrix: rank(Θ_k) = r + mn. This condition ensures that both the critic and the actor can be uniquely identified from the data. The paper shows that this condition actually decomposes into two separate conditions.
The first is a "critic rank condition," which ensures that the value matrix can be uniquely estimated. This is the component that imposes the largest burden, because r grows quadratically with the state dimension n. For a system with 10 states, the critic alone has 55 independent entries; for 20 states, it's 210 entries.
The second is an "actor rank condition," which ensures that the improved feedback gain can be uniquely identified. This condition involves only mn entries and grows linearly with both state and input dimensions. The actor rank condition is exactly the projected rank condition that the new algorithm requires.
The critic-free method eliminates the first condition entirely. It never attempts to identify the value matrix, so it doesn't need the data to support such identification. The actor rank condition remains essential—no learning algorithm can identify parameters from uninformative data—but it's a weaker requirement than the full-rank condition.
This decomposition also clarifies what information each method extracts from the data. The conventional approach uses the data to learn both what the current policy is worth and how to improve it. The new approach uses the data only to determine how to improve the policy, relying on the anchored formulation to handle policy evaluation implicitly.
Computational Implications
The practical impact of removing the critic is substantial for computation. The regression problem at each iteration shrinks from solving for r + mn unknowns to solving for only mn unknowns.
To appreciate what this means, consider the numbers. For a system with n = 10 states and m = 2 control inputs, the conventional method must solve for r + mn = 55 + 20 = 75 unknowns at each iteration. The new method solves for only mn = 20 unknowns. The regression matrix is roughly 60% smaller in each dimension, and the computation time scales worse than linearly in the problem size.
Per-Iteration Runtime Comparison (seconds)
| Label | Value |
|---|---|
| n=2, m=1 | 0.3 |
| n=4, m=2 | 0.6 |
| n=6, m=2 | 1 |
| n=8, m=3 | 1.8 |
| n=10, m=3 | 2.5 |
| n=12, m=4 | 4 |
The paper's simulations confirm these savings in practice. For systems of different sizes, the per-iteration runtime of the new algorithm is consistently lower than the conventional method, with the gap widening as the dimensions increase. This reflects the reduced regression dimension: solving a smaller least-squares problem requires less computation.
For real-time applications, where controllers must update quickly to respond to changing conditions, this speedup matters. The computational budget freed by eliminating the critic could be used for other purposes: longer data collection, more sophisticated exploration strategies, or faster overall iteration. Or it could simply enable the method to work within the strict time constraints of high-performance control systems.
The Role of Exploration
One might wonder whether the eliminated data requirements simply reappear elsewhere—perhaps the new method requires more exploratory data to compensate for not estimating the critic. The paper's analysis suggests this is not the case.
The key insight is that the information lost by not estimating the critic is not needed for policy improvement. The critic tells you how good the current policy is, which is useful if you're trying to decide whether to stop iterating. But the improvement step—the computation that produces a better controller from the current one—doesn't require knowing the critic's value. It requires knowing how to change the controller to reduce future cost, and this information is preserved in the projected data equation.
The exploration requirements for the actor are the same as they would be for the full method. You need data that excites the relevant dynamics of the system, so that the projected data matrix has full column rank. But the actor's rank condition is strictly weaker than the full-rank condition, because it doesn't require the data to support critic identification.
In practice, this means the method could work with shorter data batches or less excitation than the conventional approach, though the paper doesn't provide experimental results on this specific point. The theoretical equivalence ensures that when the actor rank condition is satisfied, the resulting controller is optimal.
Relationship to Prior Work
The paper positions itself at the intersection of several research threads in data-driven control and reinforcement learning.
On-policy integral reinforcement learning, as developed by Vrabie and colleagues, uses data collected while applying the current policy to estimate the value matrix. This requires the policy to be stabilizing and uses the closed-loop data directly. The off-policy methods that followed, notably the work of Jiang and Yu, relaxed this requirement by using data from arbitrary behavior policies. But they maintained the joint critic-actor structure.
The new method builds on this foundation but removes the critic requirement. The key technical tools—null-space projection to eliminate endpoint terms and the anchored policy-space formulation—do not appear in prior work. The equivalence to Kleinman iteration, in particular, is a new result.
Previous policy-space approaches, such as direct gradient methods and policy algebraic equations, had shown that critic-free characterizations of optimal policies were possible when the model is known. The paper extends this perspective to the data-driven, off-policy setting, completing the bridge between model-based and model-free optimal control.
The paper also relates to Q-learning for continuous systems, which reformulates the value function to avoid certain matrix inversions. In the discrete-time case, Q-learning is actor-critic by design: the critic is the action-value function, and the actor is the policy. The continuous-time version faces similar challenges. The new method sidesteps these by never estimating a critic at all.
Why This Matters for Engineering Practice
Understanding the theoretical contribution is valuable, but the practical implications are what will determine the paper's impact.
The elimination of the critic makes data-driven optimal control more tractable for high-dimensional systems. Applications with many states—power grids with numerous generators, chemical plants with many unit operations, autonomous vehicles tracking many variables—face disproportionate computational burdens with conventional methods. The new approach reduces this burden by eliminating the component that grows quadratically with state dimension.
Real-time adaptation becomes more feasible. When controllers must update quickly—say, to respond to changing operating conditions or unexpected disturbances—the reduced computation time of each iteration could be the difference between meeting and missing deadlines. The freed computational budget could support more frequent updates or more sophisticated adaptation logic.
The weaker data requirement may enable applications where data is scarce or expensive to collect. Systems that are expensive to operate, dangerous to excite, or subject to strict experimental protocols might not provide enough data to satisfy the full-rank condition. If the actor rank condition is easier to meet, these systems might become amenable to data-driven optimal control.
The method also simplifies the implementation and analysis of data-driven control systems. The joint critic-actor structure requires managing two interdependent estimates, with convergence depending on both components. The actor-only structure is conceptually simpler: there's one thing to identify, and the update is exactly Kleinman iteration. Engineers trying to apply these methods might find the new formulation easier to understand and verify.
Caveats and Open Questions
The paper establishes a strong theoretical result, but several practical questions remain open.
The method requires a stabilizing anchor controller. The paper assumes this is available, which is often reasonable in practice. But for some applications, finding any stabilizing controller might be non-trivial. The method doesn't specify how to obtain the anchor or what happens if the available anchor is poorly conditioned.
The rank condition on the projected data matrix must be verified in practice. The paper shows this condition is necessary and sufficient for the algorithm to recover the optimal controller, but it doesn't provide explicit conditions on the system dynamics and exploration signal that guarantee the condition holds. Understanding what exploration strategies reliably satisfy the rank condition is an important direction for future work.
The convergence rate of the algorithm is inherited from Kleinman iteration, which is locally linear. For some systems, this might be slower than alternatives. The paper doesn't explore whether the critic-free structure enables new acceleration techniques or modifications that would improve practical convergence.
The analysis assumes the data is exact and noise-free. Real sensor measurements are noisy. Understanding how the method behaves with noisy data—and whether the elimination of the critic improves or degrades noise robustness—would be valuable for practical application.
Finally, the method applies specifically to the linear quadratic regulator. Extensions to nonlinear systems, partially observable settings, or constrained control problems are not addressed. Whether the critic-free insight generalizes beyond the linear-quadratic case is an open question.
What Comes Next
The paper opens several promising directions for future research.
The most immediate is investigating whether the actor rank condition is strictly weaker than the full-rank condition in practical scenarios. If there are systems where the full-rank condition fails but the actor rank condition holds, the new method would enable data-driven control where the conventional approach could not. Systematic characterization of this gap would clarify the practical benefits.
The connection to Kleinman iteration suggests exploring modifications and extensions of the classical method through the data-driven lens. Kleinman iteration can be accelerated by various techniques; whether these carry over to the data-driven setting is worth investigating. The policy-space perspective might also suggest new variants that are more robust to noise or more efficient in their data use.
Distributed and multi-agent extensions are natural candidates for the new framework. In cooperative control problems where multiple controllers must coordinate, the computational burden of joint critic-actor estimation is even more acute. The actor-only structure could enable more scalable algorithms for these settings.
The method's conceptual simplicity might also benefit education and software implementation. The critic-actor structure, while mathematically elegant, can be confusing for practitioners. An implementation based directly on Kleinman iteration, with the data-driven aspect handled by the projection procedure, might be easier to understand, verify, and debug.
Finally, the broader insight that unnecessary components in learning algorithms might impose hidden costs deserves further exploration. If the critic was unnecessary for policy improvement in this setting, what other common structures might be re-examined? The answer might lie in carefully analyzing what information each component of an algorithm actually uses versus what it appears to require.
The Bigger Picture
The arc of development in data-driven control has followed a predictable pattern: start with methods that require complete model knowledge, then gradually relax the assumptions, adding complexity to handle less information. The result has been increasingly powerful methods that are also increasingly complex.
This paper runs counter to that pattern. Rather than adding complexity to handle uncertainty, it removes a component that was thought to be necessary but wasn't. The result is a simpler, more efficient algorithm with stronger theoretical guarantees.
The lesson is not that all complexity is unnecessary. Rather, it's that each component of an algorithm should be questioned: what does this actually do, what information does it actually require, and is that requirement fundamental or incidental? The answer might be surprising. In this case, fifty years of work on data-driven optimal control proceeded on an assumption that turned out to be false.
For engineers building control systems in an uncertain world, the practical benefits are real: faster computation, weaker data requirements, simpler implementation. But there's a deeper satisfaction in the result. The optimal controller, the solution to a problem posed in 1960 and solved elegantly by Kleinman in 1968, can be recovered from data without ever estimating how good your current controller is. You only need to know how to improve it. Sometimes, the question you thought you were asking is not the question you needed to answer.