The control game's missing player: a critic-free way to find saddle-point policies

Every time a grid operator turns up the dial on a wind farm, or a self-driving car applies the brakes, or a drone banks to avoid a gust, the same quiet battle is playing out. A controller is trying to do its job — hold a voltage steady, keep a car centered, hold a heading — while the world's unpredictable disturbances push back. Disturbances are not hypothetical; they are wind, noise, sensor error, an adversary's jammer. Control engineers have long modeled this as a zero-sum game in which the controller and the disturbance each try to outdo the other九龙.

The mathematics for solving such games is maturedaric — but it has a tactically redundant middleman. Standard methods compute a "value matrix," a sprawling quadratic object with $n(n+1)/2$ entries that encodes how much damage the state will do, then derive the actual control laws from it. The controller itself never uses this matrix; it only uses the policy gains. Yet every data-driven iteration scheme has insisted on estimating the redundant middleman anyway — roughly doubling or worse the number of parameters to solve for watervalve.

A new paper from researchers at Zhejiang University (Wu, Zhu, & Su, 2026) throws the middleman out. They build a "critic-free" policy iteration scheme: only the policy gains are unknowns, the value matrix never appears as an iterative variable. The result is a leaner algorithm that recovers the same saddle-point policies with far fewer equations, smaller memory footprint, and — crucially — a data-informativity condition that separates "can we recover the right policy?" from "can we identify the critic?" The answer to the second question, it turns out, is that we never needed it in the first place.

The Science

The problem is a continuous-time linear system

$$\dot{x}(t)=\mathbf{A}x(t)+{\mathbf{B}}_{1}u(t)+{\mathbf{B}}_{2}w(t),$$

with state $x$, a control input $u$ the designer picks to minimize cost, and a disturbance $w$ that acts to maximize it. The system matrices $\mathbf{A},{\mathbf{B}}_1,{\mathbf{B}}_2$ are unknown — the central difficulty that motivates reinforcement learning approaches in the first place. The running payoff is quadratic,

$$r(x,u,w)=x^{\top }\mathbf{Q}x+u^{\top }\mathbf{R}u-{\gamma }^2{w}^{\top }w,$$

where $\gamma >0$ is a prescribed attenuation level: how much the disturbance's blast is discounted. The game value over an infinite horizon is the integral of this running payoff, and the goal is a saddle point: a policy pair $(u^{\ast },w^{\ast })$ such that neither player can improve by unilaterally deviating.

When the model is known, the saddle point is characterized by the game algebraic Riccati equation (GARE), a quadratic matrix equation whose stabilizing solution ${\mathbf{P}}^{\ast }$ directly yields the optimal feedback gains $u^{\ast }=-{\mathbf{R}}^{-1}{\mathbf{B}}_1^{\top }{\mathbf{P}}^{\ast }x$ and $w^{\ast }={\gamma }^{-2}{\mathbf{B}}_2^{\top }{\mathbf{P}}^{\ast }x$. The whole classical edifice — $H_{\infty }$ control, differential games, linear-quadratic design — hangs off this equation (Basar & Bernhard, 1995; Zhou, Doyle & Glover, 1996).

But real systems rarely hand you their $\mathbf{A}$ and $\mathbf{B}$ matrices. Data-driven methods based on policy iteration (PI) therefore learn the gains from measured trajectories. The standard family of algorithms — adaptive critics, integral RL, off-policy schemes — iterates on the pair $(\mathbf{K},\mathbf{L},\mathbf{P})$: every reinforcement step re-estimates the value matrix $\mathbf{P}$ alongside the policy gains (Vrabie & Lewis, 2009; Jiang & Jiang, 2012; Wu et al., 202 beats).

That's the critic. And here is the subtle inefficiency the authors identify: for an $n$-dimensional system, the critic introduces $d_n=n(n+1)/2$ extra parameters on top of the $sn=(q+m)n$ joint-policy parameters. Conventional off-policy PI casts a regression over all $d_n+sn$ unknowns at once, and demands a full-rank condition on the combined actor–critic regression.

The paper's central trick is to reformulate everything in joint policy space alone. They bundle control and disturbance gains into a single matrix $\mathbf{\Theta }=\mathrm{col}(\mathbf{L},-\mathbf{K})$, and — through what they call a Sylvester lifting — construct a policy game Riccati equation (PGRE) whose only unknowns are policy gains. A theorem establishes a one-to-one correspondence between PGRE solutions and the symmetric solutions of the classical GARE: nothing is lost by dropping the critic.

The catch is that this clean construction requires an open-loop spectral condition — the matrix $\mathbf{A}$ must not share an eigenvalue with its own negative. That's a real restrictionting, so the paper introduces a stabilizing anchor: a fixed, known-stable policy ${\mathbf{\Theta }}_a$ that recenters the problem and removes the spectral requirement. Around this anchor, Newton updates proceed directly in actor space. A key technical result is that the Jacobian of the actor-space residual is nonsingular at every stabilizing policy, which makes each update well-posed — and, remarkably, the model-based policy sequence produced by this critic-free iteration coincides exactly with that of the classical simultaneous policy update algorithm (SPUA). They're computing the same trajectory of policies, but each step is far cheaper.

What They Found

The most striking finding is compositional: dropping the critic changes the type of object being iterated, not just the count of parameters. The PGRE is a system of $(m+q)n$ scalar polynomial equations of degree at most two in exactly that many unknowns — no more (Section III). For a power-systems frequency-regulation example, the paper reports substantial reductions in both computational time and memory requirements relative to the actor–critic baseline, with the advantage growing with problem size (Wu et al., 2026).

The second major result concerns unknown dynamics. When $\mathbf{A},{\mathbf{B}}_1,{\mathbf{B}}_2$ are unavailable, the authors turn the integral policy-evaluation identity into a data-driven actor-only regression. The mechanism is endpoint projection: they take a single batch of measured state trajectoriescation, build an integral identity, and project it onto the nullspace of the endpoint increments (Figure 2,

). This projection annihilates the value-matrix term before any regression happens — eliminating the critic before the least-squares step, not after. The resulting update involves only the joint policy gains.

The third, and conceptually deepest, result is about data informativity — when does data suffice to pin down the right answer? The paper derives a necessary and sufficient rank condition for uniquely recovering the policy from data, and proves it hinges only on the actor variables, entirely separate from whether the critic would be identifiable. Even more, this rank condition is iteration-invariant: if it holds at the first step, it holds at every step of the iteration. The dramatic upshot is stated plainly: critic identifiability is unnecessary for policy recovery. A value matrix that cannot be uniquely identified from the data can still leave the saddle-point policy perfectly pinned down.

This separation is the kind of structural insight that recasts an entire literature. Classic analysis (Lewis, Vrabie & Syrmos, 2012; Engwerda, 2005) treated actor and critic as locked together — you can't have one without the other, and the rank condition applied to the whole bloated regression. The Zhejiang group shows the two questions were always separable: you can ask "can I find the right gain?" without ever asking "can I find the value matrix?" — and the answer to the first can be yes even when the answer to the second is no.

Joint-policy unknowns of the critic-free scheme

Label	Value
n = 4	32 joint policy params
n = 8	128 joint policy params
n = 12	288 joint policy params

Joint policy unknowns s×n = n(n+1)/2×... for m+q=n; critic would add d_n = n(n+1)/2 more. Source: Wu et al. (2026), Sections I & III.

shows how the policy-gain unknowns scale with system dimension, and

Computational/memory reduction vs actor-critic

Label	Value
Classical actor-critic	100 relative memory
Critic-free PI	45 relative memory

Illustrative relative scaling from the paper's scalability tests on the frequency-regulation example (Wu et al., 2026), Section VI.

the associated computational reduction.

Why This Changes Things

The efficiency claim is not cosmetic. In reinforcement learning for control, the dominant cost in many deployments is the off-policy regression, and its memory footprint scales with the square of the parameter count. By cutting the unknowns from $d_n+sn$ to $sn$, the paper roughly halves the regression dimension in typical games — before the endpoint-projection and anchor tricks, which reduce it further. For an $n$-dimensional system with $m+q=n$ inputs, that's a reduction in the least-squares unknowns from $n^2$ down to $n^2/2$ — a constant factor that, in memory-bound embedded control (drones, satellites, grid inverters), translates directly into hardware-simpler requirements.

But the deeper contribution is conceptual, and it touches a live tension in modern control theory. Off-policy reinforcement learning works by reusing a fixed batch of behavior data to evaluate policies it never executed. The classical guarantee demanded that this data be persistently exciting enough to identify the full value function. The paper's iteration-invariant rank condition shows that the data only needs to be exciting in the policy directions that matter — a markedly weaker requirement. In the language of the paper, unique policy recovery "does not require the value matrix to be identifiable" (Wu et al., 2026).

There's resonance here with a broader recent trend in control: policy-only algebraic formulations for the single-player LQR setting (Fazel et al., 2018; Bu et al., 2019) and for $H_{\infty }$ control (Lin et al., 2024). But the zero-sum game is genuinely harder than the single-player problem in a specific technical sense: the weighting matrix ${\mathbf{R}}_g=\mathrm{diag}(-{\gamma }^2{\mathbf{I}}_q,\mathbf{R})$ is indefinite — it has both a negative block (the disturbance is rewarded for pushing) and a positive block (the control is punished for pulling). Indefiniteness breaks most of the monotonicity arguments that make single-player Riccati iteration flow smoothly. The authors' contribution is to show the policy-space machinery still works, via the anchored construction and the well-posedness of the actor Jacobian despite the indefinite weighting. That is a nontrivial technical lift.

The power-systems frequency-regulation example matters for another reason: it's not a toy. Frequency regulation is the real-time balancing act of the electricity grid — keeping the grid's 50 or 60 Hz signal steady as load and generation fluctuate constantly. Grid dynamics are large, distributed, and only partially modeled, which makes them a natural stress test for data-driven, model-free control. The paper's convergence and policy-recovery results on this example suggest the critic-free pipeline survives contact with a realistic engineering domain, not just textbook matrices.

Convergence of critic-free policy iteration

Label	Value
Iteration 1	1 policy error
Iteration 2	0.08 policy error
Iteration 3	0.006 policy error
Iteration 4	0.0004 policy error

Representative convergence trend in the power-systems frequency-regulation example; critic-free PI reproduces the model-based SPUA sequence (Wu et al., 2026), Sections IV & VI.

summarizes the convergence behavior across iterations.

What's Next

The one-to-one correspondence between PGRE and GARE solutions is established for the symmetric GARE solution set. But the GARE can have non-symmetric solutions too — the paper's theorem covers the symmetric ones, and the stabilizing saddle-point solution is the symmetric star of the show. Whether the full (non-symmetric) solution structure can also be lifted into policy space is a natural next question.

The iteration-invariant rank condition is a statement about offline data collected in one batch. The natural extension is to online or adaptive implementations, where the data informativity condition must be maintained as new measurements stream in and old data is dropped. The paper's Theorem shows the condition is invariant across stabilizing policy iterations; whether it survives the policy drift of online exploration is an open thread the authors gesture toward.

The most provocative avenue is the identifiability separation. If policy recovery genuinely does not require critic identifiability, then the standard practice of insisting on richly excited exploration — often expensive or physically intrusive in real plants, where aggressive probing risks instability or wears out actuators — may be over-conservative. A controller may need far less probing than classical persistence-of-excitation theory suggested, provided its exploration is aligned with the policy-relevant directions. That's a testable, practical hypothesis worth pushing into lab-scale experiments on real hardware.

There are also broader generalization questions. The machinery here is linear-quadratic; the real world is nonlinear. But the pattern — define the object you actually deploy (the policy), lift it into a self-contained equation via an anchor, and manufacture a data-driven regression that eliminates the redundant middleman by projection — is a template that may transfer to nonlinear value-free policy iteration, or to min-max formulations in multiplayer and multiobjective settings beyond two-player zero-sum games.

For now, the achievement is clean and the message is clear. In a zero-sum game between a controller and its disturbance, the winner never needed to carry the scorecard around. The policies — the actual decisions — were self-sufficient all along.

Figures

illustrates the endpoint-projection mechanism: state trajectories over sampling intervals (a), the nullspace-direction coefficients that annihilate the endpoint increments (b–c), and the cumulative critic endpoint terms that vanish once all intervals are combined (d) — the mechanical proof that the value matrix can be eliminated before regression.