The WWII Slot-Machine Puzzle That Still Runs Adaptive Control

During World War II, the Allied war effort met its match in a slot machine. Not the literal kind—an abstract, mathematical one. British statisticians wrestling with how to allocate limited resources across uncertain options became so consumed by the puzzle that, according to the mathematician Peter Whittle, someone suggested "the problem be dropped over Germany, as the ultimate instrument of intellectual sabotage." The problem was the multi-armed bandit: you have several levers, each paying out at an unknown rate, and you must decide how to split your pulls between figuring out which lever is best and simply pulling the best one you've found so far.

That dilemma—explore or exploit—underpins everything from clinical trials to how animals forage for food. And in a new review by Tomas Meijer and Anders Rantzer of Lund University, slated for the Annual Review of Control, Robotics, and Autonomous Systems, it takes a sharper, more engineering-specific form. Their subject is dual control: the problem of controlling a physical system whose behavior you don't yet fully understand, where every control action both shapes the system and reveals information about it. It is a near-century-old puzzle that engineers still haven't cracked in full generality, and their review maps the four intellectual traditions that have tried.

The Science

Dual control is best understood through a concrete image. Imagine steering a ship whose steering response you only vaguely know—how much does it turn per degree of rudder, and how much does it drift? Every turn of the wheel is simultaneously an attempt to reach the destination and a measurement of the ship's behavior. Probe too aggressively and you waste fuel and risk overshooting; steer too cautiously and you never learn how the ship actually responds, so you remain lost.

The mathematics of this was first formalized in the early 1960s by the Russian scientist Alexander Feldbaum, who coined the term "dual control" for the way a controller must simultaneously regulate a plant and probe it to reduce uncertainty (Meijer & Rantzer, 2026). He understood that learning must often be active: without probing, you never learn how the system responds to your inputs. This makes dual control radically different from classical communication theory, where information arrives whether you ask for it or not. Here, information has a cost baked into every action.

The setup is deceptively simple (

). You have a linear time-invariant system:

$$x(t+1)=Ax(t)+Bu(t)+w(t)$$

where $x$ is the state, $u$ the control input, and $w$ a disturbance. The catch is that the matrices $(A,B)$ are initially unknown—they belong to some model class $\mathcal{M}$—and the controller must learn them from measurements while simultaneously regulating the system. A causal dual controller generates each input $u(t)$ from the history of measurements and past inputs, using that history to both act and learn.

Feldbaum attacked this using Bellman's dynamic programming, combining the physical state with an "information state" that tracked what the controller knew. But he immediately recognized that solving Bellman's equation exactly was computationally intractable, and that practical progress would require clever approximations. That computational wall has stood for sixty years.

What They Found

The review organizes the field into four distinct research traditions, each with its own assumptions, its own metrics, and its own blind spots (

).

The multi-armed bandit is the oldest and most theoretically mature strand. Formulated mathematically by William Thompson in 1933 for clinical trials, it asks how to choose among $N$ arms with unknown reward probabilities ${\theta }_i$. Two great machine approaches emerged. The Bayesian one, due to Gittins & Jones, assigns each arm a "Gittins index"—think of it as an effective per-unit-time reward under the assumption you can stop anytime—and proves that always pulling the arm with the highest index is optimal. Elegant, but computationally demanding.

The frequentist line, initiated by Lai & Robbins, introduced the concept of regret: the performance loss relative to a hypothetical policy that already knows the true parameters. Their landmark result proved that regret must grow at least as fast as $\log T$, and they built policies achieving that rate asymptotically. Later work by Agrawal simplified their "upper confidence bound" to a beautifully transparent form:

$${\hat{\theta }}_i(t)+\sqrt{(2\log t)/n_i(t)}$$

where ${\hat{\theta }}_i(t)$ is the empirical mean reward of arm $i$ and $n_i(t)$ the number of times it's been pulled. The square-root term is a measure of uncertainty—the less you've tried an arm, the larger the bonus it gets. Picking the arm maximizing this bound embodies "optimism in the face of uncertainty": you explore most when you know the least. Auer, Cesa-Bianchi & Fischer proved this simple scheme achieves logarithmic regret uniformly, not just in the asymptotic limit.

The self-tuning regulator, introduced by Åström & Wittenmark, is the engineering workhorse. Its idea is deliberately crude and effective: take the known minimum-variance controller for a system with known parameters, and simply swap in recursively-updated parameter estimates. The core recursion is least-squares estimation:

$$\hat{\theta }(t)=\hat{\theta }(t-1)+\frac{P(t-1)\varphi (t-1)}{1+{\varphi }^{\top }(t)P(t-1)\varphi (t)}\left[y(t)-{\varphi }^{\top }(t-1)\hat{\theta }(t-1)\right]$$

This became the dominant paradigm of adaptive control through the 1970s and 1980s and shipped in commercial products. But it wasn't designed to optimize the exploration–exploitation tradeoff—it just emerged as a side effect—and the review dissects where that shows. Minimum-variance control is a "dead-beat" scheme that cancels disturbances as fast as possible; at high sampling rates that produces overly aggressive control. It can even be internally unstable: in one simple example, the minimum-variance controller drives the input to grow exponentially even as it perfectly cancels the output disturbance. And if noise is colored rather than white, parameter estimates become biased, forcing extra conditions—like Ljung's positive-real requirement—for convergence.

Regret rate minimization for linear-quadratic control brings the statistical rigor of the bandit literature to full dynamical systems. The central insight, which the review credits to Lai & Wei, is that logarithmic regret is the generic outcome whenever the data you collect is rich enough to identify the optimal controller. But the interesting cases—those where genuine exploration is needed—produce a different curve: regret that grows as $\sqrt{T}$, not $\log T$. The distinction is fundamental. Logarithmic regret means your mistakes compound slowly and information accumulates efficiently; square-root regret means the price of learning compounds. This is the frontier where the exploration–exploitation tradeoff is most punishing.

Minimax optimal dual control is the fourth direction and, the authors argue, the most promising. Instead of assuming noise is stochastic with known statistics, it treats disturbances as adversarial and worst-case, framed as a zero-sum dynamic game between the controller and nature. This yields controllers that are robust to the kinds of colored noise and unmodelled dynamics that break the other approaches. But it's also the youngest field, and the review is candid that a fully general theory is still open.

Why This Changes Things

What makes this review more than a survey of old math is the way it reframes a surprisingly broad swath of modern technology. Every time an algorithm learns by doing—and every modern adaptive system does—it's solving a dual control problem.

Consider the self-tuning regulator's central practical lesson: persistence of excitation. A key insight, formalized by Åström & Bohlin in 1966, is that parameter estimation only works if the input data is rich enough. And here's the trap: if you use only the optimal controller, you collect almost no information about the alternatives. A perfectly tuned "exploiter" is an information vacuum. This is exactly the failure mode that stalks adaptive systems of every kind, from recommendation algorithms that get stuck showing you the same content to autonomous vehicles that never encounter the corner cases their models missed. Persistence of excitation is the mathematical name for the requirement that you keep some randomness in your actions or you stop learning.

The review's honesty about what self-tuning regulators get wrong is valuable. The "dead-beat" minimum-variance objective can make a closed-loop system internally unstable even when the open-loop plant is fine—an object lesson in how optimizing the wrong metric, or ignoring the plant's own poles and zeros, can quietly destroy robustness. And the colored-noise problem shows that the assumptions baked into your model are not decorations; they decide whether your controller converges or drifts into biased, suboptimal behavior.

The $\sqrt{T}$ versus $\log T$ distinction is the real intellectual payoff for the generalist. It's the difference between learning that accumulates gracefully and learning that costs you a measurable fraction of your performance no matter how long you run. When a controller can identify the optimal policy from intrinsic noise alone—no explicit probing needed—you get logarithmic regret, and the price of learning fades to nothing over time. But when you must deliberately sacrifice current performance to gain information, regret grows as $\sqrt{T}$, and the tradeoff never disappears. Identifying which regime a system falls into tells you whether exploration is a nuisance or the whole game.

The minimax direction speaks to a contemporary anxiety. Self-tuning and regret-minimizing controllers assume noise with known statistics; the real world delivers disturbances of unknown character. Worst-case, adversarial models of disturbance—borrowed from game theory—are the design philosophy behind systems that must work even when reality doesn't cooperate with your probability model. It's the intellectual cousin of robust control, and it's where the authors place their bets for the future.

What's Next

The review is forward-looking precisely because the fourth direction is unfinished. The state of the art is a set of powerful but partial tools: bandit theory that's complete only for static, memoryless problems; self-tuning regulators with proven convergence only under restrictive positive-real conditions; regret bounds that distinguish logarithmic from square-root growth but don't yet handle the full interplay of colored noise and required exploration; and a minimax framework in search of a complete theory.

The natural next steps, the authors suggest, lie in unifying these strands—borrowing the statistical concentration bounds from the regret literature and applying them within the worst-case minimax framework, and extending the minimax approach to the full linear-quadratic setting with concurrent estimation and regulation. The open questions are concrete: How much excitation is necessary, not just sufficient? Can logarithmic regret be recovered in the regimes where exploration is unavoidable? Does the minimax controller beat its stochastic-disturbance cousins on real, non-adversarial plants without giving up too much average performance?

There's also a bridge to modern practice that the review only gestures at. Dual model-predictive control—mentioned but not covered—already uses receding-horizon optimization to approximate the exploration–exploitation tradeoff while handling safety and actuator constraints. The theory in this review provides the foundation those practitioners are implicitly building on, and the terms of trade—how much safety you give up for learning, how much current reward for future information—are exactly what dual control makes precise.

Nearly a century after Thompson formalized the bandit problem, and more than sixty years after Feldbaum named dual control, the field remains at once mature and unfinished. What's satisfying is the coherence the authors reveal across four seemingly separate literatures. Ecologists watching animals decide whether to keep grazing or move on, statisticians allocating clinical trial patients, and control engineers steering uncertain ships are all solving versions of the same problem: how much to give up today so that tomorrow you know a little more. The review's real contribution is showing that these aren't parallel fields that happen to rhyme—they're four dialects of one language, and the frontier is where the fourth dialect is still being invented.