When Momentum Hits Its Stride: Control Theory's New Proof for AI's Most Important Algorithm

A Controller's Eye on the Algorithm That Trains AI

Imagine you're trying to find the lowest point in a vast, fog-shrouded landscape. You can measure the slope beneath your feet, but you can't see the terrain ahead. You know that if you always move downhill, you'll eventually reach a valley—but how quickly you descend depends on whether you stumble carefully or charge recklessly downhill.

This is, in essence, the optimization problem at the heart of machine learning. When a neural network learns to recognize faces or translate languages, it's solving an optimization problem: finding the combination of weights that minimizes prediction error. The algorithm that performs this search—the one that decides how aggressively to step downhill and whether to carry momentum from previous steps—has been studied for decades.

But a pair of researchers from the University of Michigan and McGill University have now offered a surprising new perspective on these algorithms. Their work, published in August 2026, reframes the problem of proving convergence for momentum-based optimization methods through the lens of control theory—a field that studies how systems maintain stability and respond to inputs. By viewing optimization algorithms as feedback control systems, Sepehr Moalemi and James Richard Forbes have derived explicit conditions for when these algorithms will successfully find their target. Their results are more general than previous analyses, apply to a broader class of functions, and reveal that even well-studied algorithms like Nesterov's accelerated gradient still had unexplored theoretical territory.

The finding that will make practitioners lean forward: the researchers prove convergence guarantees for methods like Nesterov's accelerated gradient—a cornerstone of modern machine learning—applied to functions that were previously beyond the reach of such guarantees. These are functions with no curvature requirement at their minimum, meaning the landscape can flatten out in ways that traditionally caused algorithms to stumble. The team shows that under explicit conditions on stepsize and momentum, these algorithms will still find their target. And crucially, they can tell you exactly when to stop searching for bugs when your algorithm fails to converge.

The Language of Optimization

Before diving into the results, let's establish the vocabulary. An optimization problem, in its simplest form, asks: find the point x that minimizes a function f(x). In machine learning, f(x) measures prediction error, and x represents the neural network's weights.

The function f is assumed to be continuously differentiable—meaning we can compute its gradient, the vector pointing in the direction of steepest ascent. To minimize f, we want to move opposite to the gradient: downhill.

The most basic algorithm is gradient descent (GD): take a step proportional to the negative gradient. If the step is too large, you'll overshoot the minimum; too small, and convergence crawls. The stepsize parameter α controls this trade-off.

More sophisticated methods add momentum. The intuition is physical: if you've been rolling downhill, you have inertia. You don't want to change direction abruptly just because the slope shifted slightly. Polyak's heavy-ball method (HB) stores information from previous steps and carries it forward, like a ball accumulating speed as it rolls. The momentum parameter β controls how much previous direction influences the current step.

Nesterov's accelerated gradient (NAG), introduced by Yurii Nesterov in 1983, takes this further. Before computing the gradient, it first peeks ahead by taking a momentum step, then computes the gradient at that lookahead point. It's as if the ball looks a bit ahead to anticipate what's coming, rather than reacting to what's directly beneath it.

All three methods can be written in a unified framework that the paper calls the generalized accelerated gradient (GAG) method:

$${\mathbf{x}}^{k+1}={\mathbf{x}}^k+\beta ({\mathbf{x}}^k-{\mathbf{x}}^{k-1})-\alpha \nabla f({\mathbf{y}}^k)$$

where $\mathbf{y}^{k} = \mathbf{x}^{k} + \gamma(\mathbf{x}^{k} - \mathbf{x}^{k-1})$ is the point at which the gradient is evaluated. Setting β = γ = 0 recovers gradient descent; β > 0 and γ = 0 gives heavy-ball; β > 0 and γ = β gives Nesterov's method.

The Assumptions We Make About Functions

A crucial part of optimization theory is specifying what kinds of functions we're willing to consider. The difficulty of finding a minimum depends enormously on the landscape's shape.

The simplest case is L-smooth functions: those whose gradient changes at most as fast as a rate L. Picture a landscape where slopes can't become arbitrarily steep—no cliffs or spikes. This is a mild assumption that holds for most practical problems.

Strong convexity adds a second constraint: the function must be at least as curved as a parabola near any point, with parameter m capturing this minimum curvature. Intuitively, strong convexity ensures the function doesn't flatten out near minima—a crucial property because flat regions can cause gradient-based methods to slow to a crawl or oscillate.

Together, these assumptions define the class $\mathcal{S}_{m,L}$ of m-strongly convex, L-smooth functions. Most theoretical guarantees for momentum methods have required $m > 0$.

But here's the problem: in real machine learning, we often can't guarantee strong convexity. Neural network loss landscapes are notoriously non-convex and can have flat regions. Requiring m > 0 is a significant restriction.

Moalemi and Forbes work with a broader class of functions: those whose gradient satisfies something called sector bounds—mathematically denoted $\mathcal{F}{m,L}$. This class includes non-convex functions and is genuinely more general than $\mathcal{S}{m,L}$. The authors specifically tackle the most challenging case: $m = 0$, meaning no lower bound on curvature. The function can flatten arbitrarily close to its minimum.

They further introduce a subclass $\hat{\mathcal{F}}{0,L}$ with two additional properties: a unique stationary point (one global minimum, not multiple valleys), and the gradient doesn't become arbitrarily small far from that point. This second condition—formally $\lim\inf{|\mathbf{x}| \to \infty} |\nabla f(\mathbf{x})| > 0$—prevents the pathological case where the algorithm could drift infinitely far while the gradient still appears to be vanishing. It's a reasonable condition: as you get very far from the minimum, the function should eventually start climbing again, making the gradient non-negligible.

Reframing Optimization as Control

The paper's central insight is reframing the optimization algorithm as a feedback control system. This perspective has deep roots—the connection between optimization algorithms and control theory has been explored since the 1960s—but Moalemi and Forbes push it further.

Consider the optimization algorithm as a controller $\mathcal{G}$ that decides how to update the iterate $\mathbf{x}^k$. The function f, through its gradient, acts as a nonlinearity $\Delta$ that maps inputs to outputs. The two are connected in a feedback loop: the controller's output affects what gradient we compute, which affects the next controller action, and so on.

This formulation lets us import powerful tools from control theory. In particular, the paper uses passivity theory—a branch of control concerned with whether systems dissipate energy rather than amplifying it.

A system is passive if it can't generate more energy than it receives. An output strictly passive (OSP) system dissipates energy at some positive rate. These properties are desirable because interconnected passive systems tend to be stable: you can't build an amplifier that grows without bound from passive components.

The key technical contribution involves a loop transformation. The controller $\mathcal{G}$ in the optimization framework is