← News
Tech for Good Tech for Good Frontiers

A small change in risk preference can turn a learning loop into a cycle

A small change in risk preference can turn a learning loop into a cycle
2x2 And 2x3 Games Game type studied
Conditional Value-At-Risk Risk measure tested
Learning Can Fail To Converge Key finding

In a game with only two actions for one player and three for the other, a learning process can keep cycling—even in continuous time—when the two-action player cares about downside risk. That is the paper’s sharp warning: the reassuring convergence results familiar from the smallest games can break after a modest increase in strategic complexity, and changing how an agent judges uncertainty can be enough to expose the break (Liang et al., 2026).

The result concerns fictitious play, a classic model of how independent agents learn while repeatedly interacting. Each player tracks how often the opponent has chosen each action, then best responds to those observed frequencies. There is no central coordinator and no need to know the opponent’s payoff table. In the standard version, “best” means the action with the highest expected payoff—the average payoff over the opponent’s empirical behavior.

Liang, Mohammadi, and Zhang ask what happens when agents keep the same learning rule but evaluate the possible payoffs differently. An agent might care especially about bad outcomes, not just the average. The authors call this nonlinear fictitious play, or NL-FP. Their analysis shows both sides of the change: in two-by-two games, useful convergence geometry often survives; in a two-by-three game, a risk-sensitive agent can make it fail.

The Science

A game is represented by its available actions and the payoffs each combination produces. In a two-by-two game, each player has two actions. In a two-by-n game, the first player has two and the second has n. The paper focuses on these settings because they are simple enough to analyze geometrically, yet rich enough to reveal when learning settles down and when it does not.

In classical fictitious play, a player looks at the empirical distribution of the opponent’s past actions. If the opponent has played left in 60 percent of observed rounds, for example, the player treats left as having probability 0.6 when evaluating a response. The empirical belief updates after every round, giving less weight to each new observation as the history grows. Both players perform this process independently.

NL-FP leaves that belief update alone. It changes only how an action’s payoff distribution is scored. Instead of taking its mean, a player can apply a nonlinear evaluation functional: a rule that maps a distribution of possible payoffs to a single evaluation. Conditional value-at-risk (CVaR) is one example. In this paper’s reward-maximization convention, CVaR emphasizes the lower tail of outcomes: it can favor a more reliable action over one with a higher average but an especially painful downside.

The authors define a corresponding equilibrium, called a ρ-Nash equilibrium. It is a strategy profile in which every player’s chosen action or mixture is a best response under that player’s own evaluation rule. When both evaluation rules are ordinary expectations, this reduces to standard Nash equilibrium. With nonlinear evaluations, however, a familiar equilibrium theorem cannot simply be assumed to apply: the way probabilities affect evaluations may change the best-response structure.

The main tool is geometry. In a two-by-two game, each player’s choice can be summarized by an advantage: the evaluated value of one action minus the evaluated value of the other. The sign of that advantage determines which action is preferred; zero means indifference. As the opponent’s observed frequency moves from zero to one, the advantage may cross zero, changing the best response.

The single-crossing property (SCP) restricts how this switching happens. Informally, it prevents the advantage from repeatedly changing sign as the opponent’s action probability varies. That preserves the broad shape of the classical two-by-two learning dynamics. The authors also introduce a weaker finite-crossing idea: even if the advantage does not obey single-crossing, it crosses the relevant thresholds only a finite number of times. Their continuous-time analysis follows the resulting regions of the strategy square, while the discrete-time analysis tracks the empirical-frequency updates on a finite planar grid.

What They Found

The first important result is positive. When advantage functions satisfy the single-crossing property, nonlinear fictitious play retains the classical convergence geometry in two-by-two games. The paper proves convergence results for both continuous-time dynamics and discrete-time learning, with the assumptions and selection rules specified in the relevant propositions. The researchers also provide sufficient conditions on evaluation functionals that guarantee single-crossing, while noting that common nonlinear criteria do not always meet those conditions.

That limitation does not end the convergence story. The authors show that in two-by-two games, a finite-crossing condition is enough for convergence of the selected continuous-time dynamics and of the discrete-time iterates. This reaches beyond the cases where single-crossing holds. Put simply, an agent’s evaluation can behave in a more complicated way than the classical theory allows, yet convergence can still follow if that complication changes best responses only finitely many times.

The result is not a guarantee for all nonlinear preferences. The paper’s counterexample moves to a two-by-three game. The player with two actions evaluates outcomes using CVaR, while the player with three actions is risk-neutral. In this setting, continuous-time NL-FP need not converge. The paper’s summary table marks discrete-time convergence in this particular counterexample as not established—not as a proven failure. That distinction matters: the result demonstrates a possible continuous-time breakdown, not every possible form of cycling in every version of the algorithm.

When convergence is established—and when it can fail

Summary of the paper’s stated convergence results by game size and condition. ✓ means convergence under the stated assumptions; ✗ means nonconvergence can occur; — means not established here.

When convergence is established—and when it can fail
LabelValue
2×2, strict single-crossing2 continuous and discrete results
2×2, single-crossing2 continuous and discrete results
2×2, finite-crossing2 continuous and discrete results
2×3, CVaR counterexample-1 continuous and discrete results
2×n, finite-crossing + nondegeneracy2 continuous and discrete results

The paper also extends positive conditions beyond two-by-two games. In two-by-n settings, when the row player’s best response depends on a linear statistic of the other player’s strategy, a finite-crossing condition can ensure convergence. The stated general result adds nondegeneracy assumptions and covers both continuous and discrete dynamics. The geometric analysis identifies basins of attraction—regions of starting beliefs that lead toward a particular equilibrium—and separatrices, boundaries dividing those regions.

For classical expected-utility fictitious play, this geometric account offers a new perspective on two-by-n games. Under stronger, relatively simple assumptions about how the column player’s best response switches in the interior, the authors obtain an alternative proof of the established convergence result for nondegenerate games. The contribution is therefore not only about risk-sensitive agents; it also reframes the landscape of a standard learning rule.

Why This Changes Things

The paper’s most useful lesson is that learning behavior depends on more than the update rule. Two agents can keep exactly the same records and respond to exactly the same empirical frequencies, yet reach different long-run behavior because they score risky outcomes differently. In practical terms, a model of adaptive decision-making that assumes everyone maximizes average payoff may miss whether behavior settles or cycles.

This matters wherever repeated decisions involve uncertainty and strategic feedback. In markets, an institution may judge a strategy by its downside exposure. In automated negotiation or multi-agent systems, a policy may be trained to avoid rare failures rather than maximize average reward. In such settings, the opponent’s behavior affects one’s payoff distribution, and one’s response feeds back into what the opponent observes. The paper provides a mathematical lens for asking whether that loop is likely to settle.

But the findings are theoretical, not evidence that a particular market or deployed AI system will cycle. The paper studies finite games with specified payoff structures and evaluation rules. Real applications may have noisy observations, changing payoffs, exploration, communication, or learning algorithms unlike fictitious play. The results identify mechanisms and conditions; they do not diagnose a real-world system without further modeling.

The contrast between two-by-two and two-by-three games is especially instructive. In the smallest setting, each player’s strategic response can be organized around a single threshold-like switch. Adding a third action gives one player more ways to reshape the other’s payoff distribution. A risk measure such as CVaR can then produce response patterns that undermine the simple geometry supporting convergence. This is not a claim that every extra action causes instability. It is a proof that the two-action intuition is not automatically safe to extrapolate.

Scope of the paper’s convergence map

Number of action profiles in the listed game dimensions, derived directly from their stated dimensions.

Scope of the paper’s convergence map
LabelValue
2×24 profiles
2×36 profiles
2×n2 profiles

The work also clarifies what equilibrium means when agents do not share expected-utility preferences. A risk-sensitive agent is not necessarily making an error by rejecting the highest-mean action; it is optimizing a different criterion. The corresponding ρ-equilibrium captures mutual best responses under those criteria. That is conceptually important for designing learning systems: stability should be judged relative to the objective agents actually use, rather than an assumed common average-payoff objective.

What’s Next

The paper leaves practical and theoretical questions open. Its counterexample establishes that continuous-time convergence can fail in a two-by-three setting, but the supplied summary does not quantify how often such failures arise across payoff matrices, or how sensitive the cycles are to the CVaR level and other details. Mapping out those boundaries would help distinguish a structural possibility from a common concern in applications.

The finite-crossing conditions are sufficient conditions, not a complete classification. A natural next step is to identify sharper tests: when can one tell directly from a payoff table and an evaluation functional whether the response advantage has finitely many crossings? Which commonly used risk measures satisfy the assumptions across broad classes of games, and which do not? The authors’ results suggest that the shape of the evaluation rule matters at least as much as its label.

Another open question is robustness. Real agents do not always select exact best responses, and empirical beliefs may be noisy or incomplete. It would be useful to know whether the convergence results persist under approximate responses, random tie-breaking, or perturbations of payoffs. Likewise, the paper studies a two-player structure; extensions to more players could reveal whether the geometric conditions survive when beliefs and feedback become higher-dimensional.

The forward-looking point is neither that risk-sensitive learning is doomed nor that convergence can be guaranteed with one elegant condition. It is that preference and dynamics are coupled. By showing precisely where classical geometry survives—and where a small expansion of the action set can break it—this paper gives researchers a way to ask a more grounded question before deploying a learning rule: not only what do agents want, but what shape does that desire give the feedback loop?