The AI That Decides When to Fix a Broken Server

The Problem No One Talks About When Your Favorite App Stays Up

Every time you check your bank balance, stream a show, or send a message, invisible machinery is working to keep that service running. Somewhere in a data center, tens or hundreds of computers are running copies of the same software simultaneously. This redundancy—running multiple identical copies, or replicas, of a service—is how the internet stays online despite constant hardware failures, cyberattacks, and the occasional caffeine-spilled-on-a-server incident.

The mathematics behind this redundancy are elegant: if you have N replicas and no more than f of them fail at once (where f < N), the system keeps humming along. Clients never notice. The magic, though, is in what happens after something goes wrong. A replica crashes. A cyberattack corrupts memory. A power outage in one geographic region takes out two machines at once. Now you need to fix those broken replicas before enough of them fail that the whole system collapses.

This is called recovery control, and it sounds straightforward until you try to do it well. Do you fix replicas immediately? Wait until you've accumulated several failures? Check on them periodically, say, every Tuesday at 3 AM? Each approach has obvious flaws. Fix too aggressively and you waste resources recovering replicas that weren't actually broken. Fix too slowly and you risk cascading failures—a chain reaction where one crashed replica increases the stress on the others, making them more likely to crash too. Add to this the fact that you often can't even see what state a replica is in with perfect clarity, and you have a genuinely hard problem that the industry mostly solves with blunt heuristics.

Kim Hammar of Imperial College London and Yuchao Li of Arizona State University have a better idea. In a new paper, they formulate replica recovery as a type of sequential decision-making problem that mathematics has tools to optimize—but they add a crucial twist that makes it actually work in real systems. Their approach, called autonomous multiagent rollout with signaling, reduces recovery costs while scaling to systems with dozens of replicas. In experiments across both simulations and a physical testbed, their method outperformed the recovery policies that most companies use today.

The result isn't just an academic exercise. As our infrastructure increasingly relies on distributed systems that must stay online continuously—think autonomous vehicles coordinating in real-time, or AI services answering queries with sub-second latency—the stakes of getting recovery right keep climbing. This paper offers a principled path forward.

The Science: Modeling the Invisible Machine

What is a Replicated System, Really?

To understand Hammar and Li's contribution, it helps to build a mental model of what's actually happening when you access a "cloud service." In the simplest setup, you have N servers, each running the same software. When you make a request—say, to check your email—the system queries all N servers and relies on a consensus protocol to determine the correct answer. The consensus protocol is essentially a voting mechanism: as long as fewer than half the servers are malfunctioning (or, more precisely, fewer than N are Byzantine—that is, behaving arbitrarily badly), the system can determine the truth and respond correctly to you.

This setup is called Byzantine fault tolerance, named after a thought experiment in distributed computing about generals coordinating an attack. In practice, it means the system is robust to a wide range of failures: a server that stops responding entirely, one that's been compromised by malware and sends back gibberish, or one that's simply running outdated code. As long as the number of such faulty replicas stays below the threshold f, you—the client—never know anything went wrong.

The challenge arises when faulty replicas accumulate faster than they can be fixed. The system can tolerate f simultaneous failures, but if new failures outpace recoveries, the system eventually hits a tipping point where service degrades or stops entirely. The research question, then, is: when should you initiate recovery of a given replica?

Why This Is Surprisingly Hard

If you could observe everything perfectly, recovery control might be straightforward. If replica #7 just crashed, recover it now. If it's been running fine for a year, don't bother. But here's the complication that makes the problem genuinely hard: you can't see the full picture.

Each replica's control agent receives only partial observations—failure alerts from anomaly detectors, network timeouts, error logs. These signals are noisy. A replica might be healthy but generating false-positive alerts due to a misconfigured monitoring system. Or a compromised replica might be cleverly masking its faults, sending back responses that look valid to the monitoring system but are actually incorrect. Distinguishing a truly faulty replica from a healthy one that just had network hiccups, or from a malicious replica that's gaming the system, requires inference under uncertainty.

And failures aren't independent events. This is a crucial point that many existing approaches miss. Two replicas might be running on hardware in the same data center, so a power outage or cooling failure takes both down simultaneously. They might run the same operating system, so a zero-day exploit affects both at once. They might be part of the same software update rollout, so a buggy patch corrupts both. The authors capture this via a failure dependency matrix A, where entry A_ji = 1 means that replica j's failure makes replica i more likely to fail. (See Figure 3 in the paper, which visualizes how dependencies between replicas affect failure probabilities over time.)

This interdependence creates cascading risk. If you observe that replica #3 has failed, you now know that replica #4—which shares a network switch with #3—is more likely to fail soon too. Smart recovery decisions must account for these elevated risks.

Framing It as a POMDP

Hammar and Li's first contribution is formal: they show that the recovery control problem can be precisely formulated as a partially observable Markov decision problem (POMDP) with a multiagent structure. If you're not familiar with POMDPs, here's what you need to know: it's a mathematical framework for making decisions when you can't observe the full state of the world, but where you can maintain a belief—a probability distribution over possible states, updated rationally as you receive new observations.

The authors define:

• State: A vector x_k = (x_k^1, ..., x_k^N) where x_k^i = 1 if replica i is faulty and 0 otherwise. (So the "state" is just which replicas are currently broken.)

• Control: A vector u_k = (u_k^1, ..., u_k^N) where u_k^i = 1 means "recover replica i" and u_k^i = 0 means "wait." Each replica's local agent makes this decision for its replica.

• Observation: Each agent receives z_k^i, the number of failure alerts related to its replica at time k. This is an imperfect signal about whether the replica is actually faulty.

• Cost function: The objective trades off three types of costs:

  • Failure cost: When a replica is faulty (x=1) and you don't recover it (u=0), you incur η per time step
  • Recovery cost: When you recover a replica that's actually healthy (x=0, u=1), you incur a cost
  • Disruption cost: If too many replicas are simultaneously faulty or being recovered—so the total exceeds the tolerance threshold f—the entire service is disrupted, incurring a large cost λ

The cost function (Equation 2 in the paper) balances these competing pressures. You want to recover faulty replicas before they cascade into system-wide disruption, but you don't want to waste resources recovering healthy replicas or create disruption by taking too many replicas offline for recovery simultaneously.

The transition dynamics (how the state evolves) encode the failure dependencies: the probability that replica i fails at the next time step is min{p_F(1 + Σ_j A_ji x_j^k), 1}, where p_F is a base failure rate and the sum captures how other replicas' failures increase i's risk.

From POMDP to Multiagent Rollout

Formulating the problem as a POMDP is satisfying because it establishes the correct mathematical structure, but it doesn't solve anything. The optimal policy for a POMDP with N replicas would require exploring a control space of size 2^N (every possible combination of recover/wait decisions across all replicas) and a belief space with 2^N dimensions (every possible combination of which replicas are faulty). For a system with 10 replicas, that's 1,024 possible controls to consider. For 20 replicas, it's over a million. For 50 replicas, it's more than the number of atoms in the observable universe.

The standard approach to handling POMDPs is rollout, which starts with a base policy (even a simple one, like "recover every replica every Tuesday") and improves it by simulating futures. The rollout policy evaluates what would happen if you chose each possible action now, using the base policy to approximate what you'd do later, and picks the action with the lowest expected total cost. Bertsekas developed this approach in the 1990s, and it has a theoretical guarantee: the rollout policy is always at least as good as the base policy, and typically better.

The problem with standard rollout for this setting is that computing the optimal control requires centralized coordination. To decide whether to recover replica #7, you'd need to know what decisions you're making for replicas #1 through #6 and #8 through #N simultaneously, because all those decisions interact—recovering too many at once creates disruption, or recovering a cluster of interdependent replicas might miss a cascade.

Hammar and Li's solution is multiagent rollout, which decomposes the centralized problem into N separate subproblems, one per replica. Here's how it works:

1. The agents decide sequentially: agent 1 picks u^1, then agent 2 picks u^2 given what agent 1 chose, then agent 3 picks u^3 given both previous choices, and so on.

2. Each agent performs a lookahead optimization: it simulates what would happen over the next ℓ time steps if it chose u^i = 0 versus u^i = 1, using the base policy to approximate the other agents' future decisions.

3. The agent picks the action (recover or wait) with the lower expected cost.

The sequential decomposition reduces the computational complexity from O(2^N) to O(N). For 10 replicas, that's 10 decisions instead of 1,024. For 20 replicas, 20 decisions instead of over a million. This is a massive practical improvement.

But there's a catch: this sequential approach requires agents to wait for previous agents' decisions before computing their own. In a system with 70 replicas, that could mean 70 sequential rounds of computation per time step—impractical if decisions need to be made quickly.

The Signaling Innovation

This is where the paper's key innovation comes in: autonomous multiagent rollout with signaling. The insight is that if agents could know what the previous agent decided without waiting for explicit communication, they could compute their actions in parallel.

Hammar and Li achieve this by precomputing a signaling structure. Before runtime, the system computes a mapping from each agent's observation (the failure alerts it receives) to a prediction of what the previous agents would do under the base policy. At runtime, each agent observes its signals, looks up its precomputed prediction, and proceeds as if it knew the previous decisions—without actually needing to wait for those decisions to arrive.

This is the "autonomous" in autonomous multiagent rollout: each agent acts independently based on its local observation, guided by precomputed information. The coordination is implicit rather than explicit, which dramatically reduces communication overhead and enables parallelism.

What They Found: Numbers That Matter

The paper validates the approach through both extensive simulation and experiments on a physical testbed. The testbed consists of physical servers running service replicas in virtual containers, with the researchers able to inject realistic failures—cyberattacks, network partitions, power outages—and measure the resulting failure alerts. They calibrated their observation model (how likely an agent is to receive 0, 1, 2, ... alerts given the true underlying state) from real measurements on this hardware. (See Figure 6 in the paper for the empirical observation distributions, which vary across replicas based on their individual characteristics.)

Scalability Results

The most striking result is about scalability. Using multiagent rollout with signaling, the authors were able to compute recovery decisions for systems with up to 70 replicas—something that would be computationally intractable with standard rollout.

Exponential Growth of Control Space Size

Label	Value
10	1,024 replicas
20	1,048,576 replicas
30	1,073,741,824 replicas
40	1,099,511,627,776 replicas
50	1,125,899,906,842,624 replicas

Source: Figure 4 from Hammar & Li (2026) — Size of control space = 2^N. Standard rollout requires exploring this entire space; multiagent rollout reduces this to O(N) sequential decisions.

Figure 4 in the paper illustrates why this matters. The state space of a system with N replicas has 2^N possible states (every combination of healthy/faulty), and the control space has 2^N possible actions (every combination of recover/wait decisions). Standard rollout requires exploring a tree of possible futures with branching factor 2^N |Z|, where |Z| is the number of possible observations. For even modest N, this becomes astronomically large.

Multiagent rollout with signaling sidesteps this by having each agent consider only its own action (binary: recover or wait) rather than the full joint action space. The precomputed signaling allows agents to approximate what other agents would do without exploring all 2^N possibilities.

Figure 8 in the paper directly compares compute times across different rollout methods. The key result is that multiagent rollout with signaling scales linearly in the number of replicas, while standard rollout (even the sequential variant) scales exponentially. For 70 replicas, standard approaches would take hours or days to compute a single decision. With signaling, the computation takes seconds.

Cost Reduction vs. Current Practice

The more important practical question is: does this method actually outperform the recovery policies currently used in production systems? The answer, according to both simulation and testbed experiments, is yes.

The authors compared their multiagent rollout policy against two baselines:

• Periodic recovery: Every replica is recovered on a fixed schedule (e.g., every week), regardless of actual health state. This is a common approach in practice because it's simple to implement.

• Threshold-based recovery: Replicas are recovered when some monitoring metric crosses a threshold (e.g., when a replica generates more than K error messages in an hour).

The results show that multiagent rollout achieves lower total cost across a range of failure intensities and system sizes. The cost reduction comes from two sources:

1. Avoiding unnecessary recoveries: Periodic recovery wastes resources on healthy replicas. The multiagent policy initiates recovery only when the expected benefit—reduced failure risk, faster return to full capacity—exceeds the cost of recovery.

2. Proactive cascade prevention: By accounting for failure dependencies, the policy can anticipate and prevent cascading failures. If replica #3 just failed and it's strongly dependent on replica #4, the policy might prioritize recovering #4 before it fails too, even if #4's current health indicators look acceptable.

The authors report specific cost improvements, but the exact numbers depend on the weighting parameters η (cost of an unmitigated failure) and λ (cost of service disruption). In scenarios with high failure rates and high disruption costs, the improvement over periodic recovery can be substantial—reducing total operational costs by tens of percent or more.

Theoretical Guarantees

The paper also provides theoretical conditions under which the multiagent rollout policy is guaranteed to improve upon the base policy. The key result (Proposition 1 in the paper) states that the rollout policy approximately improves the base policy with error at most α^ℓ ε, where α is the discount factor (how much the future is valued relative to the present), ℓ is the lookahead horizon (how many steps ahead the agent simulates), and ε is a parameter capturing how well the cost-to-go is approximated.

This is important because rollout methods have a theoretical property called person-by-person optimality: no single agent can improve its local decision without making others worse off, given what the other agents actually do. This doesn't guarantee global optimality (you might get stuck in a local optimum where everyone could improve if they coordinated), but it does guarantee that the solution is stable and no individual agent has an incentive to deviate.

Why This Changes Things: From Heuristics to Principles

The State of Practice Today

Most production systems today use variations on two themes for recovery control: periodic recovery or threshold-based recovery. These approaches are simple to implement and easy to understand, which is why they're pervasive. An operator sets a schedule (recover every replica every Sunday at 2 AM) or configures a threshold (alert me if error rate > 5%), and the system runs.

But these approaches have fundamental limitations that become increasingly costly as systems scale and failure modes become more complex.

Periodic recovery wastes resources by recovering healthy replicas. It also responds too slowly to actual failures—a replica that fails on Monday isn't recovered until Sunday. And it can't account for correlated failures: if you're in the middle of a DDoS attack that's taking down replicas across your system, recovering them on a fixed weekly schedule is useless.

Threshold-based recovery is more responsive but still reactive rather than predictive. A threshold is crossed after something goes wrong. Moreover, threshold tuning is notoriously difficult: set it too low and you get false positives (recovering healthy replicas); set it too high and you miss real failures until they've cascaded.

Neither approach accounts for failure dependencies. If your replica #12 and replica #13 are on the same power circuit, a circuit breaker trip takes both down simultaneously. A threshold-based policy would see two independent "medium-severity" failures and respond with two independent recoveries. A smarter policy would recognize the correlation and recover them together, or prioritize based on their shared vulnerability.

What Changes with Optimal-ish Recovery

Hammar and Li's approach represents a shift from heuristic to principled recovery control. Instead of rules of thumb, you have an optimization framework that:

1. Accounts for uncertainty in failure detection
2. Models dependencies between replica failures
3. Balances multiple costs (failure, recovery, disruption) according to configurable weights
4. Adapts to varying failure patterns rather than following a fixed schedule

For large-scale systems, this principled approach can yield substantial cost savings. But the more important benefit might be robustness: when failure patterns shift (a new cyberattack vector emerges, a software update introduces correlated bugs, a supply chain issue affects a specific hardware model), a principled approach adapts while heuristics continue making the same mistakes.

Consider a concrete scenario: a cloud provider runs a service on 50 replicas across multiple data centers. A zero-day exploit is discovered in a widely-used library. Within hours, replicas start failing—but they're not failing independently. The exploit affects replicas running the vulnerable version of the library, which happens to be all of them in one geographic region.

A periodic recovery policy still recovers replicas on Tuesday morning, regardless. A threshold-based policy waits for each replica's error count to exceed its threshold individually. A multiagent rollout policy, knowing that replicas in the affected region are interdependent and that the failure of one increases the probability of others, might immediately prioritize recovery of all vulnerable replicas—preventing a cascade before it starts.

Limitations and Caveats

The paper is careful to acknowledge limitations, and a responsible digest should highlight them too.

First, the method requires an accurate model of failure dynamics and observation probabilities. In practice, these models are approximations. The failure dependency matrix A must be calibrated somehow—either from historical data, domain expertise, or online learning. The observation model p(z|x) was calibrated from testbed measurements in this paper, but real-world systems may have different alert characteristics.

Second, the theoretical guarantee (approximate cost improvement) depends on the quality of the cost-to-go approximation J~. If this approximation is poor, the guarantee degrades. The paper suggests several approaches for obtaining this approximation (aggregation methods, deep learning), but tuning these approximations for specific systems remains an engineering challenge.

Third, the paper assumes a partially synchronous communication model—a standard assumption in distributed systems, but one that may not hold in extremely adversarial network conditions. If network partitions persist longer than expected, the signaling precomputation may become stale.

Fourth, the experiments were conducted on systems with up to 70 replicas. Whether the approach scales to systems with hundreds or thousands of replicas—an increasingly common scale for major cloud services—remains an open question. The authors note that for very large systems, additional approximations or hierarchical decomposition would likely be needed.

Finally, the paper focuses on recovery control and doesn't address recovery mechanisms—how to actually execute a recovery (restarting with a new OS, re-provisioning from a golden image, etc.). The method assumes that recoveries can be initiated quickly and reliably, which may not hold for all system architectures.

What's Next: Open Questions and Future Directions

Closing the Gap Between Theory and Production

The most immediate next step is deploying this approach in real production systems and measuring its performance in vivo. Testbeds can only simulate a subset of real-world failure modes, and production traffic has dynamics that are hard to replicate in the lab. A deployment study would also provide data for refining the failure and observation models—potentially using online learning to adapt them to the specific system's characteristics over time.

Handling Larger Systems

Seventy replicas is impressive, but major cloud services routinely run on thousands of nodes. Extending the approach to these scales likely requires hierarchical decomposition: grouping replicas into clusters, applying multiagent rollout within clusters, and coordinating at the cluster level. The authors hint at this direction but don't pursue it in the current paper.

Another approach is to relax the requirement for precomputed signaling across all replicas. Perhaps some replicas (those that rarely fail, or whose failures don't affect others) can use simple policies, while only the critical replicas employ the full multiagent rollout framework.

Learning Failure Dependencies

The current formulation assumes the failure dependency matrix A is known. In practice, this may not be the case—or the dependencies may change over time as the system evolves. An important extension would be to learn the dependency structure from data, using techniques from causal inference or graphical models. This would make the approach more robust and adaptive.

Integration with Consensus Protocols

The paper assumes a consensus protocol is already in place and focuses on recovery control above that layer. But there's potential for tighter integration: if the consensus protocol signals that replicas are struggling to reach agreement (perhaps because several are slow or sending inconsistent messages), this could inform recovery priorities. Similarly, if recovery control knows something about upcoming consensus operations (e.g., a scheduled configuration change that will require consensus), it could plan recoveries to avoid disrupting critical operations.

Extensions Beyond Recovery

While the paper focuses on recovery control, the multiagent rollout with signaling framework is general and could apply to other decision problems in replicated systems. The authors mention that similar techniques have been applied to warehouse robot coordination, fleet repair scheduling, taxi routing, and antenna tilt optimization. For replicated computing systems, relevant extensions might include:

• Load balancing: deciding how to route client requests across healthy replicas
• Scaling: when to add or remove replicas based on demand forecasts
• Maintenance scheduling: coordinating updates to replicas without disrupting service

Each of these has similar structure—multiple agents making local decisions with partial information, with system-level costs that depend on the joint action.

The Broader Significance

Hammar and Li's work sits at the intersection of distributed systems, control theory, and machine learning—a place where progress can have broad impact. As our society increasingly relies on always-on digital infrastructure, the stakes of keeping that infrastructure running correctly keep growing. A hospital's AI diagnostic system that goes down loses more than a server; it potentially delays patient care. An autonomous vehicle whose perception system lags loses more than a subscription; it risks a collision.

The traditional approach to reliability has been to throw hardware at the problem: more replicas, more geographic redundancy, more monitoring. This works up to a point, but it has diminishing returns and significant cost. The alternative is to make the software that manages that hardware smarter—to use principled decision-making rather than blunt heuristics.

This paper is a step in that direction. It shows that the recovery control problem has enough structure to be modeled precisely and solved approximately with strong theoretical guarantees. It demonstrates that the solution scales to realistic system sizes and outperforms current practice in experiments. And it does so with an approach that's extensible—multiagent rollout with signaling can accommodate different failure models, cost structures, and system architectures.

The gap from this paper to production deployment is still significant—model calibration, integration with existing systems, operational engineering. But the foundation is solid. For the invisible machinery that keeps the internet running, this is a promising development.

The next time your favorite app stays up during a major outage—while other services crumble around it—somewhere in the background, a well-designed recovery policy might be doing exactly what Hammar and Li described: making smart decisions about when to act, based on partial information, in a system where the stakes of getting it wrong are very high.

---

This digest is based on "Recovery Control in Replicated Systems through Autonomous Multiagent Rollout" by Kim Hammar and Yuchao Li, published on arXiv in July 2026. The full paper is available at [arXiv reference pending]. Code and testbed measurements are available at https://github.com/Kim- Hammar/csle.