Why Checkpoint Storms Don't Actually Storm: The Dynamics of Shared Storage Among Identical Jobs

Two things get called a checkpoint storm. One, a single training job serializing thousands of GPUs at the same instant, is a design problem with known solutions — stagger the shards, drain asynchronously, add a buffer. The other, independent jobs colliding in shared storage without any explicit alignment, is the one that keeps operations teams up at night: bursts of correlated I/O tens of gigabytes strong, power transients that swing megawatts in sub-second windows, and the creeping worry that this is self-reinforcing — that jobs which once collide keep colliding, drifting into alignment the way pendulums on the same shelf eventually do.

A new paper shows that the second kind of storm isn't a storm at all — not in the way that matters. In a minimal mathematical model of identical jobs sharing anonymous resources, the mechanism everyone assumes is driving synchronization — contention — doesn't actually couple the jobs pairwise. The phase gap between two jobs stays exactly where it started, cycle after cycle, under storage contention, under a shared power cap, and under both at once. Two bodies that should attract or repel instead pass like ships. What's more, the synchronous state isn't an attractor: it's a fixed point with multiple expanding directions, and nearby trajectories fly apart from it rather than toward it.

"The two-body interaction on which the Kuramoto and Mirollo-Strogatz frameworks are built is not weak here but absent," write the authors. "Call a resource anonymous if the rate it delivers to an active user depends on how many users are active and not on which. For identical jobs whose write is shorter than their compute interval, an anonymous resource produces no pairwise coupling at all."

This is a paper about what doesn't happen — and what survives when the expected coupling vanishes.

The Science

The setting is a cluster running large-scale distributed training, where every job periodically checkpoints: it halts computation, serializes tens to hundreds of gigabytes of model state, and writes it to a parallel file system shared with dozens of other jobs. The fabric has finite bandwidth. The power infrastructure has finite headroom. And when multiple jobs checkpoint simultaneously — as they inevitably do by chance — they contend.

The operational assumption, baked into scheduling practice, is that this contention creates a feedback loop: a job that catches a slow write this cycle arrives at its next checkpoint later, and therefore closer to its neighbors; proximity breeds more contention; contention breeds more proximity, like traffic merging onto a highway. The proposed fix is to deliberately stagger jobs, to break the alignment before it starts. But if the system has an implicit attractor pulling jobs into phase, staggering is futile — contention will drag them back.

The paper asks the dynamical question that this assumption raises: is the synchronized state an attractor? If it is, the engineers are right to worry. If it isn't, they may be solving a problem that doesn't exist.

The tool is the language of phase-coupled oscillators. Each job is an integrate-and-fire unit: its phase is its progress through the inter-checkpoint interval, and its discharge is the checkpoint write. The jobs are coupled through the shared resource — storage bandwidth or power headroom — and the question becomes whether that coupling pushes them toward or away from synchrony.

The model is deliberately minimal. It contains three assumptions. First, blocking checkpoints: while a job writes, it doesn't compute. This is the standard synchronous checkpointing mode, the baseline that async mechanisms are designed to improve on. Second, anonymous equal sharing: each writer receives a share of the bandwidth that depends only on how many writers are active, not which ones. Storage bandwidth divided uniformly across concurrent streams. Power divided uniformly across a homogeneous fleet. Third, deterministic per-cycle work: the compute time between checkpoints is fixed and repeatable.

An anonymous resource is the key abstraction. The rate you get doesn't depend on which job you are; it depends only on how many jobs are talking. This is both a simplification and, the paper argues, a reasonable idealization of a well-configured parallel file system serving concurrent streams without priorities.

The analysis proceeds in orders. Order two examines two identical jobs in isolation. Order three examines all N jobs when their write windows fully overlap. Higher orders follow numerically.

What They Found

The two-body result: no coupling, no drift

The core theorem requires only that a job's write is shorter than its compute interval — that the checkpoint takes less time than the work between checkpoints, which is true for essentially all real training workloads. Under that condition, the return map of the phase gap is the identity: the gap at cycle k+1 equals the gap at cycle k, exactly, under storage contention, under a power cap, and under both simultaneously.

The proof rests on an accounting identity. Because phases are exclusive (blocking checkpoints), each job's time is partitioned into writing and computing. The key insight is that on intervals where one job writes and the other computes, the writing job drains at full rate (it has the fabric to itself) while the computing job advances at full rate (it has the compute to itself). Anonymity ensures that which job is alone doesn't matter — the rates are the same either way.

This symmetry means the difference in accumulated work between the two jobs depends only on their write offset, not on where they are in the cycle. And because the write offset is preserved — neither job gains a write on the other — the phase gap is preserved too. The two jobs translate rigidly through phase space; they never rotate toward or away from each other.

The authors call this phase-blindness: not weak coupling, but coupling that is identically zero. The function that measures the increment to the phase gap — denoted g(δ) — is constant, and exchange symmetry between identical jobs forces that constant to zero.

This holds regardless of the power cap. Whether the cap binds or not, whether one or both resources are shared, the pairwise term vanishes.

The three-body result: the map that survives

With three or more jobs, a third-order effect remains. When all N write windows overlap — which happens automatically if the jobs fire close enough together — the return map in interval coordinates is diagonal. The intervals between consecutive write starts evolve according to:

$$a_j^{\prime }=\frac{N-j}{j}{a}_j$$

where a_j is the gap between the j-th and (j+1)-st firing times, and N is the fleet size.

The map stretches the leading gaps (small j) and squeezes the trailing gaps (large j). The product of all eigenvalues is one, which follows from a conservation law built into anonymity: the total write time available per cycle is fixed, and the map preserves it. But this conservation is fragile. The product being one means the determinant of the linearized map is one — and a map with determinant one has exactly as many expanding directions as contracting ones.

For N identical jobs, the eigenvalues are λ_j = (N-j)/j. For a fleet of five, that's λ₁=4, λ₂=3/2, λ₃=2/3, λ₄=1/4. Two expanding, two contracting. The synchronous state — where all gaps are zero — is a fixed point with ceiling(N/2)−1 expanding directions. It is not an attractor. It is a saddle.

Eigenvalue spectrum of the N-writer return map (N=5)

Label	Value
λ₁	4
λ₂	1.5
λ₃	0.667
λ₄	0.25

Expanding directions in blue; contracting in gray. Reciprocal pairing about 1 preserves product = 1.

Figure 1 illustrates the mechanism for N=3. Jobs enter their write phase at intervals a₁ and a₂. Because a job that entered earlier banked a lead at a lower level of concurrency (when fewer jobs were writing) and cashes it out at a higher one (when more jobs finish), the exit intervals become a'₁ = (N-1)a₁ and a'₂ = a₂/(N-1). The leading gap stretches; the trailing gap squeezes; their product is preserved.

The eigenvalues are not just theoretical. The authors compute them numerically from the exact event-driven integrator and verify they match the prediction (N-j)/j precisely — including at the point where the linear approximation breaks down and the full nonlinear map must be used.

The invariant order: no trajectory reaches synchrony from outside

Beyond the dynamics of the gaps, the firing order itself is frozen. For any fleet size, any cap, any anonymous sharing rule, the cyclic order in which jobs fire is a constant of the motion. Jobs cannot pass each other. Distinct phases stay distinct.

This means no trajectory that starts away from synchrony can ever reach it. Synchrony is reachable only from synchrony. The basin of attraction is the synchronous state itself, and nothing else.

What the numbers say: trajectories don't cluster

The authors run numerical experiments across a range of fleet sizes (N=4 to N=16) and loads (L, the total write demand per unit compute time). "A fleet launched at random neither locks nor clusters." After hundreds of cycles:

• No decay of the smallest gap is resolved.
• The gap of the pair tightest at launch opens by a factor of 6 to 33 — not shrinks.
• Daido moments up to m=4 show no coherence that a change of null doesn't remove.
• The distance to the launch configuration (7–14% of a cycle) equals the distance to an independent launch relabelled into the same order sector, cell by cell and within a standard error. The 22–26% gap between unrelabelled configurations measures a permutation the dynamics cannot perform.

Trajectories remember only the invariant order, and nothing else of their launch.

Trajectory separation after 800 cycles

Label	Value
N=4, L=0.4	0.9
N=4, L=0.7	2.8
N=8, L=0.4	2.3
N=8, L=0.9	7.3
N=16, L=0.6	8.7

Separation grows with fleet size N and load L. Dashed line marks Lyapunov exponent = 0.

Separation of two trajectories launched 10⁻⁹ apart in one job's phase tells the same story. Over the cycles plotted, the separation gains 8.7 decades at N=16, L=0.6 and 7.3 at N=8, L=0.9, against 0.9 at N=4. The growth rises with fleet size and load. This is not the divergence you see near a stable fixed point — it's the separation from an unstable saddle, trajectories flying apart on the expanding eigenvectors.

The upper tail: bursts survive even where locking doesn't

Here is the result that complicates the headline. Absence of locking is not absence of bursts. The upper tail of the number of concurrent writers stays above its independent-phase value in every cell measured.

The three things that scheduling practice wants to prevent — asymptotic phase locking, transient clustering, and the extreme upper tail of concurrent writers — are not the same thing. Phase locking doesn't happen. Trajectories don't cluster. But the tail stays heavy. A fleet that neither locks nor clusters still has more moments where many jobs write simultaneously than you would expect from random phases.

"Storm" in the sense of the power transient — the extreme tail event that concerns facility stability — may survive even where "storm" in the sense of the dynamical attractor does not.

With jitter: the margin that erodes

Introduce per-cycle stochasticity σ to the compute time, and the deterministic picture softens. Each gap of a staggered fleet becomes a driftless random walk. A margin m survives 0.11 to 0.24(m/σ)² cycles, depending on the fleet.

Stagger survival under per-cycle jitter (m/σ = 10)

Label	Value
N=4	1.1
N=8	1.5
N=16	2.1

Survival cycles scale as 0.11–0.24(m/σ)² across two decades. Larger fleets are more robust.

Across two decades of the jitter budget, the scaling holds: the surviving cycles scale quadratically with the margin-to-noise ratio. The coefficient decreases with fleet size — larger fleets are more robust to jitter, because the gaps are larger relative to the noise. Small fleets with tight staggering are fragile.

This is the one place where the deterministic picture genuinely changes: randomness erodes the stagger. But it does so slowly. Even with m/σ = 10, a median fleet survives several hundred cycles before the smallest gap closes.

The boundary: where the result stops generalizing

The statement generalizes to identical jobs. It stops at heterogeneous jobs behind a binding cap.

When two jobs differ in write volume or compute work, and the power cap binds, they acquire a genuine pairwise coupling. The phase response is supported on gaps below the detuning. The coupling is real, not an artifact — and it is a property of the controller, not of the sharing itself.

A companion paper, on a shared power envelope with a delayed throttle, finds exactly this: a coupling whose sign is set by the control lag. The present paper removes the lag and the memory, keeps the sharing, and finds the coupling vanishes. The two are reconciled: the coupling in the throttle-constrained system is a property of the controller, not of the anonymous sharing.

Why This Changes Things

The assumption baked into checkpoint-storm mitigation — that collisions are self-reinforcing, that jobs drift into alignment — rests on a dynamical picture that the paper shows is wrong for the canonical case. Identical jobs behind anonymous resources don't couple pairwise. The feedback loop isn't there.

This matters for how teams think about scheduling. If contention doesn't drag jobs into phase, deliberate staggering isn't correcting a natural drift — it's imposing an order that the system would never achieve on its own. That may still be valuable, if the goal is to reduce the upper tail of concurrent writers (the burst probability). But it's not correcting a instability; it's fighting noise.

The distinction between three phenomena — phase locking, transient clustering, and burst tails — is the paper's most operationally important contribution. They come apart. A fleet can have heavy bursts without locking, locking without clustering, clustering without either. Mitigation strategies that treat them as the same problem may be solving the wrong one.

The power transient story is subtler. The extreme tail of concurrent writers — the thing that swings megawatts in sub-second windows — is not produced by phase locking in this model. It is produced by random coincidence, amplified by the fact that random coincidence is more likely when jobs are many and write windows are long. The mitigation target isn't an attractor; it's a heavy tail.

But "not an attractor" doesn't mean "ignore it." The upper tail stays above the independent-phase value even when locking is absent. Real fleets have power caps that bind, jobs that differ, jitter that accumulates. The model isolates the anonymous-sharing channel; real systems have others.

The three-order hierarchy is conceptually important. Pairwise coupling vanishes; the surviving effect is third-order, appearing only when three or more write windows overlap. The Mirollo-Strogatz framework, which proves synchronization for pulse-coupled oscillators with excitatory coupling, doesn't apply because the coupling isn't pairwise. The order of the interaction is what differs, and it differs for a reason rooted in the structure of anonymous sharing.

What's Next

The model is minimal by design. Three assumptions, proved where possible, verified numerically, stated explicitly where they fail. The most consequential is blocking checkpoints (Assumption 1): synchronous checkpointing, where the training loop halts while state is flushed. Asynchronous mechanisms, which decouple the write from the compute, are increasingly common — and the paper notes they have "nothing here bearing on" them. The dynamical question for async checkpointing is open.

The heterogeneous case behind a binding cap is the paper's own negative result, and it's a natural direction for future work: what does the coupling look like, quantitatively, when it exists? How does it scale with the cap severity, with the heterogeneity, with the fleet size?

The upper tail result is the most operationally relevant and the least theoretically developed. Why does the tail stay heavy even as locking disappears? The paper documents it but doesn't fully explain it — the mechanism seems to be that anonymous sharing doesn't suppress bursts, only randomize their distribution across cycles. Understanding the tail analytically, and characterizing when it matters for facility stability, is an open problem.

Finally, the gap between model and machine. The paper claims that symmetric contention between identical jobs cannot be the cause of synchronization in this model. It doesn't claim that real fleets never synchronize — only that this channel, in isolation, doesn't produce it. Real fleets have job placement, gang scheduling, thermal throttling, stragglers, and a dozen other channels. The model says the checkpoint-storm narrative, as usually stated, is incomplete. What the complete story looks like is a question for data.