The Synchronization Trap: How Your Data Center's Power Cap Might Be Turning AI Training Jobs Against Each Other

When Your Data Center's Power Management Could Wake Up Like a Swarm of Fireflies

Imagine a data center running a hundred AI training jobs. Each job — a large language model being fine-tuned, a vision system retrained on new data, a foundation model scaling up — cycles through the same pattern: a burst of heavy computation, then a pause to exchange gradients across thousands of accelerators, then another compute burst. The grid sees this as a quasi-periodic load, pulsing up and down like a heartbeat.

Now here's what nobody had properly asked before: when all these independent jobs share one power envelope — when the facility has, say, 500 megawatts of power rights but a thousand megawatts worth of GPUs crammed into the building — what happens to their cycles? Do they stay scrambled relative to each other, their power swings averaging out like waves from different boats on a lake? Or can they somehow fall into step with each other, their pulses aligning until the whole facility swings in unison?

The answer, according to a new paper by Brieuc Le Roux Tardif, is that they can absolutely synchronize — and the mechanism is hiding in plain sight inside the facility's own power-management stack. The jobs don't couple through the grid's frequency (which would be the obvious analogy to power-line synchronization). They couple through throttling. When the aggregate load gets too high and the power cap bites, every job slows down a little. That slowdown, acting only during compute-bound phases, creates a feedback loop that can pull jobs toward each other. Pulled hard enough, they phase-lock. And once locked, the facility's power swings stop being the manageable √N ripple of independent cycles and become a coherent N-sized wave crashing against the envelope — potentially triggering protection circuits, violating interconnection agreements, or even feeding instabilities back into the grid.

The striking part: the default behavior is actually the opposite of dangerous. At short control delays — the fast regime of electrical power capping — the coupling is repulsive. Jobs push each other apart. This is good. The system naturally spreads them out. Danger emerges only when the total phase lag in the control loop exceeds half an iteration cycle, at which point the coupling flips sign and starts pulling jobs together. The transition to the locked state is often first-order and hysteretic: a nominally safe fleet can be nudged into synchronization by a brief surge, and once there, it stays.

This matters enormously as data centers scale toward gigawatt campuses filled with fleets of near-identical training runs. The paper provides operators with something concrete: a dimensional analysis of whether they're in the safe or dangerous regime, a monitoring protocol using quantities computable from existing telemetry, and a software-only mitigation that costs only the small throughput hit of deliberately detuning the jobs' phases apart.

The Science

The setup

Modern AI training is power-hungry in a specific way. A cluster running a large training job doesn't draw constant power; it pulses. Tens of thousands of GPUs execute optimization steps in near-lockstep: a compute-bound interval where every chip is blazing through matrix multiplications at thermal design power (call it P^hi), followed by a communication-bound interval where they stall on collective operations — gradient reductions and all-reduce operations — and draw much less (P^lo). The cycle repeats roughly every 2 to 66 seconds for the duration of the run, sometimes millions of iterations.

The contrast is significant: P^hi can be two to three times P^lo. At hyperscale — a facility running hundreds to thousands of GPUs — the peak-to-peak swing reaches tens to hundreds of megawatts, with ramp rates exceeding 10 MW per second.

Two trends make this a pressing concern. Single campuses are scaling toward the gigawatt level. And those campuses are increasingly filled with near-identical workloads — replicated foundation-model training runs, fleets of fine-tunes on the same architecture, experiments with slight hyperparameter variations. The same iteration waveform, repeated across the site.

The existing literature treats each facility as an exogenous periodic forcing on the electrical grid. Proposed mitigations act either on the grid side (adding storage to absorb the swings) or on the facility side (reshaping each job's power profile individually through workload insertion or millisecond-scale GPU power smoothing). Tardif takes the other seat. He asks: before the aggregate swing ever reaches a grid mode, what happens inside the operator's own infrastructure — the oversubscribed power envelope, the power-delivery hardware sized for a certain profile, the interconnection agreement that bounds ramp and swing at the point of common coupling?

The implicit assumption in most capacity planning — that when N jobs coexist, their forcing terms superpose with statistically independent phases — is not a modeling convenience. It's a capacity-planning premise. Under it, aggregate fluctuation grows only as √N, partial cancellation keeps swings manageable even at large N, and oversubscription is safe. The paper asks whether this premise holds, and finds it doesn't — not because of something external, but because of the operator's own power-management stack.

The modeling approach

Tardif frames the problem in the language of nonlinear dynamics and synchronization theory, specifically the Kuramoto paradigm. This is the same mathematical framework used to describe synchronization in pendulum clocks, firefly swarms, Josephson junction arrays, and power grids. The basic idea: each job is modeled as a phase oscillator with a natural frequency (its iteration rate). The collective state is measured by an order parameter r, ranging from 0 (phases uniformly spread, incoherent) to 1 (all phases aligned, perfectly synchronized). Synchronization occurs when the coupling between oscillators is strong enough to overcome their natural frequency differences.

The paper builds a model with several layers of abstraction, each justified and its cost acknowledged.

First, the two-level waveform: each job alternates between P^hi and P^lo at duty cycle δ, the fraction of the iteration spent compute-bound. This is a relaxation oscillation — essentially a square wave — as opposed to the sinusoidal limit cycles for which the classical Kuramoto model is exact. The square wave carries rich harmonic content; each Fourier component couples independently and can have a different sign. This turns out to be consequential.

Second, the coupling mechanism. Tardif shows that grid frequency cannot couple the jobs: accelerator clocks sit behind rectification and regulation, insulated from line frequency variations. The operative channel is instead load-dependent throttling: enforced power caps, voltage droop, and shared cooling all slow computation when aggregate demand is high. This makes each job's step rate a function of every other job's instantaneous power — a closed loop that the operator owns and can observe.

Third, the reduction to a generalized Kuramoto system. The throttling response is linearized around the operating point, yielding a Sakaguchi-Daido coupling function whose shape is fixed by the training waveform convolved with the throttling response. The analysis focuses on the weak-coupling, slow-phase regime, valid when the cap binds a modest fraction of the time and throttling is a small perturbation.

The key equation governing each job's phase evolution is:

$${\dot{\phi }}_i={\omega }_i+\frac{K}{N}\sum _{j=1}^N\Gamma ({\phi }_j-{\phi }_i)$$

where Γ is the coupling function encoding the throttling mechanism. The coupling strength K scales with the cap stiffness and the power contrast; the coupling function's Fourier coefficients determine whether interaction is attractive or repulsive.

The numerical study

The analysis is analytical and simulation-based, with no facility data. The numerical study uses direct simulation of the unaveraged delay-differential equation (DDE) model, testing the leading-order analytical predictions against the full nonlinear dynamics. Seven experiments probe different aspects: the per-harmonic sign law, coherence scaling, splay-state stability, the first-order onset and its hysteresis, higher-harmonic cluster locking, the effect of rate heterogeneity, and the phase-scattering mitigation. Parameters are chosen to bracket realistic operating regimes; the focus is on falsifiability rather than exhaustive parameter sweeps.

What They Found

The stake, made quantitative

The most consequential result is the scaling. Under statistical independence — the benign assumption that jobs' phases are uncorrelated — the aggregate power fluctuation grows as √N. The residual coherence in a large incoherent fleet scales down as 1/√N, so adding more jobs actually helps: the aggregate gets smoother relative to the mean load. This is why oversubscription works in the first place, and why N^0.5 is the number operators implicitly trust when they size their power envelopes.

But if the jobs synchronize — if the order parameter r climbs toward 1 — the coherent part of the aggregate swings with amplitude proportional to N, not √N. The relative swing amplification is r√N: a fleet of a thousand jobs with r=0.5 would swing with roughly sixteen times the amplitude of the incoherent baseline. At the locked state (r=1), the amplification is √N. This isn't a modeling artifact; it's an exact consequence of how coherent addition works.

The paper quantifies this in

Scaling of Aggregate Power Fluctuation with Fleet Size

Label	Value
Independent phases (K=0)	0.48 relative fluctuation
Repulsive coupling	0.35 relative fluctuation
Attractive coupling (coherent)	1 relative fluctuation

Figure regenerated from simulation data (e3_amplification.csv). Repulsive coupling is 16–35% below the uncoupled control; attractive coupling scales linearly with N.

. The independent-phases case produces a ripple that averages out; the locked case produces a coherent wave with amplitude proportional to the number of jobs. The dangerous state is the one where everything moves together.

Scaling of Aggregate Power Fluctuation with Fleet Size

Label	Value
Independent phases (K=0)	0.48 relative fluctuation
Repulsive coupling	0.35 relative fluctuation
Attractive coupling (coherent)	1 relative fluctuation

Figure regenerated from simulation data (e3_amplification.csv). Repulsive coupling is 16–35% below the uncoupled control; attractive coupling scales linearly with N.

The sign law: why the default is safe

At leading order, the coupling is repulsive. This is the most counterintuitive and most important finding. Jobs don't naturally synchronize — they push each other apart, favoring splay states where phases are evenly distributed around the cycle. In a splay state, when some jobs are in their compute-bound (high-power) phase, others are in their communication-bound (low-power) phase. The aggregate stays near the mean, and the facility never sees the full peak-to-peak swing from all jobs simultaneously.

Repulsion arises from the interaction being gated by compute-bound phases. The throttle signal — slowing everything when aggregate demand is high — acts only on jobs that are currently in their compute phase (the phase gate χ_i(φ_i) = 1). A job that's idling on collectives feels the throttle signal but doesn't respond to it, because it's already in the low-power state. This creates a phase-dependent coupling that, at fundamental frequencies, tends to spread jobs apart rather than bunch them.

The repulsive default is fragile, however. The sign of the fundamental coupling coefficient a_1 depends on the control delay τ through the relationship:

$$a^{(1)}(\tau )\propto -\sin (\bar{\omega }\tau )$$

where ω̄ is the mean iteration frequency. At short delays — the fast electrical capping regime, where ω̄τ ≈ 0 — the sine is approximately zero and the coupling is weakly repulsive. As τ grows, the sine becomes more negative, repulsion strengthens, then at τ* = T/2 (half an iteration cycle), it passes through -1 and begins reversing. Beyond τ*, the coupling becomes attractive. The sign flips at:

$${\tau }^{\ast }=\frac{T}{2}=\frac{\pi }{\bar{\omega }}$$

This delay budget is partitioned among several stages: polling and enforcement intervals, sample-and-hold, actuation latency, and cascaded slow stages (thermal, cooling). Fast electrical stages contribute phase bounded by π/2 and cannot invert the sign on their own. Dead time accumulates phase without bound and can cross τ*. The bottom line: if your throttling control loop has accumulated enough delay to exceed half an iteration cycle, the coupling flips from protective to dangerous.

Figure 2 illustrates this per-harmonic sign law. At ω̄τ = π/2 (left dotted line), the fundamental is maximally repulsive while the third harmonic is maximally attractive. At ω̄τ = π (right dotted line), the fundamental reverses sign and becomes attractive. No delay is repulsive at every mode simultaneously.

Higher harmonics: when repulsion isn't enough

The square-ish waveform of training loads carries harmonics beyond the fundamental. And each harmonic has its own sign law:

$$a^{(m)}(\tau )\propto -\sin (m\bar{\omega }\tau )$$

This means a fleet can be repulsive at the fundamental frequency (so jobs avoid locking in sync) but attractive at higher harmonics. If a fleet is sufficiently homogeneous — all jobs running at nearly the same iteration rate — it can lock into a cluster state where every job is synchronized, not at the fundamental period, but at a submultiple. For example, a second-harmonic lock would have all jobs aligned at twice the iteration frequency: they'd be synchronized but completing two complete power cycles per training iteration. This is still dangerous, because the aggregate still swings with amplitude proportional to N.

The mode thresholds for harmonic locking are:

$$K_c^{(m)}=\frac{2}{\pi a^{(m)}(\tau )g(0)}$$

Rate diversity is required to avoid this: a heterogeneous fleet (different iteration times due to different model sizes, parallelization strategies, or batch sizes) has a broader distribution of natural frequencies, which raises all mode thresholds. No choice of control delay can substitute for diversity. The paper explicitly states this as a design requirement.

Synchronization Thresholds: Repulsion vs. Attraction and Hysteresis

Label	Value
Repulsive (ω̄τ = π/4)	0 K/Kc
Attractive (ω̄τ = 3π/2)	1 K/Kc
Near-identical (γ=0.05)	1 K/Kc
Diverse (γ=0.20) - Forward	1 K/Kc
Diverse (γ=0.20) - Backward	0.6 K/Kc

Figure regenerated from simulation data (e7_loop_gamma*.csv). Diverse fleets show a bistable window (shaded region) with hysteresis spanning K/Kc from 0.6 to 1.0.

The onset: first-order and hysteretic

The transition from incoherent to locked is not always gradual. For sufficiently diverse fleets with moderate control delays (γτ = O(1)), the onset becomes first-order — a discontinuous jump — and hysteretic.

The mechanism is frequency-correlated frustration. The phase lag of the control loop creates a frustration term α_j = ω_jτ + π/2 in each oscillator's dynamics. This frustration is frequency-dependent: jobs at different iteration rates experience different effective phases. When this frequency-correlated frustration is strong enough, it bends the bifurcation branch subcritical. Instead of smoothly growing from r=0, the system can have two stable states — incoherent (r≈0) and locked (r≈1) — coexisting over a range of coupling strengths. A fleet below threshold can be tipped into the locked state by a transient perturbation, and once there, it stays even if the perturbation is removed.

This is illustrated in

. In the supercritical case (short delays), r grows continuously from zero as coupling increases. In the subcritical case (moderate delays and diverse rates), the bifurcation diagram shows a fold: the incoherent branch loses stability at one coupling strength, the locked branch at a lower one, and in between, both states are stable. The fleet can jump from incoherent to locked, or from locked to incoherent, depending on history.

The hysteresis is intrinsic — it requires no inertia, which is fortunate given that data-center loads carry negligible mechanical inertia. The frequency-correlated frustration is enough. The hysteresis loop width, measured in simulations, reaches up to 68% of the critical coupling in diverse fleets — a substantial bistable region where the system's fate depends on initial conditions and perturbations rather than on parameters alone.

The scaling experiments

Two numerical experiments illustrate the stakes most clearly.

Experiment 1 (E1): The order parameter r_1 was measured across one period of the control delay for a diverse fleet (γ=0.20), comparing the unaveraged DDE model against the derived coupling coefficient a_1(τ) = -sin(ω̄τ). The result: the fleet was incoherent across the repulsive window (where a_1 < 0) and locked throughout the attractive window (where a_1 > 0), with the reversal occurring at the predicted τ* = T/2. Two elevated features at the exactly-reactive points (τ=0, π) were explained by boundary-layer effects where the fast tail of the rate distribution nucleates coherence ahead of the bulk.

Experiment 3 (E3): Aggregate power fluctuation was divided by √N and plotted against fleet size for three cases: an attractive coupling case (τ=3π/2), a repulsive coupling case (τ=π/4), and an uncoupled K=0 control. The attractive fleet followed the coherent ∝N guide, confirming linear scaling. The uncoupled control sat just under the theoretical independent-phases line, as expected. The repulsive fleet was flat in N and 16–35% below the control — meaning it was actively suppressing coherence below the independent baseline, not merely failing to amplify it.

Why This Changes Things

The implications for data-center design

Data centers are increasingly designed around oversubscription. An operator with 500 MW of power rights doesn't buy 500 MW worth of GPUs; they buy more, banking on the statistical argument that not all jobs will hit peak power simultaneously. This works — it has always worked — because the implicit assumption was that jobs are independent.

Tardif's analysis shows that this independence cannot be taken for granted. The power-management stack that enforces the cap is also the coupling channel. When it has enough delay, it can flip from naturally spreading jobs apart to actively pulling them together. The conditions for this flip are specific: enough accumulated dead time in the control loop, enough homogeneity in the job mix, and enough tightness of the oversubscription (so the cap binds frequently enough to maintain the coupling).

The gigawatt-scale facility — which is coming — amplifies both the opportunity and the risk. More jobs means more coherent amplitude if they lock. But more importantly, it means more near-identical waveforms: foundation-model runs at the same scale on the same architecture, fine-tunes that share the same parallelization strategy. The same waveform repeated N times is exactly the input that maximizes the order parameter once even weak coupling locks in.

What this means for grid stability

The paper defers the detailed grid implications to companion work, but it notes one concern: if a locked fleet swings coherently at its iteration frequency, and if that frequency falls near a grid inter-area mode, the coherent forcing could excite the mode. This is speculative — the grid interaction is complex and depends on the facility's location, the grid topology, and the relative impedance. But it's a coupling direction the existing literature hasn't considered, because the literature treats each facility as an independent exogenous forcing. If the forcing is coherent — if every facility on the same power domain is swinging together — the aggregate forcing is larger and more structured than assumed.

The more immediate concern is the point of common coupling: the interconnection agreement that bounds ramp and swing at the facility's grid connection. A locked fleet swinging with amplitude ∝ N instead of ∝ √N can violate these bounds even when the mean load is unchanged. The violation would be intermittent — appearing whenever the fleet happens to synchronize — which makes it harder to diagnose and mitigate than a steady overload.

The shift from grid problem to operator problem

Perhaps the most significant reframing is conceptual. The existing literature treats the facility as a source of grid instability; the proposed fixes live on the grid side (storage, grid-forming inverters) or act on individual jobs (per-job power smoothing). Tardif relocates the problem. The coupling mechanism — the reason jobs can synchronize — lives entirely within the operator's domain. Every element is owned, configured, and observable: cap policy, control-loop bandwidth, cooling infrastructure, job mix.

This is simultaneously more alarming and more tractable. Alarming, because the operator may be engineering their own instability. Tractable, because the same knowledge that identifies the hazard identifies the levers to control it. You can't modify grid topology or install grid-scale storage from inside a data center. But you can look at your control-loop delays, tune your job mix, and implement phase-scattering in the scheduler.

What's Next

The falsification test

The paper is explicit that its predictions are falsifiable. The most direct test is a two-job co-capped measurement: run two training jobs behind the same power cap, measure their phases over time, and compute the order parameter. If the coupling is repulsive (as predicted at short delays), the phase difference should wander uniformly and the order parameter should stay near zero. If the coupling is attractive (as predicted at long delays), the phase difference should lock and the order parameter should approach one. A controlled measurement of two jobs is simple enough to be feasible and sufficiently unambiguous to be conclusive.

Monitoring obligations

For operators, the paper specifies what to watch. The order parameter r and its first few harmonics r_m are computable from per-job power telemetry — the power draw of each job is observable, and from it one can estimate the phases and their coherence. The monitoring obligation after transients (job starts, stops, and reschedules) is to ensure r settles back toward zero, not toward one. A fleet that stays coherent after a perturbation is in the locked state, and unless the coupling is reduced or the job mix diversified, it will remain there.

The hysteresis means the question isn't just "are we below threshold?" but "have we been tipped into the locked state?" A fleet at K = 0.6 K_c may be nominally safe — below the forward critical coupling — but if it was previously pushed above the backward critical coupling, it can remain locked below threshold. This converts a static sizing criterion into a monitoring obligation.

The mitigation: phase-scattering

The paper proposes phase-scattering scheduling as a software-only mitigation. The idea is to deliberately detune and offset training jobs — imposing fixed phase differences — so that even if the coupling is attractive, the fleet sits in a stable splay state rather than a locked one. At first order, the throughput cost is:

$$\Delta C=\frac{\pi }{8}\frac{{\kappa }^2}{K^2}\bar{\omega }$$

where κ is the scattering strength. This is a small cost for modest scattering: if you impose phase offsets with a spread of κ/K in the coupling units, the cost scales as (κ/K)^2. For a fleet that is close to but not past the critical coupling, scattering keeps it below threshold at a modest efficiency cost.

Phase-scattering acts on the relative phases of the fleet, not on individual power profiles. This makes it complementary to existing software mitigations like per-job GPU smoothing. It lives at the scheduler — the operator already runs one — and needs no additional hardware.

Open questions and extensions

The analysis deliberately stays at leading order, in the mean-field limit, with a two-level waveform and retention of the fundamental harmonic. Each of these simplifications has a cost:

Waveform realism: Modern frameworks overlap collectives with computation, raising the low-power level and shrinking the power contrast. This rescales the coupling strength downward — which is safer — but multi-pulse structure (several collectives per step, micro-batching) can inject harmonics that change the higher-mode dynamics. The square wave is a genuine upper bound at m=1, but at m≥2, the bounds are waveform-specific.

Topology: The mean-field assumption (all-to-all coupling through a shared cap) is a tractable worst case. Real facilities are hierarchical — racks, PDUs, data halls, campus — with local caps and global aggregation. Block-local coupling dilutes synchronization compared to mean-field, but modular topologies can support local cluster resonances that a mean-field analysis misses. Whether hierarchical coupling is generically safer or can introduce new instabilities is an open question.

Amplitude dynamics: The paper models only the phase channel — throttling affects speed, not power. But throttling simultaneously reduces power draw, which is a negative feedback on the aggregate that should damp high swings. This is heuristically conservative (the analysis may overstate the risk), but the amplitude channel carries its own lag and could shift the effective frustration. A full treatment is left to future work.

The amplitude channel interaction: The paper notes that including the power-reduction side of throttling is an open extension. The heuristic expectation is that it would damp aggregate swings, but the expectation isn't a proof.

Grid interaction: The paper defers detailed analysis of how a locked fleet might force grid inter-area modes. This is the most consequential extension, because it would move the problem from the operator's domain back to the grid's — but with the twist that the grid sees not a collection of independent oscillators but a coherent multi-megawatt load.

The bigger picture

There is something quietly profound about a facility's power-management system accidentally implementing a form of coupled oscillation. The Kuramoto model was developed to explain synchronization in physical and biological systems — fireflies blinking in unison, superconducting Josephson junctions phase-locking, power grids with synchronous generators. The last case is the most directly relevant: grids synchronize because generators share the line frequency, and deviations from that frequency carry information about load imbalances. The corrective torque acts on rotor angles, pulling machines toward sync.

Data centers don't share line frequency with their GPUs. But they share something else — a power cap, a cooling system, a voltage bus — and the throttling that enforces shared resources creates a different kind of coupling. It's slower (iteration timescales rather than grid cycles), more structured (the training waveform's duty cycle and harmonics), and more controllable (every element is a configuration choice). The analogy to the Kuramoto model isn't loose; the mathematical structure is genuinely similar, and the tools developed for that setting transfer.

What changes is the agency. Grid synchronization is a physical necessity; you can't opt out of sharing frequency with every other generator on the network. Data-center synchronization is a design artifact — an emergent property of how oversubscription, throttling, and job homogeneity happen to interact. That means it can be designed away. Phase-scattering is one approach; others might involve architectural choices (hierarchical caps, asymmetric throttling), control design (faster loops, predictive enforcement), or operational practice (deliberate rate diversity in job scheduling).

The paper's framing — operator's question, not grid's question — points toward a future where data centers are understood not just as loads to be managed but as complex dynamical systems with their own internal synchronization modes. The analogy to power grids cuts both ways: just as grids can have blackouts triggered by synchronization failures, data centers can have stability issues triggered by phase-locking. The good news is that the data-center operator has more levers, because the coupling lives entirely within their domain. The challenge is knowing which levers to pull, and when. That is what this analysis provides.

---

Key takeaways:

• When multiple AI training jobs share a power cap, they can spontaneously synchronize their power cycles through load-dependent throttling — the power-management stack itself creates the coupling.
• The default coupling is repulsive (jobs naturally spread apart), flipping to attractive only when control-loop delay exceeds half an iteration cycle; this means fast electrical capping is safe, but cascaded slow stages can push into the dangerous regime.
• Synchronization changes aggregate power swing scaling from √N to N — a fleet of 1,000 jobs can swing 30× more than the incoherent baseline once locked.
• The transition is often first-order and hysteretic: a nominally safe fleet can be tipped into the locked state and stay there, requiring monitoring of the order parameter after transients.
• Phase-scattering scheduling — deliberately detuning job phases — raises all mode thresholds at once and offers a software-only mitigation at modest throughput cost.
• Rate diversity is not optional: a homogeneous fleet can lock into higher-harmonic cluster states even when the fundamental coupling is repulsive.