A shortcut through time could make climate models easier to teach
Training a physical simulation through time can consume so much memory that the model’s learning window has to be shortened. In tests spanning a chaotic toy system and models of turbulent ocean-like flows, a new approach cut peak memory by as much as an order of magnitude and runtime by as much as fourfold, while still supporting training at comparable accuracy. The key idea is not to stop gradients from reaching back through time. It is to calculate the exact derivatives at only a sparse set of points, then approximate what happens between them.
That distinction matters for geophysical models, where the parts we can resolve on a computer are only a fraction of the processes shaping weather and ocean circulation. Researchers increasingly combine physical equations with neural networks that fill in unresolved dynamics or correct a model’s bias. Such hybrid models need to be trained while coupled to the simulated physics. But doing so over long trajectories can demand a prohibitively large memory budget. González-Zamora and colleagues propose a way to ease that constraint without simply severing the model’s connection to its distant past.
The Science
A fluid simulation has to represent interactions across an enormous range of scales. A model may resolve broad currents or atmospheric patterns while leaving smaller eddies and turbulent exchanges unresolved. A traditional closure scheme approximates the net effect of those missing processes. A learned closure can instead use a neural network, trained from data, to estimate that effect.
The promise of a hybrid approach is that it keeps the structure and constraints of the physics-based model while letting a learned component handle what the existing equations capture poorly. The challenge is that a neural network trained alone may encounter unfamiliar conditions once coupled to a simulation. It can then behave unreliably or degrade a long-term forecast. Online learning addresses this by placing the learned component inside the dynamical model during training: the model runs forward, predictions are compared with a reference, and the network is adjusted based on the error accumulated over the simulated path (González-Zamora et al., 2026).
The adjustment relies on gradients. A gradient tells the learning algorithm how a small change in a model parameter would change the eventual error. In backpropagation through time (BPTT), the algorithm traces those effects backward from later states to earlier ones, applying the chain rule at each time step. Because the correct derivative depends on the intermediate states, exact BPTT generally needs to retain those states—or the information required to reconstruct them—through the backward calculation. For large models and long rollouts, memory grows with both the duration of the trajectory and the size of the simulated state.
One established workaround, truncated BPTT, limits how far backward the calculation reaches. It cuts a long trajectory into windows and discards sensitivity flowing across the boundaries. That reduces memory, but it also removes some long-range dependencies from the gradient. The new paper instead groups time steps into chunks and preserves the adjoint recurrence—the backward propagation of sensitivity—across the full trajectory. Within each chunk, it approximates the derivative operators, called pullbacks, that carry the sensitivities backward.
A pullback is a way of asking how a change in a later simulated state relates to changes in an earlier state or in a model parameter. The authors use exact automatic differentiation at one anchor state per chunk. Between anchors, their Zero-order method treats the state pullback as the identity: in effect, it assumes that the state sensitivity changes little over one small time step. Pullback Reuse instead applies the anchor’s pullback repeatedly throughout the chunk, reusing local dynamical information rather than recomputing it at every step.
The second ingredient is a computational shortcut. Since every time step within a chunk uses the same parameter pullback, the algorithm can add together the stepwise sensitivities first and apply the parameter derivative once to that sum. That replaces a parameter-pullback calculation at every time step with one per chunk.
The study tests the approach on three systems, from a low-dimensional incomplete Lorenz-63 model to a large-eddy simulation of two-dimensional quasi-geostrophic turbulence and a multilayer quasi-geostrophic ocean model. These are progressively more complex numerical systems, not operational forecasts of the full real-world climate. The authors compare gradient direction, training performance, runtime, and peak memory against exact BPTT and truncated BPTT.
What They Found
Across the three cases, the authors report that approximate gradients could support hybrid-model training at comparable accuracy. The largest reported reductions reached an order of magnitude in peak memory and a factor of four in runtime. Those are maximum gains across the tests, not a guarantee that every model, rollout length, or hardware setup will see the same improvement.
The central trade-off is between the frequency of exact anchor calculations and the distance between them. More anchors mean more exact pullbacks and generally a closer approximation, but less of the computational saving. Fewer anchors make the method cheaper, while increasing the opportunity for the local dynamics to change substantially between anchor states. The paper’s comparisons show that Pullback Reuse generally stays closer to the exact gradient direction than Zero-order, because it retains a frozen version of the local state dynamics instead of replacing them with the identity.
That gradient-direction comparison is useful, but it is not the same as proving that every training step improves the model. Optimization can tolerate some gradient error, and the eventual quality depends on the model, data, objective, and training procedure. The practical finding is more bounded: in these three test systems, the approximations were sufficient to train hybrid models with accuracy comparable to the exact-gradient runs, while using fewer resources.
The examples make the point concrete. In the double-periodic, single-layer quasi-geostrophic case, the learned component is a convolutional neural network (CNN) closure for unresolved turbulent effects. A figure compares the evolution of vorticity—the local spinning of the fluid—for the reference, an uncorrected model, and trained models. The Pullback Reuse result illustrates that an approximate gradient can still yield a useful learned correction over the simulated interval.
In the multilayer ocean-model case, the neural network is trained to correct model bias. The paper shows a final bottom-layer relative-vorticity field after five years for ground-truth, biased, and debiased simulations. The correction is trained using Pullback Reuse with a 12-step rollout and two anchor points, over 15 epochs. This is evidence that the method can be used in a more complex geophysical setting; it is not evidence that a five-year forecast has become reliable for real-world ocean prediction.
There is another distinction worth keeping in view: lower memory and faster backward passes can make longer training windows or finer model resolutions possible under a fixed budget, but the paper does not claim that these gains automatically translate into better forecasts. They create room to test training regimes that were otherwise too expensive. Whether that room leads to improved predictive skill is a separate scientific question.
Why This Changes Things
The most important conceptual move is to separate two ideas often bundled together in discussions of approximate gradients. One is preserving the chain of influence across the entire trajectory. The other is computing every local derivative exactly. Truncated BPTT sacrifices the first to save resources. Pullback Approximation methods retain the full backward recurrence but simplify local calculations. That can preserve long-range structure in the sensitivity even when the numerical value of the gradient is approximate.
The difference is especially relevant to geophysical simulations. Ocean and atmosphere models have huge state spaces: a state may contain values across many grid cells, layers, and physical variables. Storing all the information needed for exact backward differentiation can restrict how long a model is unrolled during training, or how fine its grid can be. If approximate pullbacks make those runs more affordable, researchers could explore hybrid models under longer horizons and at more realistic resolutions.
This is also a reminder that “exact” is not synonymous with “necessary.” Exact gradients are mathematically faithful to the specified simulation and loss, but the whole training system is itself an approximation of a complicated world. The numerical model has unresolved processes, observations have limitations, and the learned component may be imperfect. A gradient approximation that is cheaper yet sufficiently informative could be a sensible engineering choice—provided its consequences are measured rather than assumed.
Pullback Reuse appears particularly promising because it occupies a middle ground. Zero-order propagation makes the cheapest assumption: nearby state sensitivities are nearly unchanged. Reuse retains an estimate of how the dynamics transform those sensitivities, frozen at the anchor. The paper reports that this method most closely tracks the exact gradient direction among its proposed approximations. That does not mean it will always be superior; its accuracy depends on how much the local dynamics vary within a chunk.
The work also has a potentially broader use beyond geophysical modeling. Any differentiable dynamical system that is trained through long sequences faces a version of the same memory problem: a simulator, a control system, or a learned model of a physical process. The specific performance results here belong to the tested systems, but the underlying question travels: how much of an exact backward calculation can be replaced while still learning what matters?
What’s Next
The authors’ results leave several practical questions open. First, the best chunk size and anchor spacing may depend on the regime. A stable flow and a rapidly changing turbulent state need not tolerate the same amount of reuse. Adaptive methods that place anchors more densely where the dynamics change quickly could help, but would need to be evaluated against the extra complexity they introduce.
Second, comparable accuracy in three test cases is a promising demonstration, not a broad validation across operational weather and ocean models. Those systems bring larger domains, varied forcing, imperfect observations, and strong demands for stability over long integrations. Testing the method at those scales will show whether the memory savings persist and whether approximate gradients remain useful when errors compound over much longer periods.
Third, a gradient that points in roughly the right direction at a given stage may still interact differently with the optimizer over many updates. Future work should examine robustness across random initializations, training objectives, rollout lengths, and parameter choices, as well as the consequences for forecast skill and physical consistency. It will also matter how the method compares with checkpointing, which trades extra recomputation for lower storage, and with other ways of managing the backward pass.
For now, the paper offers a carefully scoped but consequential result: online learning for hybrid simulations may not need a perfectly exact derivative at every time step. By computing exact pullbacks at sparse anchors and carrying approximate sensitivities between them, researchers can preserve the long trajectory’s influence while using substantially less memory and time in the tested systems. That does not solve the challenge of modeling the atmosphere or ocean. It may, however, make it easier to train the models that attempt to do so.