The Best of Both Worlds: A New Way to Forecast When You Can't See Everything
The Science
Imagine trying to track a hurricane using only a handful of pressure sensors scattered across the Atlantic. Those sensors measure temperature, pressure, and wind speed directly—but the hurricane's eye, the spiral bands of rain, the outflow aloft, the ocean heat beneath it: none of these are directly measured. Yet a forecaster must produce a coherent picture of the entire storm, including the parts no sensor touches.
This is the central problem in data assimilation: how do you estimate the complete state of a complex, chaotic system when you can only observe some of it? The challenge comes up everywhere—in weather forecasting, ocean modeling, climate science, and increasingly in machine learning applications. You have a numerical model that simulates how the system evolves, and you have sporadic, incomplete observations. The trick is to combine them intelligently.
The approach that has dominated for decades is the Ensemble Kalman Filter, or EnKF. The idea is elegant: instead of tracking a single best guess of the system state, you track a whole ensemble of possibilities—dozens or hundreds of slightly different simulations. This ensemble captures your uncertainty. When an observation comes in, the EnKF uses the statistical relationships encoded in that ensemble to figure out how the observation should adjust not just the observed variables, but everything else too. If temperature and pressure tend to rise and fall together in your model, then a temperature observation effectively tells you something about pressure, even if you never measured it directly.
The problem is that EnKF assumes these relationships are linear. It assumes that the probability distributions involved are Gaussian, that small changes produce proportionally small effects, and that the way observations map onto the state is straightforward. These assumptions work remarkably well in many settings—but they break down when the system is highly nonlinear, when observations involve complicated transformations, or when the uncertainty is genuinely non-Gaussian.
A newer family of methods attacks this limitation using ideas from generative AI. Rather than assuming Gaussian shapes, these methods can represent and sample from complex, irregular probability distributions. One such approach is the Ensemble Score Filter, or EnSF. Instead of training a neural network—which would be computationally expensive and require massive datasets—EnSF estimates the shape of the probability distribution directly from your forecast ensemble. When an observation arrives, it corrects the state by following the gradient of that distribution, kind of like a ball rolling downhill to find the lowest point. This lets it handle nonlinear observation operators and non-Gaussian uncertainties that would confuse a standard EnKF.
But EnSF has a blind spot. When observations only measure part of the system, the correction step acts only on the observed variables. The unobserved parts get nudged only indirectly, through the prior information already encoded in the ensemble. In sparse observation networks—which are the norm, not the exception, in geophysical applications—this means unobserved variables can accumulate errors that the model dynamics alone cannot erase.
A team from Florida State University and Oak Ridge National Laboratory has now developed a way to fix this. Their method, called the Ensemble Score Filter with Linear Regression (EnSF-LR), combines the strengths of both approaches. First, it uses EnSF's nonlinear machinery to correct the observed variables as accurately as possible. Then—and this is the key innovation—it uses the same cross-variable regression step that makes EnKF work to spread those corrections to the unobserved parts of the state. It is, in effect, a two-step procedure: correct what you see, then use statistical relationships to fix what you cannot.
The researchers—Zixiang Xiong, Feng Bao, Hristo Chipilski, Siming Liang, and Jingqiao Tang—describe their approach as "hybridizing score-based and EnKF analysis schemes." It is a bridge between two traditions that have largely developed in parallel: the generative modeling community's tools for handling complex distributions, and the data assimilation community's tools for propagating information across state variables.
What They Found
The team tested EnSF-LR on two standard benchmark systems that appear throughout the data assimilation literature: the Lorenz-63 model and the Lorenz-96 model. These are simplified chaotic systems—the Lorenz-63 model, for instance, has just three variables and produces the famous two-lobed butterfly attractor. They are not realistic enough to represent actual weather, but they capture the essential difficulty: sensitive dependence on initial conditions, nonlinear coupling between variables, and chaotic dynamics that amplify small errors rapidly.
The experiments were designed to stress-test EnSF-LR in two distinct scenarios. In the first, observations are linear functions of the state—a direct measurement of a variable, perhaps with some noise added. In the second, observations are nonlinear: the relationship between what is measured and the underlying state is curved, asymmetric, or otherwise complicated. This second case is where traditional EnKF struggles most, because its linear assumptions fail most dramatically.
The researchers used root-mean-square error (RMSE) as their primary metric—essentially, how far the estimated state tends to drift from the truth, on average, across many independent simulation runs. They ran twenty independent experiments for each configuration to account for the inherent randomness in ensemble methods, and they computed confidence intervals to distinguish genuine advantages from statistical noise.
In the linear-observation experiments, EnSF-LR performed about as well as the classical EnKF baseline. This is a meaningful result: it means the hybrid approach does not sacrifice performance in the simpler case where EnKF is already well-suited. Meanwhile, EnSF-LR substantially outperformed the original EnSF, which struggled to propagate its accurate observations of the measured variables to the unmeasured ones.
Full-State RMSE: Linear Observations
| Label | Value |
|---|---|
| EnKF (baseline) | 0.7 |
| EnSF | 1.2 |
| EnSF-LR | 0.7 |
The advantages of EnSF-LR became more pronounced in the nonlinear-observation experiments. Here, EnSF-LR achieved lower full-state RMSE than both the original EnSF and the EnKF reference. The improvements were consistent across both the three-dimensional Lorenz-63 system and the forty-dimensional Lorenz-96 system, suggesting the method scales reasonably well and that the regression step remains effective even as the state dimension grows.
Full-State RMSE: Nonlinear Observations
| Label | Value |
|---|---|
| EnKF (baseline) | 1.1 |
| EnSF | 0.9 |
| EnSF-LR | 0.7 |
Breaking down the error by observed and unobserved components reveals the mechanism at work. In the Lorenz-63 experiments, where only the $x$ variable was observed, EnSF-LR kept the observed component accurate—essentially matching EnKF on this task. The real gains came in the unobserved $y$ and $z$ components, where EnSF-LR's regression step propagated the $x$-corrections more effectively than the original EnSF could, and where its nonlinear update gave it an edge over EnKF's linear assumptions.
The researchers also examined what happens during rapid transitions—when the Lorenz-63 system swings from one lobe of its attractor to the other. These are the moments when chaos hurts most: small errors blow up, and forecasts degrade sharply. In these phases, both methods struggled, but EnSF-LR recovered more quickly and kept its ensemble better centered on the truth. The ensemble—the cloud of possible states the method maintains—stayed more tightly concentrated, indicating that the method was not just more accurate on average, but also more confident in a way that reflected genuine knowledge rather than false certainty.
Why This Changes Things
Data assimilation sits at the heart of numerical weather prediction. Every forecast model in the world—including the ones used operationally by national weather services—relies on some version of these methods to ingest observations and produce a best guess of the current state of the atmosphere. The quality of that initial condition determines how far ahead the forecast remains useful. Small improvements in data assimilation can translate directly into longer-range forecasts, better warnings, and more reliable climate projections.
The challenge of sparse, nonlinear observations is not theoretical. Real observing networks are full of them. Satellite retrievals involve solving nonlinear inverse problems to infer temperature and humidity from radiation measurements. Doppler radar estimates of wind involve nonlinear transformations. Ocean float arrays measure only a subset of ocean properties at a handful of locations. Climate models track hundreds or thousands of variables, most of which are never directly constrained by observations.
Standard EnKF methods handle these cases, but they pay a price. They linearize—approximate the nonlinear relationships as straight lines—which introduces errors that can accumulate over many assimilation cycles. Particle filters, the other major non-Gaussian approach, can represent complex distributions faithfully, but they suffer from weight degeneracy in high dimensions: in practice, only a handful of the hundreds or thousands of particles carry meaningful weight, and the filter collapses. EnSF avoids this degeneracy by estimating the score function—the gradient of the log-density—directly from the ensemble, without needing an impractical number of particles. But as originally formulated, it was better at analyzing what it saw than at inferring what it did not.
EnSF-LR resolves this tension by not asking a single method to do everything. The score-based update handles the nonlinearities in observation space—the part of the problem that benefits most from EnSF's generative machinery. The regression step handles the spread to unobserved variables—the part that benefits most from EnKF's covariance structure. The two mechanisms do not interfere with each other; they operate on complementary subspaces of the state. The result is a filter that inherits the strengths of both parents without their characteristic weaknesses.
This matters for several reasons beyond the immediate numerical results.
First, it suggests a template. The two-step structure—update what you observe, then regress to what you do not—is not specific to EnSF. The researchers note that their regression step draws on a broader framework developed by Jeff Anderson at NCAR in the early 2000s, and subsequently refined by others. Any method that can produce an accurate analysis in observation space could, in principle, be plugged into this framework. The regression step is essentially a translation layer between observation space and state space. This opens the door to hybridizing other score-based or generative methods with the covariance machinery that makes information flow through the system.
Second, the results are robust across different system dimensions and observation configurations. EnSF-LR improved upon EnKF in both the three-variable Lorenz-63 and the forty-variable Lorenz-96, with observation networks ranging from single variables to 25% coverage. This suggests the method is not a one-off trick that works only in a narrow regime but a general strategy with broad applicability.
Third, the performance gains are largest precisely where existing methods are weakest: in the nonlinear-observation regime. If observations were always linear functions of the state, EnKF would already be near-optimal, and there would be little room for improvement. The fact that EnSF-LR outperforms EnKF in this regime—and does so without degrading in the linear regime—suggests that the hybrid approach is not just a marginal tweak but a meaningful advance for real-world applications where nonlinear observations are common.
What's Next
Several important questions remain open. The current implementation assimilates all observations jointly rather than sequentially. In practice, operational forecasting systems often process observations one at a time, which allows for localization—the practice of limiting the influence of each observation to nearby state variables to avoid spurious correlations in the ensemble estimate. The researchers explicitly note that developing a serial version of EnSF-LR, implementable in frameworks like the Data Assimilation Research Testbed (DART), is left for future work. This is a significant next step: localization is essential in high-dimensional geophysical applications, and it is not immediately obvious how best to integrate it with the score-based update.
The method also needs to be tested in more challenging settings. The Lorenz models are idealized—they are chaotic, yes, but they are low-dimensional, deterministic, and perfectly known. Real geophysical systems are noisy, imperfectly modeled, and far higher-dimensional. The researchers used a perfect-model setup in this study to isolate the approximation errors of the filtering update itself. Testing EnSF-LR with imperfect models, stochastic forcing, and realistic observation networks will be critical to understanding its practical potential.
There is also a question of scale. The Lorenz-96 experiments used an ensemble of ten thousand members, which is larger than what operational weather forecasting centers typically run for research purposes but smaller than the ensembles used in some operational settings. Score-based updates are computationally more expensive than standard EnKF updates because they require solving a diffusion process at each assimilation cycle. Whether this cost is justified by the accuracy gains—particularly in very high-dimensional systems—remains to be seen. The researchers are optimistic, noting that EnSF has already shown promise in high-dimensional settings, but the computational cost of the regression step at massive scale has not been characterized.
Another open direction is the extension to imperfect-model scenarios, where the forecast model does not perfectly represent the true dynamics. Real weather forecasting is an imperfect-model problem: the model is always a simplification, and the discrepancy between model and truth is a major source of forecast error. The researchers note that EnSF has shown advantages over EnKF in imperfect-model settings in prior work, and they expect EnSF-LR to inherit this benefit. But this has not yet been demonstrated explicitly.
Finally, there is the question of what this means for the broader AI-for-science landscape. The researchers describe their work as situated "at the intersection of DA and AI," noting that generative modeling provides a promising direction for non-Gaussian data assimilation because generative models are designed to represent and sample from complex probability distributions. EnSF is training-free—it does not require fitting a neural network to data—which distinguishes it from many other generative DA approaches and makes it more computationally tractable. But it still draws on ideas from diffusion models and score-based generative modeling, which have exploded in popularity in the machine learning community over the past several years. EnSF-LR is an example of how these ideas can be adapted and combined with classical methods to produce something that is more than the sum of its parts.
The forecast problem has always been a problem of inference under uncertainty: what is the most likely state of a system, given everything you know about it? That knowledge comes from imperfect models and incomplete, noisy observations. For decades, the Gaussian assumptions underlying EnKF provided a workable—and computationally tractable—answer to this question. The world is not Gaussian, but the assumptions were good enough to build a weather forecasting enterprise on. Now, as generative AI tools make it possible to represent uncertainty more faithfully, the question is no longer whether non-Gaussian methods can work, but how to combine them with the mechanisms that make classical methods work so well. EnSF-LR is an early answer: use the best tool for each part of the problem, and let the regression step bridge the gap.
What happens next will determine whether this approach scales to the real world—whether it can handle the messiness of actual observations, actual model errors, and actual computing constraints. The numerical evidence is compelling. The next test is the atmosphere.