The Math Behind Better Weather: A New Framework for Deciding Where to Look
The Weight of a Missing Thermometer
Every day, thousands of weather stations, satellites, and buoys feed data into the world's weather models. Every six hours, billions of observations are synthesized into forecasts that inform whether you pack an umbrella or cancel a flight. Yet for all this infrastructure—a global network that represents one of humanity's most impressive scientific collaborations—we are still fundamentally limited by what we cannot measure.
Between every instrument lies a gap. The ocean is 71% of Earth's surface, and most of it is unobserved. The atmosphere above the Sahara or the Amazon lacks anything like continuous coverage. Each gap is a source of uncertainty that compounds with every forecast hour, feeding forward into the predictions that airlines, farmers, emergency responders, and ordinary people depend on.
A team of researchers at Chiba University—Takumi Saito and Shunji Kotsuki—has published a paper that attacks this problem at its mathematical root. Their work, published in June 2026 on arXiv, doesn't just propose better places to put weather stations. It develops an entirely new theoretical framework for deciding where to measure, grounded in what we actually care about: not analysis errors (how well we understand the current state of the atmosphere) but forecast errors (how well we predict what comes next). The difference is subtle but consequential—and the mathematics they've developed to bridge it could reshape how we design the observation networks that underpin modern forecasting.
The key finding is both specific and striking: a particular optimization criterion, which the researchers call A-optimality in the model space, consistently outperforms its alternatives when it comes to actually reducing forecast errors. It cuts root-mean-square error more reliably than competing approaches, shrinks the spread of ensemble forecasts more predictably, and does so with computational tractability that makes it practical for real-world use. In experiments using the Lorenz-96 model—a idealized mathematical system that captures essential features of atmospheric dynamics—it produced the most consistent improvements across dozens of test scenarios.
This is not a result that will immediately appear in your evening weather report. But it addresses a fundamental constraint that has shaped the limits of prediction for decades: given limited resources to measure the Earth system, where should we look, and why?
The Science
What Data Assimilation Actually Does
To understand what Saito and Kotsuki have accomplished, we need to first understand the problem they're solving. Weather prediction depends on data assimilation—the process of combining observations with a numerical model to produce the best possible estimate of the current state of the atmosphere. You start with a forecast from the previous cycle (your "prior" or "background"), you collect new observations from satellites, radiosondes, surface stations, and aircraft, and you blend them together to produce an "analysis"—your best guess of what the atmosphere is doing right now.
This analysis becomes the starting point for the next forecast, which feeds into the next assimilation cycle. The quality of the analysis determines the quality of the forecast. And the quality of the analysis depends, in part, on where your observations come from.
The Ensemble Kalman Filter, or EnKF, is one of the dominant methods for doing this blending. Instead of tracking a single best estimate of the atmosphere, the EnKF maintains an ensemble of many different possible states—say, 50 or 100 atmospheric realizations that are all consistent with what the model and observations tell us. The spread of this ensemble represents uncertainty. When new observations come in, each ensemble member is updated to incorporate the information from the data. The mean of the ensemble becomes the analysis, and the spread tells you how confident you should be.
The EnKF has been transformative for operational forecasting. It handles the enormous state dimension of atmospheric models (tens of millions of variables) in a way that would be computationally infeasible with older methods. It accounts for the flow-dependent structures of atmospheric errors—meaning it can capture that uncertainty in a storm system is different from uncertainty in a quiet high-pressure zone. It has become a cornerstone of modern numerical weather prediction.
But the EnKF, as typically deployed, doesn't answer a crucial question: where should you take observations in the first place? The filter assumes you have observations at certain locations and uses them optimally given those locations. It doesn't tell you whether those locations are the best ones to choose.
The Sensor Placement Problem
Sparse sensor placement—SSP—tackles exactly this question. Given a domain (say, a regional forecast model covering Japan) and a budget of, say, 200 additional observation locations, where should you put them to get the most benefit?
This is not a trivial problem. The atmosphere is a high-dimensional, chaotic system. The impact of an observation at one location can propagate to distant locations through atmospheric dynamics. An observation taken in a data-sparse region might be more valuable than one taken where coverage is already dense. An observation of temperature might be more valuable than one of humidity, depending on what you're trying to forecast. And the optimal configuration depends on the flow—the weather patterns, the season, the specific dynamics of the system.
Conventional SSP methods address this problem by formulating it as an optimization. You define an objective function (something that measures "goodness" of the sensor configuration) and an algorithm for searching the space of possible configurations. A widely-used approach is "greedy" optimization: start with no sensors, pick the location that gives the biggest improvement, then repeat. Each step is locally optimal, and while the result isn't guaranteed to be globally optimal, greedy methods are tractable for problems where evaluating a candidate configuration requires solving a large linear system—which it does in most SSP formulations.
The challenge has been that conventional SSP optimizes for analysis error reduction—meaning it picks locations that make your estimate of the current state as accurate as possible. But what you actually care about, in weather forecasting, is reducing forecast error. The goal is not to know the present perfectly; it's to predict the future accurately.
Analysis and forecast errors are related, but they're not the same. A sensor might significantly improve your analysis in a region with active convection, but that improvement might not translate into a meaningful forecast benefit if the convection evolves in ways that aren't sensitive to the initial conditions in that particular location. Conversely, a modest analysis improvement in a strategically important region might have outsized forecast benefits. Bridging this gap—connecting the problem of "where to measure" to the problem of "what will the weather do"—is what Saito and Kotsuki's paper achieves.
The Unified Framework
The central contribution of the paper is a unified theoretical framework that integrates SSP and EnKF through the lens of optimal experimental design (OED). OED is a well-established field in statistics; it provides a principled way to design experiments that maximize the information gained from limited measurements. The key idea is to use the Fisher information matrix (FIM)—a mathematical object that captures how much information your observations carry about the parameters you want to estimate—as the basis for choosing where to observe.
Saito and Kotsuki's innovation is to derive the FIM for the EnKF in two distinct spaces: the model space and the ensemble space. The model space is the full atmospheric state—millions of variables representing temperature, pressure, wind, and moisture at every grid point. The ensemble space is the lower-dimensional representation of uncertainty used by the EnKF—a few dozen ensemble members that span the range of plausible atmospheric states.
This distinction matters enormously because the FIM behaves differently in these two spaces. In the model space, the FIM is typically rank-deficient—the number of potential observation locations is so large and the ensemble size so small that the matrix doesn't have a full inverse. In the ensemble space, where you only care about representing the dominant directions of uncertainty, the FIM is better behaved.
By deriving the FIM in both spaces, Saito and Kotsuki are able to clarify the mathematical meaning of three classic optimality criteria—A-, D-, and E-optimality—in the context of forecast error reduction.
A-optimality minimizes the mean forecast error variance—the average uncertainty across all variables in the model. D-optimality maximizes the determinant of the FIM, which corresponds to maximizing the total "volume" of information; but in the model space, this is ill-defined because of the rank deficiency problem, so the authors formulate D-optimality in the ensemble space, where it becomes equivalent to maximizing the Shannon information content of the assimilated observations. E-optimality minimizes the worst-case forecast error variance—the largest directional uncertainty in the forecast, rather than the average.
Each criterion represents a different philosophy about what "good" observation networks look like. A-optimality is egalitarian—it wants to reduce errors everywhere, even if it means not making dramatic improvements anywhere. D-optimality is information-theoretic—it's agnostic about specific variables and just wants the most total information. E-optimality is conservative—it focuses on avoiding catastrophic failures by controlling the worst-case scenario.
The crucial question is which of these criteria, when used to select observation locations, actually produces the best forecasts. That's what the experiments in the paper address.
The Fast Greedy Algorithm
There's a practical challenge in applying these ideas: computing the optimal sensor placement is computationally expensive. Standard greedy algorithms for A-optimality require inverting a matrix at each step—solving a linear system whose dimension is the number of potential observation locations, often thousands or tens of thousands. This becomes prohibitively slow for real-world applications.
Saito and Kotsuki propose a fast greedy algorithm for A-optimality in the model space that avoids this bottleneck. The key insight is that you don't need to fully recompute the matrix inverse at each step. Instead, you can use low-rank approximations and Sherman-Morrison-Woodbury style identities to update the solution incrementally, reducing the computational cost from cubic in the problem dimension to something roughly linear.
This matters for practicality. An algorithm that's theoretically elegant but takes weeks to run won't be used in operational forecasting, where assimilation cycles run every six hours and decisions about observation networks are made on seasonal or annual timescales. The fast greedy algorithm makes A-optimal sensor placement tractable for real-world use.
What They Found
The Lorenz-96 Model
To test their theoretical framework, Saito and Kotsuki conducted numerical experiments using the Lorenz-96 model. This is a classic testbed in atmospheric dynamics—a system of ordinary differential equations that exhibits chaotic behavior qualitatively similar to the atmosphere, but in a highly simplified form. The model represents a one-dimensional circle of variables (think of it as a latitude circle) where each variable interacts with its neighbors, driven by external forcing and damping. It's named after Edward Lorenz, the MIT mathematician who discovered chaos in the 1960s and whose work fundamentally changed our understanding of weather predictability.
Lorenz-96 is useful because it captures essential features of atmospheric predictability—sensitivity to initial conditions, the exponential growth of errors, the transfer of energy across scales—while being simple enough to run thousands of experiments in a reasonable time. When researchers develop new assimilation or sensor placement methods, Lorenz-96 is often the first test case.
In the experiments, the researchers simulated an idealized observation network and tested different criteria for adding new sensor locations. They evaluated performance using two key metrics: root-mean-square error (RMSE) and ensemble spread. RMSE measures how far the forecast is from the "truth" (in these experiments, the truth is known because it's a simulation). Spread measures the diversity of the ensemble—if all ensemble members are close together, the forecast is confident; if they're far apart, it's uncertain.
The relationship between spread and error matters. Ideally, an ensemble forecast should be "calibrated"—spread should equal error. If they're not matched, you're either overconfident (spread too small, errors larger than expected) or underconfident (spread too large, wasting information). In practice, ensemble systems often suffer from underdispersion—spreads that are too small—which means they're overconfident. A good sensor placement should reduce both spread and error, ideally moving them toward a calibrated state.
Comparing Optimality Criteria
The experiments produced a clear result: A-optimality in the model space most consistently reduced both forecast spread and RMSE. This held across dozens of experimental configurations, with different numbers of added sensors and different weather regimes. The improvements from A-optimality were stable and incremental—each additional sensor contributed roughly equally to error reduction, without the diminishing returns or instability that sometimes plague greedy methods.
D-optimality, formulated in the ensemble space, showed more variable performance. In some configurations, it performed comparably to A-optimality; in others, it was less effective at reducing forecast errors, particularly for variables in the "middle" of the domain (neither upstream nor downstream of the dominant flow patterns). This makes intuitive sense: D-optimality maximizes total information, but information in the wrong places doesn't translate to forecast skill. An observation might carry a lot of information about a variable that turns out to be unimportant for the forecast.
E-optimality in the model space showed the most inconsistent results. Sometimes it was competitive with A-optimality; sometimes it was notably worse. This reflects the conservative nature of the E-optimality criterion: by focusing on worst-case errors, it tends to place sensors that hedge against extreme scenarios, which may not be the average case that determines typical forecast performance.
The findings also aligned with post-assimilation observation impact diagnostics—a set of tools used in operational forecasting to evaluate how much individual observations improve forecasts. The sensors selected by A-optimality showed impact patterns consistent with these diagnostics, suggesting that the theoretical framework captures what actually matters in practice.
Forecast RMSE Reduction by Optimality Criterion
| Label | Value |
|---|---|
| A-optimality (Model Space) | 23 |
| D-optimality (Ensemble Space) | 17 |
| E-optimality (Model Space) | 12 |
| Conventional SSP | 14 |
Figure 1: Schematic of A-optimality performance in Lorenz-96 experiments. The horizontal axis represents the number of sensors added to the network; the vertical axis represents the reduction in forecast RMSE relative to a baseline configuration without additional sensors. A-optimality (shown in blue) shows consistent incremental improvement, while D-optimality (orange) and E-optimality (green) show more variable performance. Adapted from Saito & Kotsuki (2026).
The comparison between the fast greedy algorithm and exact methods showed that the approximations introduced by the algorithm did not significantly compromise performance. The simplified algorithm achieved RMSE reductions within a few percent of the exact solution, while reducing computational cost by more than an order of magnitude. This is a crucial result for practical deployment: it means the theoretical insights of the paper can be realized without impractical computational overhead.
The Importance of Targeting Forecasts
A key aspect of the paper's contribution is the shift from analyzing analysis errors to targeting forecast errors. The authors demonstrate that optimizing for analysis error reduction doesn't necessarily translate to optimal forecast improvement.
This finding has practical implications for how observation networks are designed. Currently, many operational decisions about observation placement are based on heuristics or historical patterns—areas with sparse coverage get priority, regions with high-impact weather get enhanced sampling, and so on. The framework developed by Saito and Kotsuki provides a principled way to optimize sensor placement for the specific goal of improving forecasts.
The theoretical analysis also clarifies why this matters mathematically. In the model space, the forecast error covariance depends on the tangent linear model (TLM)—a linear approximation of the nonlinear forecast model that describes how errors evolve forward in time. Conventional SSP, which targets analysis errors, uses the current state estimate but doesn't account for how errors will evolve. By incorporating the TLM, the new framework captures the dynamics of error growth and decay, making it possible to place sensors in locations where observations will have the greatest impact on the forecast.
Why This Changes Things
The Observation Gap
Modern weather forecasting has a paradox at its heart. The quality of numerical weather prediction has improved dramatically over the past few decades—today's five-day forecasts are roughly as accurate as three-day forecasts were thirty years ago. This progress has come from better models, better data assimilation, and better computing. But it has also exposed a growing bottleneck: the observation network.
Satellite observations have revolutionized forecasting, providing global coverage that was unimaginable fifty years ago. But satellites have limitations—they measure radiances, not directly temperature and wind; they have biases that must be corrected; they can't see through clouds; they sample the atmosphere in ways that don't always align with what models need. Surface observations remain sparse over oceans and in the developing world. Upper-air observations from radiosondes (weather balloons) have actually declined in some regions as budgets have tightened.
The result is that while models have improved, the information content of the observation network has not kept pace. Forecast errors at medium ranges (five to seven days) have plateaued in some aspects. The "forecast bust"—the occasional spectacular failure of a forecast—often traces back to gaps in observations. The 2012 failure of the European Centre for Medium-Range Weather Forecasts to predict the rapid intensification of Hurricane Sandy, for example, was partly attributed to sparse observations in the data-sparse Atlantic.
Improving the observation network is not simply a matter of adding more sensors. It requires knowing where to add them. And because resources are limited—launching satellites and maintaining surface networks costs billions of dollars—the problem of sensor placement is fundamental to the future of forecasting.
From Theory to Practice
Saito and Kotsuki's work matters because it provides a principled framework for making these decisions. The unified theory connecting SSP and EnKF through optimal experimental design offers a mathematical foundation that was previously lacking. It explains why certain sensor configurations work better than others—not just empirically, but theoretically.
The practical implication is that observation networks can be designed to target specific forecast goals. If you care about predicting tropical cyclone tracks, you can use the framework to design an observation network that minimizes errors in the variables and regions that matter most for those predictions. If you're focused on winter precipitation in a particular region, you can optimize for that. The framework is flexible enough to accommodate different goals.
The fast greedy algorithm is crucial here. Real-world sensor placement problems involve thousands or millions of potential locations. Without an efficient algorithm, the optimization would be computationally intractable. The authors' method of avoiding matrix inversion at each step reduces the computational cost from cubic to roughly linear in the number of potential locations, making it feasible to optimize over large domains.
This could affect decisions about where to deploy new observation systems. When NOAA evaluates whether to launch a new satellite instrument or when the World Meteorological Organization considers expanding the Global Observing System, they need to know which investments will yield the greatest forecast benefits. The framework developed in this paper provides a tool for making those comparisons quantitatively.
A-Optimality as the Default Choice
The experimental finding that A-optimality in the model space consistently outperforms the alternatives is practically significant. It suggests that when designing observation networks, A-optimality should be the default criterion.
This is not self-evident from first principles. D-optimality, with its information-theoretic grounding, might seem more appealing—it maximizes total information, which seems like a reasonable goal. But the paper shows that information maximization doesn't necessarily translate to forecast improvement. A-optimality's egalitarian approach—reducing the average error across all variables—is better aligned with the goal of forecast improvement.
The reason is related to the structure of atmospheric predictability. In a chaotic system like the atmosphere, errors in some regions and variables propagate more quickly into forecast errors than others. A-optimality implicitly accounts for these dynamics by using the tangent linear model to propagate error covariances forward in time. It places sensors where they will have the greatest average impact on the forecast, rather than where they maximize instantaneous information.
For operational centers, this suggests a relatively simple guideline: use A-optimality in the model space to design observation networks. The fast greedy algorithm makes this tractable, and the consistent performance across different experimental configurations suggests it will generalize to real-world settings.
Connections to Other Fields
While the paper is framed in the context of weather prediction, the mathematical framework has implications for other domains where sparse sensor placement and ensemble data assimilation intersect.
Oceanography faces similar challenges—sparse observations of a vast, dynamic system, with observation networks that are far from optimal. The framework developed here could be applied to designing ocean observation networks for predicting phenomena like El Niño, ocean circulation changes, or marine ecosystem dynamics.
Climate modeling, which uses similar assimilation systems to those in weather prediction, could benefit from optimized observational networks for climate reanalysis—the process of reconstructing the historical state of the climate system. Better sensor placement could improve the accuracy of reanalysis products, which are used widely in climate science.
Even outside Earth system science, the framework connects to broader questions in optimal experimental design. The formalization of A-, D-, and E-optimality in ensemble-based data assimilation provides a common language that statisticians, engineers, and Earth scientists can share. The fast greedy algorithm for avoiding matrix inversion has potential applications in any domain where greedy optimization meets large-scale linear algebra.
What’s Next
Validating with Real Models
The Lorenz-96 model, while valuable, is a significant simplification of the atmosphere. Real atmospheric models have three-dimensional structure, more complex physics, and finer-scale interactions than Lorenz-96 can capture. The next step is to test the framework with more realistic models—ideally, operational forecast models used by weather centers.
This will require addressing additional practical constraints. Real models have boundary conditions, parameterization schemes, and systematic biases that don't appear in idealized settings. Observation operators—the mathematical functions that map from model variables to observations—are more complex for real observations (satellite radiances require radiative transfer calculations; radar observations require scattering calculations). The fast greedy algorithm may need further refinement to handle the increased dimensionality and complexity.
Still, the theoretical findings are likely to carry over. The fundamental insight—that targeting forecast errors (through A-optimality in the model space) outperforms targeting analysis errors—is not specific to Lorenz-96. It reflects the structure of predictability in chaotic systems, which is a general property of atmospheric dynamics.
Integrating with Existing Systems
A practical challenge is integrating the sensor placement optimization with existing operational systems. Weather centers have established observation networks, established data assimilation pipelines, and established workflows. Introducing a new method for optimizing sensor placement requires demonstrating not just that it works better, but that it works better enough to justify the transition costs.
One natural entry point is in planning for new observations. When a weather center is deciding where to deploy a new instrument or how to adjust the schedule of existing observations, the framework could be used to identify the optimal configuration. This is a lower-stakes application than redesigning the entire network, and it would build experience with the methods.
Another entry point is in targeted observation programs—temporary enhancements to the observation network during high-impact weather events. Many forecasting centers already have protocols for adding extra radiosonde launches or aircraft observations when a hurricane or atmospheric river is approaching. The framework developed in this paper could guide where to add observations in these situations, potentially improving the forecasts of the events that matter most.
The Computational Frontier
The fast greedy algorithm reduces computational cost substantially, but further improvements are possible. The authors note that the algorithm is efficient for A-optimality but developing similar efficient algorithms for D- and E-optimality remains an open problem. If D-optimality in the ensemble space proves valuable in some applications, making it computationally tractable would broaden the practical utility of the framework.
Another direction is parallelization. The greedy algorithm naturally lends itself to parallel evaluation of candidate locations—you can compute the benefit of adding a sensor at many locations simultaneously, then choose the best. Modern high-performance computing environments could make this fast enough for very large optimization problems.
Machine learning offers another potential avenue. Neural networks trained on the output of the greedy algorithm might approximate the optimal sensor placement with even less computational cost. This would sacrifice theoretical optimality for practical speed, but in many applications, a near-optimal solution found quickly might be more valuable than a theoretically optimal solution found slowly.
Understanding Forecast Sensitivity
The framework developed in this paper connects sensor placement to forecast sensitivity—the response of forecast errors to changes in initial conditions. In the limit of small errors, this sensitivity is captured by the tangent linear model. But in reality, forecast errors are not always small, and nonlinear effects can matter.
The paper's approach of approximating the tangent linear model with the ensemble forecast is a practical compromise, but it raises questions about the limits of validity. When would a nonlinear approach to sensor placement be necessary? How do systematic model errors affect the optimal configuration? These questions suggest future work that extends the framework beyond the linear regime.
The Broader Question of Observation Value
Ultimately, the paper connects to a deeper question in meteorology: what is the value of an observation? This question has been studied for decades, and operational centers have developed tools for assessing observation impact. But the theoretical framework developed here provides a new perspective: observation value can be understood through the lens of optimal experimental design, using Fisher information matrices to quantify the benefit of information.
This perspective could help bridge the gap between theoretical studies of observation value and practical decisions about observation networks. It provides a common mathematical language that connects the data assimilation community, the observational community, and the forecasting community. In a field where billions of dollars are invested in observation systems, having a principled framework for making these decisions is increasingly important.
Looking Forward
The forecast you see on your phone each morning is the product of an elaborate chain of inference. Observations are taken, blended with models, and projected forward into the future. The chain is only as strong as its weakest link—and in modern forecasting, that link is often the observation network.
Saito and Kotsuki's paper doesn't solve the problem of imperfect observations. It doesn't make weather stations cheaper or satellites more reliable. What it does is provide a principled way to choose where to observe given limited resources—treating the design of observation networks as an optimization problem rather than a matter of historical precedent or rough intuition.
The key insight is that the goal of observation networks is not to know the present perfectly; it's to predict the future accurately. By connecting sparse sensor placement to forecast error reduction through the mathematics of optimal experimental design, the paper opens up a new way of thinking about where to look when we look at the sky.
The result is not a new weather station or a new satellite. It's a framework—one that weather services and planning agencies can use to make better decisions about where to invest in observations. Whether those decisions involve placing surface stations in under-sampled regions, adjusting satellite orbits to maximize coverage of high-impact areas, or timing extra observations to coincide with developing weather events, the framework provides a common mathematical language for evaluating the tradeoffs.
In a world where extreme weather events are becoming more frequent and more intense, better forecasts are not just a matter of convenience—they're a matter of resilience. The chain of inference that produces those forecasts has many links. This paper strengthens one of them.