A Race to the Top for Weather Forecasts — Before It’s Too Late

90% chance of rain next week. That’s the message millions of smallholder farmers could soon receive—delivered via SMS, radio, or app—thanks to a quiet revolution in artificial intelligence. But here’s what they won’t know: that “90%” might reflect a forecast trained on the very weather it’s predicting, or that it refers to a trace of drizzle, not a downpour. It might sound accurate, even scientific. But if the model behind it was improperly validated, that number could mislead farmers into planting too early, harvesting too late, or flooding their fields unnecessarily.

This isn’t hypothetical. As the authors of a new paper warn, we’re on the brink of a ‘race to the bottom’ in weather forecasting—one where the cheapest, flashiest models win contracts, while accurate but less aggressively marketed systems get ignored. The stakes? Hundreds of millions of livelihoods hanging on forecasts that may be worse than useless.

And yet, there’s hope. The same AI advances enabling this risk also make it possible to democratize forecasting at an unprecedented scale. Open-source AI weather models now match or exceed traditional supercomputer-driven systems at a fraction of the cost. Startups, academics, and nonprofits can now generate hyper-local predictions for regions long ignored by national meteorological services. The question is no longer whether we can deliver these forecasts—but whether we should, and under what guardrails.

The answer, according to Colin Aitken and colleagues (Aitken et al., 2026), lies not in slowing innovation, but in accelerating accountability. They propose a framework of four principles—Transparency, Relevant Evaluation, Demonstrated Out-of-Sample Skill, and Decision-Relevance—that together form a checklist for credible, agriculturally useful forecasts. These aren’t abstract ideals. They’re designed to be implemented by small teams, audited by non-experts, and scaled into a global standard.

The Science

Weather forecasting has entered a new era. For decades, accurate predictions required massive computational power—supercomputers running complex fluid dynamics simulations across global grids. These Numerical Weather Prediction (NWP) models remain gold standards, but they’re expensive, centralized, and slow to adapt to local conditions.

Enter AI Weather Prediction (AIWP) models. Trained on decades of atmospheric reanalysis data, these deep learning systems learn patterns in temperature, pressure, humidity, and wind to predict future states. Models like GraphCast (Lam et al., 2023) and PanguWeather (Bi et al., 2023) now rival or surpass traditional NWPs in global skill metrics, while running on a single GPU. This leap in efficiency has opened the floodgates: anyone with modest resources can now tailor forecasts to specific crops, regions, or decisions.

But with accessibility comes chaos. Governments and NGOs eager to support smallholder farmers face a dizzying array of forecast providers—each claiming superior accuracy. Without a common language for evaluating quality, buyers are left vulnerable to misleading claims. Worse, the very design of AI models introduces new risks: overfitting to historical data, subtle data leakage between training and test sets, and the illusion of precision in hyper-local predictions that can’t be verified.

To address this, Aitken et al. draw from meteorology, statistics, computer science, and development economics to build a verification framework. Their goal isn’t to dictate how forecasts should be made, but to ensure that claims about their quality can be trusted. The result is a set of principles backed by concrete protocols—actionable steps that forecasters can follow and evaluators can check.

The paper focuses on agriculture because the consequences of failure are immediate and severe. Unlike a missed commute forecast, an inaccurate prediction about monsoon onset or dry spells can lead to crop failure, debt, and hunger. Yet most existing forecast evaluations focus on generic metrics like Root Mean Squared Error (RMSE) for rainfall—useful for model developers, but meaningless to a farmer deciding when to sow maize.

By anchoring their framework in real-world decisions, the authors shift the burden from technical prowess to practical value. A forecast isn’t good because it scores well on a benchmark—it’s good because it helps farmers make better choices.

What They Found

The core insight of the paper is simple: forecast quality is multidimensional, and current evaluation practices fail to capture it. The authors illustrate this through two cautionary tales.

In Illustrative Example 1, a government adopts a leading AI model to help farmers avoid harvesting during rain. The model shows strong overall skill on standard metrics. But it suffers from a “drizzle bias”—it frequently predicts light rain on days that turn out to be completely dry. While this barely affects RMSE (since light rain is close to zero), it doubles the number of predicted wet days. To a farmer, that means half as many usable harvest windows. The forecast looks good on paper. In practice, it’s harmful.

In Illustrative Example 2, a startup claims its model predicted the start of the last three major floods within one day. Impressive—until you realize it also predicted seven other floods that never happened. As economist Paul Samuelson once quipped about economic indicators, they’ve predicted “nine of the last five recessions.” Without reporting false alarm rates, such claims are meaningless.

Forecast Skill Claims vs. Real-World Performance

Label	Value
Group A	0
Group B	0

Group B claims a 30% improvement but achieves only 8% in practice due to flawed evaluation (e.g., training on test data). Group A's modest 5% claim reflects rigorous out-of-sample testing and matches real-world results.

captures the danger: Group B claims a 30% improvement over climatology, while Group A claims only 5%. A funder, seeing the bigger number, chooses Group B. But in reality, Group B’s model performs at just 8% improvement—its inflated claim due to using test data during model selection. Group A’s modest claim, grounded in rigorous out-of-sample testing, matches its real-world performance.

The paper’s proposed principles directly counter these failures:

1. Transparency: Forecasters must disclose their methods, data, and code. If an evaluator can’t replicate the result, the claim lacks credibility. This includes sharing the exact text sent to farmers—not just the raw model output—since simplification can distort meaning.

2. Relevant Evaluation: Benchmarks must reflect real alternatives. A model isn’t good because it beats a weak baseline; it’s good because it beats freely available open-source models like GraphCast or ECMWF’s public forecasts. Metrics must also match the decision: predicting total weekly rainfall isn’t the same as predicting monsoon onset.

3. Demonstrated Out-of-Sample Skill: The model must be tested on data it never saw during training—or even during model selection. Even tuning hyperparameters on test data inflates performance. The paper insists on clear documentation of data splits and any post-hoc changes to thresholds or regions.

4. Decision-Relevance: Forecasts must be tested with users. A/B tests and focus groups can reveal whether farmers understand a “70% chance of rain” as probability or intensity. One study cited in the paper found that probabilistic forecasts were often interpreted as deterministic commands—“70%” became “it will rain.”

Four Principles for Evaluating Agricultural Weather Forecasts

Label	Value
Transparency	0
Relevant Evaluation	0
Out-of-Sample Skill	0
Decision-Relevance	0

Each principle is scored for feasibility, impact, and necessity in ensuring forecast credibility and farmer benefit.

visualizes how these principles balance technical and human factors. Transparency and out-of-sample skill score highest on feasibility and impact, while decision-relevance—though slightly harder to implement—is equally critical for real-world benefit.

One protocol stands out: archiving forecasts in real-time. Once dissemination begins, forecasters should register their predictions on platforms like Zenodo before outcomes are known. For-profit companies unwilling to publish openly can use third-party escrow or cryptographic hashing to preserve verifiability without sacrificing IP. This prevents retroactive adjustments and builds long-term trust.

Why This Changes Things

At first glance, this reads like a technical checklist. But its implications are deeply political, economic, and ethical.

Consider the market for agricultural forecasts today. Governments and NGOs want to help farmers. They lack expertise to judge forecast quality. Vendors—some legitimate, some not—compete for contracts. In this environment, the cheapest bid often wins. If accurate models cost more to develop and validate, they lose out to cheaper, shoddier alternatives. Over time, bad forecasts crowd out good ones—a classic market failure driven by information asymmetry.

What Aitken et al. propose is a way to break this cycle. By establishing shared standards, they aim to create a race to the top, where accuracy and transparency become competitive advantages. Funders can require adherence to the principles as a condition of funding. Meteorological agencies can certify compliant models. Over time, farmers and governments learn to trust only certified forecasts.

This mirrors movements in other fields. In medicine, randomized controlled trials and peer review separate evidence-based treatments from snake oil. In climate modeling, the Coupled Model Intercomparison Project (CMIP) allows scientists to compare models on equal footing. In software, open-source licenses and reproducibility badges build trust.

But unlike those domains, weather forecasting for smallholder farmers operates in a context of extreme vulnerability. A wrong forecast doesn’t just waste money—it can destroy a season’s work. That’s why the authors emphasize decision-relevance. It’s not enough for a forecast to be skillful in isolation; it must be useful to the farmer who receives it.

Take the example of monsoon onset prediction. Many models focus on minimizing error in onset date. But farmers don’t need a precise date—they need confidence that rains have truly started, so they can commit seeds and labor. A forecast that says “monsoon likely began on June 12 ± 3 days” is more actionable than one that pins it to June 10 with false precision.

The framework also anticipates the rise of AI-generated science. As frontier models begin designing their own forecasting systems, humans may lose visibility into how models were trained. Protocol 1b explicitly requires that humans be able to certify principles were followed—even if AI did the work. This guards against “reward hacking,” where an AI subtly cheats to maximize a metric without improving real skill.

Perhaps most importantly, the proposal is pragmatic. It doesn’t demand perfection. It acknowledges that open-source models may have regional biases, that ground truth data is sparse in rural Africa, and that probabilistic forecasts are hard to communicate. Instead of waiting for ideal conditions, it offers a minimum viable standard—something that can be adopted now, improved later.

What’s Next

The authors position this as a starting point, not a final answer. They call for collaboration among forecast developers, meteorological agencies, agricultural ministries, and funders to refine and institutionalize the principles. Regional adaptations will be needed: a forecast useful for rice farmers in Bangladesh may not serve pastoralists in Kenya.

One immediate next step is pilot certification. A consortium of donors could agree to fund only forecasts that meet the four principles, creating demand for compliance. The World Meteorological Organization (WMO) or regional bodies like ACMAD (African Centre of Meteorological Applications for Development) could host a verification platform, offering seals of approval.

Another challenge is scaling user testing. While A/B tests and focus groups are ideal, they’re resource-intensive. The paper suggests lightweight alternatives: embedding comprehension questions in SMS surveys, or using natural language processing to analyze farmer feedback.

There are also unresolved tensions. How much transparency is too much? Should proprietary models be required to release code? The authors suggest compromises: escrow for commercial models, or publishing model architectures without weights. The goal isn’t full openness, but sufficient auditability.

Finally, the framework must evolve with the science. As AI models grow more capable, new risks will emerge. Today’s concern is data leakage. Tomorrow’s might be adversarial manipulation or emergent reasoning flaws. The principles must be living documents, updated through consensus.

The clock is ticking. AI-powered forecasts are already being deployed—from India to Ethiopia—at scale. Without standards, early failures could poison the well. Farmers burned by inaccurate predictions may reject all future forecasts, even good ones. Governments may retreat from digital agriculture initiatives altogether.

But if we act now, we can flip the script. We can ensure that the AI revolution in weather forecasting doesn’t repeat the mistakes of social media or fintech—where innovation outpaced accountability. Instead, we can build a system where the best models win not because they’re the loudest, but because they’re the most trustworthy.

As the authors write: “It is not enough for a forecast to be skillful in isolation—it must be useful to the farmer who receives it.” That sentence should be etched above every meteorologist’s desk, every developer’s screen, every policymaker’s door. Because in the end, forecasting isn’t about data or algorithms. It’s about people.