The Algorithm That Could Unlock Better Renewable Energy Planning

The Algorithm That Could Unlock Better Renewable Energy Planning

Somewhere on a server at ETH Zurich, a supercomputer is wrestling with an impossible problem. It's trying to figure out the optimal mix of solar farms, wind turbines, batteries, and transmission lines for an entire continent's power grid—not just for next week, but for the next thirty years, under thousands of possible weather scenarios. The math is sound. The data is comprehensive. But the problem is so vast that conventional solvers simply run out of memory, or take so long that the answer arrives too late to be useful.

This is the fundamental bottleneck in modern energy planning. We have the ambition to design renewable energy systems that are cost-efficient, reliable, and resilient. We have the models that could tell us how to get there. But we lack the computational muscle to run those models at the resolution they need. The result is a cruel irony: our best tools for navigating the energy transition are themselves stuck in a transition—unable to cross from theoretical elegance into practical usefulness.

A new paper from researchers at ETH Zurich's Reliability and Risk Engineering Laboratory offers a way forward. Leonard Göke and Giovanni Sansavini have developed an algorithmic improvement called Double Interior-Point Regularization—DIP-Set for short—that can solve these massive planning problems 30 to 50 percent faster than existing methods. That might sound incremental. But in a field where the largest problems have been essentially unsolvable, a 30 percent improvement is transformative. It means problems that would have required a week of computing time can now be finished in three or four days. Problems that were structurally off-limits become tractable. The boundary of what's possible shifts.

The key insight behind DIP-Set is surprisingly human: sometimes the fastest route through a maze isn't to head straight for the exit, but to stay near the walls and feel your way along. Or, in mathematical terms, the fastest route to the optimal solution isn't always to head directly toward it, but to traverse the interior of the solution space while remaining close to solutions you've already found to be promising. Göke and Sansavini have figured out how to make an optimization algorithm do exactly that, combining two previously separate approaches in a way that amplifies their strengths.

To understand why this matters, you first need to understand the problem it's trying to solve.

The Science

Capacity expansion modeling is, at its core, a thought experiment: given everything we know about future energy demand, technology costs, and resource availability, what combination of power plants, storage facilities, and transmission infrastructure should we build to meet that demand at minimum cost while maintaining reliability? It's a question that sounds straightforward but explodes in complexity the moment you try to answer it rigorously.

Consider what you'd need to capture. Weather-dependent generation from wind and solar varies hour by hour and year by year. A windy year in northern Germany doesn't look like a windy year in Spain, and neither looks like a year of persistent calm. Capturing this variability requires modeling thousands of hours across multiple weather years. Storage introduces its own complications: batteries can shift energy within a day, but longer-duration storage—pumped hydro, hydrogen, or seasonal thermal storage—needs to track energy flows across months or even years. The grid itself ties everything together, with power flowing between regions based on generation and demand at each moment. And as electrification expands to heating, transportation, and industry, the power sector couples with other parts of the energy system in ways that create additional interdependencies.

The result is an optimization problem of staggering scale. In linear programming terms—the mathematical language these models speak—memory requirements grow exponentially with the number of variables and constraints. A model with hourly resolution across an entire continent, multiple weather years, and sector coupling might involve hundreds of millions of variables. Even with the sparsity inherent in real-world systems (most variables touch only a few constraints), the computational demands push against the limits of established solvers.

This is where Benders Decomposition comes in. Developed by Jacques Benders in 1962, this solution algorithm takes a monolithic optimization problem and breaks it apart. Think of it like planning a road trip by first deciding which cities to visit, then separately figuring out the best routes between them. The "master problem" makes the big decisions—in energy planning, these are the capacity investments: how many gigawatts of solar to build in Spain, how much offshore wind in the North Sea. The "sub-problems" work out the operational implications: given these capacities, what would the grid do hour by hour? Can it meet demand? At what cost? What are the stress points?

The decomposition strategy is both a necessity and a source of elegance. By breaking the problem into smaller pieces, each sub-problem fits into memory and solves quickly. The sub-problems can be solved in parallel, exploiting the fact that they're independent—each weather scenario, for example, is a self-contained story about what the next decade might look like. The master problem stays lean, focusing on the strategic decisions rather than operational details.

But there's a catch. The sub-problems don't return simple answers; they return functions—mathematical objects that describe how operational costs change as you vary the capacity decisions. These functions are themselves complex and unknown, so Benders Decomposition approximates them using "cutting planes": linear constraints that form a piecewise linear lower bound on the true function. Each time you solve the sub-problems, you learn a little more about the shape of that function, and you add another cutting plane to refine your approximation.

The algorithm iterates. The master problem proposes a capacity mix. The sub-problems evaluate it. The cutting planes update. The master problem proposes again, guided by a better understanding of the landscape. Eventually, the approximation converges to the true optimum, and you have your answer.

It works beautifully in theory. In practice, it can crawl.

The issue is that the approximation is imperfect, especially early in the process when few cutting planes have been added. The master problem sees a distorted view of the true objective and proposes solutions that overshoot or oscillate—jumping from one extreme to another as it chases a moving target. This behavior isn't just slow; it can become pathological, with convergence stalling long before reaching the optimum.

This is where regularization enters the picture.

What Was Wrong With Previous Approaches

If Benders Decomposition tends to oscillate wildly, regularization is the algorithm learning to calm down. The core idea is to add constraints that prevent the master problem from proposing solutions that are too far from what it already knows to be good. It's like giving the algorithm a tendency toward conservatism: don't jump too far from your current best guess.

The existing approaches each embody a different philosophy about what "conservative" means.

Trust-region regularization is the most intuitive: add a constraint that directly limits the distance between any new candidate solution and the current best solution. The master problem can only propose capacities within a hypersphere of a certain radius around where you've already been. This works—it prevents oscillation—but it can be overly restrictive. The algorithm moves slowly, taking many small steps rather than bold leaps.

Level-set regularization takes a different angle. Rather than constraining where the solution can be, it constrains what the objective value can be. The algorithm is told: propose any capacities you want, as long as the expected operational cost stays below some threshold. The threshold starts loose and tightens as iterations progress. This approach allows more exploratory moves, but it provides no direct guidance about where in the solution space to look.

Interior-point level-set goes a step further. It abandons the idea of minimizing distance altogether, instead treating the optimization as a pure feasibility problem: find any solution with an objective value below the threshold. This sounds counterintuitive—why would you stop trying to minimize?—but there's a method to it. By not forcing the solution toward any particular point, the algorithm naturally finds solutions in the interior of the feasible region, rather than at extreme corners. Interior solutions tend to be more robust, less sensitive to small perturbations. And in the context of Benders Decomposition, moving through the interior can improve convergence because you're not fighting against the geometry of the cutting plane approximation.

The problem is that interior-point level-set doesn't use information about what you already know. It's exploring freely, without leveraging the accumulated wisdom of previous iterations. It finds interior solutions, but not necessarily good interior solutions—ones that are both interior and close to the best solution found so far.

Each approach had strengths. Neither had both.

The DIP-Set Solution

Göke and Sansavini's contribution is to recognize that you don't have to choose. The insight behind DIP-Set is that trust-region regularization and interior-point level-set regularization address different failure modes, and combining them might address both.

The trust-region keeps the algorithm close to known good solutions, preventing the oscillation that otherwise afflicts Benders Decomposition. The interior-point approach keeps the algorithm moving through the interior of the solution space, avoiding the jagged geometry near extreme points where convergence typically stalls. Separately, each is incomplete. Together, they're complementary.

The mathematical formulation is elegant in its simplicity. DIP-Set adds two constraints to the master problem. The first is the level-set constraint, bounding the objective value below the current threshold. The second is the trust-region constraint, bounding the distance from the current best solution. The algorithm now has to find a point that satisfies both constraints simultaneously—it has to be interior and close to known good territory.

Formally, the regularized master problem becomes:

$$\underset{x}{\min }0$$$$\text{s.t.}{c}^{\top }x+\sum _{s\in S}{\tilde{\phi }}_s(x)\le {\lambda }_i$$$$\Vert x-x_r{\Vert }_2\le r_i$$

Where the first constraint enforces the level-set (the objective must stay below $\lambda_i$, a threshold that tightens as iterations progress) and the second constraint enforces the trust-region (the solution must stay within radius $r_i$ of the reference solution $x_r$). The objective function is zero because we're treating this as a feasibility problem—we just want a point that satisfies both constraints, not a point that minimizes anything.

The hard part is that combining two regularizations means managing two dynamic parameters: $\lambda_i$ (the objective threshold) and $r_i$ (the trust-region radius). Both need to be set carefully. If they're too loose, the regularized problem is under-constrained and the algorithm reverts to wild oscillation. If they're too tight, the problem becomes infeasible, or the algorithm gets stuck in a tiny region and never makes progress.

Göke and Sansavini propose two solutions. The direct implementation couples the trust-region radius to the optimality gap: as the algorithm gets closer to convergence, the radius shrinks automatically. This is intuitive—you can afford to be more aggressive (move further from the reference) early on, but as you near the optimum, you want to be careful not to overshoot.

The indirect implementation takes a different tack. Rather than directly parameterizing the trust-region radius, it reformulates the problem so that the two constraints indirectly influence each other. The level-set constraint determines where in the solution space you're looking; the trust-region constraint ensures you don't stray too far. By carefully linking the parameters, both constraints become binding at the same time, which is the condition for effective regularization.

Both implementations are applicable beyond Benders Decomposition. The same ideas can be applied to other decomposition-based methods or first-order optimization algorithms, though Göke and Sansavini focus their benchmarking on the Benders context.

What They Found

The proof, as always, is in the benchmarking.

Göke and Sansavini test DIP-Set on two case studies. The first is a power-sector problem: a model of the European grid with 33 regions, hourly resolution, and perfect foresight about future operations. This is a large problem by any measure, but it's structured enough to serve as a controlled test case.

The second is an energy-system problem: a more comprehensive model that couples the power sector with heating, transportation, and industry, incorporating both short-duration storage (batteries, limited to daily cycling) and long-duration storage (hydrogen, seasonal thermal). This problem is harder because the interdependencies are more complex, and because the foresight is limited—the algorithm can't see the entire year at once, but must plan in chunks (two months at a time) and accept some uncertainty about how storage decisions made in January will play out in June.

The results are striking. DIP-Set outperforms all competing regularization methods across both case studies and all problem sizes. The speed-up isn't dramatic for small problems—maybe 10 to 15 percent faster than trust-region regularization. But as problems grow larger, the advantage compounds. For the largest problems in the benchmark—the ones that were previously at or beyond the limits of solvability—DIP-Set achieves a 30 to 50 percent speed-up. This isn't incremental improvement. It's the difference between a problem being tractable and being a computational black hole.

The convergence plots tell the story clearly. Under standard trust-region regularization, the optimality gap decreases steadily early in the process but then plateaus, declining glacially as the algorithm approaches the optimum. This is the characteristic behavior of Benders Decomposition: the cutting planes are accurate enough to get you close, but the final refinement is agonizingly slow. DIP-Set doesn't eliminate this plateau entirely, but it substantially reduces it. The algorithm maintains momentum deeper into the convergence process, reaching tighter tolerances in fewer iterations.

Why does this happen? The key mechanism, according to Göke and Sansavini, is that DIP-Set mitigates the sharp decrease in convergence that afflicts standard Benders Decomposition as it approaches the optimum. Near the optimum, the objective function is nearly flat—the difference between the current solution and the true optimum is small in terms of cost, but large in terms of the capacity decisions that produce it. Standard regularization methods tend to stall in this regime because the geometry is unfavorable: the cutting planes are oriented such that any new proposal overshoots. Interior-point methods navigate this geometry better because they're not constrained to be close to any particular point; they can find the direction of improvement even when it's counterintuitive.

The comparison between perfect foresight and limited foresight is also revealing. Perfect foresight—knowing the entire year in advance—is computationally easier because the problem has more structure, but it's also less realistic. Real planning operates under uncertainty. DIP-Set's advantages appear in both settings, but they're especially notable under limited foresight, where the problem is harder and the stakes for efficient convergence are higher.

Why This Changes Things

Thirty to fifty percent faster. It sounds modest when stated baldly. But consider what these models are for.

Capacity expansion models are not academic exercises. They're the analytical backbone of infrastructure planning. The scenarios they produce inform investment decisions worth billions of dollars. They shape the policy recommendations that governments and utilities use to navigate the energy transition. The speed at which these models can be solved directly affects the speed at which society can plan.

The current state of the art has a ceiling. Multi-sector models with continental scope and hourly resolution already push the limits of established interior-point solvers. The practical implication is that researchers and planners make choices: reduce spatial resolution, or collapse multiple years into a single weather scenario, or drop sector coupling to keep the problem manageable. These compromises introduce uncertainty. They create blind spots. They mean that the "optimal" solution the model produces is optimal only within a constrained version of reality.

DIP-Set doesn't eliminate the ceiling, but it raises it. Problems that were structurally off-limits become tractable. Compromises that were accepted as necessary become optional. A planning team that previously had to choose between modeling the power sector with high resolution or including sector coupling with low resolution can now have both.

The implications cascade. More detailed models are more credible models. They're harder to dismiss, harder to argue with. When a capacity expansion model shows that a particular transmission corridor is cost-effective, or that long-duration storage is essential for reliability above a certain renewable penetration, that conclusion carries more weight if the model captured the relevant dynamics accurately. DIP-Set helps make the case for investment in renewable infrastructure, storage, and grid modernization—investment that faces enough political and economic obstacles without being undermined by analytical limitations.

There's also a subtler benefit: iteration speed affects model design. A model that takes a week to run invites caution. Researchers hesitate to add new features, explore alternative scenarios, or push for higher resolution. They accept the first plausible result rather than probing further. A model that takes three days invites experimentation. The researchers can test robustness, explore edge cases, and build confidence in the results. Better algorithms enable better science.

The Bigger Picture: Planning the Impossible

The energy transition is, at its core, a planning problem. Decarbonizing the power sector while electrifying transportation, heating, and industry requires not just deploying existing technologies but deploying them in a coherent system. Solar panels in the wrong places are wasted capital. Transmission lines built for yesterday's grid topology don't serve tomorrow's flows. Storage that's sized for weekly cycling fails when drought reduces hydropower for months.

The mistakes of the fossil fuel era—overbuilding here, underbuilding there, infrastructure that locks in path dependencies for decades—are expensive and politically contentious to undo. The hope of capacity expansion modeling is to avoid repeating those mistakes. To plan intelligently. To make decisions with full awareness of their consequences.

But that hope has been constrained by computational reality. The models we've built are more sophisticated than the computers we have to run them on. We've known for years what we should be modeling; we simply couldn't afford to. DIP-Set is a crack in that constraint. Not a breakthrough, not a revolution—just a 30 to 50 percent improvement for the hardest problems. But in a field where that gap was the difference between possible and impossible, it's significant.

The method isn't magic. It won't turn a problem with a billion variables into one that solves in milliseconds. It won't eliminate the need for model simplification or scenario reduction. What it does is shift the frontier. Problems that were too large for state-of-the-art methods become solvable. Problems that were solvable become fast to solve. The modeler's latitude expands.

There are limits to celebrate carefully. DIP-Set still requires the problem to be decomposable—structured enough that Benders Decomposition applies. For truly unstructured problems, or those with strong non-convexities, other approaches may be needed. The method's performance depends on parameter choices that aren't fully automated; practitioners will need experience and judgment to tune it effectively. And the 30 to 50 percent speed-up, while substantial, doesn't mean the largest problems now solve instantly. A 70 percent reduction in a multi-day runtime is still a multi-day runtime.

But the trajectory matters. Algorithmic improvements compound. What DIP-Set enables today becomes the baseline for further innovation tomorrow. The researchers who dismissed a feature as computationally infeasible can now reconsider. The planners who accepted a coarse resolution can now refine. The boundaries of tractability shift.

What's Next

The paper opens several doors.

First, there's the question of how DIP-Set performs with integer variables. Göke and Sansavini focus on continuous problems, but capacity expansion models often include discrete decisions—build a plant or don't, site a facility here or there. The integer case introduces combinatorial complexity that can exacerbate the convergence problems DIP-Set is designed to address. Early evidence suggests interior-point approaches work well for mixed-integer problems, but systematic benchmarking hasn't been done.

Second, there's the question of adaptive parameter setting. The performance of DIP-Set depends on how the dynamic parameters $\lambda_i$ and $r_i$ are chosen. Göke and Sansavini's strategies work well in their test cases, but the literature lacks generalizable insights into optimal parameter schedules. A learning-based approach—where the algorithm adapts its parameters based on observed convergence behavior—could improve performance further and reduce the burden on practitioners.

Third, there's the question of scale. The test cases in the paper are large by current standards, but the trajectory of energy modeling is toward even larger problems: finer temporal resolution (sub-hourly to capture fast dynamics), larger spatial scope (continental to global), more scenarios (to capture climate uncertainty), and deeper sector coupling (to capture system integration effects). DIP-Set will need to scale with this ambition.

Fourth, there's the question of adoption. Academic papers describe methods; practitioners decide whether to implement them. DIP-Set is a modification to an existing algorithm (Benders Decomposition) and can be implemented in off-the-shelf optimization solvers with modest effort. The code is presumably available from the authors. But the energy modeling community is fragmented, with many groups using proprietary or semi-proprietary frameworks. Widespread adoption will require outreach, documentation, and perhaps reference implementations in common modeling platforms.

Finally, there's the question of what this enables beyond what's already described. Every algorithmic improvement opens research directions that weren't previously thinkable. DIP-Set may enable stochastic capacity expansion models with more scenarios, or robust optimization formulations that guarantee performance across a wider range of uncertainties, or higher-resolution sector-coupled models that capture the detailed interactions between electricity, heat, and transport. The paper doesn't explore these possibilities, but that's the nature of enabling technologies: the most important applications are often the ones you can't anticipate.

The Takeaway

The energy transition requires planning. Planning requires models. Models require computation. Computation has limits.

Göke and Sansavini have pushed those limits. Their method—DIP-Set—solves large-scale capacity expansion problems 30 to 50 percent faster than existing approaches, with the greatest gains for the hardest problems. It does this by combining two regularization strategies: trust-region (keeping solutions close to known good territory) and interior-point (moving through the interior of the solution space). Each approach alone is incomplete. Together, they're more than the sum of their parts.

The immediate benefit is practical: models that ran for days can now run for hours or finish overnight. The broader implication is strategic: problems that were structurally off-limits become tractable. Researchers and planners can work at higher resolution, with more scenarios, and with deeper sector coupling. The models can capture more of the real world's complexity because the computational cost of doing so is lower.

None of this is revolutionary. The ceiling hasn't been lifted; it's been raised a few inches. But in a field where those inches were the difference between possible and impossible, they're worth having. And they point toward further improvements—further lifting of the ceiling—as the method is refined, adopted, and built upon.

The supercomputer at ETH Zurich is still wrestling with impossible problems. But it's winning more of them now.