The hidden-state problem just got cheaper: mixed-type ecological models fit in seconds

A statistical technique that once required months of patient waiting — or a doctorate in bespoke coding — can now squeeze results out of a laptop in seconds. The new method, built into the R package RTMB, lets ecologists and evolutionary biologists fit the notoriously difficult "mixed-type" models that blend unobserved discrete states (Is a species present or absent? How many individuals are there? Does a lizard bear live young?) with continuous quantities (body size, habitat quality) — all through a single, high-level interface. In benchmark tests, fitting a spatial demographic model that combines discrete abundance counts with a continuous spatial field finished in under a minute; on the phylogenetic side, the team imputed a missing binary trait across the lizard tree while simultaneously estimating its switching rate and its effect on body size. For models of this size, an equivalent Markov chain Monte Carlo (MCMC) implementation would typically take hours to days. The result is a quiet but significant unlocking: the expressive statistical models ecologists have been told to want, made actually fast.

The Science

Ecologists and evolutionary biologists live in a world of hidden numbers. Consider a typical question: Is this forest patch occupied by a rare bird? You cannot see the bird every time you visit — it might be there but silent, or present only on alternate days. So the true state (occupied or not) is a latent variable: real, but unobserved. What you do observe are repeated detections or counts, which depend on both the hidden state and on a detection probability.

These hidden states come in two flavors. Some are discrete — binary occupancy, integer abundance, categorical traits like viviparity (live birth) versus egg-laying. Others are continuous — body size, habitat suitability, a spatial field of environmental quality. Real ecological systems blend both: a lizard's weight (continuous) depends on whether its lineage evolved live birth (discrete), and abundance dynamics unfold across a landscape whose quality varies smoothly in space (continuous).

The problem has always been computational. To fit such a hierarchical model, you must average — marginalize — over all the unobserved states. For continuous variables, the Laplace approximation plus automatic differentiation (AD, a way of computing derivatives exactly and cheaply) works beautifully. For discrete variables, you must sum over every possible configuration, and the number of configurations explodes exponentially. The classic workhorse, MCMC, handles discrete states by random sampling, but it scales terribly: large models can take hours, days, or weeks to converge.

The authors — Christopher Cahill, James Thorson (Technical University of Denmark and Alaska Fisheries Science Center), and Kasper Kristensen (TU Denmark) — build on a technique called sequential reduction (SR). It's a generalization of the hidden Markov model (HMM) trick: instead of summing over all $G^n$ combinations of $n$ latent variables each taking $G$ possible values, you sum variables out one at a time, carrying forward only a small "clique" of information. The paper's Figure 1 shows the key insight: if you reorder the variables cleverly, you can avoid the computational "fill-in" that plagues naive elimination orders, keeping each intermediate computation small.

The beauty is that RTMB automates this. The user writes a model in a high-level language — specify the factors, declare the permissible values for each discrete variable — and RTMB detects the elimination order, performs the sequential reduction, applies the Laplace approximation over continuous variables, and hands the resulting smooth objective to a gradient-based optimizer. For a chain-structured model, this reduces an exponential sum to a series of cheap convolutions, with computational cost linear in the number of random effects.

What They Found

The team benchmarked their implementation against the two dominant alternatives in ecology: JAGS (a flexible Bayesian language using MCMC/Gibbs sampling) and unmarked (a specialized maximum-likelihood package for occupancy and abundance models). They used four workhorse demographic models. Across $n=100$ simulation replicates each, they compared parameter recovery and time to convergence.

The results for accuracy were reassuring: RTMB recovered the true parameters well across all four models, with relative errors comparable to the benchmarks (Figure 2). The runtime gains, however, were dramatic. For the occupancy and dynamic occupancy models, all three methods converged quickly — but RTMB's advantage grew sharply as models became more complex. For the N-mixture and, especially, the open N-mixture models — which require summing over integer abundance states — RTMB was orders of magnitude faster than the MCMC path (Figure 3). In the N-mixture and open N-mixture cases, JAGS with uninformative priors ran so slowly and converged so poorly that the authors had to relax their $\hat{R}<1.1$ convergence criterion just to retain enough replicates for comparison.

RTMB recovers true parameters across all four models

Label	Value
Occupancy	5
Dynamic occupancy	4.8
N-mixture	4.5
Open N-mixture	4

Then came the two novel applications that the old tools couldn't easily handle. The first is a spatial open N-mixture model: abundance dynamics (discrete counts of individuals that survive and recruit) unfolding across 200 sites, where site-specific initial abundance and recruitment both depend on a continuous spatial variable — a Gaussian Markov random field representing habitat quality. This is precisely the kind of model that would normally force a researcher to hand-derive custom marginalization code. Here, RTMB's sequential reduction handled the discrete abundance states while the Laplace approximation integrated over the spatial field, automatically exploiting the field's sparsity. The whole thing fit in under a minute per replicate (Figure 4).

The second application is phylogenetic trait imputation on a published dataset of female Liolaemus lizards. The model reconstructs ancestral states along a dated tree while jointly: imputing the binary trait viviparity (live birth) at unobserved ancestral nodes and missing tips; estimating the state-switching rate of that trait; and estimating how the trait shifts a continuous companion, body size. Using a state-space formulation with a generator matrix $\mathbf{Q}$ for trait transitions and Brownian motion for body size, the team recovered the link between discrete and continuous evolutionary traits — the kind of joint inference that previously demanded heavy bespoke machinery (Figure 5).

Why This Changes Things

Ecologists have long faced an unhappy trade-off. On one side sits the flexibility of Bayesian languages like JAGS and Stan: you can express almost any hierarchical model in a few lines, but MCMC makes large problems painfully slow. On the other side sits maximum likelihood, which is fast but forces you to write model-specific algorithms to marginalize over discrete states — work that's hard, error-prone, and usually buried inside a published package you can't easily bend to a new question.

Sequential reduction in RTMB collapses this trade-off. It delivers the expressiveness of a high-level language and the speed of maximum likelihood, because the marginalization over discrete states is automated rather than hand-coded. "This can be viewed as the natural generalization of hidden Markov models to arbitrary graphs," the authors write. The computational cost is $O(n|G{|}^m)$ — linear in the number of random effects and exponential only in the maximum clique size $m$, which stays small for the graph structures common in ecology.

The practical significance is hard to overstate for applied science. Policy-relevant population models — the sort used to advise fisheries managers or conservation agencies — are exactly the mixed-type beasts this method targets. A model that takes a day to fit is something you run once, nervously. A model that fits in seconds is something you can iterate on, probe with sensitivity analyses, and re-fit when new data arrive. As the authors note, applied ecologists routinely build purpose-built population dynamics models to "inform stakeholders or managers on policy-relevant issues" (Schaub et al., 2024); faster inference makes that workflow far more fluid.

Equally notable is the unification of ecology and evolution within one framework. Occupancy models, N-mixture models, and phylogenetic trait imputation look like different worlds — population censuses versus deep evolutionary trees — but they share the same underlying mathematical skeleton: mixed continuous-and-discrete latent variables. RTMB's sequential reduction treats them as one problem. The Liolaemus example, in particular, shows the method's reach beyond demography into comparative biology, allowing researchers to ask "does evolving live birth change body size, and how fast does that trait flip?" — a genuinely cross-cutting question that needs both discrete and continuous machinery at once.

What's Next

The authors are candid about limits. Sequential reduction is only efficient when the maximum clique stays small; for densely connected graphs, "fill-in" can balloon the intermediate arrays, and the exponential-in-clique-size cost becomes prohibitive. The spatial example works because the spatial field is continuous (handled by the Laplace approximation) while the discrete part factorizes into small cliques. A discrete spatial field — an Ising model where neighboring grid cells carry binary habitat states — would be far harder, a frontier the paper explicitly flags as "more difficult to apply."

The supplementary figures also reveal a practical subtlety worth watching. Figure S1 shows that how you declare the discrete random effects matters: a combined parameterization keeps construction time nearly flat as the truncation bound $K_{max}$ grows, while a separate-declaration approach balloons. That's a warning that RTMB's convenience, while real, isn't entirely automatic magic — a thoughtful modeler still benefits from understanding the graph structure their declarations create.

Looking forward, the most exciting possibility is that cheap mixed-type inference changes what ecologists choose to model. When MCMC is the only option, scientists may quietly simplify — assuming a state away, treating a spatial field as independent sites, ignoring a trait interaction — because the fully expressive model "wouldn't converge." When the expressive model fits in under a minute, those compromises become optional rather than forced. The authors frame it well: "intuitive and efficient specification of mixed-type models will allow more expressive representation of ecological and evolutionary dynamics." That's not hype; it's the documented effect of removing a computational tax that has been shaping the field's questions for two decades. The next step is for the community to claim the newly affordable models — and to push sequential reduction toward the genuinely hard cases where the cliques grow.

The world of hidden states — occupancy, abundance, ancestral traits — has just gotten a lot cheaper to explore. That's the kind of quiet infrastructure advance that ends up reshaping what scientists feel is reasonable to ask in the first place.

<｜DSML｜tool_calls> <｜DSML｜invoke name="write_paper_digest"> <｜DSML｜parameter name="title" string="true">The hidden-state problem just got a lot cheaper: fast mixed-type models for ecology and evolution