Teaching Algorithms to Model Epidemics From Imperfect Data
In 2013, the Caribbean reported its first case of chikungunya—and within eighteen months, more than a million people across the Americas had been infected. The virus, transmitted by Aedes mosquitoes, causes fever, incapacitating joint pain, and sometimes persistent arthritis that lasts months or years. It spreads fast because most infected people don't show symptoms immediately, and those who do may never know they have chikungunya versus dengue or Zika without laboratory confirmation. By the time public health officials recognize an outbreak, it has often already escaped containment.
This gap—between what surveillance data can tell us and what we need to know to act—defines one of the hardest problems in epidemic modeling. The mathematical models epidemiologists rely on to forecast outbreaks and evaluate interventions are elegant in theory. They divide populations into compartments—susceptible, exposed, infectious, recovered—and write down differential equations describing how people flow between them. In principle, you can solve those equations and predict peak infections, final epidemic sizes, and the effects of vaccinating or spraying. In practice, the models demand precise values for parameters that no one actually knows: transmission rates, incubation periods, the probability that a mosquito bite leads to infection. Real surveillance data only samples some of the compartments, at irregular intervals, with measurement error. The result is a toolkit that looks powerful but often produces unreliable forecasts.
A team of mathematicians at Utah State University has now proposed a different approach. Bernard Asamoah Afful, Changhong Mou, and Luis Gordillo have developed a hybrid method that combines two powerful computational techniques—Sparse Identification of Nonlinear Dynamics (SINDy) and the Ensemble Kalman Filter (EnKF)—to learn the structure of an epidemic model directly from imperfect data, correct it as new observations arrive, and infer information about compartments that no one is actually measuring. In numerical experiments simulating a chikungunya outbreak, their framework recovered the correct governing equations when the data were clean, degraded gracefully as noise increased, and—with the Kalman filter working in tandem—reconstructed unobserved compartments with reasonable accuracy even when the underlying model equations were only imperfectly known. It is, in essence, a machine-learning system that learns how a disease spreads from the data, then continuously updates its best guess as new, noisy, incomplete information arrives.
The work matters not because it solves epidemiology's parameter-estimation problem overnight, but because it points toward a genuinely new way of thinking about epidemic modeling: one that does not require researchers to know the right equations in advance, does not collapse when data are messy, and can work with the partial, delayed, error-prone surveillance that is the reality in most of the world.
The Science
Chikungunya and the Compartmental Model
Before building their data-driven framework, the researchers needed a detailed mechanistic model to serve as both a benchmark and a source of synthetic data. Chikungunya virus (CHIKV) spreads through a host-vector interaction: humans carry the virus, mosquitoes bite them, and infected mosquitoes transmit it to the next person they bite. The model the researchers built reflects this dual-population structure.
The human population is divided into seven compartments. Susceptible hosts \((S_h)\) are those who can become infected. When a susceptible person is bitten by an infectious mosquito, they move into the exposed compartment \((E_h)\), carrying the virus but not yet infectious themselves. After an incubation period of roughly two to twelve days, they transition into one of two infectious states: symptomatically infectious hosts \((I_h)\) experience the characteristic fever and arthralgia, while asymptomatically infectious hosts \((J_h)\) carry and transmit the virus without obvious illness. Some symptomatic individuals receive treatment \((T_h)\), and eventually all infectious individuals recover \((R_h)\) with at least temporary immunity. A seventh compartment, vaccinated hosts \((V_h)\), accounts for a portion of the population with vaccine-derived protection, which wanes over time and can return individuals to the susceptible class.
The mosquito population is simpler—just three compartments. Susceptible vectors \((S_v)\) can acquire the virus by biting an infectious human. They then enter an exposed phase \((E_v)\) during which the virus replicates in the mosquito's body. Once the virus reaches the mosquito's salivary glands, the vector becomes infectious \((I_v)\) and remains so for the rest of its short life.
The full model consists of ten coupled ordinary differential equations, one for each compartment, plus equations describing the total host and vector populations. These equations track recruitment (births of new susceptible individuals), mortality (both natural and disease-induced), vaccination, waning immunity, transmission through mosquito bites, incubation, recovery, and treatment. The transmission term follows the law of mass action—a foundational principle in epidemic modeling that treats infection as a chemical reaction between infectious mosquitoes and susceptible humans, scaled by the transmission probability per bite. The model also incorporates vaccine efficacy: vaccinated individuals can still become infected, but at a reduced rate that reflects partial protection.
To generate synthetic data for their experiments, the researchers set initial conditions representing a realistic outbreak scenario: 1,000 susceptible humans, 50 initially exposed, 10 symptomatic infections, 5 asymptomatic infections, and a mosquito population of roughly 20,000 (20,000 susceptible, 100 exposed, 50 infectious). They integrated the equations using a fourth-order Runge-Kutta method with a fine time step to produce reference trajectories—the "ground truth" against which they would later evaluate their data-driven reconstructions.
Figure 1. Illustration of the Chikungunya virus (CHIKV) model, showing the seven host compartments (susceptible, vaccinated, exposed, symptomatic infectious, asymptomatic infectious, treated, and recovered) and the three vector compartments (susceptible, exposed, and infectious mosquitoes), with arrows indicating transitions between states driven by infection, incubation, treatment, recovery, vaccination, and mortality.
SINDy: Discovering Equations from Data
The first half of the hybrid framework is Sparse Identification of Nonlinear Dynamics, or SINDy. Developed by Steven Brunton, Joshua Proctor, and J. Nathan Kutz in 2016, SINDy is a method for discovering the governing equations of a dynamical system directly from observed data. The idea is surprisingly elegant.
Imagine you have time-series measurements of a system's state—say, the number of susceptible, exposed, and infectious individuals at many points in time. SINDy assumes that the equations governing the system's evolution can be written as a sparse combination of simple candidate functions. Rather than trying to fit a predetermined model structure, it constructs a library of candidate terms—constants, linear terms like \((S_h)\), and nonlinear interactions like \((S_h \cdot I_v)\)—and then asks which combination of these candidates best explains the observed rates of change.
Mathematically, if \(\mathbf{x}(t)\) is the state vector containing all compartment populations, SINDy seeks a sparse coefficient matrix \(\boldsymbol{\Xi}\) such that
where \(\boldsymbol{\Theta}(\mathbf{x})\) is the candidate library—a matrix whose columns evaluate each candidate function at every time point. The coefficient matrix \(\boldsymbol{\Xi}\) tells you which terms actually appear in each equation and with what weights.
To find the sparse solution, SINDy uses an algorithm called Sequentially Thresholded Least Squares (STLSQ). It starts with a standard least-squares fit, then zeroes out any coefficients below a chosen threshold \(\lambda\), solves the reduced problem, and repeats. The threshold acts as a sparsity knob: larger values produce simpler models with fewer terms; smaller values retain more terms and fit the data more closely. The goal is to find the simplest model that still accurately captures the dynamics—a principle known as Occam's Razor applied to differential equations.
In the clean-data experiment, the researchers constructed their candidate library from all linear terms (each compartment variable on its own) and all pairwise bilinear products (terms like \((S_h \cdot I_v)\), \((E_h \cdot I_h)\)), which matches the structure of mass-action epidemic models. With a very small threshold \(\lambda = 10^{-10}\), the algorithm recovered the correct equations for all ten compartments—the reconstructed dynamics were essentially indistinguishable from the ground truth, with component-wise root mean square errors below 1 for every compartment.
The Noise Problem
The clean-data result is encouraging but misleading in an important way. Real surveillance data is never clean. Case counts are underreported (especially for a disease like chikungunya, where asymptomatic transmission is common), sampled irregularly (reporting delays mean you learn about infections days or weeks after they occurred), and corrupted by measurement error. More fundamentally, SINDy requires numerical derivatives—estimates of \(\frac{d\mathbf{x}}{dt}\) computed from the data—and differentiation amplifies noise in a way that regression cannot easily filter. Small errors in the measurements become large errors in the estimated derivatives, which then corrupt the sparse regression. The result can be spurious terms (noise artifacts that look like real dynamics), missing terms (real dynamics that drown in derivative noise), or complete failure of the identification.
The researchers quantified this degradation systematically. They added increasing levels of Gaussian noise to the clean trajectories—simulating 5%, 10%, 15%, and 25% noise—then reapplied SINDy to the corrupted data. As the noise level rose, the reconstructed equations increasingly diverged from the true ones. The errors propagated into forward simulation, causing the predicted trajectories to wander away from reality. At 25% noise, the discrepancy was substantial enough that standalone SINDy could no longer be trusted.
This is not a new discovery—SINDy's noise sensitivity has been documented in fluid dynamics, neuroscience, and other fields. But it is a critical limitation for epidemiology, where noisy, incomplete surveillance data is the norm rather than the exception. If the data-driven model itself becomes unreliable precisely when data are most imperfect, the practical value of SINDy for epidemic forecasting is limited.
The Ensemble Kalman Filter: Inferring the Unseen
The second half of the framework addresses this limitation. The Ensemble Kalman Filter (EnKF) is a sequential data-assimilation method originally developed for geophysical applications—forecasting weather, tracking ocean currents, and monitoring groundwater systems. It works by maintaining an ensemble of model states, propagating them forward through time, and updating them when observations become available.
The intuition is straightforward. You start with an ensemble of plausible initial states, spread around your initial uncertainty. You then integrate each member of the ensemble forward in time using the model equations, generating a forecast distribution of where the system might be at the next time step. When an observation arrives, you compare it to what each ensemble member predicted, and you adjust every member of the ensemble accordingly—pulling them toward the observation in proportion to how well each member predicted it. Ensemble members that were closer to the observation are pulled less; those that were farther are corrected more. The correction also propagates across variables: if you only observe some compartments but not others, the cross-variable correlations in the ensemble tell you how a correction to one compartment should adjust your estimate of unobserved compartments.
Mathematically, the EnKF approximates two key quantities at each step. The forecast step computes the prior ensemble mean \(\bar{\mathbf{x}}{n+1|n}\) and covariance matrix \(\mathbf{P}{n+1|n}\) from the ensemble spread. The analysis step computes the Kalman gain \(\mathbf{K}_n\)—a matrix that determines how much each observation should influence each state estimate—and applies it to update each ensemble member:
where \(\mathbf{z}_{n+1}^{(j)}\) is a perturbed observation drawn for ensemble member \((j)\), and \((H)\) is the observation operator mapping from the full state space to the observed subspace.
The result is an updated ensemble whose mean provides a filtered state estimate and whose spread quantifies uncertainty. Critically, the EnKF can infer unobserved compartments purely from their statistical relationships with observed ones. If infectious human count and exposed human count tend to move together in the model (because exposed individuals become infectious), then when you correct an error in your estimate of infectious individuals, the filter automatically adjusts your estimate of exposed individuals in a way that reflects their learned coupling. You do not need to measure \((E_h)\) directly to estimate it—you just need enough observations to learn the cross-correlation structure.
This capability makes the EnKF particularly well-suited for epidemiological applications. Surveillance systems typically observe symptomatic case counts \((I_h)\) and sometimes treated counts \((T_h)\), but they almost never observe the exposed compartment \((E_h)\), the asymptomatic compartment \((J_h)\), or any of the vector compartments. Yet understanding the exposed population is crucial for early-warning systems: if you know there is a large exposed reservoir building up, you know the epidemic is about to surge even before the case counts reflect it.
The Hybrid Framework
The researchers' key innovation is to embed the SINDy-derived model inside the EnKF framework, combining the two methods in a way that compensates for each other's weaknesses.
In their setup, SINDy is first applied to clean or moderately noisy data to identify the structure of the governing equations—the sparse coefficient matrix that tells you which terms belong in each compartment's equation. This identified model then serves as the forward operator for the EnKF: it is the rule used to propagate each ensemble member from one time step to the next. The EnKF, in turn, takes the SINDy model as a starting point but continuously corrects it using noisy observations, prevents the accumulation of forecast errors, and reconstructs unobserved compartments from cross-variable correlations.
This is a fundamentally different architecture from previous hybrid approaches. In EKF-SINDy, a related method developed for structural engineering applications, the SINDy model and Extended Kalman Filter are tightly coupled for joint state and parameter estimation—the SINDy coefficients themselves are corrected by the filter in real time. The Utah State framework keeps the SINDy model structure fixed (the identified equations do not change) but allows the EnKF to correct the state estimates as observations arrive. This separation of concerns—SINDy handles equation discovery, EnKF handles state estimation—is cleaner and potentially more robust, since it does not try to simultaneously calibrate the model and estimate the state.
The researchers also explored how frequently observations need to arrive for the system to work. They tested observation intervals ranging from 0.1 to 10 time units (where the simulation time step was 0.01) and found that the EnKF performed best with more frequent observations, as expected, but maintained reasonable accuracy even when observations were sparse—provided the partial observability was informative enough to constrain the unobserved compartments.
Figure 2. Illustration of the proposed learning framework, showing how SINDy identifies the governing equations from time-series data, which then serve as the forward model for the Ensemble Kalman Filter. The EnKF ingests noisy, partially observed measurements and produces corrected state estimates for both observed and unobserved compartments.
What They Found
SINDy Recovers the Truth When Data Are Clean
In the noise-free baseline experiment, SINDy performed exceptionally well. The recovered equations exactly matched the structure of the original model for all ten compartments. When the researchers simulated the identified equations forward in time from the same initial conditions, the trajectories overlaid the ground truth nearly perfectly—component-wise root mean square errors below 1 for both host and vector compartments. The epidemic's initial growth phase, its peak, and its eventual approach to equilibrium were all reproduced with high fidelity.
This result confirms that for clean data, SINDy is capable of faithfully discovering the governing equations of a complex host-vector epidemic model. The candidate library—constants, linear terms, and bilinear products—contained the correct structure, and the sparse regression correctly identified which terms were active. It is an existence proof: data-driven epidemic modeling is at least possible in principle.
SINDy Degrades Under Noise
When the researchers added increasing levels of Gaussian noise to the trajectories, SINDy's performance declined systematically. At 5% noise, some compartments were still tracked reasonably well, but others began showing visible divergence between the reconstructed and true trajectories. At 10% and 15% noise, the errors grew more pronounced, with the identified equations sometimes retaining spurious interaction terms or missing weak but real dynamics. At 25% noise—which is not unusual for real surveillance data, where underreporting and diagnostic uncertainty can easily exceed this magnitude—the reconstructed model produced trajectories that bore only qualitative resemblance to the truth.
The mechanism of failure is instructive. The numerical differentiation step—computing \(\frac{d\mathbf{x}}{dt}\) from the noisy state measurements—is the bottleneck. Differentiation amplifies high-frequency noise into apparent high-magnitude derivatives, which then corrupt the sparse regression. Even a few misestimated derivative values can shift the coefficient estimates enough to introduce spurious terms or suppress genuine ones.
The Hybrid Framework Restores Accuracy
The combined SINDy-EnKF system is where the results become genuinely interesting. When the researchers used the SINDy-identified model as the forward operator within the EnKF, and then assimilated noisy observations with the Ensemble Kalman Filter, the filtered trajectories tracked the true clean dynamics substantially better than standalone SINDy did. The EnKF corrected the forecast errors that accumulated in the SINDy model's forward integration, pulling the state estimates back toward the true trajectory whenever new observations arrived.
Figure 8. EnKF reconstruction of observed host compartments under 25% noise. The plot compares the noisy observations (used as input), the raw SINDy trajectory (prior to filtering), the filtered posterior mean from the EnKF, and the true clean trajectory for the observed compartments: vaccinated hosts \((V_h)\), symptomatic infectious hosts \((I_h)\), asymptomatic infectious hosts \((J_h)\), and treated hosts \((T_h)\). The filtered estimates closely track the true dynamics despite severe observation noise.
Perhaps more importantly, the EnKF reconstructed unobserved compartments with meaningful accuracy. In the partial-observability experiments, the researchers assumed that only four of the ten compartments were observed—vaccinated hosts, symptomatic and asymptomatic infectious hosts, and treated hosts. The remaining six compartments (susceptible humans, exposed humans, recovered humans, susceptible vectors, exposed vectors, and infectious vectors) were treated as unobserved, as they would be in a real surveillance scenario. The EnKF used the cross-compartment correlations learned from the ensemble to infer these unobserved states.
The filter's performance depended on the observation frequency. More frequent observations led to tighter estimates, as the correction step ran more often and kept the ensemble tightly constrained. But even with observations spaced 1 to 2 time units apart (out of a simulation horizon that extended to hundreds of time units), the reconstruction of both observed and unobserved compartments remained reasonably accurate. The researchers reported normalized root mean square errors and temporal correlations indicating that the EnKF successfully recovered the true dynamics for most compartments.
Initial Conditions for Host Compartments
Initial population distribution across the seven host compartments at the start of the simulated outbreak.
| Label | Value |
|---|---|
| Susceptible hosts (Sh) | 1,000 individuals |
| Exposed hosts (Eh) | 50 individuals |
| Symptomatic infected (Ih) | 10 individuals |
| Asymptomatic infected (Jh) | 5 individuals |
| Treated (Th) | 0 individuals |
| Recovered (Rh) | 0 individuals |
Initial Conditions for Vector Compartments
Initial population distribution across the three vector compartments at the start of the simulated outbreak.
| Label | Value |
|---|---|
| Susceptible vectors (Sv) | 20,000 mosquitoes |
| Exposed vectors (Ev) | 100 mosquitoes |
| Infectious vectors (Iv) | 50 mosquitoes |
Why This Changes Things
Epidemiologists have long understood that mechanistic models are only as good as their parameters, and that parameters estimated in one outbreak often fail to generalize to the next. The traditional response has been to build increasingly elaborate models—adding compartments, estimating more parameters, calibrating to more data—and to hope that structural realism compensates for parameter uncertainty. It rarely does. A model with fifty parameters and imperfect data typically produces forecasts with large confidence intervals, and those intervals widen further as you project further into the future.
The hybrid SINDy-EnKF framework takes a different approach. Rather than postulating a detailed mechanistic model and then searching for parameters to fit it, it asks: what can the data itself tell us about the structure of the dynamics? SINDy does this by finding the sparse combination of candidate functions that best explains the observed rates of change. The EnKF then provides a way to use the identified model for forecasting and reconstruction under realistic observational constraints.
There are several reasons this matters for the future of epidemic modeling.
It reduces reliance on assumed mechanisms. Traditional epidemic models encode assumptions about how diseases spread—in the case of chikungunya, that transmission occurs through mosquito bites, that incubation is a single-stage process, that recovered individuals have temporary immunity. These are reasonable assumptions backed by biological evidence, but they are still assumptions. SINDy can recover the correct structure from data without being told what the structure should be, which means it might identify genuine deviations from the assumed mechanism—unexpected transmission routes, age structure effects, spatial heterogeneity—that would be invisible to a model that was locked into the wrong structure from the start.
It works with the data you actually have. Most countries do not have the surveillance infrastructure to measure all compartments of an epidemic model in real time. They observe symptomatic cases, perhaps some hospitalization data, and maybe a sample of serological surveys. The EnKF's ability to infer unobserved compartments from observed ones—using cross-correlation structure rather than direct measurement—is potentially transformative for regions where comprehensive surveillance is not feasible. If a health ministry can observe case counts and use the filter to estimate the size of the exposed reservoir, they have actionable intelligence for early warning without needing to sample the exposed population directly.
It is computationally tractable. The EnKF is Monte Carlo-based: it runs an ensemble of model integrations in parallel and uses their spread to estimate uncertainty. This makes it scalable to high-dimensional systems without the computational cost of traditional Kalman filtering or the curse of dimensionality that affects many other data-assimilation methods. For a ten-compartment model like the chikungunya system, an ensemble of moderate size (tens to hundreds of members) is sufficient, and the forward model—either the SINDy-identified equations or the original mechanistic model—is cheap to integrate.
It provides uncertainty quantification. Many machine-learning methods for time-series forecasting produce point estimates without confidence intervals. The EnKF, by maintaining an ensemble, naturally provides a distribution over possible states. Public health decision-makers dealing with an outbreak need to know not just what the best estimate is, but how uncertain that estimate is—which models call for immediate action and which warrant a wait-and-see approach.
The comparison with existing SINDy variants is also worth noting. Ensemble-SINDy, which uses bootstrap aggregation to improve noise robustness, and SINDy-PI, which reformulates the identification as a convex optimization to handle implicit dynamics, are both impressive advances. But they address the noise problem within the identification step itself. The hybrid approach here takes a different route: it does not try to make SINDy more robust to noise. Instead, it accepts that SINDy's identified model will be imperfect and lets the EnKF compensate. This separation of roles—one method discovers the structure, another corrects the state—may prove more robust in practice than approaches that try to do both things within a single algorithm.
For chikungunya specifically, the results are promising but preliminary. The numerical experiments use synthetic data generated from a known model, which means the researchers know the ground truth. Real chikungunya surveillance data would introduce additional complexities: reporting delays (you learn about infections days after they occur, not on the day they start), systematic underreporting (actual cases may outnumber reported cases by a factor of ten or more), spatial structure (outbreaks are not uniform across a region), and temporal changes in transmission rates due to seasonal mosquito abundance, human behavior, or intervention campaigns. None of these complications are addressed in the current paper.
What's Next
The obvious next step is to test the framework on real chikungunya surveillance data. The researchers' numerical experiments establish proof of concept, but whether the method can handle the additional layers of complexity in real epidemiological data remains to be seen. Ideally, this would involve historical outbreak data from regions with reasonably good surveillance—Brazil, Thailand, and several Indian states have reported major chikungunya outbreaks—and would compare the hybrid framework's forecast accuracy against established methods like SEIR models with maximum-likelihood parameter estimation.
There are also several methodological extensions worth exploring. The current framework uses a fixed SINDy model: once the governing equations are identified, they do not change during the assimilation. An adaptive version that allows the model structure to evolve as new data arrives—incorporating time-varying transmission rates or seasonal forcing—would be more realistic for real-world applications. The EKF-SINDy approach, which jointly updates state and parameters, offers one path toward this capability, though it comes with the risk of overfitting the model structure to noisy data.
Another important direction is quantifying the sensitivity to model form. SINDy's success depends on the candidate library containing the true terms. If the actual dynamics include a term that is absent from the library—say, a nonlinear saturation effect in transmission at high prevalence—the identified model will be misspecified, and the EnKF will be trying to correct a fundamentally wrong forecast. One could mitigate this by using adaptive libraries that grow as needed, or by using the residual analysis from the EnKF to detect model misspecification and flag the analyst.
The partial-observability results are perhaps the most practically significant part of the paper, but they also raise the most questions. The experiments assumed that four specific compartments were observed. In practice, which compartments are observed varies by country, by disease, and by reporting system. Understanding which observation subsets are most informative for reconstructing the full epidemic state—which compartments carry the most information about the others—would help surveillance systems prioritize their data collection. It would also help identify which surveillance gaps are most consequential for forecast accuracy.
Finally, the framework is currently validated on a single disease. Chikungunya's transmission dynamics share important features with dengue, Zika, and yellow fever—arboviral diseases with asymptomatic transmission, vector-borne spread, and similar surveillance challenges. Testing whether the approach generalizes to these related diseases, and whether the identified equations transfer across geographic regions with different mosquito populations and human behaviors, would establish whether this is a broadly applicable tool or a niche method for specific contexts.
The broader significance of this work is that it demonstrates, in a controlled setting, that data-driven epidemic modeling can work even under realistic conditions of noise and partial observability. It is not a magic bullet. The framework has real limitations: it requires sufficient data to identify the model structure, it depends on the candidate library containing the true terms, and it has not yet been validated against real surveillance data. But it points toward a future in which epidemic models are not constructed purely from prior biological knowledge and then fitted to data, but are actively learned from the data as it arrives—in real time, under uncertainty, with the humility that comes from knowing your model is probably wrong in some way and building in correction mechanisms accordingly.
Whether that future arrives for chikungunya, for other arboviruses, or for respiratory diseases with entirely different transmission dynamics, remains to be seen. But the framework offers a template: discover the structure from data, forecast with the discovered model, correct with incoming observations, and infer what you cannot measure from what you can. In a world where surveillance data is always incomplete and the next outbreak is always coming, that is exactly the kind of tool public health needs.