When Compression Meets Filtering: Training Models to Do What You Actually Need
When you're trying to track something you can't fully see — the heat bleeding through a factory floor, the pressure rippling through an oil pipeline, the forces tugging on a robot arm — you're working with incomplete, noisy information. The math that makes this possible is called state estimation. And for sixty years, the Kalman filter has been its gold standard: a principled algorithm that predicts, measures, and refines estimates of hidden system states. But there's a catch. Real-world systems — fluid flows, power grids, chemical reactors — can have thousands or millions of interdependent variables. Running a Kalman filter on these systems means propagating enormous matrices at every time step. On an embedded chip in a drone, that's not feasible.
A new paper from researchers at IDSIA Dalle Molle Institute for Artificial Intelligence proposes a different approach. Rather than designing the reduced model and the filter as separate pieces, they train both simultaneously, optimized entirely for one goal: accurate state estimation. The result is a compressed, linear representation of a nonlinear system that can be filtered cheaply and accurately. In tests on a heat diffusion benchmark — estimating temperature across a 100-node rod from sparse measurements — their method achieved estimation errors comparable to a full-order filter running on the complete system, while operating in a latent space roughly 20 times smaller. They also provide, for the first time in this setting, probabilistic bounds on how much performance you'd sacrifice by using the compressed model instead of the full one.
The Challenge of Seeing What's Hidden
Before diving into the solution, it's worth understanding the problem at a conceptual level. Many engineering systems are governed by nonlinear dynamics: the way heat diffuses through a material, the flow of air over a wing, the oscillations in a power grid. If you know the current state of such a system and its inputs, you can predict what comes next. But you rarely have perfect knowledge of the current state — you're relying on sensors that only sample a few locations, contaminated by noise.
State estimation is the task of combining your model of how the system evolves with the incomplete, noisy data coming in from sensors, to produce the best possible guess of what's actually happening. The Kalman filter, introduced by Rudolf Kálmán in 1960, provides an optimal solution for linear systems under Gaussian noise. For nonlinear systems, variants like the Extended Kalman Filter (EKF) linearize the dynamics locally and apply the same logic. The core idea: predict the next state, compare your prediction against what sensors report, compute a correction, and update.
The trouble is computational. The Kalman filter maintains a covariance matrix — a measure of uncertainty — that scales with the square of the state dimension. A system with 1,000 state variables requires a 1,000 × 1,000 covariance matrix. At every time step, you must propagate this matrix through matrix multiplications and inversions. On systems with millions of variables, this is intractable.
The standard workaround is model reduction: compress the high-dimensional system into a low-dimensional equivalent that captures the essential behavior. But here's where conventional approaches stumble. They optimize the reduced model for something like predicting the system trajectory — not for estimating states from sparse, noisy measurements. A representation that looks good for prediction may not be good for filtering. The structure of your sensors, the noise characteristics, the specific dynamics that matter for estimation: these don't enter the model-building phase. You're solving a proxy problem and hoping the solution transfers.
Compressing the Problem, Keeping What Matters
The core innovation in the paper by Mejari, Banitalebi Dehkordi, and Piga is to flip this around. Instead of first building a model and then designing a filter on top, they train both the model and the filtering algorithm jointly, with the loss function shaped entirely by the estimation task.
The architecture has two parts. The first is an autoencoder — a neural network with an encoder and decoder. The encoder maps the high-dimensional state of the system into a compact latent representation. The decoder does the reverse: it takes a point in this low-dimensional space and lifts it back to the original high-dimensional space. If the system has 100 temperature sensors along a rod, the encoder might compress that 100-dimensional vector into, say, 5 or 10 dimensions. The decoder is trained to reconstruct the original 100-dimensional state from these 5 numbers.
The second part is a reduced-order linear time-invariant (RO-LTI) model. This is a simple state-space model — linear dynamics with constant coefficients — operating in the latent space. In continuous form, it looks like this:
where $\mathbf{z}$ is the latent state, $\mathbf{u}$ is the input, $A$ and $B$ are matrices learned from data, and $\mathbf{w}$ is process noise. The output equation couples the latent state to the measurements:
where $\mathbf{y}$ is the sensor reading and $\mathbf{v}$ is measurement noise. All of $A$, $B$, $C$, $D$, and the noise covariance matrices $Q$ and $R$ are learned — not assumed.
The latent space is small enough that a standard Kalman filter runs trivially. But here's the key: the latent model is not trained to approximate the system dynamics well in isolation. It's trained to be a good filtering model.
The End-to-End Learning Trick
To achieve this, the authors embed the Kalman filter directly into the training loop. This is where the paper's technical contribution lies. The loss function has four components, each with a clear interpretation:
Autoencoder reconstruction loss: The decoder should be able to reconstruct the original high-dimensional state from the encoder's latent representation. This ensures the latent space captures meaningful structure in the data.
Kalman filter negative log-likelihood: This is the probability of the observed measurements given the model. Maximizing this is equivalent to asking: does the latent model explain what the sensors actually see? If the model says a measurement should be 72 degrees and the sensor says 68, that's a penalty — but it's weighted by the model's own estimate of uncertainty. If the model thinks its predictions are uncertain, it penalizes mismatches less harshly.
Decoded state estimation error: After running the Kalman filter forward through the latent model, you decode the estimated latent state back to the original space and compare it to the true state. This is the actual estimation error — the number you ultimately care about.
Latent alignment loss: The encoder produces a latent state from each high-dimensional observation. The Kalman filter produces its own estimate of the latent state. These should agree. This alignment term ensures consistency between the learned representation and the filtering model.
The loss is a weighted sum of these four terms. During training, backpropagation flows through the Kalman filter — meaning the gradients tell the model exactly how to adjust its parameters to improve estimation. This is differentiable programming: the filter itself becomes a layer in a neural network.
The elegance is that everything trains together. The linear model in latent space, the encoder, the decoder, and even the noise covariances all co-evolve to serve the estimation objective. There's no sequential pipeline where you first compress the system and then design a filter. The reduced model is the filtering model, shaped entirely by the task.
Why This Beats the Conventional Approach
The traditional two-stage approach is intuitive: identify a reduced-order model that approximates the system well, then plug it into a Kalman filter. But this has a fundamental limitation. The reduced model is optimized for approximating the system's dynamics — meaning it tries to predict the trajectory accurately. But prediction and estimation are different problems.
Consider a simple analogy. Imagine trying to track the position of a bouncing ball from noisy camera frames. A model optimized for prediction might focus on accurately capturing the parabolic arcs between bounces. But a model optimized for estimation from sparse frames might focus on how the ball's position correlates with the bits of the frame you can actually see. Different objectives. The paper's approach trains directly for the second.
There's another distinction worth noting. The paper is not a Koopman operator approach — a popular family of methods that lift nonlinear dynamics into high-dimensional linear representations for prediction. Koopman methods go the opposite direction: they find a linear model in a higher-dimensional (often infinite-dimensional) space. The authors compress instead, learning a low-dimensional linear model explicitly designed for filtering.
What the Numbers Show
The validation focuses on heat diffusion: a rod with 100 temperature sensors, where heat spreads according to the partial differential equation governing conduction. This is a classic benchmark in reduced-order modeling — it's nonlinear (through boundary conditions), spatially distributed, and has correlated states. The authors generate training data by simulating the PDE with random inputs and initial conditions.
The key result appears in
, which compares three approaches across 1,000 Monte Carlo trials:
- FOEKF (Full-Order Extended Kalman Filter): Runs on the full 100-dimensional state. This is the gold standard but computationally expensive.
- SIDKF (System Identification-based Kalman Filter): A two-stage approach where a reduced-order model is first identified for prediction accuracy, then a Kalman filter is designed on top.
- ROKF (Reduced-Order Kalman Filter): The authors' proposed method.
shows the RMSE (root-mean-square error) of state estimates for all three methods. The ROKF method achieves estimation accuracy close to FOEKF, while outperforming the two-stage SIDKF approach. This validates the core claim: joint training for the filtering objective produces a better reduced model than optimizing for prediction.
provides a finer-grained look. Part (a) shows the spatial distribution of estimation errors across the rod at a particular time step. The dotted line indicates where the sparse sensors are located. You can see that errors spike near sensor locations and away from them, which is expected — the filter is most confident where it has direct observations. Part (b) shows per-node RMSE across all 100 sensors, with the dashed line indicating the average. The errors are not uniform, but they're bounded.
shows the temperature fields at different times. Part (a) is the ground truth — what the system is actually doing. Part (b) is the ROKF's estimate. The visual match is close, though you can see slight smearing in regions where sensors are sparse.
Bounding the Gap
A more novel contribution is the paper's use of conformal predictions to bound the performance gap between reduced-order and full-order filtering. Conformal prediction is a distribution-free framework for constructing prediction sets with guaranteed coverage: no assumptions about the data distribution are required.
The authors define $\Psi$ as the difference in RMSE between the reduced-order filter's estimate and the full-order filter's estimate. A positive $\Psi$ means the reduced filter is worse. They want to bound how large $\Psi$ can get.
Theorem 1 in the paper provides this bound. Given a validation set of $m$ trajectories, let $\Psi_{(k^\star)}$ be a quantile of the observed gaps — specifically the $k^\star$-th order statistic. The theorem states that with probability at least $1 - \delta$ over the validation set, the true performance gap on new data satisfies:
In plain terms: with confidence $1 - \delta$, at least a fraction $1 - \alpha$ of new test trajectories will have a performance gap no larger than $\Psi_{(k^\star)}$.
In the heat diffusion experiments, with $m = 500$ validation trajectories, $\alpha = 0.1$, and $\delta = 0.05$, the bound $\Psi_{(k^\star)} = 3.23^\circ\text{C}$ was obtained. This means: on 90% of new test trajectories, the reduced-order filter's RMSE exceeds the full-order filter's RMSE by at most 3.23°C.
visualizes this. The histogram shows the distribution of $\Psi$ across 500 independent test sets. The dashed red line marks the bound $\Psi_{(k^\star)} = 3.23^\circ\text{C}$. The green shaded region represents test sets where the gap is below the bound (49 out of 500); the pink region above the line represents test sets exceeding it (6 out of 500). That's 98.8% coverage — slightly better than the promised 90%, reflecting that the bound is conservative by design.
This is a meaningful result. In safety-critical applications, you don't just want a point estimate — you want to know worst-case behavior. The conformal framework gives you that without assuming Gaussianity or linearity.
Why This Changes Things
The practical implications are significant. Real-time state estimation in high-dimensional systems has been a bottleneck for embedded and resource-constrained applications. Drones that need to estimate airflow from a few pressure sensors. Robots operating in unstructured environments where vision is limited. Power grids where state estimation runs at millisecond intervals across thousands of nodes. The computational cost of a full-order Kalman filter has pushed many applications toward simpler, less accurate heuristics.
This work suggests you don't have to choose between accuracy and efficiency. By training the model and filter jointly with a filtering objective, you can compress the system aggressively while retaining most of the estimation performance. The latent space is small enough that the Kalman filter is cheap. The training is data-driven, so you don't need an explicit mathematical model of the system. And the conformal bounds provide confidence intervals on the trade-off.
The heat diffusion example is a proof of concept, but the framework is general. The authors frame it as applicable to fluid dynamics, power systems, and robotics — domains where systems are high-dimensional, measurements are sparse, and real-time estimation matters.
Caveats and Open Questions
This is not a solved problem. Several limitations deserve acknowledgment.
Latent dimension choice. The paper doesn't prescribe how to choose $n_z$ — the dimension of the latent space. Too small, and you lose critical information. Too large, and you defeat the purpose. This remains a hyperparameter to tune, likely based on validation performance.
Nonlinearity trade-off. The reduced-order model is linear. Nonlinearities in the original system are absorbed into the latent dynamics, but only approximately. For systems with strong nonlinearities, a linear model in latent space may be insufficient. Extended or unscented variants might be needed.
Sensor placement. The paper assumes sensors are fixed. In practice, optimal sensor placement for a given latent representation is an open problem. The autoencoder learns a compression that's useful given where you currently measure — change the sensors, and the learned representation might be suboptimal.
Scalability. The heat diffusion benchmark has 100 nodes. Real engineering systems can have millions. The autoencoder training and Kalman filter operations scale differently, and it's not clear the method extends gracefully to truly massive systems without architectural modifications.
Generalization to different systems. The method is validated on a single benchmark. How it performs on fluid flows, chemical reactions, or electrical networks is unknown. Each domain has its own structure that may or may not be captured well by a learned linear latent model.
What's Next
The authors identify several directions for future work. Extending to stochastic differential equations — where noise enters continuously rather than at discrete steps — would broaden applicability. Incorporating the measurement model selection (where sensors are placed) into the learning framework would make it more complete. Testing on higher-dimensional systems would stress-test the scalability claims.
There's also the question of theoretical foundations. The conformal prediction bounds give finite-sample guarantees, but a deeper understanding of why this joint training works — in terms of approximation theory or statistical learning — remains undeveloped. The paper demonstrates empirically that the filtering objective shapes better representations, but a formal theory connecting the choice of loss to representation quality would be valuable.
The broader trend this fits into is the ongoing marriage of physical models and machine learning. For decades, control theory and system identification operated in relatively separate intellectual space from deep learning. This paper sits at their intersection: using neural networks to learn representations, while keeping the Kalman filter — a principled, interpretable statistical engine — at the center. The filter is not replaced; it's embedded.
What makes the work compelling isn't just the architecture. It's the conceptual shift: building a model that's optimized for the job it's meant to do, rather than for a proxy task. The reduced-order model isn't a lossy compression of the original system. It's a task-specific abstraction, learned from data, shaped by what you actually need it for. In an era of ever-larger models, there's something elegant about compressing aggressively — not just to be small, but to be good at something specific.