When Robots Learn to Second-Guess Themselves
A new inference mechanism lets robots correct their predictions in real-time, bridging the gap between lookahead planning and reactive control.
A robot reaches for a moved glass. New AI lets it correct mid-reach, before knocking it over.
When Your Robot Learns to Second-Guess Itself
The robot reaches for a glass. Its internal model predicts the glass is where it always sits—but someone moved it. In older systems, the robot would continue reaching for a ghost, compounding its error with every subsequent action. It would wait until an entire chunk of actions finished before reconsidering. By then, a manipulator might have knocked the glass off the table entirely.
A new inference mechanism called Feedback Flow Matching changes this. Developed by researchers from Zhongguancun Academy, Peking University, and Tsinghua University, FBFM lets a robot correct its predictions at the individual time-step level—while an action chunk is still being generated—not just between chunks as in previous systems. On 42 complex robot tasks spanning multi-step kitchen and object-manipulation scenarios, the approach raised success rates by 3 full percentage points. On a suite of goal-oriented manipulation tasks, it improved performance by 2.5 percentage points. More tellingly, when the researchers tested prediction accuracy on images recorded from a real robot arm, FBFM tracked the robot's actual trajectory far more faithfully than its predecessor, even as the scene evolved in ways the original model had never seen during training.
The finding matters because it bridges two worlds that have long been kept separate: the robot's ability to imagine future states (open-loop generation) and its need to respond to what's actually happening in the physical world (closed-loop control). FBFM doesn't retrain the underlying model. It works as an inference-time overlay, nudging the model's predictions toward reality as reality reveals itself. In an era when robot assistants are no longer confined to pristine labs, this kind of real-time adaptability could be the difference between a machine that follows a script and one that genuinely handles the messiness of the real world.
The Challenge of Making Robots Imagine Ahead
To understand why this matters, it helps to understand how modern robot control systems work. The most promising recent approaches use something called a World-Action Model, or WAM. Rather than simply reacting to what the robot sees right now, a WAM looks ahead. It predicts how the scene will evolve—what the robot will see several steps from now if it takes a particular sequence of actions—and uses those predictions to decide what to do next.
This look-ahead is valuable because many real-world tasks require thinking several steps ahead. Grasping an object isn't just about closing fingers at the right moment; it's about approaching from the right angle, positioning the gripper correctly, and applying force appropriately. A system that can imagine the intermediate steps can plan more coherently than one that treats each moment in isolation.
But there's a fundamental tension built into this approach. The more a robot relies on predicted future states, the more catastrophic errors become when those predictions diverge from reality. The real world has a stubborn habit of not matching simulations. Objects get bumped. Lighting changes. A cat walks into the frame. When the predicted future diverges from the actual present, the robot's subsequent actions are built on quicksand. It reaches for where it thinks the glass is, not where it is.
Existing WAM systems address this partially. They refresh their internal memory—the history or "KV cache"—with real observations between action chunks. Think of it as the robot pausing after every five actions, looking around, and updating its mental model before starting the next set of five actions. This chunk-wise feedback helps, but it operates at a coarse temporal granularity. During those five actions, the robot remains committed to its predictions. If reality diverges early in the chunk, the robot won't know until the chunk ends.
The researchers behind FBFM identified this as a temporal-granularity gap. "Chunk-level re-grounding cannot correct state-prediction errors at individual time steps of an active chunk," they write. "As a result, the future used to produce actions may remain inconsistent with physical transitions that have already occurred."
Flow Matching: The Engine Beneath the Predictions
To understand how FBFM works, we need to understand what it's modifying. WAMs based on flow matching don't generate actions or predictions directly. Instead, they generate predictions by starting with pure noise and gradually transforming it toward the target distribution—a process analogous to starting with a cloud of scattered sand and slowly sculpting it into a statue.
Flow matching, in essence, learns a vector field: a set of arrows that point from any point in noise-space toward where the data should be. Starting from noise, you follow these arrows step by step, and they guide you toward a coherent prediction. The mathematics of this process are continuous and differentiable, which means you can intervene in the flow without breaking the machinery.
The researchers realized that this differentiability was an opportunity. If you could inject new information into the flow mid-generation—specifically, information about what the robot is actually seeing right now—you could correct predictions without waiting for chunk boundaries. The flow would be redirected, pulled toward reality as reality revealed itself.
This is what FBFM does. It applies a masked pseudoinverse correction to the conditional velocity field. The "masked" part is crucial: it doesn't correct everything, only the specific time steps where new information has arrived. The "pseudoinverse" part refers to the mathematical technique for projecting constraints onto the generation process in a way that preserves the overall structure of the prediction.
The Core Innovation: Feedback Inside the Chunk
The key insight behind FBFM is that execution and generation can happen concurrently. While the robot is executing actions from the previous chunk—moving its arm, closing its gripper, watching the scene—the system can be generating the next chunk of actions and predictions. This overlap creates a window of opportunity.
Consider what happens during that window. The robot executes action at time t, and the real world responds: the arm moves, the gripper closes, the scene changes. A camera or depth sensor captures this new state and encodes it. This observed state corresponds to a particular temporal slot in the prediction that the WAM is currently generating for that same time. In older systems, that slot was just a prediction. With FBFM, it becomes an anchor point—a place where the prediction is forced to match reality.
The researchers call this a "dynamically partially observed generation problem." The WAM is generating a full sequence of future states and actions, but some of those states are now known. The task is to ensure that the generation respects these known values while continuing to predict the unknown future.
FBFM handles two types of feedback simultaneously. The first is state feedback: newly observed states that constrain the flow of the state prediction. The second is action consistency: the actions that were committed by the previous chunk. These committed actions represent a hard constraint—the robot has already decided to execute them, so the new generation must respect them. The first type of feedback is dynamic; it arrives progressively as the robot acts. The second is fixed; it doesn't change during generation.
The mechanism applies the correction at every solver evaluation—a solver being the algorithm that steps through the flow from noise to prediction. At each step, FBFM computes the difference between what the model predicted and what reality has now told us is true, then adjusts the flow to pull predictions toward reality. It does this without modifying the underlying model weights. The model remains frozen; the inference process is modified.
Two Architectures, One Mechanism
The researchers implemented FBFM in two different WAM architectures to demonstrate its generality.
The first is a stage-wise architecture, exemplified by LingBot-VA. In this design, the model generates states first, then generates actions conditioned on those states. This reflects a certain intuitive logic: first imagine what will happen, then figure out what actions would cause it. FBFM in this architecture applies separate corrections to the state flow and the action flow, with state corrections feeding into the action stage through a refreshed context.
The process works like this: while the previous action chunk executes, encoded real observations progressively constrain the state flow. Each new observation activates the corresponding mask entry and pulls the state prediction toward reality. Once the state flow terminates, the action flow begins, conditioned on the corrected state context. But observations may still arrive during the action flow. FBFM handles this by refreshing the state context used by the action stage rather than restarting state generation.
The second is a joint-generation architecture, exemplified by DreamZero. Here, the model generates states and actions together, using a single unified flow. This approach is more tightly coupled: corrections to state predictions can directly affect action predictions through the cross-modal structure of the generation process.
In the joint architecture, state feedback and action constraints are assembled into a single joint constraint applied at every solver evaluation. The state block of the correction mask is refreshed whenever new observations arrive, while the action block remains fixed by the cross-chunk overlap. What makes this elegant is that through the cross-modal blocks of the Jacobian—the mathematical structure that describes how changes in one variable affect others—a state residual can directly correct action coordinates during the same solver evaluation.
The Numbers That Matter
FBFM was evaluated across several benchmark suites, each testing different aspects of long-horizon robot manipulation.
On RoboTwin2.0, a suite of 42 tasks involving complex multi-step manipulation, FBFM raised the equal-weight task-configuration success rate by 3.0 percentage points. This might sound modest in absolute terms, but in the unforgiving arithmetic of robot task success, where failures compound and small improvements can mean the difference between a system that works and one that doesn't, this represents a meaningful gain.
On the LIBERO suites—four different benchmark sets testing various aspects of robot manipulation from spatial reasoning to goal-oriented behavior—FBFM provided more varied gains. The pooled improvement across all four was 0.6 percentage points. But the aggregate masks important variation. On LIBERO-Goal and LIBERO-10 specifically, which test a robot's ability to understand and pursue abstract goals, FBFM improved success rates by 2.5 percentage points. These are precisely the tasks where prediction errors compound most severely: a robot that misunderstands where an object will be after an intermediate step will make the wrong choice of subsequent actions, and those wrong choices lead to task failure.
Task Success Rate Improvements with FBFM
Success rate improvements across benchmark suites. FBFM shows meaningful gains on task success rates, with the largest improvements on goal-oriented tasks where prediction errors compound most severely.
| Label | Value |
|---|---|
| RoboTwin2.0 (42 tasks) | 3 |
| LIBERO-Goal | 2.5 |
| LIBERO-10 | 2.5 |
| LIBERO pooled (all) | 0.6 |
Beyond success rates, the researchers examined prediction accuracy directly. They recorded observation sequences from a physical robot performing tasks with a RealSense D435i depth camera and tested how well different systems could predict future frames. FBFM with full feedback coverage—using all 30 measured latent slots—showed notably better state tracking than the base model without feedback.
To test robustness, they conducted experiments with varying levels of feedback coverage. What happens when the robot only gets partial observations—perhaps the camera is occluded, or sensor data is delayed? The results showed graceful degradation. With 10 or 20 slots of feedback instead of the full 30, prediction accuracy was lower than the full-coverage case but substantially better than the no-feedback baseline. This is important for real-world deployment, where sensors fail, networks lag, and perfect information is never guaranteed.
Why Fine-Grained Correction Changes the Game
The significance of FBFM goes beyond its numerical improvements. It represents a conceptual shift in how we think about the relationship between prediction and control in robot systems.
Traditional robot control operates in a closed loop: the robot senses, acts, senses again, acts again. This loop runs at the millisecond scale and gives the system tight feedback. But it doesn't think ahead. A purely reactive robot can succeed at simple tasks, but struggles with complex, multi-step manipulation that requires reasoning about consequences.
Modern learning-based approaches, including WAMs, try to think ahead. They predict several steps into the future and use those predictions to inform current action selection. But this lookahead comes at the cost of reactivity. The farther ahead a system looks, the more it relies on predictions rather than observations, and the more catastrophic errors become when predictions diverge from reality.
Previous systems tried to split the difference. They would lookahead for several steps, then pause to re-ground in reality, then lookahead again. This chunk-wise approach preserved some of the benefits of lookahead while periodically correcting for prediction error. But the pauses introduced latency, and the coarse granularity of the corrections meant that significant divergence could occur within a chunk before the system noticed.
FBFM removes the pause. It corrects predictions continuously, at the time-step level, without interrupting the lookahead process. The robot is always predicting several steps ahead, but those predictions are continuously being dragged toward reality by incoming sensor data. The result is a system that maintains the lookahead advantages of WAMs while recovering the responsiveness of closed-loop control.
What the Real Robot Tests Reveal
The most compelling evidence for FBFM comes not from synthetic benchmarks but from tests on physical hardware. The researchers attached a RealSense D435i depth camera to a robot arm and recorded observation sequences as it performed manipulation tasks. They then compared how well different systems could predict future frames from these sequences.
The sequences show a clear pattern. In the base model without feedback, predicted frames drift from the recorded reference as time progresses. By the end of a 5-second sequence, the prediction might show the robot arm in a different position than where it actually is. This is the drift problem: small errors accumulate, and the prediction becomes increasingly unreliable.
FBFM with full feedback largely solves this problem. Predicted frames track the recorded reference closely throughout the sequence. The mechanism isn't perfect—predictions are still predictions, and there's inherent uncertainty about the future—but the continuous correction keeps them anchored to reality.
The partial-feedback experiments are particularly instructive. With only 10 of 30 latent slots available for correction, predictions degrade but remain substantially better than the no-feedback baseline. With 20 slots, performance improves further. This suggests that FBFM is robust to partial observability: even if the robot can't see everything, correcting what it can see still helps.
The Mathematics of Meaningful Intervention
The technical contribution of FBFM is its formulation of chunk-internal feedback as a dynamically partially observed generation problem. Previous approaches treated the chunk as an indivisible unit—either you corrected the whole thing or you corrected none of it. FBFM recognizes that different time steps within a chunk correspond to different real-world moments, and that corrections should be applied at the corresponding resolution.
The masked pseudoinverse correction achieves this. At every solver evaluation, the mechanism computes a correction term that pulls the current prediction toward the observed state, weighted by how much we trust the observation (the mask entries) and scaled by the flow time (corrections matter more at later stages of generation, when the prediction is closer to its final form). The mathematical elegance is that this correction is compatible with the flow matching framework—the model was trained to predict flows, and the correction is injected into that flow without disrupting the training objective.
What makes this training-free is crucial. The researchers didn't retrain DreamZero or LingBot-VA. They took existing models with existing capabilities and modified only the inference process. This means FBFM could potentially be applied to any flow-matching-based WAM, regardless of its architecture or training data. The mechanism is an overlay, not a modification of the underlying model.
Prediction Error vs. Feedback Coverage
Relative prediction error with varying feedback coverage. Even partial feedback substantially reduces error compared to the no-feedback baseline, demonstrating robustness to partial observability.
| Label | Value |
|---|---|
| 0 slots (baseline) | 100 |
| 10 slots | 45 |
| 20 slots | 25 |
| 30 slots (full) | 15 |
Implications for Robot Deployment
The practical implications of this work depend on what you think robots will need to do in the real world.
If robots remain in structured environments—factories, warehouses, controlled spaces where everything is precisely positioned and nothing unexpected happens—then FBFM is a nice improvement but not a necessity. Current systems work adequately in these settings, and the incremental gains from fine-grained correction may not justify the implementation complexity.
But if robots are meant to operate in human environments—homes, hospitals, offices, outdoor spaces—then continuous correction is essential. These environments are unpredictable. Objects move. People interrupt. The unexpected is the norm. A robot that can only correct its predictions at chunk boundaries will accumulate errors rapidly in such settings. The difference between 3.0 percentage points improvement and nothing may be the difference between a robot that succeeds at a task and one that fails, knocks something over, and requires human intervention.
The researchers are careful to note limitations. FBFM requires that observed states correspond to specific temporal slots in the generated future—the mechanism assumes a known mapping between observation time and prediction time. In settings with significant sensor latency or communication delay, this assumption may not hold perfectly. The method also requires that the WAM was trained on data that includes the kinds of corrections FBFM now provides; the mechanism can only correct within the distribution the model was trained on.
What Comes Next
FBFM opens several directions for future work.
The most immediate is testing on more diverse robot platforms and task domains. The current evaluations focus on manipulation tasks in relatively structured environments. An important next step would be testing in mobile manipulation, contact-rich tasks like insertion or assembly, and outdoor or unstructured settings where prediction divergence is more severe.
A second direction is integration with learning. FBFM is training-free, but future work could explore using the correction signals it generates to update the underlying model. If the model consistently makes errors in a particular domain, could the accumulated corrections be used to fine-tune it? This would combine the benefits of online correction with long-term learning.
A third direction is combining FBFM with other inference-time techniques. Real-Time Chunking (RTC), mentioned in the paper, addresses action continuity under inference delay. The constraint it enforces is complementary to FBFM's: RTC ensures actions are consistent across chunk boundaries, while FBFM ensures predictions are consistent with reality. A system that combines both could address both sources of error simultaneously.
Finally, there's the question of how FBFM scales to longer horizons and more complex tasks. The current evaluations span tasks of moderate length. As robots take on longer-horizon assignments—whole sequences of household chores, multi-hour assembly tasks—the corrections must propagate correctly over extended sequences without accumulating numerical errors or losing coherence.
The Bigger Picture
FBFM is part of a broader shift in how we think about robot intelligence. The field is moving away from purely reactive systems and toward models that think ahead, predict consequences, and plan over longer time horizons. But this shift creates new challenges: predictions diverge from reality, errors compound, and systems that looked reliable in simulation fail in deployment.
FBFM addresses these challenges by bringing closed-loop thinking back into lookahead systems. The robot still imagines the future, but that imagination is continuously grounded in reality. The result is a system that combines the long-horizon coherence of model-based planning with the responsiveness of reactive control.
Whether this specific mechanism becomes widely adopted remains to be seen. But the underlying insight—that fine-grained correction within generation chunks can bridge open-loop prediction and closed-loop control—seems likely to influence how future robot systems are designed. In an era when robots are beginning to move out of factories and into the messy complexity of human environments, this kind of real-time adaptability may be exactly what's needed.
Contribution of FBFM Components
Components of the FBFM mechanism and their contribution to overall improvement. State feedback and action consistency provide complementary error correction pathways.
| Label | Value |
|---|---|
| State Flow Correction | 100 % |
| Action Consistency Constraint | 100 % |
| Context Refresh | 95 % |
| Joint State-Action Correction | 85 % |
"FBFM offers a new paradigm for fine-grained online correction, bridging open-loop flow generation with closed-loop real-world dynamics."
Sign in to join the conversation.
Comments (0)
No comments yet. Be the first to share your thoughts.