The Geometry of Robot Failure: How Scientists Are Learning to Fix AI Models from the Inside

When Robots Forget How to See

A robot reaches for a coffee mug. Its gripper hovers, uncertain, then closes on empty air. The mug hasn't moved. But the camera is three centimeters to the left. That's enough.

This is the brutal reality of today's robotic foundation models. These AI systems — trained on millions of demos, deployed in labs that sparkle with promise — still flinch at the minor indignities of the real world. A change in lighting. A shifted viewpoint. A speck of dust on the lens. Any of these can send a capable robot into failure.

The research community has documented these failures extensively. Camera changes. Gripper miscalibrations. Visual noise. These aren't exotic edge cases — they're the default conditions of deployment. And yet, the standard fix has been the same for years: collect more data, train longer, cross your fingers.

But a new paper from researchers at Georgia Tech flips this script entirely. Instead of asking "how do we collect better training data?", they ask something stranger: "what's actually breaking inside the robot's brain when it fails?"

They find that some AI models for robot control have a remarkably clean internal structure — a geometric order that, once identified, can be exploited to fix failures without touching the model's weights at all. Other models are messier, their failures harder to repair. And critically, you can predict which models will respond to intervention before you even try to fix them.

"We show that steerability is architecture-dependent," the researchers write. "Cosmos-Policy and DiT4DiT exhibit clear linear structure across multiple perturbations, whereas LingBot-VA exhibits substantially weaker separability." In plain English: some robot brains are organized; others are chaotic. The organized ones can be guided. The chaotic ones resist help.

This matters because it offers a path to robust robots without endless data collection. It suggests that before deploying a model, you can test whether its internal representations will cooperate with the interventions you want to apply. And it introduces a new control-theoretic method — WA-LQR — that gently nudges a robot's AI brain back on track when the world shifts beneath it.

The Robot's Inner Eye

To understand what Hong, Skifstad, and colleagues discovered, you need to understand what a "World Action Model" actually is — and why researchers are so excited about them despite their fragility.

Foundation models have transformed AI over the past several years. Large language models like GPT-4 can write poetry and debug code. Image generators like DALL-E can conjure photorealistic scenes from text. These models share a crucial property: they're pretrained on massive datasets and then adapted to specific tasks, giving them a kind of general knowledge that transfers across contexts.

The robotics community has been chasing the same dream. Robot foundation models — systems pretrained on huge collections of robot demonstrations, then fine-tuned for specific tasks — should inherit some of that same flexibility. A robot trained on millions of pick-and-place demos might generalize to novel objects. One pretrained on manipulation across many environments might handle new kitchens without retraining.

The problem is that most robot foundation models are reactive. They look at what they see right now and predict what action to take right now. They're brilliant at imitating the demos they trained on but struggle when the world deviates even slightly from those demos.

World Action Models take a different approach. Instead of just predicting actions, they predict the future. Given a current observation and a task description, a WAM generates a sequence of future states — what the robot will see if it executes those actions — and then decodes actions from those predicted futures. The intuition is that if you understand how the world evolves, you can plan more robustly. Changes in camera angle or lighting might shake your confidence in the current observation, but if you can predict what the future should look like, you have something to hold onto.

WAMs build on video generation backbones — the same technology that powers Sora and Runway. These diffusion transformers (DiTs) are trained to predict what video frames will look like after noise is gradually removed. WAMs hijack this capability: instead of generating realistic video of cats or cars, they generate realistic predictions of robot manipulation. The same architecture that can imagine the future of a scene can imagine the future of a grasp.

But here's the catch that the Georgia Tech team decided to investigate: these models are still brittle. Camera moves break them. Gripper offsets break them. Visual noise breaks them. The video-generation backbone that was supposed to make them robust turns out to inherit the fragility of its pretraining.

"WAMs aim to improve robustness with future-state modeling and video-based priors," the researchers note, "but still fail under shifts such as camera viewpoint and robot initial-state changes."

So the question becomes: why? What's happening inside the model when it fails? And can we fix it without collecting millions more robot demos?

Cracking Open the Black Box

The technique the researchers used is called mechanistic interpretability — the scientific study of what neural networks actually do internally. It's a bit like neuroscience, except the subject is a mathematical function and the brain scans are activation vectors.

The basic idea is this: when a neural network processes information, each layer transforms the input through a series of operations. The intermediate results — the "activations" at each layer —encode features that the network uses to make its decisions. Mechanistic interpretability tries to reverse-engineer what those features mean.

In language models, researchers have found that concepts like "the Eiffel Tower" or "emotional sentiment" or "languages other than English" have surprisingly simple geometric representations in activation space. You can often draw a line — a linear direction — that separates examples with the feature from examples without it. This is called linear separability. And if a feature is linearly separable, you can manipulate it. You can add a vector pointing in the "positive" direction to make the model behave more as if it had that feature.

This is the foundation of "activation steering" — a collection of techniques for modifying model behavior by directly editing internal activations, without retraining the model or changing its weights. It's a bit like performing brain surgery with a calculator.

The Georgia Tech team wanted to know: do WAMs have this same kind of clean geometric structure? Specifically, do features related to robustness — the ability to handle perturbations like camera shifts and noise — have linear representations that could be exploited for steering?

To find out, they needed to look inside the models.

They studied three different WAM architectures: Cosmos-Policy 2B (from NVIDIA), DiT4DiT, and LingBot-VA. Each represents a different approach to coupling action prediction with video generation. For each model, they ran it on tasks from the LIBERO-10 benchmark suite — a set of ten manipulation problems where robots must pick up objects and place them in specific locations.

Then they broke things.

Looking for Ghosts in the Machine

The researchers introduced three types of perturbations to see how each model would respond:

Camera position changes: The camera angle was shifted, simulating the kind of viewpoint variation that happens constantly in real deployment.

Gripper position offsets: The robot's starting gripper position was moved slightly, representing the small miscalibrations that accumulate in real hardware.

Gaussian visual noise: Random pixel-level noise was added to the camera images, mimicking the sensor degradation that occurs over time.

For each perturbation type, they collected the model's internal activations when processing nominal (unperturbed) inputs and compared them to activations when processing perturbed inputs. If the model was handling the perturbations gracefully, the activations might look similar. If it was failing, they might look different in informative ways.

Specifically, they looked at activations in the DiT blocks — the layers where the model does most of its computation. They averaged across spatial positions (the models process tokens representing different regions of the image) and timesteps (the models process sequences of observations over time), resulting in summarized activation vectors in a high-dimensional space.

Then came the geometric analysis. They projected these activation vectors onto their top three principal components — the directions of maximum variance — and asked a simple question: can a linear classifier tell the difference between nominal and perturbed activations?

They trained a support vector machine (SVM) on this three-dimensional projection and measured its "hinge loss" — a number that tells you how well the classifier performs. Zero means perfect separation: the two classes (nominal vs. perturbed) sit on opposite sides of a plane with room to spare. One means the classifier learned nothing; the two groups are hopelessly intertwined.

What they found was striking.

For Cosmos-Policy and DiT4DiT, the activations corresponding to robustness-critical features were remarkably well-separated. The SVM hinge loss was often near zero — these models had organized, linear representations of perturbation-related features. The geometry was clean.

For LingBot-VA, the results were dramatically different. Across tasks and perturbation types, the researchers saw "little to no separability." The representations were tangled, their geometry chaotic. A linear classifier couldn't tell nominal from perturbed activations apart any better than random chance.

Figure 2 from the paper illustrates this divergence. For Cosmos-Policy (top rows), activations corresponding to noise-perturbed and clean inputs form distinct clusters when projected into three dimensions — especially in the intermediate and final layers. The gray hyperplane shows the SVM decision boundary, and it separates the classes cleanly. For camera perturbations (bottom rows), the same pattern holds: clean and perturbed activations are geometrically distinct.

For LingBot-VA, the picture is murkier. The activations overlap substantially; no linear boundary cleanly separates nominal from perturbed. The model has no obvious geometric handle for distinguishing these conditions.

Figure 4 makes this contrast explicit. The activations from camera-perturbed and clean LingBot-VA inputs show substantial overlap across all layers — the first block, the best intermediate block, and the final block. Whatever information the model uses to handle camera changes, it's not cleanly linear.

The researchers also examined whether the same perturbation features were represented consistently across different tasks. This matters because if you want to apply steering learned on one task to a new task, the underlying representation needs to be shared. They found that for Cosmos-Policy, many task pairs shared feature representations corresponding to the same input perturbation — a kind of cross-task invariance that enables generalization.

Figure 3 shows this pairwise separability for camera perturbations. Certain task pairs have high linear separability — their activations separate cleanly along perturbation-relevant directions. This suggests that the contrastive directions learned on one task might transfer to another.

The Prediction Machine

Here is the most practically useful finding of the paper: linear separability predicts steerability.

The researchers didn't just measure separability for its own sake. They used it as a diagnostic. The hypothesis was straightforward: if a model's representations of robustness features are linearly separable, then activation steering should work. If they're entangled, steering should fail.

They validated this hypothesis empirically. After computing the separability metric for each model-task-perturbation combination, they applied activation steering and measured the resulting task success rates. The correlation was strong.

"We show that separability is strongly correlated with steerability, making it a useful diagnostic for identifying models that are amenable to activation steering," they write.

Figure 7 plots hinge loss (a measure of how separable the representations are) against steering performance (how much improvement the interventions achieve). The red line shows the correlation; the points cluster along it. Models and conditions with cleaner geometric structure (lower hinge loss) consistently benefited more from steering. The messy ones resisted.

This is valuable because it means you can test a model before deploying it. Run the separability analysis. If the geometry is clean, you know the model will respond to steering interventions. If it's tangled, you know you'll need a different approach.

The researchers also examined the dimensionality of the contrastive subspace — how many directions are needed to capture the robustness-relevant information. For camera orientation perturbations on Cosmos-Policy, they found that the top three singular vectors explained most of the variance in the contrastive directions.

Figure 6 shows the cumulative explained variance as a function of the number of singular vectors retained. The curve rises steeply and then plateaus, indicating that just a few directions capture almost everything. This means the steering intervention doesn't need to operate in the full high-dimensional activation space — a handful of dimensions suffice. The low-rank structure enables efficient controller synthesis, as we'll see.

Building the Steering Wheel

Once the researchers had established that some WAMs have clean linear structure amenable to steering, they needed to actually build the steering method. They developed two approaches: a simple baseline and a more sophisticated controller.

Activation Addition (ActAdd) is the straightforward method. Given pairs of inputs that differ primarily in the feature you care about (clean vs. noisy, nominal vs. perturbed), you compute contrastive activation directions by subtracting the hidden states from the two forward passes. For each layer and denoising timestep, this gives a vector pointing from "negative" examples toward "positive" ones. You average these vectors to get a steering direction, then add a scaled version of it to the activations at inference time.

ActAdd is training-free and weight-preserving — you never touch the model's learned parameters. But it's also open-loop: it adds the same correction regardless of what's happening in the current forward pass. It doesn't know whether the feature is already at the desired strength. It doesn't adapt to the model's actual state. This can cause oversteering or understeering.

The researchers' second method, WA-LQR (World-Action Linear Quadratic Regulator), is more sophisticated. It treats the activation dynamics as a control system and uses optimal control theory to compute a feedback controller.

The intuition is this: if the model's activations are locally linear — if small deviations from a nominal trajectory evolve in a predictable way — then we can model those dynamics with matrices (the A and B matrices from control theory) and synthesize a controller that drives the system toward a desired state. The LQR framework gives us an optimal controller in closed form, one that trades off between tracking error (deviation from the desired feature strength) and control effort (how much intervention we're applying).

There are several complications. First, the full activation space is enormous — millions of dimensions. Computing an LQR controller in that space would be intractable. Second, the model has many layers (the "L" in the DiT architecture), and the dynamics through those layers matter. Third, the model runs multiple denoising timesteps (the "T" in diffusion), and each timestep is like a separate mini-controller chained together.

WA-LQR handles these complications by projecting into a low-dimensional subspace. The researchers use the contrastive directions themselves to define this subspace — they take the top few singular vectors of the matrix of contrastive directions and work in that compressed space. The result is a reduced-order model where the dimensions that matter for robustness are retained and the irrelevant dimensions are discarded.

They define "feature setpoints" in this latent space — targets for how strong the robustness feature should be. The controller then measures the current feature strength, computes the deviation from the setpoint, and applies a correction proportional to that deviation. Unlike ActAdd, which applies a fixed offset regardless of context, WA-LQR adapts.

The controller also has a temporal structure. The researchers introduce an "action-decay schedule" over robot action chunks — early in a trajectory, when the model is most sensitive to perturbation, the steering is strong. Later, as the trajectory progresses, the steering weakens. This prevents oversteering in later timesteps where the model's own dynamics may have naturally corrected the deviation.

"Unlike open-loop activation addition, WA-LQR adapts online, steering only when activations deviate from the target feature strength while penalizing large perturbations," the researchers explain.

The math works out cleanly: for each denoising timestep and each action chunk, solving the LQR problem yields a gain matrix that can be precomputed offline and applied efficiently at runtime. The whole controller synthesis runs on CPU in polynomial time.

The Results

How well does all this actually work?

The researchers evaluated on the LIBERO-10 benchmark suite, testing robustness to camera perturbations, gripper position changes, and Gaussian visual noise. They compared WA-LQR against several baselines: unsteered models (no intervention), prompt steering (providing perturbation information in the text prompt), and activation addition (the open-loop method).

The results were consistent with the mechanistic predictions.

On Cosmos-Policy, WA-LQR improved robustness substantially across perturbation types. The method generalized contrastive directions learned on some LIBERO tasks to new tasks it hadn't seen during direction computation — a crucial test of whether the approach would work in practice. The model could leverage steering directions derived from one manipulation scenario to handle perturbations in a different scenario.

On LingBot-VA, the results were weak — consistent with the low linear separability observed in the mechanistic analysis. The model's tangled representations couldn't be fixed by the clean geometric interventions that worked on Cosmos-Policy.

Figure 1 shows the qualitative effect. In these Cosmos-Policy examples from LIBERO-10, the task is to place two objects (marked with yellow and green boxes) in a basket. Without steering (left column), the WAM fails — the gripper misses the objects or drops them outside the basket. With WA-LQR (right column), the same perturbations are handled gracefully, and the task succeeds.

Figure 8 shows more examples across perturbations, tasks, and models. Each row represents a rollout with six equally-spaced snapshots. In many cases, the unsteered model fails partway through (red border), while the WA-LQR steered model succeeds (green border). The pattern is consistent: where the geometry is clean, steering helps. Where it's messy, it doesn't.

The researchers also examined the local linearity assumption underlying WA-LQR. They computed the Jacobian matrices (the A~ matrices) that describe how activations evolve under small perturbations and checked whether the linear approximation accurately predicted actual activation dynamics.

Figure 5 addresses this. Part (a) shows the cosine similarity between the linear approximation and actual activations under camera orientation changes — the approximation tracks reality well, especially in later layers. Part (b) shows overlap between subspaces spanned by the top singular vectors across random inputs — the dynamics are consistent, not dependent on specific inputs. This validates the modeling assumption: WAMs are locally linear enough for LQR synthesis to work.

What This Changes

The standard approach to robot robustness has been brute force: more data, more diversity, more training. If the robot fails in new conditions, collect data in those conditions and retrain. Repeat until covered.

This approach has obvious limitations. You can't anticipate every possible perturbation. Real deployment will always surface conditions you haven't seen. And training on new data is slow, expensive, and can cause catastrophic forgetting of skills the model already knew.

The Georgia Tech work offers an alternative: instead of training the model to handle perturbations, you fix the model's representations at inference time. The model itself doesn't change — its weights stay the same, its training stays the same. You're just adjusting its internal activations to compensate for distribution shift.

This is valuable for several reasons. First, it's fast: once you've computed the contrastive directions and synthesized the controller, steering requires just a small computation per forward pass. Second, it's reversible: if the steering has unintended side effects, you can simply turn it off. Third, it's targeted: you're not retraining the whole model, just nudging the specific features that are relevant to the perturbation at hand.

But the most important contribution may be the diagnostic power. The paper shows that linear separability is a strong predictor of whether steering will work. This means you can evaluate a model before deployment and know whether you'll be able to improve its robustness through activation interventions. A model with clean geometry is a model that's fixable. A model with tangled geometry needs a different approach — perhaps architectural changes, different training procedures, or alternative intervention methods.

"We show that MI can diagnose WAM robustness and guide inference-_time interventions," the researchers write. "To the best of our knowledge, these are the first open- and closed-loop activation steering methods for WAMs."

The Caveats

No paper is complete, and this one has honest limitations.

First, the results are architecture-dependent. Not all WAMs have clean linear structure in their robustness representations. LingBot-VA didn't. The approach works on Cosmos-Policy and DiT4DiT, but whether it generalizes to other architectures remains to be seen. The community needs more models evaluated, more perturbations tested, more architectures compared.

Second, the perturbations studied are relatively structured. Camera shifts, gripper offsets, and Gaussian noise are common in robotics, but they're not exhaustive. Real deployment includes perturbations the researchers didn't test: lighting changes, object occlusions, cloth and deformable manipulation, contact-rich tasks where the model must sense through touch. Whether the geometric structure observed here extends to these conditions is an open question.

Third, the controller synthesis requires contrastive data — pairs of inputs that differ in the feature you want to steer. In the paper, these are generated by running the model on nominal and perturbed inputs. In practice, this means you need to know what perturbations you want to handle. You can't steer a model toward robustness to an unknown failure mode.

Fourth, the action-decay schedule has hyperparameters that must be tuned. The researchers used R_init, R_final, and τ_R to control how steering strength evolves over time. These values were set based on experimentation; the sensitivity to hyperparameter choice isn't fully characterized.

Fifth, the paper focuses on simulation. LIBERO-10 is a benchmark with realistic tasks, but it's still a simulated environment. Real robots have latencies, contact dynamics, and hardware failures that simulations abstract away. Whether the steering interventions remain effective on physical hardware is a question for future work.

What's Next

The researchers have opened several threads.

One is to explore other architectures. The finding that Cosmos-Policy and DiT4DiT have clean structure while LingBot-VA doesn't suggests that the specific design choices in a WAM — the video backbone, the action decoding mechanism, the pretraining procedure — affect the geometric organization of robustness features. Understanding why some architectures produce tidy representations and others produce messy ones would guide future model design.

Another thread is to test on more diverse perturbations. The current work focuses on camera and gripper variations, which are well-studied in robot learning. More interesting would be perturbations that are harder to model: lighting changes, background clutter, novel object categories, physical properties like friction and mass. Whether the linear separability finding extends to these conditions is an important empirical question.

A third thread is to improve the controller synthesis. The current WA-LQR assumes local linearity and works in a reduced subspace. Future work might develop controllers that handle non-linearities, work in higher-dimensional subspaces, or adapt to changing dynamics as the model processes a sequence.

Perhaps most interesting is the connection to broader questions in mechanistic interpretability. The finding that robustness features are linearly separable in some models but not others echoes similar observations in language models — some concepts are cleanly represented, others aren't. Is there a unifying principle that explains when features will have clean geometry? Does training procedure matter? Does model size? The WAM domain, with its concrete behavioral implications, provides a useful testbed for these questions.

The Robot That Doesn't Panic

Imagine a robot working in a warehouse. The morning shift begins; the lighting is different from yesterday because a bulb burned out. The camera angle shifted slightly when someone bumped into the shelf last week. There's a smudge on the lens that the maintenance team hasn't cleaned yet.

A reactive robot model might fail these conditions. It's seen clean lighting, pristine cameras, and exact angles in its training data, and small deviations from those conditions can cascade into failure.

A World Action Model should be more robust — it predicts what it will see next, and that prediction gives it something to hold onto when the current observation is noisy. But the video-generation backbones underlying current WAMs haven't fully delivered on this promise. They fail under camera shifts, gripper offsets, and visual noise.

The Georgia Tech work suggests that the problem is partly in how robustness information is represented. In some models — Cosmos-Policy, DiT4DiT — the information about how to handle perturbations is encoded in clean geometric directions. Interventions can find those directions and push along them. In other models — LingBot-VA — the information is distributed in ways that resist geometric intervention.

This means the community can work in two directions. One is to design models with better geometric structure — models whose robustness representations are organized enough to be steered. The other is to develop better steering methods for when the structure is messier.

WA-LQR is an example of the first direction. It's a minimal intervention that respects the model's own dynamics and steers only when needed. The name is no accident: it's named after the Linear Quadratic Regulator, a workhorse of optimal control theory that has been solving feedback problems in engineering for decades. The connection to control theory is deliberate. "The local linearity of the diffusion transformer dynamics in a reduced WAM activation space enables the efficient closed-form synthesis of closed-loop steering controllers," the researchers write. They're borrowing mathematical tools from aerospace, process control, and robotics and applying them to the internals of neural networks.

This is characteristic of where the field is heading. As foundation models become more capable, the community is learning to treat them not as mysterious black boxes but as dynamical systems with interpretable structure — systems that can be analyzed, diagnosed, and guided using the same mathematical language that engineers have used to control rockets and chemical plants.

The robot that reaches for a coffee mug with a shifted camera won't be fixed by data collection alone. It needs a model whose internal representations are organized enough to be understood. And it needs control-theoretic methods that can gently nudge those representations toward robustness when the world shifts beneath it.

The paper doesn't claim to have solved robot robustness. It claims something more modest but more lasting: it has shown that mechanistic interpretability can diagnose which models are fixable, and that optimal control can guide the repair. The path from diagnosis to cure is still under construction. But now we know which direction to walk.