The 8-Gigabyte Brain: What It Takes to Run a Robot on Cheap Hardware
The robot had an 8 gigabyte budget. Not for storage, not for a backup drive — for everything at once. Every camera frame, every intermediate calculation, every neural network weight needed to fit inside a single chip smaller than a paperback novel, one of the cheapest machines NVIDIA makes. And it needed to coordinate two arms, pick up a squishy beanbag, and place it in a box. All simultaneously. All without a single byte of help from the cloud.
This is what it took to run a state-of-the-art bimanual manipulation system — the kind of AI that can learn a task from human demonstration and then replicate it — on hardware that costs around $600 and draws under 30 watts. Not because anyone wanted to prove it could be done cheaply, but because the people who build these systems had never actually tried. Every benchmark, every demo, every "breakthrough" you may have read about in robotics had been quietly running on a workstation GPU that costs more than most cars. The arms might be cheap; the brain was not.
A team at the Georgia Tech Research Institute decided to find out what happens when you take the brain seriously as a constraint. What they discovered contradicted almost every assumption they started with — about memory, about which algorithm to use, and about whether the industry-standard approach to making neural networks faster even works on the transformer architectures powering modern robotics.
The Science
The system they built has a name — SO-101 — and a lineage. It belongs to a family of low-cost bimanual robots that have quietly transformed what manipulation research looks like. A decade ago, training a robot to use two arms required either pre-programmed controllers or expensive custom hardware. Now you can backdrive two arms through handheld controllers, record what you do, and let a neural network figure out how to replicate it. The platform that popularized this approach, ALOHA, demonstrated that fine, contact-rich tasks could be learned from a modest number of teleoperated demonstrations. Since then, the hardware has become cheaper, the software more standardized, and the community more collaborative. An entire ecosystem now exists — ALOHA 2, SO-101, LeRobot — for collecting manipulation data and training policies on it.
What this ecosystem had not done was seriously examine its own compute assumptions. Training and evaluation happen on workstations with discrete GPUs, tens of gigabytes of dedicated memory, and cooling fans that sound like small aircraft. This is fine for research. It is not fine for deployment. A robot that needs a tethered workstation is not, in any meaningful sense, an edge device. It is a very expensive toy that only works in a lab.
The GTRI team chose the most constrained member of NVIDIA's embedded line to make this point concrete. The Jetson Orin Nano Super Developer Kit — 8 gigabytes of LPDDR5 unified memory shared between CPU and GPU, 102 gigabytes per second of memory bandwidth, 67 INT8 trillion operations per second — is the entry-level tier. The least capable machine on which this system could plausibly run. Precisely because of that, it is the right question to ask.
Their setup observed the workspace through three USB cameras: one mounted above the arms for a global view, two mounted on the wrists for close-in tracking. At 640 by 480 resolution and 10 frames per second, each camera produces roughly as much data per second as a compressed music file. Together with four SO-101 arms (two leader, two follower), a control loop running at 10 hertz, and the neural network itself, they needed all of this to share a single 8 gigabyte memory pool — and to do it without missing a beat.
The task was bimanual pick-and-place of a deformable beanbag. Not a rigid cube, not a carefully positioned cylinder — a soft object that squishes under contact, can be grasped in many configurations, and rewards a policy that understands how two hands coordinate rather than just where each hand goes. The team recorded 100 demonstration episodes, varying the beanbag's starting position and orientation each time, and used this dataset to train two different neural network architectures.
The first was ACT — Action Chunking with Transformers. ACT predicts a chunk of future actions at once (100 actions in their configuration) rather than one action at a time. This "action chunking" smooths out noise and improves stability on fine manipulation tasks. With roughly 52 million parameters and a ResNet18 visual backbone feeding into a transformer, ACT is large enough to be capable and structured enough to be optimizable. The LeRobot framework, which the team used for training and deployment, recommends approximately 100,000 gradient steps for convergence on typical manipulation tasks.
The second was Diffusion Policy. Rather than predicting actions directly, Diffusion Policy learns to denoise — starting from random noise and iteratively refining it toward actions that match what a demonstrator would do. This approach is better at capturing multimodality: if a task can be done in several valid ways, a diffusion model can represent all of them rather than averaging them into a mediocre compromise. The cost is paid twice — at training time (because the model must learn across all noise levels) and at inference time (because producing one action chunk requires running the denoising loop repeatedly). LeRobot's reference Diffusion Policy is trained for 200,000 gradient steps, roughly twice ACT's budget, and is designed for PushT: a 2D planar pushing task with a single end-effector, a single observation view, a rigid target object, and no grasping.
The key design choice in their comparison was to train each policy at its own reference budget rather than at a matched step count. This matters. If ACT converges at 80,000 steps and Diffusion Policy needs 200,000, then a fair comparison between architectures at equal steps would always favor ACT — not because Diffusion Policy is inferior, but because it hadn't finished learning. The team wanted to know which policy could reach task competence under its own recommended training budget, on their actual task, with their actual camera configuration.
Both policies trained on the same dataset, with identical camera configurations and held-out splits, isolating architecture as the only variable. Training happened on an NVIDIA GeForce RTX 3070 — a consumer desktop GPU, used offline only, never present during deployment.
The final piece of the system was quantization: converting the trained ACT model from full 32-bit floating-point precision (FP32) to half precision (FP16) or 8-bit integer (INT8) for faster inference. This is standard practice in embedded deep learning. Smaller numbers mean less memory bandwidth, faster compute, and lower power draw. The team used TensorRT, NVIDIA's inference optimization toolkit, to build optimized engines at each precision level and ran them on the Orin Nano Super's onboard GPU — not the training machine, not a cloud instance, but the actual chip that would live inside the robot.
What They Found
The first surprise came from the camera pipeline.
The team built a GStreamer capture system backed by NVIDIA NVMM buffers — technology that keeps decoded video frames in GPU-accessible memory from the moment they're captured, eliminating the need to copy them from host memory to device memory before the neural network can process them. Their hypothesis was that three simultaneous camera streams at 640 by 480 would not fit within the 8 gigabyte budget using the conventional approach, and that zero-copy sensing would be what made the whole thing possible.
They were wrong about the problem. The conventional path fit comfortably and dropped no frames at all. Over 18,000 attempted frames across five test runs per condition, the legacy pipeline received every single one, with a 0% drop rate. Memory usage was approximately 5.3 gigabytes at peak — well within the 8 gigabyte limit. Zero-copy sensing did not save the system from a memory crisis because there was no memory crisis to begin with.
What zero-copy sensing actually recovered was CPU headroom and worst-case latency. Peak single-core CPU utilization fell from 98.0% to 77.0%. Peak all-core utilization dropped from 73.8% to 33.2%. Maximum pipeline latency — the worst case, the tail where deadlines get missed — fell from 117.31 milliseconds to 101.52 milliseconds, a 15.79 millisecond reduction in the worst observed delay. The gains appeared entirely in the tail of the latency distribution, not in the average, which changed by a negligible 0.16 milliseconds (99.25 ms to 99.09 ms).
The interpretation matters more than the numbers. The legacy pipeline was saturating a single CPU core at peak — 98% utilization — during normal operation. This is a control systems problem, not a throughput problem. A core pinned at 98% during peaks is a source of missed deadlines even when average utilization looks acceptable. Worst-case latency, not mean latency, determines whether a control loop holds its period. The 21-point reduction in peak single-core utilization is what buys the policy room to run. Without it, the CPU becomes a bottleneck even when memory is fine.
ACT Inference Latency by Precision Level
| Label | Value |
|---|---|
| FP32 (PyTorch) | 114.02 |
| FP16 (TensorRT) | 17.93 |
| INT8 (TensorRT) | 12.65 |
The second finding concerned which policy to use in the first place.
ACT trained at its 100,000-step reference budget converged to a task-competent policy, succeeding in 19 of 20 physical trials on the deformable beanbag task. Diffusion Policy, trained at its own 200,000-step reference budget — twice the steps — did not converge to a usable policy. Across 10 physical trials, every rollout produced near-stationary arm motion and terminated at the episode step limit without task progress. The arms barely moved. The policy had not learned what to do.
The team is careful not to interpret this as evidence that ACT is architecturally superior. The opposite argument could be made: their task is precisely the kind Diffusion Policy should excel at. The beanbag deforms under contact, admits many valid grasp configurations, and requires two hands to coordinate — exactly the multimodal action distribution that Diffusion Policy's denoising objective is designed to represent. Prior work (ChicGrasp, specifically) reported Diffusion Policy succeeding on deformable poultry carcass grasping at comparable demonstration counts where behavioral cloning and LSTM-GMM baselines failed entirely. A policy that works on deformable objects should work here.
But the 200,000-step reference budget was not calibrated on their task. It was calibrated on PushT — a 2D simulation with one arm, one camera, a rigid square, and no grasping. Their task is substantially harder along every axis: two arms instead of one, three cameras instead of one, a deformable object instead of a rigid one, and contact-rich bimanual coordination instead of planar pushing. A training budget sufficient for an easier task is not guaranteed sufficient for a harder one. The near-stationary failure mode — a policy collapsing toward the marginal action distribution, which for an arm at rest is approximately no motion — is consistent with the model not having converged, not with a fundamental incapacity.
The finding, then, is about training cost, not architectural quality. Under affordable compute — not unlimited cloud resources, not datacenter-class training machines — the architectures do not perform equivalently. ACT reaches task competence at its reference budget. Diffusion Policy does not reach it at twice the budget, on a substantially harder task than the budget was designed for.
CPU Utilization: Legacy vs Zero-Copy Pipeline
| Label | Value |
|---|---|
| Peak single-core | 98 |
| Peak all-core | 73.8 |
| Avg all-core | 31.5 |
The third finding was the most technically significant, and the most surprising.
ACT was converted to TensorRT and characterized at FP32 (full precision, PyTorch), FP16 (half precision, TensorRT), and INT8 (8-bit integer, TensorRT). The results were dramatic. Mean inference latency dropped from 114.02 milliseconds at FP32 to 17.93 milliseconds at FP16 — a 6.4× speedup — and to 12.65 milliseconds at INT8, a 9.0× speedup overall. Task success was preserved at all three precisions: 19/20 at FP32, 18/20 at FP16, 19/20 at INT8. The 90% success rate at FP16 represents one failed trial out of twenty and is statistically consistent with the other two conditions.
But the structural results are more interesting than the speedups.
TensorRT's INT8 calibration — the process of determining how to represent 32-bit floating-point numbers as 8-bit integers — quantizes the ResNet18 visual backbone. It accepts the vision encoder, the part of the network that processes camera images. What it does not accept is any of the 145 transformer layers. Zero. The INT8 calibration tool examined every one of ACT's transformer components — the attention mechanisms, the feedforward layers, the decoder — and rejected all of them. It fell back to higher-precision execution for the entire transformer stack.
This explains a counterintuitive result in the size data. INT8 reduced the model size by only 0.9% compared to FP16 (70.42 megabytes versus 71.09 megabytes), despite using 8-bit integers instead of 16-bit floats for internal compute. The size reduction from FP32 to FP16 was substantial — 63.9%, from 197.2 MB to 71.09 MB — because FP16 weights are literally half the size of FP32 weights. But INT8, which should theoretically be half the size of FP16 again, barely moved the needle. Because TensorRT was running the transformer in FP16 internally, the weight storage remained essentially unchanged. INT8 delivered a further 28% latency gain (12.65 ms versus 17.93 ms) because it could still use INT8 for the parts of the compute graph that accepted it — the visual backbone, the matmuls that dominate transformer inference — even while storing the weights in FP16 precision.
The practical implication is that general-purpose INT8 calibration, as implemented in TensorRT, is not effective for ACT's transformer layers. Someone seeking to deploy ACT on an INT8-only accelerator — a lower-end edge device, a microcontroller with integer-only support — would need to develop custom quantization strategies or accept that the standard tools don't work out of the box.
Physical Trial Success Rates
| Label | Value |
|---|---|
| ACT FP32 | 95 |
| ACT FP16 | 90 |
| ACT INT8 | 95 |
| Diffusion Policy | 0 |
The second structural result about quantization was subtler but equally important for deployment decisions.
Full-precision (FP32) inference was feasible at ACT's default configuration: predicting 100 actions per forward pass, executing the full chunk before re-predicting. The model fit in memory, inference completed within the 100-millisecond control period, and success rate was 95%. But if you wanted to use temporal ensembling — re-predicting more frequently to smooth out accumulated errors — the memory footprint grew beyond what the 8 gigabyte budget could accommodate at full precision. Quantization became necessary not because the hardware was too slow, but because the action-chunking configuration determined whether the model fit at all.
This is a conditional finding. Whether full-precision inference is feasible depends on a policy hyperparameter that controls how often the policy re-predicts, not on the hardware alone. An ACT deployment that looks viable at one configuration may become infeasible without quantization at another.
Why This Changes Things
The robotics field has spent the last several years celebrating the democratization of hardware. Low-cost bimanual platforms, open-source teleoperation software, standardized datasets, reference policy implementations — the message has been consistent: manipulation is now accessible to anyone with a few thousand dollars and a willingness to learn. And this is true, in a narrow sense. The arms are cheap. The software is open. The demonstrations are collectible.
But the compute story has been told from inside a bubble. Training happens on expensive workstations. Evaluation happens on expensive workstations. "Deployment" in most published results means "running in a lab with a desktop GPU nearby." The cost narrative has always been about the robot, never about the brain.
This paper makes the brain the constraint, and what it finds is that the gap between "works in the lab" and "works at the edge" is not just a matter of optimization. It spans the entire stack: which algorithm to use (ACT converges at its reference budget, Diffusion Policy doesn't on hard tasks), how to move data around (zero-copy matters for CPU headroom, not memory), and which optimization techniques actually work (general-purpose INT8 calibration fails on transformer layers, and the reason is structural, not incidental).
For the field, this is a calibration. Benchmarks that run on workstation GPUs are measuring something different from what edge deployment requires. A success rate on a datacenter-class accelerator is not a success rate on an embedded device; they are different problems with different constraints, and conflating them produces research that cannot generalize.
For practitioners, the findings are actionable. Zero-copy sensing is now a known requirement for CPU headroom, not a speculation about memory. ACT is now a known viable choice under affordable training budgets, with the caveat that "affordable" means the architecture's own reference budget, not an arbitrary step count. And INT8 optimization for transformer-based policies requires custom work — the standard tools will not handle it, and deploying an ACT variant with additional transformer components (InterACT, IACE, LAV-ACT) would face the same or greater quantization difficulty.
For the broader conversation about AI in physical systems, the beanbag result is worth dwelling on. A deformable object, grasped by two hands, carried across a workspace, and placed in a marked zone. The kind of task a four-year-old can do without thinking. The kind of task that has defeated decades of pre-programmed robotics. And now, a $600 embedded computer with 8 gigabytes of memory can run a neural network that does it 19 times out of 20.
Not because the problem is solved. Because the problem is now characterizable.
What's Next
Several threads remain open.
The Diffusion Policy non-convergence result is the most directly interesting follow-up. The task on which ACT succeeded and Diffusion Policy failed was calibrated, by design, against an architecture's reference budget rather than against a matched difficulty level. A fair comparison at equal convergence — running Diffusion Policy until it succeeds, whatever that takes — would tell a different story. The 200,000-step budget was derived from PushT, a task with one arm, one camera, and a rigid square. If Diffusion Policy requires 400,000 steps or 600,000 steps on a bimanual deformable-object task, that is a training cost finding, not an architectural indictment. But it is a cost finding the field does not yet have.
The camera configuration question is also unresolved. AV-ALOHA's work shows that using more cameras does not monotonically improve success rate — on tasks that don't require it, additional cameras degrade performance. What AV-ALOHA does not consider is the camera set as a resource cost. Running three cameras consumes CPU cycles, memory bandwidth, and inference budget. A resource-aware ablation — systematic comparison of camera configurations against success rate, latency, and CPU utilization simultaneously — would tell practitioners which configuration to use under which deployment constraint.
The INT8 finding on transformer layers is the most architecturally significant, and the most in need of follow-up. Why do ACT's 145 transformer layers reject TensorRT's general-purpose INT8 calibration? The paper speculates that the dynamic range of transformer activations — the difference between the smallest and largest values that appear during computation — is too large for 8-bit integers to represent without unacceptable accuracy loss. But this is a hypothesis, not a measurement. Characterizing the activation distributions in ACT's transformer layers, quantifying the dynamic range problem, and developing targeted quantization strategies (per-channel or per-tensor, with or without calibration data) would move this from a finding to a solution.
The conditional nature of quantization necessity — feasible at n_action_steps=100, infeasible at the per-step re-prediction temporal ensembling requires — also deserves systematic study. Different action-chunking configurations make different tradeoffs between inference frequency and inference latency. Mapping this tradeoff space under an 8-gigabyte memory constraint would give practitioners a decision framework rather than a single data point.
There is also the question of what happens on even more constrained hardware. The Orin Nano Super is the entry-level Jetson, but it is not the bottom of the embedded hierarchy. A system with 4 gigabytes, or 2 gigabytes, or an integer-only microcontroller, would face qualitatively different constraints. The findings in this paper characterize what the edge problem looks like; the full space of edge constraints remains to be mapped.
Finally, the real-world deployment context matters. A laboratory success rate of 19/20 is impressive; a factory floor success rate of 19/20 is not sufficient for most commercial applications. The gap between controlled evaluation and operational deployment includes object variation, lighting changes, wear on the grippers, and perturbations that were not present during demonstration. The beanbag result is a necessary first step. It is not a sufficient demonstration of field readiness.
What this paper does is define the question precisely. Not "can bimanual manipulation work?" — it can. Not "can it work on cheap hardware?" — it can, with caveats. The question is: what does it take, in compute, in memory, in latency budget, in algorithm choice, in data, to make it work reliably enough to matter? And the answer, derived from putting an actual system on an actual robot with an actual constraint, is more tractable than the field has assumed.
Eight gigabytes. That's the budget. Here's what's inside it.