How AI Learned to Control Robots That Feel Their Own Joints

A Robot That Feels Its Own Joints

Imagine performing surgery from thousands of miles away. Your hands move in New York; robotic arms move in Tokyo, faithfully replicating every gesture, every micro-adjustment. Now imagine a slight lag in the connection—or worse, a lag that varies unpredictably, sometimes fast, sometimes slow. Add to this the fact that those robotic arms aren't perfectly rigid; they flex and vibrate slightly with each movement, like a human wrist does. Keeping the master's motion and the slave's response synchronized under these conditions is one of the hardest problems in robotics.

A new paper by researchers at Bu-Ali Sina University in Iran tackles exactly this challenge. Their hybrid approach—combining a traditional stable controller with a deep reinforcement learning agent that learns to tune gains on the fly—achieves something remarkable: it maintains precise position tracking and suppresses unwanted vibrations even when network delays fluctuate unpredictably between zero and one second. The work bridges the gap between theory and practical deployment, offering a real solution for the messy, uncertain conditions that characterize actual teleoperation networks.

The core insight is elegant in its simplicity: don't try to solve everything with one method. Instead, let a conventional Proportional-plus-Damping controller provide a stability guarantee—a mathematical safety net—and let a learning algorithm optimize the performance on top of that foundation. The result is a system that is both provably stable and practically effective.

The Physics of Remote Manipulation

Teleoperation isn't science fiction. It's how surgeons perform minimally invasive procedures through the da Vinci system, how astronauts repair satellites from the relative safety of a space station, and how engineers handle hazardous materials in decommissioned nuclear plants. The basic setup is elegant: a human operator controls a "master" robot, whose motions are transmitted over a communication network to a "slave" robot that replicates the actions in a remote environment.

But the physical reality is far messier than this simple description suggests. Real robotic joints aren't perfectly rigid—they flex under load, store energy temporarily, and release it in ways that cause vibrations and position errors. When a surgeon moves a laparoscopic tool through the da Vinci interface, the remote instrument's tip doesn't move exactly as commanded; it lags, overshoots, and oscillates slightly due to these flexibilities in the mechanical linkage.

Compounding this problem are the communication delays inherent in any networked system. In an ideal world, a command sent from master to slave arrives instantly. In the real world, packets of data traverse routers, encounter congestion, and arrive after variable intervals. A command might take 50 milliseconds to cross the Atlantic on a good day, and 800 milliseconds during a network outage. The system must remain stable—in the mathematical sense that its energy doesn't grow unbounded—across this entire range of conditions.

The mathematical model that captures these dynamics involves two coupled systems of differential equations. For the rigid master manipulator, the equation describes how the joint positions evolve based on the operator's applied torque and the controller's corrective forces. For the flexible slave manipulator, the model splits into two interconnected parts: one describing the link position (where the tool tip actually is) and one describing the motor position (where the actuator is trying to move it to). The difference between these two positions—governed by a stiffness matrix representing the joint's elasticity—is precisely what causes the vibrations that plague teleoperation systems.

The researchers deliberately chose this formulation because it reflects real-world manipulators more accurately than the rigid-joint models assumed by earlier control approaches. A surgical robot's joints, a space manipulator's linkages, and a rehabilitation device's actuated limbs all exhibit this kind of elastic behavior. By modeling it explicitly rather than treating it as negligible, the control design can account for—and ideally suppress—its effects.

The Hybrid Architecture

The proposed control strategy rests on a clever division of labor between two complementary components. The first is a Proportional-plus-Damping (P+d) controller, a relatively simple architecture that has been proven in prior work to guarantee stability for nonlinear teleoperators even with flexible joints and time-varying delays. The second is a Twin Delayed Deep Deterministic Policy Gradient (TD3) agent that adaptively tunes the controller's gains in real time, optimizing performance beyond what any fixed-gain configuration could achieve.

The P+d controller works by applying torque proportional to the position error—how far the slave's actual position deviates from what the master commanded—plus a damping term that resists motion. Think of it like steering a car: proportional control is like turning the wheel more when you're far from your lane; damping is like the resistance that naturally slows your corrections as you approach the target. This combination is intuitive, well-understood, and analytically tractable. The researchers prove, using a Lyapunov-Krasovskii functional approach, that as long as certain inequalities relating gains to delay bounds are satisfied, the closed-loop system cannot become unstable.

The Lyapunov-Krasovskii method is a powerful tool from control theory. Rather than solving the system's differential equations directly—which is generally impossible for nonlinear systems with delays—engineers construct a "candidate function" that behaves like a system's energy. If they can show that this function always decreases (or at least never increases) along the system's trajectories, then the system is guaranteed to remain bounded. The researchers adopted the same functional used by Nuño and colleagues, adapted to handle time-varying gains produced by the learning agent.

One crucial insight in the stability analysis: the Lyapunov functional uses fixed positive weights rather than time-varying ones. This means that the derivative of the energy function contains no terms involving the rate of change of the controller gains. In practical terms, the TD3 agent can vary the gains as rapidly as it wants—updating them every control cycle, if necessary—without destabilizing the system, provided the gains themselves stay within the admissible range. This decouples the learning dynamics from the stability proof, a subtle but important engineering choice that enables aggressive optimization.

The stability condition reduces to a single inequality: the product of local and remote damping gains must exceed a threshold based on the sum of the maximum delays and the product of proportional gains, plus a small safety margin. This inequality serves as a constraint that the TD3 agent must respect when selecting gains.

On top of this stable foundation, the TD3 agent continuously adjusts the remote-side proportional and damping gains to minimize tracking errors and suppress vibrations. TD3 is an off-policy, model-free deep reinforcement learning algorithm that has proven effective across a wide range of continuous control tasks. Unlike policy-gradient methods that can be unstable during training, or value-function methods that struggle with continuous action spaces, TD3 strikes a practical balance: it uses two critic networks to reduce overestimation bias (the "twin" in its name), delays policy updates to let the value function stabilize before learning from it (the "delayed"), and adds clipped noise to target actions to smooth out policy changes (target policy smoothing).

The agent learns through standard reinforcement learning: it observes the current state of the teleoperation system, takes an action (adjusting the gains), receives a reward (based on how well the system performed), and updates its neural networks to prefer actions that led to higher rewards. Over thousands of training episodes in simulation, the agent learns a mapping from system states to gain settings that balances multiple objectives.

Learning to Feel the Network

The state vector that the TD3 agent observes is carefully designed to capture the information it needs for effective gain tuning. It includes the remote joint's current position and velocity, the delayed position of the local master (delayed because this is what the slave can actually observe through the network), the tracking error between remote and master positions, and the integral of this tracking error over time. It also includes what the researchers call the "damping error"—the difference between the motor position and the link position—which is a direct measure of how much the flexible joint is currently flexing. By observing both the integral of tracking error and the integral of damping error, the agent can detect and correct persistent biases that a purely proportional controller might miss.

The reward function captures the three things the system cares about: accurate position tracking, minimal joint flexing (which manifests as vibration), and economical use of control effort. The first term penalizes large tracking errors, weighted by a coefficient that prioritizes getting the slave to follow the master accurately. The second term penalizes large damping errors, which correspond to vibrations and oscillations in the flexible joint. The third term penalizes large actions—the magnitude of the gain changes themselves—which encourages the agent to find smooth, stable tuning policies rather than oscillating between extreme gain values.

The training process unfolds in simulation, where the agent can safely explore different gain configurations and learn from the consequences without risking damage to physical hardware. After training, the learned policy can be deployed on the real system, where it continues to adapt in real time to actual conditions. The agent's action—output by a neural network with a Softplus final activation that ensures all gains remain positive—is applied directly to the remote controller, updating the proportional and damping gains every control cycle.

The neural network architecture follows standard practice for TD3: both actor and critic networks are fully connected multilayer perceptrons with ReLU activations. The actor network's final layer uses Softplus instead of the typical linear activation, which guarantees that all gain outputs are strictly positive—negative gains would be physically meaningless and could destabilize the system. Each critic network processes both states and actions through parallel feature-extraction layers before merging them to estimate the Q-value, and two separate critics are maintained to mitigate overestimation bias.

Simulating the Surgeon's Hands

The researchers tested their approach through extensive numerical simulations in MATLAB/Simulink, modeling a 2-Degree-of-Freedom (2-DOF) bilateral teleoperation system over a 40-second simulation with 10-millisecond sampling intervals. The master was modeled as a rigid 2-DOF manipulator, while the slave was modeled as a flexible-joint 2-DOF manipulator with the stiffness characteristics discussed earlier. Joint flexibility was represented by a stiffness matrix of 100 N·m per radian, meaning the joint springs back with a torque proportional to how far the link has deflected from the motor position.

The physical parameters reflect typical laboratory-scale robotic manipulators: 0.38-meter link lengths, masses of 0.5 kg for the first link and 0.35 kg for the second, and motor inertias of 0.3 kg·m². The human operator was modeled as a spring-damper system that attempts to move the master toward a sinusoidal reference trajectory, simulating a human operator tracing a smooth path. The environment was set to free-space operation (no external forces on the slave), which isolates the tracking performance from contact dynamics that might otherwise confound the results.

The communication delays on both forward and backward channels were bounded by one second—severe enough to challenge any controller but realistic for internet-based teleoperation across continental distances. The actual delay varied over time, as network conditions changed, creating the time-varying scenario that motivated the research.

During training, the TD3 agent learned to adjust the remote gains in response to the observed system states. The simulation results show the learned gains evolving over time, converging toward values that depend on the current phase of the human operator's commanded motion. Unlike a fixed-gain controller, which uses the same parameters regardless of what the operator is doing, the adaptive approach modulates gains based on the current dynamics: higher proportional gains when rapid corrections are needed, higher damping when oscillations threaten to develop.

Position and Vibration Control Performance

Label	Value
Joint 1 tracking	0.05
Joint 2 tracking	0.04
Joint 1 damping error	0.03
Joint 2 damping error	0.025

Steady-state errors demonstrate the adaptive controller's ability to minimize both position deviation and mechanical vibration across multiple degrees of freedom.

The time response of the system demonstrates the effectiveness of this approach. The remote joint positions track the delayed local positions closely, with errors settling rapidly after any transient disturbance. The tracking error—which represents the difference between what the master commanded and where the slave actually moved—remains bounded and generally small throughout the simulation, demonstrating that the adaptive gain tuning successfully maintains coordination between master and slave despite the time-varying delays.

Perhaps more importantly, the link-joint damping error (the difference between where the motor is and where the link actually is) remains well-controlled. This error directly corresponds to the joint flexing that causes vibrations and oscillations in rigid-body manipulators. By adaptively adjusting the damping gain based on observed dynamics, the TD3 agent suppresses these vibrations more effectively than any fixed-gain configuration could, since the optimal damping level depends on the current velocity and acceleration of the system.

Communication Channel Delay Profile

Label	Value
0-5s	0.2
5-10s	0.45
10-15s	0.15
15-20s	0.6
20-25s	0.85
25-30s	0.3
30-35s	0.7
35-40s	0.4

The delay profile shows that network conditions change rapidly and unpredictably, with the system required to maintain stability and performance across the full range from 0.15s to 0.85s latency.

The communication delay profile illustrates the challenge the system faces. Delays vary unpredictably over time, sometimes increasing sharply and sometimes decreasing, never staying constant for long. A controller tuned for a particular delay value would perform poorly when conditions change, but the adaptive approach continuously adjusts to maintain performance across the entire range of possible delays.

The system maintains stability not by avoiding the effects of delay variation but by respecting the inequality constraint that relates gains to the maximum possible delay. As long as the current gain settings satisfy this constraint, the Lyapunov stability guarantee holds regardless of what the delay is doing at the moment. The TD3 agent learns to respect this constraint while optimizing for tracking performance and vibration suppression.

Why This Matters for Real-World Deployment

The contribution of this work extends beyond the specific results demonstrated in simulation. It addresses a fundamental limitation of traditional teleoperation control: the tradeoff between stability and performance that arises from uncertain, time-varying delays.

Classical approaches to teleoperation delay fall into two categories. Methods like scattering transformation and wave variables achieve guaranteed stability by treating the communication channel as a passive element, but they do so at the cost of transparency—the operator feels resistance that doesn't correspond to the remote environment. Passivity-based approaches face similar tradeoffs: ensuring robustness against arbitrary delay values requires conservative gain settings that sacrifice tracking accuracy.

More recent adaptive control methods can adjust to constant or slowly-varying delays, but they struggle with the rapid fluctuations characteristic of internet-based teleoperation. When the delay changes from 200 milliseconds to 600 milliseconds within a single second, a controller designed for one delay regime may perform poorly or even become unstable.

The hybrid approach proposed here sidesteps this tradeoff by separating stability and performance concerns. The P+d controller ensures that the closed-loop system remains stable for any bounded time-varying delay—a mathematical guarantee backed by the Lyapunov-Krasovskii analysis. The TD3 agent then optimizes the actual performance (tracking accuracy, vibration suppression) within the space of gain settings that preserve this stability guarantee.

This separation is what makes the approach practical for deployment. Engineers can trust that the system won't become unstable—theoretically, no matter what the network does—while still benefiting from the performance improvements that adaptive gain tuning provides. The learning agent acts as a intelligent tuning layer sitting atop a proven stable foundation.

The Path Forward

Several limitations of the current work suggest directions for future research. The study focuses exclusively on the remote (slave) manipulator's controller, leaving the local (master) controller's gains fixed. Extending the adaptive approach to both sides of the teleoperation link could yield further performance improvements, though it would also complicate the stability analysis by introducing additional learning dynamics.

The experiments were conducted entirely in simulation, which raises natural questions about whether the learned policy would transfer to physical hardware. Simulation necessarily omits effects like friction, backlash, sensor noise, and unmodeled dynamics that can degrade performance on real systems. Future work should validate the approach on experimental testbeds, and potentially employ techniques like domain randomization during training to improve robustness to modeling errors.

The human operator and environment are modeled with simple linear dynamics in this study. Real humans exhibit complex, time-varying impedance characteristics that depend on posture, fatigue, and task demands. Real environments can exert forces that are difficult to predict and model. Adapting to these sources of uncertainty—while maintaining the stability guarantee—remains an open challenge.

From a machine learning perspective, the current approach trains the TD3 agent offline in simulation before deploying the learned policy. An online learning variant that continues to adapt based on interaction with the actual system could potentially achieve better performance by tailoring the control policy to the specific characteristics of the deployed hardware and network. However, online learning in safety-critical systems raises additional concerns about stability during the exploration phase.

The Bigger Picture

Teleoperation sits at the intersection of robotics, control theory, and human-computer interaction—a nexus that will only grow more important as networks become faster, robots become more capable, and applications become more demanding.

Consider telesurgery. The da Vinci system, which dominates the field, currently requires dedicated high-bandwidth, low-latency communication links to function safely. This limits its applicability to well-equipped hospitals and restricts the ability to provide remote expertise during emergencies. If control systems could reliably maintain stability and performance over ordinary internet connections—with their variable delays and occasional congestion—the scope of telesurgery could expand dramatically.

Space exploration offers another compelling application. Astronauts aboard the International Space Station already control robotic arms to capture visiting cargo vehicles, and future missions may involve teleoperation of rovers on Mars or moons from Earth. The communication delays involved—up to 20 minutes round-trip for Mars—fall far outside the framework considered here, but the same principles of adaptive gain tuning could be extended to handle longer delays with appropriate modifications.

Rehabilitation robotics represents perhaps the most intimate application. Remote physical therapy, guided by a therapist hundreds of miles away, could address critical shortages of specialized care in rural and underserved communities. A patient with a flexible exoskeleton could receive real-time guidance from a therapist whose commands traverse the internet, adaptive controllers ensuring that the robot assists rather than startles, supports rather than destabilizes.

The work by Attarzadeh and colleagues doesn't solve all of these challenges—it doesn't even attempt to. But it takes a meaningful step toward the adaptive, robust control systems that such applications will require. By combining the mathematical rigor of Lyapunov-based stability analysis with the practical flexibility of deep reinforcement learning, the research demonstrates a design pattern that could prove fruitful across many domains where uncertainty and variation challenge traditional control approaches.

The robot that can feel its own joints, that learns to compensate for the network's vagaries, that maintains its composure when the world won't stay still—this robot is getting closer to reality.

System Parameters and Operating Conditions

Label	Value
Stiffness (Sr)	100 N·m/rad
Link length (l₁=l₂)	0.38 N·m/rad
Link 1 mass	0.5 N·m/rad
Link 2 mass	0.35 N·m/rad
Motor inertia (Jr)	0.3 N·m/rad
Max delay bound (Tl*, Tr*)	1 N·m/rad
Local proportional gain (Kl)	25 N·m/rad
Local damping gain (Bl)	10 N·m/rad

Simulation parameters reflect typical laboratory-scale robotic manipulators with meaningful joint flexibility and severe (up to 1s) communication delays.