Reinforcement Learning Framework for Improving Real-World Performance of the Bipedal Robot SUBO-2 with Low-Backdrivability

This paper proposes a deep reinforcement learning framework to improve the Sim-to-real performance of the bipedal robot SUBO-2. This robot exhibits low-backdrivability characteristics due to its actuators, which are composed of harmonic gears with a high reduction ratio (up to 240:1), electric motors, and timing belt-pulley mechanisms. Key features of this study include the application of an unsupervised learning-based Unsupervised Actuator Network (UAN). By modeling the nonlinear dynamics of the actuators, UAN enables the realization of similar behaviors in both simulation and real-world environments, even when using identical joint PD gains. An experimental environment was established to fix the robot's torso for training data acquisition. During UAN training, a joint angular velocity tracking reward was added to improve both joint angle and angular velocity tracking performance. However, when applying the trained UAN to walking policy training, discontinuities in the policy output increase action jerk, which degrades learning stability and walking performance. To resolve this, Lipschitz-Constrained Policy (LCP) is applied to constrain the policy gradient, reducing action jerks and ensuring learning stability. The proposed UAN-LCP framework was verified through 0.3 m/s forward walking simulation experiments, which confirmed reductions in action jerk, joint position jerk, joint angular velocity, torque, and power consumption compared to conventional smoothing methods. Furthermore, in treadmill walking experiments with the real robot SUBO-2, both joint position jerk and power consumption were reduced. A comparison of body orientation demonstrated that the proposed deep reinforcement learning framework achieves stable and energy-efficient walking. Finally, it was experimentally verified that the framework enables diverse locomotion controls even on soft ground, such as a carpet, which is challenging for bipedal walking.