Humanoid Lab
University of Tsukuba - Cooperative Graduate School System

Vision-based Belt Manipulation by Humanoid Robot

Deformable objects are very common around us in our daily life. Because they have infinitely many degrees of freedom, they present a challenging problem in robotics. Inspired by practical industrial applications, we present our research on using a humanoid robot to take a long, thin and flexible belt out of a bobbin and pick up the bending part of the belt from the ground. By proposing a novel non-prehensile manipulation strategy “scraping” which utilizes the friction between the gripper and the surface of the belt, efficient manipulation can be achieved. In addition, a 3D shape detection algorithm for deformable objects is used during manipulation process. By integrating the novel “scraping” motion and the shape detection algorithm into our multi-objective QP-based controller, we show experimentally humanoid robots can complete this complex task.

sim2real: Learning Humanoids Locomotion using RL

Recent advances in deep reinforcement learning (RL) based techniques combined with training in simulation have offered a new approach to developing control policies for legged robots. However, the application of such approaches to real hardware has largely been limited to quadrupedal robots with direct-drive actuators and light-weight bipedal robots with low gear-ratio transmission systems. Application to life-sized humanoid robots has been elusive due to the large sim2real gap arising from their large size, heavier limbs, and a high gear-ratio transmission systems.

In this work, we investigate methods for effectively overcoming the sim2real gap issue for large-humanoid robots for the goal of deploying RL policies trained in simulation to the real hardware.

Link to YouTube video: here.

Enhanced Visual Feedback with Decoupled Viewpoint Control in Immersive Teleoperation using SLAM

During humanoid robot teleoperation, there is a noticeable delay between the motion of the operator’s and robot’s head. This latency could cause the lag in visual feedback, which decreases the immersion of the system, may cause some dizziness and reduce the efficiency of interaction in teleoperation since operator needs to wait for the real-time visual feedback. To solve this problem, we developed a decoupled viewpoint control solution which allows the operator to obtain the visual feedback changes with low-latency in VR and to increase the reachable visibility range. Besides, we propose a complementary SLAM solution which uses the reconstructed mesh to complement the blank area that is not covered by the real-time robot’s point cloud visual feedback. The operator could sense the robot head’s real-time orientation by observing the pose of the point cloud.

Multi-Contact Motions Using the Elbow or Knee of a Humanoid Robot via External Force Estimation

Humanoid robots are expected to work like humans. However, there are still some limitations that make this goal difficult. We focus on the limitation of using intermediate control points such as elbows or knees. While humans use their intermediate body parts, humanoid robots cannot do so. One reason is that humanoid robots lack force sensors there and typically have force sensors only on the end-effectors. To perform control, it is necessary to measure the output. One solution to this problem is to distribute sensors over the whole body, but this is unrealistic due to mechanical and electrical complexity. We adopt external force estimation based on the “Kinetics Observer.” The Kinetics Observer uses IMUs, joint encoders, and force sensors on the end-effectors as measurements, which are standard equipment in many humanoid robots. By applying the external force estimation from the observer, it becomes possible to perform force control without direct force measurement. The picture shows simulation and experimental scene demonstrating multi-contact motion without direct force measurement.

Monocular Visual SLAM that works with Pre-built Point Cloud Maps using Point and Line Features

Monocular Visual SLAM is attractive for its low cost and portability, but it suffers from scale ambiguity and typically cannot localize directly in heterogeneous prior maps such as dense LiDAR/3D-reconstruction point clouds. This project enables a monocular point-line SLAM pipeline to perform absolute localization in a pre-built dense point cloud map. We first extract 3D structural line segments from the dense point cloud offline, and then detect 2D line segments in incoming images online. By establishing robust 2D-3D line correspondences and solving a Perspective-n-Line (PnL) optimization, the system estimates metric absolute camera poses during initialization, recovering real-world scale without requiring additional sensors. After monocular initialization, multiple PnL poses are accumulated and aligned with the SLAM trajectory to compute a fixed Sim(3) transformation, which is then propagated throughout tracking and mapping to maintain globally consistent poses in the dense map coordinate frame.