Search

Imitation learning is one of the methods for reproducing human demonstration adaptively in robots. So far, it has been found that generalization ability of the imitation learning enables the robots to perform tasks adaptably in untrained environments. However, motion styles such as motion trajectory and the amount of force applied depend largely on the dataset of human demonstration, and settle down to an average motion style. In this study, we propose a method that adds parametric bias to the conventional imitation learning network and can add constraints to the motion style. By experiments using PR2 and the musculoskeletal humanoid MusashiLarm, we show that it is possible to perform tasks by changing its motion style as intended with constraints on joint velocity, muscle length velocity, and muscle tension., Comment: Accepted at IEEE Robotics and Automation Letters

The estimation and management of motor temperature are important for the continuous movements of robots. In this study, we propose an online learning method of thermal model parameters of motors for an accurate estimation of motor core temperature. Also, we propose a management method of motor core temperature using the updated model and anomaly detection method of motors. Finally, we apply this method to the muscles of the musculoskeletal humanoid and verify the ability of continuous movements., Comment: Accepted at IEEE Robotics and Automation Letters

Various robotic tool manipulation methods have been developed so far. However, to our knowledge, none of them have taken into account the fact that the grasping state such as grasping position and tool angle can change at any time during the tool manipulation. In addition, there are few studies that can handle deformable tools. In this study, we develop a method for estimating the position of a tool-tip, controlling the tool-tip, and handling online adaptation to changes in the relationship between the body and the tool, using a neural network including parametric bias. We demonstrate the effectiveness of our method for online change in grasping state and for deformable tools, in experiments using two different types of robots: axis-driven robot PR2 and tendon-driven robot MusashiLarm., Comment: Accepted at IEEE Robotics and Automation Letters

The flexible musculoskeletal hand is difficult to modelize, and its model can change constantly due to deterioration over time, irreproducibility of initialization, etc. Also, for object recognition, contact detection, and contact control using the hand, it is desirable not to use a neural network trained for each task, but to use only one integrated network. Therefore, we develop a method to acquire a sensor state equation of the musculoskeletal hand using a recurrent neural network with parametric bias. By using this network, the hand can realize recognition of the grasped object, contact simulation, detection, and control, and can cope with deterioration over time, irreproducibility of initialization, etc. by updating parametric bias. We apply this study to the hand of the musculoskeletal humanoid Musashi and show its effectiveness., Comment: Accepted at IEEE Robotics and Automation Letters

The flexible under-actuated musculoskeletal hand is superior in its adaptability and impact resistance. On the other hand, since the relationship between sensors and actuators cannot be uniquely determined, almost all its controls are based on feedforward controls. When grasping and using a tool, the contact state of the hand gradually changes due to the inertia of the tool or impact of action, and the initial contact state is hardly kept. In this study, we propose a system that trains the predictive network of sensor state transition using the actual robot sensor information, and keeps the initial contact state by a feedback control using the network. We conduct experiments of hammer hitting, vacuuming, and brooming, and verify the effectiveness of this study., Comment: Accepted at ICRA2020

While the musculoskeletal humanoid has various biomimetic benefits, the modeling of its complex structure is difficult, and many learning-based systems have been developed so far. There are various methods, such as control methods using acquired relationships between joints and muscles represented by a data table or neural network, and state estimation methods using Extended Kalman Filter or table search. In this study, we construct a Musculoskeletal AutoEncoder representing the relationship among joint angles, muscle tensions, and muscle lengths, and propose a unified method of state estimation, control, and simulation of musculoskeletal humanoids using it. By updating the Musculoskeletal AutoEncoder online using the actual robot sensor information, we can continuously conduct more accurate state estimation, control, and simulation than before the online learning. We conducted several experiments using the musculoskeletal humanoid Musashi, and verified the effectiveness of this study., Comment: Accepted at IEEE Robotics and Automation Letters

This paper summarizes an autonomous driving project by musculoskeletal humanoids. The musculoskeletal humanoid, which mimics the human body in detail, has redundant sensors and a flexible body structure. These characteristics are suitable for motions with complex environmental contact, and the robot is expected to sit down on the car seat, step on the acceleration and brake pedals, and operate the steering wheel by both arms. We reconsider the developed hardware and software of the musculoskeletal humanoid Musashi in the context of autonomous driving. The respective components of autonomous driving are conducted using the benefits of the hardware and software. Finally, Musashi succeeded in the pedal and steering wheel operations with recognition., Comment: Accepted at IEEE Robotics and Automation Magazine

Manipulation performance improvement is crucial for aerial robots. For aerial manipulators, the baselink position and attitude errors directly affect the precision at the end effector. To address this stability problem, fixed-body approaches such as perching on the environment using the rotor suction force are useful. Additionally, conventional arm-equipped multirotors, called rotor-concentrated manipulators (RCMs), find it difficult to generate a large wrench at the end effector due to joint torque limitations. Using distributed rotors to each link, the thrust can support each link weight, decreasing the arm joints' torque. Based on this approach, rotor-distributed manipulators (RDMs) can increase feasible wrench and reachability of the end-effector. This paper introduces a minimal configuration of a rotor-distributed manipulator that can perch on surfaces, especially ceilings, using a part of their body. First, we design a minimal rotor-distributed arm considering the flight and end-effector performance. Second, a flight controller is proposed for this minimal RDM along with a perching controller adaptable for various types of aerial robots. Third, we propose a motion planning method based on inverse kinematics (IK), considering specific constraints to the proposed RDMs such as perching force. Finally, we evaluate flight and perching motions and \revise{confirm} that the proposed manipulator can significantly improve the manipulation performance., Comment: IEEE Transactions on Robotics (2023)

The musculoskeletal humanoid is difficult to modelize due to the flexibility and redundancy of its body, whose state can change over time, and so balance control of its legs is challenging. There are some cases where ordinary PID controls may cause instability. In this study, to solve these problems, we propose a method of learning a correlation model among the joint angle, muscle tension, and muscle length of the ankle and the zero moment point to perform balance control. In addition, information on the changing body state is embedded in the model using parametric bias, and the model estimates and adapts to the current body state by learning this information online. This makes it possible to adapt to changes in upper body posture that are not directly taken into account in the model, since it is difficult to learn the complete dynamics of the whole body considering the amount of data and computation. The model can also adapt to changes in body state, such as the change in footwear and change in the joint origin due to recalibration. The effectiveness of this method is verified by a simulation and by using an actual musculoskeletal humanoid, Musashi., Comment: Accepted at IROS2022

While the musculoskeletal humanoid has various biomimetic benefits, its complex modeling is difficult, and many learning control methods have been developed. However, for the actual robot, the hysteresis of its joint angle tracking is still an obstacle, and realizing target posture quickly and accurately has been difficult. Therefore, we develop a feedback control method considering the hysteresis. To solve the problem in feedback controls caused by the closed-link structure of the musculoskeletal body, we update a neural network representing the relationship between the error of joint angles and the change in target muscle lengths online, and realize target joint angles accurately in a few trials. We compare the performance of several configurations with various network structures and loss definitions, and verify the effectiveness of this study on an actual musculoskeletal humanoid, Musashi., Comment: Accepted at IROS2022

In this study, we investigate object grasping by visual servoing in a low-rigidity robot. It is difficult for a low-rigidity robot to handle its own body as intended compared to a rigid robot, and calibration between vision and body takes some time. In addition, the robot must constantly adapt to changes in its body, such as the change in camera position and change in joints due to aging. Therefore, we develop a method for a low-rigidity robot to autonomously learn visual servoing of its body. We also develop a mechanism that can adaptively change its visual servoing according to temporal body changes. We apply our method to a low-rigidity 6-axis arm, MyCobot, and confirm its effectiveness by conducting object grasping experiments based on visual servoing., Comment: Accepted at IEEE Robotics and Automation Letters

Various robots have been developed so far; however, we face challenges in modeling the low-rigidity bodies of some robots. In particular, the deflection of the body changes during tool-use due to object grasping, resulting in significant shifts in the tool-tip position and the body's center of gravity. Moreover, this deflection varies depending on the weight and length of the tool, making these models exceptionally complex. However, there is currently no control or learning method that takes all of these effects into account. In this study, we propose a method for constructing a neural network that describes the mutual relationship among joint angle, visual information, and tactile information from the feet. We aim to train this network using the actual robot data and utilize it for tool-tip control. Additionally, we employ Parametric Bias to capture changes in this mutual relationship caused by variations in the weight and length of tools, enabling us to understand the characteristics of the grasped tool from the current sensor information. We apply this approach to the whole-body tool-use on KXR, a low-rigidity plastic-made humanoid robot, to validate its effectiveness., Comment: Accepted at ICRA2024, website - https://haraduka.github.io/wholebody-tooluse/

In this study, we present an implementation strategy for a robot that performs peg transfer tasks in Fundamentals of Laparoscopic Surgery (FLS) via imitation learning, aimed at the development of an autonomous robot for laparoscopic surgery. Robotic laparoscopic surgery presents two main challenges: (1) the need to manipulate forceps using ports established on the body surface as fulcrums, and (2) difficulty in perceiving depth information when working with a monocular camera that displays its images on a monitor. Especially, regarding issue (2), most prior research has assumed the availability of depth images or models of a target to be operated on. Therefore, in this study, we achieve more accurate imitation learning with only monocular images by extracting motion constraints from one exemplary motion of skilled operators, collecting data based on these constraints, and conducting imitation learning based on the collected data. We implemented an overall system using two Franka Emika Panda Robot Arms and validated its effectiveness., Comment: Accepted at ICRA2024, website - https://haraduka.github.io/fls-imitation

Library-based methods are known to be very effective for fast motion planning by adapting an experience retrieved from a precomputed library. This article presents CoverLib, a principled approach for constructing and utilizing such a library. CoverLib iteratively adds an experience-classifier-pair to the library, where each classifier corresponds to an adaptable region of the experience within the problem space. This iterative process is an active procedure, as it selects the next experience based on its ability to effectively cover the uncovered region. During the query phase, these classifiers are utilized to select an experience that is expected to be adaptable for a given problem. Experimental results demonstrate that CoverLib effectively mitigates the trade-off between plannability and speed observed in global (e.g. sampling-based) and local (e.g. optimization-based) methods. As a result, it achieves both fast planning and high success rates over the problem domain. Moreover, due to its adaptation-algorithm-agnostic nature, CoverLib seamlessly integrates with various adaptation methods, including nonlinear programming-based and sampling-based algorithms.

When a robot executes a task, it is necessary to model the relationship among its body, target objects, tools, and environment, and to control its body to realize the target state. However, it is difficult to model them using classical methods if the relationship is complex. In addition, when the relationship changes with time, it is necessary to deal with the temporal changes of the model. In this study, we have developed Deep Predictive Model with Parametric Bias (DPMPB) as a more human-like adaptive intelligence to deal with these modeling difficulties and temporal model changes. We categorize and summarize the theory of DPMPB and various task experiments on the actual robots, and discuss the effectiveness of DPMPB., Comment: Accepted at Robotics and Automation Magazine (RAM)

Tendon-driven musculoskeletal humanoids typically have complex structures similar to those of human beings, such as ball joints and the scapula, in which encoders cannot be installed. Therefore, joint angles cannot be directly obtained and need to be estimated using the changes in muscle lengths. In previous studies, methods using table-search and extended kalman filter have been developed. These methods express the joint-muscle mapping, which is the nonlinear relationship between joint angles and muscle lengths, by using a data table, polynomials, or a neural network. However, due to computational complexity, these methods cannot consider the effects of polyarticular muscles. In this study, considering the limitation of the computational cost, we reduce unnecessary degrees of freedom, divide joints and muscles into several groups, and formulate a joint angle estimation method that takes into account polyarticular muscles. Also, we extend the estimation method to propose a joint angle estimation method using only the relative changes in muscle lengths. By this extension, which does not use absolute muscle lengths, we do not need to execute a difficult calibration of muscle lengths for tendon-driven musculoskeletal humanoids. Finally, we conduct experiments in simulation and actual environments, and verify the effectiveness of this study., Comment: Accepted at Humanoids2018

By the recent spread of machine learning in the robotics field, a humanoid that can act, perceive, and learn in the real world through contact with the environment needs to be developed. In this study, as one of the choices, we propose a novel humanoid TWIMP, which combines a human mimetic musculoskeletal upper limb with a two-wheel inverted pendulum. By combining the benefit of a musculoskeletal humanoid, which can achieve soft contact with the external environment, and the benefit of a two-wheel inverted pendulum with a small footprint and high mobility, we can easily investigate learning control systems in environments with contact and sudden impact. We reveal our whole concept and system details of TWIMP, and execute several preliminary experiments to show its potential ability., Comment: Accepted at Humanoids2018

Modular self-reconfigurable robots (MSRRs) offer enhanced task flexibility by constructing various structures suitable for each task. However, conventional terrestrial MSRRs equipped with wheels face critical challenges, including limitations in the size of constructible structures and system robustness due to elevated wrench loads applied to each module. In this work, we introduce an Aerial MSRR (A-MSRR) system named BEATLE, capable of merging and separating in-flight. BEATLE can merge without applying wrench loads to adjacent modules, thereby expanding the scalability and robustness of conventional terrestrial MSRRs. In this article, we propose a system configuration for BEATLE, including mechanical design, a control framework for multi-connected flight, and a motion planner for reconfiguration motion. The design of a docking mechanism and housing structure aims to balance the durability of the constructed structure with ease of separation. Furthermore, the proposed flight control framework achieves stable multi-connected flight based on contact wrench control. Moreover, the proposed motion planner based on a finite state machine (FSM) achieves precise and robust reconfiguration motion. We also introduce the actual implementation of the prototype and validate the robustness and scalability of the proposed system design through experiments and simulation studies.

The human joint is an open-type joint composed of bones, cartilage, ligaments, synovial fluid, and joint capsule, having advantages of flexibility and impact resistance. However, replicating this structure in robots introduces friction challenges due to the absence of bearings. To address this, our study focuses on mimicking the fluid-exuding function of human cartilage. We employ a rubber-based 3D printing technique combined with absorbent materials to create a versatile and easily designed cartilage sheet for biomimetic robots. We evaluate both the fluid-exuding function and friction coefficient of the fabricated flat cartilage sheet. Furthermore, we practically create a piece of curved cartilage and an open-type biomimetic ball joint in combination with bones, ligaments, synovial fluid, and joint capsule to demonstrate the utility of the proposed cartilage sheet in the construction of such joints., Comment: Accepted at RoboSoft2024

The body structures of tendon-driven musculoskeletal humanoids are complex, and accurate modeling is difficult, because they are made by imitating the body structures of human beings. For this reason, we have not been able to move them accurately like ordinary humanoids driven by actuators in each axis, and large internal muscle tension and slack of tendon wires have emerged by the model error between its geometric model and the actual robot. Therefore, we construct a joint-muscle mapping (JMM) using a neural network (NN), which expresses a nonlinear relationship between joint angles and muscle lengths, and aim to move tendon-driven musculoskeletal humanoids accurately by updating the JMM online from data of the actual robot. In this study, the JMM is updated online by using the vision of the robot so that it moves to the correct position (Vision Updater). Also, we execute another update to modify muscle antagonisms correctly (Antagonism Updater). By using these two updaters, the error between the target and actual joint angles decrease to about 40% in 5 minutes, and we show through a manipulation experiment that the tendon-driven musculoskeletal humanoid Kengoro becomes able to move as intended. This novel system can adapt to the state change and growth of robots, because it updates the JMM online successively., Comment: Accepted at IEEE Robotics and Automation Letters, 2018

The tendon-driven musculoskeletal humanoid has many benefits that human beings have, but the modeling of its complex muscle and bone structures is difficult and conventional model-based controls cannot realize intended movements. Therefore, a learning control mechanism that acquires nonlinear relationships between joint angles, muscle tensions, and muscle lengths from the actual robot is necessary. In this study, we propose a system which runs the learning control mechanism for a long time to keep the self-body image of the musculoskeletal humanoid correct at all times. Also, we show that the musculoskeletal humanoid can conduct position control, torque control, and variable stiffness control using this self-body image. We conduct a long-time self-body image acquisition experiment lasting 3 hours, evaluate variable stiffness control using the self-body image, etc., and discuss the superiority and practicality of the self-body image acquisition of musculoskeletal structures, comprehensively., Comment: Accepted at IEEE Robotics and Automation Letters, 2019

Tendon-driven musculoskeletal humanoids have many benefits in terms of the flexible spine, multiple degrees of freedom, and variable stiffness. At the same time, because of its body complexity, there are problems in controllability. First, due to the large difference between the actual robot and its geometric model, it cannot move as intended and large internal muscle tension may emerge. Second, movements which do not appear as changes in muscle lengths may emerge, because of the muscle route changes caused by softness of body tissue. To solve these problems, we construct two models: ideal joint-muscle model and muscle-route change model, using a neural network. We initialize these models by a man-made geometric model and update them online using the sensor information of the actual robot. We validate that the tendon-driven musculoskeletal humanoid Kengoro is able to obtain a correct self-body image through several experiments., Comment: Accepted at IROS2018

In this study, a chair-type asymmetric tripedal low-rigidity robot was designed based on the three-legged chair character in the movie "Suzume" and its gait was generated. Its body structure consists of three legs that are asymmetric to the body, so it cannot be easily balanced. In addition, the actuator is a servo motor that can only feed-forward rotational angle commands and the sensor can only sense the robot's posture quaternion. In such an asymmetric and imperfect body structure, we analyzed how gait is generated in walking and stand-up motions by generating gaits with two different methods: a method using linear completion to connect the postures necessary for the gait discovered through trial and error using the actual robot, and a method using the gait generated by reinforcement learning in the simulator and reflecting it to the actual robot. Both methods were able to generate gait that realized walking and stand-up motions, and interesting gait patterns were observed, which differed depending on the method, and were confirmed on the actual robot. Our code and demonstration videos are available here: https://github.com/shin0805/Chair-TypeAsymmetricalTripedalRobot.git, Comment: Accepted at RoboSoft2024, website - https://shin0805.github.io/chair-type-tripedal-robot/ , YouTube - https://youtu.be/-f8LDlhmdBg

In this study, seated walk, a movement of walking while sitting on a chair with casters, is realized on a musculoskeletal humanoid from human teaching. The body is balanced by using buttock-contact sensors implemented on the planar interskeletal structure of the human mimetic musculoskeletal robot. Also, we develop a constrained teaching method in which one-dimensional control command, its transition, and a transition condition are described for each state in advance, and a threshold value for each transition condition such as joint angles and foot contact sensor values is determined based on human teaching. Complex behaviors can be easily generated from simple inputs. In the musculoskeletal humanoid MusashiOLegs, forward, backward, and rotational movements of seated walk are realized., Comment: accepted at IROS2022

Musculoskeletal humanoids have been developed by imitating humans and expected to perform natural and dynamic motions as well as humans. To achieve desired motions stably in current musculoskeletal humanoids is not easy because they cannot maintain the sufficient moment arm of muscles in various postures. In this research, we discuss planar structures that spread across joint structures such as ligament and planar muscles and the application of planar interskeletal structures to humanoid robots. Next, we develop MusashiOLegs, a musculoskeletal legs which has planar interskeletal structures and conducts several experiments to verify the importance of planar interskeletal structures., Comment: accepted at Humanoids2020

Human can not only support their body during standing or walking, but also support them by hand, so that they can dangle a bar and others. But most humanoid robots support their body only in the foot and they use their hand just to manipulate objects because their hands are too weak to support their body. Strong hands are supposed to enable humanoid robots to act in much broader scene. Therefore, we developed new life-size five-fingered hand that can support the body of life-size humanoid robot. It is tendon-driven and underactuated hand and actuators in forearms produce large gripping force. This hand has flexible joints using machined springs, which can be designed integrally with the attachment. Thus, it has both structural strength and impact resistance in spite of small size. As other characteristics, this hand has force sensors to measure external force and the fingers can be flexed along objects though the number of actuators to flex fingers is less than that of fingers. We installed the developed hand on musculoskeletal humanoid "Kengoro" and achieved two self-weight supporting motions: push-up motion and dangling motion., Comment: accepted at IROS2017

Human hands can not only grasp objects of various shape and size and manipulate them in hands but also exert such a large gripping force that they can support the body in the situations such as dangling a bar and climbing a ladder. On the other hand, it is difficult for most robot hands to manage both. Therefore in this paper we developed the hand which can grasp various objects and exert large gripping force. To develop such hand, we focused on the thumb CM joint with wide range of motion and the MP joints of four fingers with the DOF of abduction and adduction. Based on the hand with large gripping force and flexibility using machined spring, we applied above mentioned joint mechanism to the hand. The thumb CM joint has wide range of motion because of the combination of three machined springs and MP joints of four fingers have variable rigidity mechanism instead of driving each joint independently in order to move joint in limited space and by limited actuators. Using the developed hand, we achieved the grasping of various objects, supporting a large load and several motions with an arm., Comment: accepted at IROS2018

Various musculoskeletal humanoids have been developed so far. While these humanoids have the advantage of their flexible and redundant bodies that mimic the human body, they are still far from being applied to real-world tasks. One of the reasons for this is the difficulty of bipedal walking in a flexible body. Thus, we developed a musculoskeletal wheeled robot, Musashi-W, by combining a wheeled base and musculoskeletal upper limbs for real-world applications. Also, we constructed its software system by combining static and dynamic body schema learning, reflex control, and visual recognition. We show that the hardware and software of Musashi-W can make the most of the advantages of the musculoskeletal upper limbs, through several tasks of cleaning by human teaching, carrying a heavy object considering muscle addition, and setting a table through dynamic cloth manipulation with variable stiffness., Comment: Accepted at Humanoids2022

We have developed a parallel wire-driven monopedal robot, RAMIEL, which has both speed and power due to the parallel wire mechanism and a long acceleration distance. RAMIEL is capable of jumping high and continuously, and so has high performance in traveling. On the other hand, one of the drawbacks of a minimal parallel wire-driven robot without joint encoders is that the current joint velocities estimated from the wire lengths oscillate due to the elongation of the wires, making the values unreliable. Therefore, despite its high performance, the control of the robot is unstable, and in 10 out of 16 jumps, the robot could only jump up to two times continuously. In this study, we propose a method to realize a continuous jumping motion by reinforcement learning in simulation, and its application to the actual robot. Because the joint velocities oscillate with the elongation of the wires, they are not used directly, but instead are inferred from the time series of joint angles. At the same time, noise that imitates the vibration caused by the elongation of the wires is added for transfer to the actual robot. The results show that the system can be applied to the actual robot RAMIEL as well as to the stable continuous jumping motion in simulation., Comment: Accepted at Humanoids2022

Wiping behavior is a task of tracing the surface of an object while feeling the force with the palm of the hand. It is necessary to adjust the force and posture appropriately considering the various contact conditions felt by the hand. Several studies have been conducted on the wiping motion, however, these studies have only dealt with a single surface material, and have only considered the application of the amount of appropriate force, lacking intelligent movements to ensure that the force is applied either evenly to the entire surface or to a certain area. Depending on the surface material, the hand posture and pressing force should be varied appropriately, and this is highly dependent on the definition of the task. Also, most of the movements are executed by high-rigidity robots that are easy to model, and few movements are executed by robots that are low-rigidity but therefore have a small risk of damage due to excessive contact. So, in this study, we develop a method of motion generation based on the learned prediction of contact force during the wiping motion of a low-rigidity robot. We show that MyCobot, which is made of low-rigidity resin, can appropriately perform wiping behaviors on a plane with multiple surface materials based on various task definitions., Comment: Accepted at Humanoids2022

The state recognition of the environment and objects by robots is generally based on the judgement of the current state as a classification problem. On the other hand, state changes of food in cooking happen continuously and need to be captured not only at a certain time point but also continuously over time. In addition, the state changes of food are complex and cannot be easily described by manual programming. Therefore, we propose a method to recognize the continuous state changes of food for cooking robots through the spoken language using pre-trained large-scale vision-language models. By using models that can compute the similarity between images and texts continuously over time, we can capture the state changes of food while cooking. We also show that by adjusting the weighting of each text prompt based on fitting the similarity changes to a sigmoid function and then performing black-box optimization, more accurate and robust continuous state recognition can be achieved. We demonstrate the effectiveness and limitations of this method by performing the recognition of water boiling, butter melting, egg cooking, and onion stir-frying., Comment: accepted at IEEE Robotics and Automation Letters (RA-L), website - https://haraduka.github.io/continuous-state-recognition/

In recent years, multiple types of locomotion methods for robots have been developed and enabled to adapt to multiple domains. In particular, aerial robots are useful for exploration in several situations, taking advantage of its three-dimensional mobility. Moreover, some aerial robots have achieved manipulation tasks in the air. However, energy consumption for flight is large and thus locomotion ability on the ground is also necessary for aerial robots to do tasks for long time. Therefore, in this work, we aim to develop deformable multirotor robot capable of rolling movement with its entire body and achieve motions on the ground and in the air. In this paper, we first describe the design methodology of a deformable multilinked air-ground hybrid multirotor. We also introduce its mechanical design and rotor configuration based on control stability. Then, thrust control method for locomotion in air and ground domains is described. Finally, we show the implemented prototype of the proposed robot and evaluate through experiments in air and terrestrial domains. To the best of our knowledge, this is the first time to achieve the rolling locomotion by multilink structured mutltrotor., Comment: 8 pages, 15 figures

Improving the safety of collaborative manipulators necessitates the reduction of inertia in the moving part. Within this paper, we introduce a novel approach in the form of a passive 3D wire aligner, serving as a lightweight and low-friction power transmission mechanism, thus achieving the desired low inertia in the manipulator's operation. Through the utilization of this innovation, the consolidation of hefty actuators onto the root link becomes feasible, consequently enabling a supple drive characterized by minimal friction. To demonstrate the efficacy of this device, we fabricate an ultralight 7 degrees of freedom (DoF) manipulator named SAQIEL, boasting a mere 1.5 kg weight for its moving components. Notably, to mitigate friction within SAQIEL's actuation system, we employ a distinctive mechanism that directly winds wires using motors, obviating the need for traditional gear or belt-based speed reduction mechanisms. Through a series of empirical trials, we substantiate that SAQIEL adeptly strikes balance between lightweight design, substantial payload capacity, elevated velocity, precision, and adaptability., Comment: accepted at IEEE Robotics and Automation Letters (RA-L), website -https://tenrobo18.github.io/saqiel-ral2023-webpage/

One of the most important features of tendon-driven robots is the ease of wire arrangement and the degree of freedom it affords, enabling the construction of a body that satisfies the desired characteristics by modifying the wire arrangement. Various wire arrangement optimization methods have been proposed, but they have simplified the configuration by assuming that the moment arm of wires to joints are constant, or by disregarding wire arrangements that span multiple joints and include relay points. In this study, we formulate a more flexible wire arrangement optimization problem in which each wire is represented by a start point, multiple relay points, and an end point, and achieve the desired physical performance based on black-box optimization. We consider a multi-objective optimization which simultaneously takes into account both the feasible operational force space and velocity space, and discuss the optimization results obtained from various configurations., Comment: accepted at IEEE Robotics and Automation Letters (RA-L), website - https://haraduka.github.io/muscle-arrange-optimization/

In this study, we develop a simple daily assistive robot that controls its own vision according to linguistic instructions. The robot performs several daily tasks such as recording a user's face, hands, or screen, and remotely capturing images of desired locations. To construct such a robot, we combine a pre-trained large-scale vision-language model with a low-cost low-rigidity robot arm. The correlation between the robot's physical and visual information is learned probabilistically using a neural network, and changes in the probability distribution based on changes in time and environment are considered by parametric bias, which is a learnable network input variable. We demonstrate the effectiveness of this learning method by open-vocabulary view control experiments with an actual robot arm, MyCobot., Comment: accepted at Humanoids2023

Legged robots with high locomotive performance have been extensively studied, and various leg structures have been proposed. Especially, a leg structure that can achieve both continuous and high jumps is advantageous for moving around in a three-dimensional environment. In this study, we propose a parallel wire-driven leg structure, which has one DoF of linear motion and two DoFs of rotation and is controlled by six wires, as a structure that can achieve both continuous jumping and high jumping. The proposed structure can simultaneously achieve high controllability on each DoF, long acceleration distance and high power required for jumping. In order to verify the jumping performance of the parallel wire-driven leg structure, we have developed a parallel wire-driven monopedal robot, RAMIEL. RAMIEL is equipped with quasi-direct drive, high power wire winding mechanisms and a lightweight leg, and can achieve a maximum jumping height of 1.6 m and a maximum of seven continuous jumps., Comment: accepted at IROS2022 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, website - https://tenrobo18.github.io/ramiel-iros2022/

Recognition of the current state is indispensable for the operation of a robot. There are various states to be recognized, such as whether an elevator door is open or closed, whether an object has been grasped correctly, and whether the TV is turned on or off. Until now, these states have been recognized by programmatically describing the state of a point cloud or raw image, by annotating and learning images, by using special sensors, etc. In contrast to these methods, we apply Visual Question Answering (VQA) from a Pre-Trained Vision-Language Model (PTVLM) trained on a large-scale dataset, to such binary state recognition. This idea allows us to intuitively describe state recognition in language without any re-training, thereby improving the recognition ability of robots in a simple and general way. We summarize various techniques in questioning methods and image processing, and clarify their properties through experiments.

It is important for daily life support robots to detect changes in their environment and perform tasks. In the field of anomaly detection in computer vision, probabilistic and deep learning methods have been used to calculate the image distance. These methods calculate distances by focusing on image pixels. In contrast, this study aims to detect semantic changes in the daily life environment using the current development of large-scale vision-language models. Using its Visual Question Answering (VQA) model, we propose a method to detect semantic changes by applying multiple questions to a reference image and a current image and obtaining answers in the form of sentences. Unlike deep learning-based methods in anomaly detection, this method does not require any training or fine-tuning, is not affected by noise, and is sensitive to semantic state changes in the real world. In our experiments, we demonstrated the effectiveness of this method by applying it to a patrol task in a real-life environment using a mobile robot, Fetch Mobile Manipulator. In the future, it may be possible to add explanatory power to changes in the daily life environment through spoken language., Comment: Accepted to 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2023)

Imitation learning has been actively studied in recent years. In particular, skill acquisition by a robot with a fixed body, whose root link position and posture and camera angle of view do not change, has been realized in many cases. On the other hand, imitation of the behavior of robots with floating links, such as humanoid robots, is still a difficult task. In this study, we develop an imitation learning system using a biped robot with a floating link. There are two main problems in developing such a system. The first is a teleoperation device for humanoids, and the second is a control system that can withstand heavy workloads and long-term data collection. For the first point, we use the whole body control device TABLIS. It can control not only the arms but also the legs and can perform bilateral control with the robot. By connecting this TABLIS with the high-power humanoid robot JAXON, we construct a control system for imitation learning. For the second point, we will build a system that can collect long-term data based on posture optimization, and can simultaneously move the robot's limbs. We combine high-cycle posture generation with posture optimization methods, including whole-body joint torque minimization and contact force optimization. We designed an integrated system with the above two features to achieve various tasks through imitation learning. Finally, we demonstrate the effectiveness of this system by experiments of manipulating flexible fabrics such that not only the hands but also the head and waist move simultaneously, manipulating objects using legs characteristic of humanoids, and lifting heavy objects that require large forces., Comment: accepted at IROS2023

The range of robot activities is expanding from industries with fixed environments to diverse and changing environments, such as nursing care support and daily life support. In particular, autonomous construction of robots that are personalized for each user and task is required. Therefore, we develop an actuator module that can be reconfigured to various link configurations, can carry heavy objects using a locking mechanism, and can be easily operated by human teaching using a releasing mechanism. Given multiple target coordinates, a modular robot configuration that satisfies these coordinates and minimizes the required torque is automatically generated by Tree-structured Parzen Estimator (TPE), a type of black-box optimization. Based on the obtained results, we show that the robot can be reconfigured to perform various functions such as moving monitors and lights, serving food, and so on., Comment: Accepted at IROS2023, website - https://haraduka.github.io/auto-modular-design/

Transferring human motion skills to humanoid robots remains a significant challenge. In this study, we introduce a Wasserstein adversarial imitation learning system, allowing humanoid robots to replicate natural whole-body locomotion patterns and execute seamless transitions by mimicking human motions. First, we present a unified primitive-skeleton motion retargeting to mitigate morphological differences between arbitrary human demonstrators and humanoid robots. An adversarial critic component is integrated with Reinforcement Learning (RL) to guide the control policy to produce behaviors aligned with the data distribution of mixed reference motions. Additionally, we employ a specific Integral Probabilistic Metric (IPM), namely the Wasserstein-1 distance with a novel soft boundary constraint to stabilize the training process and prevent mode collapse. Our system is evaluated on a full-sized humanoid JAXON in the simulator. The resulting control policy demonstrates a wide range of locomotion patterns, including standing, push-recovery, squat walking, human-like straight-leg walking, and dynamic running. Notably, even in the absence of transition motions in the demonstration dataset, robots showcase an emerging ability to transit naturally between distinct locomotion patterns as desired speed changes.

In this study, we examined scene selection methods and emotion-based descriptions for a robot's daily diary. We proposed a scene selection method and an emotion description method that take into account semantic and affective information, and created several types of diaries. Experiments were conducted to examine the change in sentiment values and preference of each diary, and it was found that the robot's feelings and impressions changed more from date to date when scenes were selected using the affective captions. Furthermore, we found that the robot's emotion generally improves the preference of the robot's diary regardless of the scene it describes. However, presenting negative or mixed emotions at once may decrease the preference of the diary or reduce the robot's robot-likeness, and thus the method of presenting emotions still needs further investigation., Comment: 6 pages, 5 figures, ROMAN 2023

In this study, we propose an automatic diary generation system that uses information from past joint experiences with the aim of increasing the favorability for robots through shared experiences between humans and robots. For the verbalization of the robot's memory, the system applies a large-scale language model, which is a rapidly developing field. Since this model does not have memories of experiences, it generates a diary by receiving information from joint experiences. As an experiment, a robot and a human went for a walk and generated a diary with interaction and dialogue history. The proposed diary achieved high scores in comfort and performance in the evaluation of the robot's impression. In the survey of diaries giving more favorable impressions, diaries with information on joint experiences were selected higher than diaries without such information, because diaries with information on joint experiences showed more cooperation between the robot and the human and more intimacy from the robot., Comment: 12 pages, 5 figures, IAS-18

Cooking tasks are characterized by large changes in the state of the food, which is one of the major challenges in robot execution of cooking tasks. In particular, cooking using a stove to apply heat to the foodstuff causes many special state changes that are not seen in other tasks, making it difficult to design a recognizer. In this study, we propose a unified method for recognizing changes in the cooking state of robots by using the vision-language model that can discriminate open-vocabulary objects in a time-series manner. We collected data on four typical state changes in cooking using a real robot and confirmed the effectiveness of the proposed method. We also compared the conditions and discussed the types of natural language prompts and the image regions that are suitable for recognizing the state changes., Comment: Accepted at IAS18-2023

This paper describes a strategy for implementing a robotic system capable of performing General Purpose Service Robot (GPSR) tasks in robocup@home. The GPSR task is that a real robot hears a variety of commands in spoken language and executes a task in a daily life environment. To achieve the task, we integrate foundation models based inference system and a state machine task executable. The foundation models plan the task and detect objects with open vocabulary, and a state machine task executable manages each robot's actions. This system works stable, and we took first place in the RoboCup@home Japan Open 2022's GPSR with 130 points, more than 85 points ahead of the other teams., Comment: In review

In this paper, we propose a method for online estimation of the robot's posture. Our method uses von Mises and Bingham distributions as probability distributions of joint angles and 3D orientation, which are used in directional statistics. We constructed a particle filter using these distributions and configured a system to estimate the robot's posture from various sensor information (e.g., joint encoders, IMU sensors, and cameras). Furthermore, unlike tangent space approximations, these distributions can handle global features and represent sensor characteristics as observation noises. As an application, we show that the yaw drift of a 6-axis IMU sensor can be represented probabilistically to prevent adverse effects on attitude estimation. For the estimation, we used an approximate model that assumes the actual robot posture can be reproduced by correcting the joint angles of a rigid body model. In the experiment part, we tested the estimator's effectiveness by examining that the joint angles generated with the approximate model can be estimated using the link pose of the same model. We then applied the estimator to the actual robot and confirmed that the gripper position could be estimated, thereby verifying the validity of the approximate model in our situation., Comment: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

Humanoids are versatile robotic platforms owing to their limbs with multiple degrees of freedom. Although humanoids can walk like humans, they are relatively slow, and cannot run over large barriers. To address these limitations, we aim to achieve rapid terrestrial locomotion ability and simultaneously expand the locomotion domain to the air by utilizing thrust for propulsion. In this paper, we first describe an optimized construction method for a humanoid robot equipped with wheels and a flight unit to achieve these abilities. Then, we describe the integrated control framework of the proposed flying humanoid for each locomotion mode: aerial, legged, and wheeled locomotion. Finally, we achieved multimodal locomotion and aerial manipulation experiments using the proposed robot platform. To the best of our knowledge, this is the first time that a single humanoid has simultaneously achieved three different types of locomotion, including flight., Comment: 8 pages, 17 figures. Accepted by IEEE Robotics and Automation Letters

State recognition of objects and environment in robots has been conducted in various ways. In most cases, this is executed by processing point clouds, learning images with annotations, and using specialized sensors. In contrast, in this study, we propose a state recognition method that applies Visual Question Answering (VQA) in a Pre-Trained Vision-Language Model (PTVLM) trained from a large-scale dataset. By using VQA, it is possible to intuitively describe robotic state recognition in the spoken language. On the other hand, there are various possible ways to ask about the same event, and the performance of state recognition differs depending on the question. Therefore, in order to improve the performance of state recognition using VQA, we search for an appropriate combination of questions using a genetic algorithm. We show that our system can recognize not only the open/closed of a refrigerator door and the on/off of a display, but also the open/closed of a transparent door and the state of water, which have been difficult to recognize., Comment: Accepted at ICRA2023

In recent years, a number of models that learn the relations between vision and language from large datasets have been released. These models perform a variety of tasks, such as answering questions about images, retrieving sentences that best correspond to images, and finding regions in images that correspond to phrases. Although there are some examples, the connection between these pre-trained vision-language models and robotics is still weak. If they are directly connected to robot motions, they lose their versatility due to the embodiment of the robot and the difficulty of data collection, and become inapplicable to a wide range of bodies and situations. Therefore, in this study, we categorize and summarize the methods to utilize the pre-trained vision-language models flexibly and easily in a way that the robot can understand, without directly connecting them to robot motions. We discuss how to use these models for robot motion selection and motion planning without re-training the models. We consider five types of methods to extract information understandable for robots, and show the results of state recognition, object recognition, affordance recognition, relation recognition, and anomaly detection based on the combination of these five methods. We expect that this study will add flexibility and ease-of-use, as well as new applications, to the recognition behavior of existing robots., Comment: Accepted at Humanoids2023