Educational Reform for Robots – Interactive Imitation Learning Instead of Programming

The supervisor demonstrates, the apprentice watches, tries, makes mistakes, gets corrected – and after a few repetitions, the movement is stored in the muscle memory. For centuries, master craftsmen and women have passed on their knowledge by showing apprentices how it’s done. The “learning by doing” principle has shaped entire generations. Today, the laboratory of the Institute for Material Handling and Logistics (IFL) at KIT has an “apprentice” that has no muscles and no experience. It is merely a high-tech robot system made up of sensors, motorized joints and algorithms. Yet, it is supposed to learn just like in a training workshop. “Our vision is that a robot can be trained in a few hours, and can easily and quickly adapt to any new task or changes in the dynamic manufacturing process,” says Professor Rania Rayyes, who leads the AI & Robotics research group at IFL.

The New School of Robotics

What sounds like a human-machine illusion is already becoming reality at KIT. In the HaptXDeep project, Rayyes and her team are developing an AI system for robots that no longer requires laborious programming of new skills, but where the skills can be learned intuitively and immediately by imitation – just like an apprentice. A teleoperated glove, controlled by humans, replaces the theory on a blackboard and a state-of-the-art robotic hand replaces the workbench. Doctoral researcher Edgar Welte demonstrates: he slides the HaptX glove over his hand and moves his fingers – the robot hand, equipped with tactile sensors, mirrors the motion. Grasp, rotate, swing, place. After only a few demonstrations, the gripping robot can act autonomously and complete the task. This newly developed system for adaptive robot automation is built on a learning process that is astonishingly close to the human one: learning by imitating and repeating. “If the robot makes mistakes, humans can intervene and correct them. These corrections are particularly valuable for learning,” emphasizes Welte.

Hands that Shape the Future

The controlling glove not only transfers the human hand´s movements to the robot, it also sends valuable touch information back with physical feedback. “When the robot hand encounters resistance, the human feels what the robot feels – a kind of ‘tactile dialogue’ between human and machine. We use all this information in the interaction to train our AI system so that the robot can evaluate the tactile feedback generated by the objects and adapt its grip accordingly,” explains researcher Welte. “An important research question to be answered: what is the most intuitive way for a person to teach a robot?” Rayyes said. To find out, the team runs user studies that compare different teaching interfaces – such as a haptic glove, camera-based hand tracking, and virtual reality setups. During these user studies, the participants are tasked with performing different manipulation tasks with the robot in a teleoperation setting, where the robot is fully controlled by the user, effectively showing the task to the robot. “The objective of this comparison is not only the speed or success rate of the task fulfillment. We also measure how quickly people learn the interface, and how safe and comfortable it feels,” Welte explains.

Industry under Reform Pressure

As with any reform, the starting point is not a machine, but a problem larger than technology: Industry is facing a growing shortage of skilled workers, the shift towards a circular economy and increasingly diverse products. “We need AI-enabled robots that seamlessly integrate flexibility and intelligence to support dynamic manufacturing processes without an expert having to reprogram everything each time, rather robots should be able to adapt quickly to the changes in the environments, the tasks or the products. 

New Skills for Model Students

Many robots in industrial automation are model students of an old school: they can execute a task perfectly as long as nothing changes. But in modern factories, everything is in flux. A used component returns to the line: scratches, deformations, different generations, unknown conditions. Every recycled product is unique. “Currently, most of the robots in industry are still preprogrammed in the traditional way. In circular productions like remanufacturing, we cannot reprogram the robot each time to adapt to every single change, and this is what we are currently researching within the SFB DFG project ‘Circular Factory’, a big collaborative project at KIT.” says Rayyes.

Why Humans Learn Fast – Robots Don’t (Yet)

A human does not need to know every screwdriver on the planet to understand how to hold and use it. Our brain is a master of generalization: a handful of examples are enough to infer the underlying rules. For AI this is still a major hurdle. Many systems require massive data sets before they grasp what is obvious to people: that blue is similar to green, that a used machine feels different from a new one, yet still demands the same task. Robotics now faces the challenge of shortening these long learning phases without compromising safety and reliability. “A new type of learning, as we are researching at HaptXDeep, could change that – and finally make scalable automation possible for smaller businesses,” Welte affirms.

From Human Hand to Robotic Gripper

Many complex manipulation tasks are demonstrated with highly dexterous robot hands, while factories often rely on simple, rugged parallel-jaw grippers. The team at KIT is working to close that gap. Recent work uses largescale vision models trained on internet video to learn to grasp a new object from a short RGB video of a human demonstration. By combining knowledge of how humans grasp objects and which parts of the hand are in contact with them, the system can propose realistic grasp poses for a parallel-jaw gripper. Building on this, future research will focus on methods that transfer manipulation skills from dexterous robot hands to industrial grippers – so that skills learned in the lab can be transferred to reliable factory hardware with minimal extra effort.

Ahead of the Curriculum

“We use uncertainty as a learning signal. This approach, known as uncertainty-aware learning, primarily helps robots to act more safely, robustly, and reliably because they always take their own uncertainty into account in addition to an action or perception,” Rayyes explains, “The AI systems we developed within the SFB project Circular Factory, show that the robot can adapt to new objects and changes in the task very quickly”. In situations of high uncertainty, robots can fail, slow down or stop. It is a closed-loop learning system in which the robot, in a sense, develops its own sense of body awareness. And perhaps one day it will be taken for granted that robots are no longer programmed but trained.

Further Links:

• Further Information
• Video

Images:

• Midjourney
• Daniel Kubus / KIT
• Amadeus Brahmsiepe / KIT