Assembly may seem simple in modern factories, but it is one of the most challenging manufacturing tasks. People complete assembly by combining hand eye coordination with years of practical experience. Robots, however, often struggle with the same work. Small alignment errors, contact forces during insertion, deformation of flexible parts, and changes in tool angle or force can all cause assembly failures.

 

How can robots move beyond simple pick and place tasks and learn the skills needed for complex assembly? A research team led by Professor Jie Hu and Associate Researcher Jin Qi from the Institute of Mechanical Design and Knowledge Engineering at School of Mechanical Engineering has developed the Human in the Loop Assembly Imitation Learning System, or HAIL. The system provides a new platform for studying complex robotic assembly in real industrial environments. Their work was published in the Journal of Manufacturing Systems under the title HAIL: Human in the Loop Assembly Imitation Learning System. Doctoral student Qianyou Zhao is the first author of the paper.

 

HAIL：Human-in-the-loop Assembly Imitation Learning System

 

Many previous studies on robot learning focused on simulation or simple pick and place tasks. Real industrial assembly is much more difficult. Robots need to know how to approach a part, align it accurately, apply force at the right time, and recover from failure when problems occur. These skills are difficult to obtain through predefined programs or simple task evaluations.

To address this challenge, the research team developed HAIL. The system brings human experience directly into robot learning. With human in the loop and mixed reality technologies, human operators can collect data, demonstrate assembly skills, and correct robot actions during training. Instead of relying on repeated trial and error, robots learn from experienced human guidance. This allows them to understand not only how to perform an action but also why the action works.

 

Visual illustration of the eight benchmark tasks and their meta action sequences

 

The team also created a realistic benchmark for robot assembly. It includes representative industrial tasks such as high precision assembly, tool based manipulation, constrained assembly and disassembly, and flexible object handling. These tasks reflect common manufacturing operations and capture the challenges of real assembly. They involve frequent contact, strong physical constraints, long operation sequences, and accumulated errors.

 

The researchers further divided complex assembly into several basic action units. These include grasping, placing, rotating, moving, and combined moving and rotating actions. This approach makes it easier to identify where a robot succeeds or fails during a task. It also helps researchers understand the physical reasons behind each failure.

 

For example, insertion tasks often fail because small errors accumulate during several stages. These stages include approaching the target, alignment, contact, and insertion. The final result may be jamming, misalignment, or wedging. HAIL therefore evaluates not only whether a robot completes a task but also why it succeeds or fails.

 

Failure Analysis of the Insertion Task

 

This evaluation method shifts robot assembly research beyond simple success rates. It provides a deeper understanding of robot capabilities and failure mechanisms. HAIL serves as both a research platform and a diagnostic tool. It helps researchers identify weak points in robot assembly, locate the most difficult action steps, and understand which physical constraints have the greatest impact on task performance. These insights support future improvements in learning algorithms and robotic systems.

 

The study connects the entire research process, from data collection and policy learning to realistic performance evaluation. It provides a reproducible platform for research on complex industrial assembly. As embodied intelligence and intelligent manufacturing continue to develop, systems such as HAIL are expected to support advanced applications in precision electronics assembly, human robot collaboration, flexible manufacturing, and complex equipment production. These systems will help robots move beyond executing programmed instructions toward understanding tasks, learning from experience, and adapting to changing environments.

 

This research was supported by the National Natural Science Foundation of China.

 

Paper Link：https://doi.org/10.1016/j.jmsy.2026.04.020