Fraunhofer IWU: Perfect human-machine cooperation with NeurOSmart

Robot technology that not only performs simple tasks, but also supports humans in all tasks, is one of the key technologies in industrial production. However, this requires robots to master complex motion sequences, execute commands and keep a safe distance from humans at all times. Fraunhofer researchers have now made this form of collaboration between humans and machines possible in the NeurOSmart lead project. 

The researchers are combining several technologies: a sensor system with a LIDAR laser that continuously monitors the workplace of humans and computers. AI-supported chips that evaluate the signals directly on the sensor system; neuromorphic chip technology that works like the human brain and is therefore very energy-efficient. 

In addition to the Fraunhofer Institute for Silicon Technology ISIT, the Fraunhofer Institute for Photonic Microsystems IPMS, the Fraunhofer Institute for Microelectronic Circuits IMS, the Fraunhofer Institute for Machine Tools and Forming Technology IWU and the Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS were also involved in the project. 

NeurOSmart was coordinated by Fraunhofer ISIT. “Together with the project partners, we have further developed the technologies and integrated them into an overall system. Cooperation with the machine poses no risk to humans,” says Prof. Shanshan Gu-Stoppel, Group Leader Optical Systems at Fraunhofer ISIT and Honorary Professor of Microtechnology at the West Coast University of Applied Sciences in Heide.

Sensor system with LIDAR laser

The sensor system monitors the area in which humans and robots move from a bird’s eye view. The researchers use a LIDAR laser (Light Detection and Ranging). This type of laser emits short pulses in the near-infrared range and uses the reflections for 3D distance measurement. Movable MEMS (Micro-Electro-Mechanical Systems) mirrors send the laser across the entire work surface and generate a high-resolution 3D image. The researchers have succeeded in increasing the performance and energy efficiency of the mirrors. “We use piezoelectric aluminum scandium nitride (AlScN) with a layer thickness of just 1 micrometer for the MEMS mirrors,” explains Gu-Stoppel. 

A key feature of NeurOSmart is the direct integration of data processing into the sensor system. First, the large amount of image data generated by the sensor’s wide field of view is pre-processed. AI-supported algorithms developed by Fraunhofer IMS bundle the incoming signals and recognize the areas in the scene that are of particular interest. This allows the sensor to be precisely aligned for the subsequent analyses, which saves power and reduces the data rate.

Neuromorphic computing

The next step, the actual evaluation of the data, which serves as the basis for controlling the robot, also takes place directly in the sensor system. Here, the researchers at Fraunhofer IPMS, led by Prof. Thomas Kämpfe, Head of Business Unit at the Center Nanoelectronic Technologies, are relying on the concept of neuromorphic computing. They have developed a special accelerator chip. The processor consists of many small computing units that are interconnected on a wafer in a matrix. Each chip acts as a “thinking cell” and makes its own decisions. The technology is based on the way the human brain works – hence the term neuromorphic computing. 

The highly specialized AI models required for this are being developed by Fraunhofer IAIS. It only takes a few milliseconds from receiving the signals and evaluating them to the mechanical reaction of the robot arm. This enables safe cooperation even with heavy-duty robots, which are slowed down or stopped by the AI if a human gets too close. Thanks to the simulation of the entire robot cell, Fraunhofer IAIS was able to simulate dangerous situations for training that cannot be reproduced in real life. “Neuromorphic computing is a major step towards a hardware architecture that not only processes resource-intensive AI applications quickly, but also very energy-efficiently,” says Thomas Kämpfe. Fraunhofer ISIT researcher Gu-Stoppel adds: “The technologies of NeurOSmart not only make industrial production more agile and efficient, they also make the factory floor a safe, creative and people-friendly place for all employees.”

Customized applications for industry

The components and technologies of NeurOSmart form a standardized technology platform. Industrial customers can build their own applications for their individual scenarios. The energy-efficient technologies of NeurOSmart with the resulting longer battery life could also be used in drones or sensor systems in agriculture in the future. 

Fraunhofer flagship project NeurOSmart

Project goal: Sensor-related data processing with high performance and energy efficiency
Applications: Robotic systems in industry, mobile systems
Project partners:

• Fraunhofer Institute for Silicon Technology ISIT – project coordination, development of piezo- and ferroelectric materials, MEMS mirrors 
• Fraunhofer Institute for Photonic Microsystems IPMS – High-Performance Chips (HPC), memory technologies 
• Fraunhofer Institute for Microelectronic Circuits IMS – AI-supported data preprocessing, system integration, sensor system with LIDAR laser 
• Fraunhofer Institute for Machine Tools and Forming Technology IWU – Evaluation of the novel sensor system in the application scenario human-robot collaboration 
• Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS – Development of highly efficient AI models for person recognition specialized for the deployment of neuromorphic hardware

– – – – –

Further links

👉 www.iwu.fraunhofer.de  

Photo: Fraunhofer ISIT