Specialists from the Don State Technical University, together with colleagues from MSTU "Stankin", have created domestic software for controlling robots using gestures, the press service of the Ministry of Education and Science of the Russian Federation reported. Now Russian industrial manipulators can recognize operator commands in the same way as foreign counterparts, but without the use of imported solutions.
The development is based on a combination of data from a regular camera and a depth sensor. The camera captures the image, and the sensor determines the distance to objects. Together, they compensate for each other's shortcomings (for example, in poor lighting or loss of depth), providing more accurate and reliable recognition. Based on the data obtained, scientists have developed a descriptor — a feature vector that fully describes the operator's movements.
The system is built on computer vision algorithms and neural network technologies. It recognizes gesture commands (the program is trained on ten gestures) and transmits data to the robot controller, where the corresponding actions of the manipulator are pre-programmed. The development allows automating loading, assembly operations, painting, laser processing, marking and other joint operations of humans and robots.