Aprendizagem profunda para reconhecimento de gestos da mão usando imagens e esqueletos com aplicações em Libras

Abstract

In this work we present methodologies based on Deep Learning for the recognition of static and dynamic gestures of the hand, with applications in signs of Libras (Brazilian Sign Language). Through data captured by the Leap Motion device, including both images and skeletons of the palm, we evaluated several architectures of Neural Networks to recognize gestures, with emphasis on signs of Libras. The methodologies can be described in three stages. In the first one, we sought to recognize static gestures (poses) using multilayer perceptron networks (MLP) for skeletal data, convolutional networks (CNN) for images, and multiple input networks using both types of information. In the second, we individually classify gestures that include motion, and for this we include recurrent Long Short-Term Memory (LSTM) layers in our architectures. To make the process even more precise, we apply Learning Transfer to the convolutional blocks, bringing the previously trained parameters with the static poses into the network designed for the dynamic gestures, and evaluate the result with and without the transfer. Finally, we present a new algorithm that allows us to recognize online the same dynamic gestures from the previous step, but executed sequentially, without pauses, and without having information about the beginning and end of the execution of each gesture.