Reconhecimento de gestos customizados da mão em tempo real usando aprendizado de métricas e grafos de ação

Abstract

The recent popularization of depth and motion sensors, human gestures recognition has become an alternative for human-computer interaction. In this Masters dissertation we focus on hand gesture recognition in real time using the Leap Motion sensor. We consider that a dynamic gesture can be represented by a small finite sequence of static poses, which are trained and recognized by a novel classifier based on metric learning called Minimal Mean Distances Classifier (MMD). To learn metrics in the pose space, we experimented the Large Margin Nearest Neighbor (LMNN) algorithm in both linear and non-linear variants. To recognize dynamic gestures, we present a classifier based on the Hidden Markov Model (HMM) and action graphs. Using this probabilistic approach, we show that dynamic hand gestures can be recognized in real-time and online (not requiring pauses or segmentation information to the user). Finally, we experimented the proposed method in four applications, highlighting the recognition of signals of the Brazilian Sign Language (LIBRAS). The presented results show the robustness of all variations of the proposed method.