Gesture Recognition & Synthesis

Gesture recognition and synthesis in the ISUE Lab has focused on applications and the development of accurate, efficient, and practical methods for translating human hand and body movements into meaningful commands for interactive systems. More recently we explored deep learning approaches as well as synthetic data generation for training robust classifiers.

Deep Learning for Gesture Recognition and Synthesis

  • Architecture of DeepGAN and DeepNAG (generator only) for non-adversarial gesture synthesis.
  • Left: Original samples from $1 dataset; Right: Samples synthesized with AVC, which maximize classification accuracy.

DeepGRU is a deep recurrent neural network with attention mechanisms designed for variable-length gesture recognition. It has been shown to be effective across a variety of datasets and modalities, including hand skeleton data. uDeepGRU extends the architecture to support real-time segmentation and classification of continuous gesture input. We have also explored adversarial and non-adversarial approahes to full-body gesture synthesis in DeepNAG, as well as data augmentation techniques for improving the performance of stroke gesture recognition with RNNs.

Publications

  • Veldhuijzen, B., Veltkamp, R., Ikne, O., Allaert, B., Wannous, H., Emporio, M., Giachetti, A., LaViola, J., He, R., Benhabiles, H., Cabani, A., Fleury, A., Hammoudi, K., Gavalas, K., Vlachos, C., Papanikolaou, A., Romanelis, I., Fotis, V., Arvanitis, G., Moustakas, K., Hanik, M., Nava-Yazdani, E., and von Tycowicz, C. "SHREC 2024: Recognition of Dynamic Hand Motions Molding Clay", Computers and Graphics, Volume 123, Article 104012 (11 pages), November 2024.
  • Maslych, M., Taranta, E., Aldilati, M., and LaViola, J. "Effective 2D Stroke-based Gesture Augmentation for RNNs", Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23), Article 282, 13 pages, April 2023.
  • Caputo, A., Giachetti, A., Soso, S., Pintani, D., D'Eusanio, A., Pini, S., Borghi, G., Simoni, A., Vezzani, R., Cucchiara, R., Ranieri, A., Giannini, F., Lupinetti, K., Monti, M., Maghoumi, M., LaViola, J., Le, M., Nguyen, H., and Tran, M. "SHREC 2021: Skeleton-based Hand Gesture Recognition in the Wild", Computers and Graphics, Volume 99, 201-211, October 2021.
  • Maghoumi, M., Taranta, E., and LaViola, J. "DeepNAG: Deep Non-Adversarial Gesture Generation", Proceedings of the 26th International Conference on Intelligent User Interfaces (IUI '21), 213-223, April 2021.
  • Maghoumi, M. "Deep Recurrent Networks for Gesture Recognition and Synthesis", PhD Dissertation, University of Central Florida, Department of Computer Science, December 2020.
  • Maghoumi, M. and LaViola, J. "DeepGRU: Deep Gesture Recognition Utility", Proceedings of the Fourteenth International Symposium on Visual Computing 2019 (ISCV 2019), 14 pages, October 2019.

Efficient Gesture Recognition for Real-Time Applications

  • The Dollar General classifies custom gestures using Jackknife after Machete identifies candidate segments using rejection thresholds estimated by GPSR and VKM.

kNN-based segmentation and recognition approaches using dynamic/continuous time warping and euclidean distance or vector product are also effective and more computationally efficient, especially on resource-constrained platforms. Machete uses continuous time warping to efficiently segment high-activity motion sequences and applies Jackknife, a robust recognizer based on dynamic time warping, to classify the segmented gestures. To find an optimal threshold, Machete uses the Voight-Kampff Machine and stochastic resampling to form score distributions for both gesture and non-gesture input. This approach allows for accurate gesture recognition even with minimal training data and is well-suited for real-time applications and is described fully in Dr. Eugene Taranta's dissertation.

Publications

Our work also studies how people can use expressive hand, body, and stroke input as a practical control channel for immersive and interactive systems, including dynamic motion benchmarks and effectiveness of different types of metrics to evaluate the performance of gesture recognition systems.

Publications