Gesture recognition is an area of research and development in computer science and language technology concerned with the recognition and interpretation of human gestures. A subdiscipline of computer vision, it employs mathematical algorithms to interpret gestures.
Gesture recognition offers a path for computers to begin to better understand and interpret human body language, previously not possible through text or unenhanced graphical user interfaces (GUIs).
Gestures can originate from any bodily motion or state, but commonly originate from the face or hand. One area of the field is emotion recognition derived from facial expressions and hand gestures. Users can make simple gestures to control or interact with devices without physically touching them.
Many approaches have been made using cameras and computer vision algorithms to interpret sign language, however, the identification and recognition of posture, gait, proxemics, and human behaviors is also the subject of gesture recognition techniques.Matthias Rehm, Nikolaus Bee, Elisabeth André, Wave Like an Egyptian – Accelerometer Based Gesture Recognition for Culture Specific Interactions, British Computer Society, 2007
Overview
Gesture recognition has application in such areas as: Automobiles Consumer electronics Transit Gaming Handheld devices Defense Home automation Automated sign language translationChai, Xiujuan, et al. "Sign language recognition and translation with kinect ." IEEE Conf. on AFGR. Vol. 655. 2013.
Gesture recognition can be conducted with techniques from computer vision and image processing.Sultana A, Rajapuspha T (2012), "Vision Based Gesture Recognition for Alphabetical Hand Gestures Using the SVM Classifier", International Journal of Computer Science & Engineering Technology (IJCSET)., 2012
The literature includes ongoing work in the computer vision field on capturing gestures or more general human pose and movements by cameras connected to a computer.Pavlovic, V., Sharma, R. & Huang, T. (1997), "Visual interpretation of hand gestures for human-computer interaction: A review", IEEE Transactions on Pattern Analysis and Machine Intelligence, July, 1997. Vol. 19(7), pp. 677 -695.R. Cipolla and A. Pentland, Computer Vision for Human-Machine Interaction, Cambridge University Press, 1998, Ying Wu and Thomas S. Huang, "Vision-Based Gesture Recognition: A Review" , In: Gesture-Based Communication in Human-Computer Interaction, Volume 1739 of Springer Lecture Notes in Computer Science, pages 103-115, 1999, , Alejandro Jaimes and Nicu Sebe, Multimodal human–computer interaction: A survey , Computer Vision and Image Understanding Volume 108, Issues 1-2, October–November 2007, Pages 116-134 Special Issue on Vision for Human-Computer Interaction,
The term "gesture recognition" has been used to refer more narrowly to non-text-input handwriting symbols, such as inking on a graphics tablet, multi-touch gestures, and mouse gesture recognition. This is computer interaction through the drawing of symbols with a pointing device cursor.Dopertchouk, Oleg; "Recognition of Handwriting Gestures", gamedev.net, January 9, 2004Chen, Shijie; "Gesture Recognition Techniques in Handwriting Recognition Application", Frontiers in Handwriting Recognition p 142-147 November 2010Balaji, R; Deepu, V; Madhvanath, Sriganesh; Prabhakaran, Jayasree "Handwritten Gesture Recognition for Gesture Keyboard" , Hewlett-Packard Laboratories Pen computing expands digital gesture recognition beyond traditional input devices such as keyboards and mice, and reduces the hardware impact of a system.
Gesture types In computer interfaces, two types of gestures are distinguished:Dietrich Kammer, Mandy Keck, Georg Freitag, Markus Wacker, Taxonomy and Overview of Multi-touch Frameworks: Architecture, Scope, and Features We consider online gestures, which can also be regarded as direct manipulations like scaling and rotating, and in contrast, offline gestures are usually processed after the interaction is finished; e. g. a circle is drawn to activate a context menu. Offline gestures: Those gestures that are processed after the user's interaction with the object. An example is a gesture to activate a menu. Online gestures: Direct manipulation gestures. They are used to scale or rotate a tangible object.
Touchless interface A touchless user interface (TUI) is an emerging type of technology wherein a device is controlled via body motion and gestures without touching a keyboard, mouse, or screen.
Types of touchless technology There are several devices utilizing this type of interface such as smartphones, laptops, games, TVs, and music equipment.
One type of touchless interface uses the Bluetooth connectivity of a smartphone to activate a company's visitor management system. This eliminates having to touch an interface, for convenience or to avoid a potential source of contamination as during the COVID-19 pandemic.
Input devices The ability to track a person's movements and determine what gestures they may be performing can be achieved through various tools. Kinetic user interfaces (KUIs) are an emerging type of user interfaces that allow users to interact with computing devices through the motion of objects and bodies. Examples of KUIs include tangible user interfaces and motion-aware games such as Wii and Microsoft's Kinect, and other interactive projects.
Although there is a large amount of research done in image/video-based gesture recognition, there is some variation in the tools and environments used between implementations. Wired gloves. These can provide input to the computer about the position and rotation of the hands using magnetic or inertial tracking devices. Furthermore, some gloves can detect finger bending with a high degree of accuracy (5-10 degrees), or even provide haptic feedback to the user, which is a simulation of the sense of touch. The first commercially available hand-tracking glove-type device was the DataGlove,Thomas G. Zimmerman, Jaron Lanier, Chuck Blanchard, Steve Bryson, and Young Harvill. http://portal.acm.org. "A HAND GESTURE INTERFACE DEVICE ." http://portal.acm.org. a glove-type device that could detect hand position, movement and finger bending. This uses fiber optic cables running down the back of the hand. Light pulses are created and when the fingers are bent, light leaks through small cracks and the loss is registered, giving an approximation of the hand pose. Depth-aware cameras. Using specialized cameras such as structured light or time-of-flight cameras, one can generate a depth map of what is being seen through the camera at a short-range, and use this data to approximate a 3D representation of what is being seen. These can be effective for the detection of hand gestures due to their short-range capabilities.Yang Liu, Yunde Jia, A Robust Hand Tracking and Gesture Recognition Method for Wearable Visual Interfaces and Its Applications, Proceedings of the Third International Conference on Image and Graphics (ICIG'04), 2004 Stereo cameras. Using two cameras whose relations to one another are known, a 3D representation can be approximated by the output of the cameras. To get the cameras' relations, one can use a positioning reference such as a lexian-stripe or infrared emitter.Kue-Bum Lee, Jung-Hyun Kim, Kwang-Seok Hong, An Implementation of Multi-Modal Game Interface Based on PDAs, Fifth International Conference on Software Engineering Research, Management and Applications, 2007 In combination with direct motion measurement (6D-Vision) gestures can directly be detected. Gesture-based controllers. These controllers act as an extension of the body so that when gestures are performed, some of their motion can be conveniently captured by the software. An example of emerging gesture-based motion capture is skeletal hand tracking, which is being developed for augmented reality and virtual reality applications. An example of this technology is shown by tracking companies uSens and Gestigon, which allow users to interact with their surroundings without controllers. Wi-Fi sensing Mouse gesture tracking, where the motion of the mouse is correlated to a symbol being drawn by a person's hand which can study changes in acceleration over time to represent gestures.Per Malmestig, Sofie Sundberg, SignWiiver – implementation of sign language technology Thomas Schlomer, Benjamin Poppinga, Niels Henze, Susanne Boll, Gesture Recognition with a Wii Controller , Proceedings of the 2nd international Conference on Tangible and Embedded interaction, 2008AiLive Inc., LiveMove White Paper , 2006 The software also compensates for human tremor and inadvertent movement.Electronic Design September 8, 2011. William Wong. Natural User Interface Employs Sensor Integration.Cable & Satellite International September/October, 2011. Stephen Cousins. A view to a thrill. TechJournal South January 7, 2008. The sensors of these smart light-emitting cubes can be used to sense hands and fingers as well as other objects nearby, and can be used to process data. Most applications are in music and sound synthesis,Percussa AudioCubes Blog October 4, 2012. Gestural Control in Sound Synthesis. but can be applied to other fields. Single camera. A standard 2D camera can be used for gesture recognition where the resources/environment would not be convenient for other forms of image-based recognition. Earlier it was thought that a single camera may not be as effective as stereo or depth-aware cameras, but some companies are challenging this theory. Software-based gesture recognition technology using a standard 2D camera that can detect robust hand gestures.
Algorithms
Depending on the type of input data, the approach for interpreting a gesture could be done in different ways. However, most of the techniques rely on key pointers represented in a 3D coordinate system. Based on the relative motion of these, the gesture can be detected with high accuracy, depending on the quality of the input and the algorithm's approach.
In order to interpret movements of the body, one has to classify them according to common properties and the message the movements may express. For example, in sign language, each gesture represents a word or phrase.
Some literature differentiates 2 different approaches in gesture recognition: a 3D model-based and an appearance-based.Vladimir I. Pavlovic, Rajeev Sharma, Thomas S. Huang, Visual Interpretation of Hand Gestures for Human-Computer Interaction; A Review, IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997 The foremost method makes use of 3D information on key elements of the body parts in order to obtain several important parameters, like palm position or joint angles. Approaches derived from it such as the volumetric models have proven to be very intensive in terms of computational power and require further technological developments in order to be implemented for real-time analysis. Alternately, appearance-based systems use images or videos for direct interpretation. Such models are easier to process, but usually lack the generality required for human-computer interaction.
3D model-based algorithms
The 3D model approach can use volumetric or skeletal models or even a combination of the two. Volumetric approaches have been heavily used in the computer animation industry and for computer vision purposes. The models are generally created from complicated 3D surfaces, like NURBS or polygon meshes.