Typealike postures are postures formed using the left or right hand, an open or closed hand form, and different wrist orientations, all further distinguished by hand position when on, just beside, or just below the keyboard.

Different postures can trigger different interactions, for example: (a) an open left hand at 90° wrist rotation beside the keyboard can decrease the volume; (b) an open left hand at 0° on the keyboard can open a screen capture snipping tool; (c) a closed right hand 0° posture below the keyboard could turn the car in a racing game; (d) an open right hand at 0° on the keyboard could advance a document page; and (e) a right open 90° hand posture beside the keyboard could increase the volume.

Quick Facts

Overall, our work contributes:

A set of near-keyboard hand postures that are acceptable to users, recognized reliably, and fast to perform.
A dataset of nearly 350K posture images captured under different lighting conditions and two keyboard backgrounds

We evaluate our approach by answering three questions: Is this style of postures preferable by users? Can they be recognized reliably? Are they quick and easy to perform?

Abstract

We propose a style of hand postures to trigger commands on a laptop. The key idea is to perform hand-postures while keeping the hands on, beside, or below the keyboard, to align with natural laptop usage. 36 hand-posture variations are explored considering three resting locations, left or right hand, open or closed hand, and three wrist rotation angles. A 30-participant formative study measures posture preferences and generates a dataset of nearly 350K images under different lighting conditions and backgrounds. A deep learning recognizer achieves over 97% accuracy when classifying all 36 postures with 2 additional non-posture classes for typing and non-typing. A second experiment with 20 participants validates the recognizer under real-time usage and compares posture invocation time with keyboard shortcuts. Results find low error rates and fast formation time, indicating postures are close to current typing and pointing postures. Finally, practical use case demonstrations are presented, and further extensions discussed.

Design Considerations for the Interaction Vocabulary

Our posture set was chosen to support our goal of a discrete command invoking method that aligns with natural laptop usage. The postures are variations of hand posture styles, all based on three design considerations:

Recognition without a key press:
Postures should not conflict with touch-typing hand motor behaviour when pressing keys and “disentangle” command execution from key-press events
Performed on or near the keyboard surface:
Postures should be compatible with maintaining a near-resting arm posture where the wrist is in constant contact with the laptop case.
Simple and easy to perform:
Postures should be naturally comfortable to perform for the user but still distinguishable for detection in order to be practical.

Technique

With these design considerations in mind, we created a style of postures that use different hand forms, kinematics of wrist rotation, and distinct locations on or near the keyboard that expands the input interaction space around the laptop.

This gives a design space of 36 postures that can be fully defined by variations along four parameters:

Form (open, closed)
Orientation (0°, 90°, 180°)
Surface (on, beside, below)
Hand (dominant, non-dominant)

Finally, all postures are characterized with a slightly extended thumb to appear unique for recognition without a keypress.

Experiments

We conducted 2 experiments to validate the technique. Both the experiments contained a game-like task representing one of four types of interaction: typing, clicking, shortcuts, and postures. During the experiments, participants played a game where they typed a word, clicked a button, pressed a shortcut key, or formed a certain posture.

Experiment I: User Preference & Data Gathering

The first experiment was performed to identify most popular postures and gather training data to create a recognizer. We recruited 30 participants, 19 to 31 years old who had extensive experience with laptop computers with daily usage between 4 to 16 hours. Participants were instructed to complete the task as accurately as possible. Video of the hands and keyboard was recorded at 30 FPS in 1280×720 resolution using the built-in laptop camera and a mirror reflector Through this experiment, we measured user-preferences for the postures and gathered the images to construct dataset to build the recognizer. To collect more samples, we used online data augmentation by asking participants to follow hand movements described in the task during posture formation.

Recognition using Deep Learning

We used Transfer Learning on ResNet-50 architecture with weights pre-trained on the ImageNet dataset.
Smith’s one-cycle policy was used to vary the learning rate and achieve faster convergence.
Monte Carlo cross-validation with 10 random splits was used for model evaluation. Confusion matrices obtained from each run were aggregate to obtain a final confusion matrix as shown in the figure.

Experiment II: Technique Evaluation

For the second experiment, we used a similar game as in the first experiment except this time, when participants formed the required posture, it was recognized and they immediately continued to the next trial. Participants were instructed to complete the task as accurately and as quickly as possible. Through this experiment, we measured the task completion time as well as the transition times between different interaction techniques.

What Do We Learn?

180° wrist rotations were significantly disliked for comfort and preference.
Below the keyboard was most preferred location for posture formation.

Open-hand postures were significantly faster than close-hands.
90° wrist rotation was significantly faster than 0° and 180° wrist rotations.
Area below the keyboard was significantly faster for posture formation compared to beside or on

Postures are significantly faster when coming from Typing or Clicking reference task
Shortcuts are faster when going back to Typing task.

Dataset

Some descriptive stats about the dataset

Kaggle Dataset

Publication

Nalin Chhibber, Hemant Bhaskar Surale, Fabrice Matulic, and Daniel Vogel. 2021. Typealike: Near-Keyboard Hand Postures for Expanded Laptop Interaction. Proc. ACM Human Computer Interaction. 5, ISS, Article 486 (November 2021), 20 pages. DOI:https://doi.org/10.1145/3486952

BibTeX

			
			@article{chhibber2021typealike,
			   title={Typealike: Near-Keyboard Hand Postures for Expanded Laptop Interaction},
			   author={Chhibber, Nalin and Surale, Hemant Bhaskar and Matulic, Fabrice and Vogel, Daniel},
			   journal={Proceedings of the ACM on Human-Computer Interaction},
			   volume={5},
			   number={ISS},
			   pages={1--20},
			   year={2021},
			   publisher={ACM New York, NY, USA}
			}