Smartphone-Mediated Augmented Reality: Extended Fitts' Law and Immersive Interaction Metrics via Hand Tracking

Abstract

Augmented Reality (AR) is rapidly emerging as a revolutionary technology capable of transforming education and training by making highly interactive and immersive experiences possible. Efficient user interface (UI) design for AR, particularly on ubiquitous smartphone platforms, demands for greater insight into the manner in which users interact with virtual content in real-world settings. This study addresses inherent shortcomings in the application of Fitts's Law, a classic model of human-computer interaction, to explain and predict user performance in hand gesture interaction in smartphone-based AR, a context that presents unique challenges to dominant HCI principles. Specifically, current applications of Fitts's Law and its 3D extensions often fail to adequately address several factors critical to smartphone-based AR. They encompass the dynamic frame of reference movements inherent in hand-held AR interaction, where device and user hand are both in motion, the variable quality of tracking provided by the smartphone camera, impacting the reliability of gesture recognition, perceptual challenges inherent in accurate depth perception on non-stereoscopic screens, and the individual performance signatures introduced by different hand gesture paradigms, such as pinch and point gestures. These factors introduce complexities not fully explained by typical Fitts's Law models. To address these knowledge gaps and transcend the limitations of existing models, this study goes beyond typical Fitts's Law research in that it collects high-fidelity performance information like device movement, gesture confidence, occlusions, and depth errors in addition to typical movement time and target distance. Additionally, this study aims to contribute empirically cognizant guidance and theoretical advancement to help immersive field with respect to design for more effective and user-centric AR applications, particularly learning and training on an easily available platform like smartphones.