Development of an On-device XR Workspace System Using Real-time Hand Gesture Recognition With a Mobile Two-dimensional Camera

This paper presents a mobile on-device extended reality workspace system. Unlike conventional augmented reality systems, which offer limited interaction capabilities, the proposed system enables users to intuitively interact with virtual objects in an extended reality environment using hand gestures. The system uses the Unity engine for creating the virtual environment, integrates a deep learning model, and builds an Android application. The deep learning model tracks the user's hand from the video feed, estimates hand joint coordinates, and classifies hand gestures based on these coordinates. The recognized gestures are then converted into control signals for manipulating objects in the virtual environment. The model focuses on efficiency, allowing real-time processing in resource-limited, on-device environments and operates reliably on mobile devices with limited hardware. This study addresses the physical constraints of real-world workspaces by facilitating intuitive interactions with virtual environments, showcasing the practical application of artificial intelligence recognition models in practical scenarios.