Back to Search
Start Over
Voicify Your UI: Towards Android App Control with Voice Commands
- Source :
- Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7, no. 1 (2023): 1-22
- Publication Year :
- 2023
-
Abstract
- Nowadays, voice assistants help users complete tasks on the smartphone with voice commands, replacing traditional touchscreen interactions when such interactions are inhibited. However, the usability of those tools remains moderate due to the problems in understanding rich language variations in human commands, along with efficiency and comprehensibility issues. Therefore, we introduce Voicify, an Android virtual assistant that allows users to interact with on-screen elements in mobile apps through voice commands. Using a novel deep learning command parser, Voicify interprets human verbal input and performs matching with UI elements. In addition, the tool can directly open a specific feature from installed applications by fetching application code information to explore the set of in-app components. Our command parser achieved 90\% accuracy on the human command dataset. Furthermore, the direct feature invocation module achieves better feature coverage in comparison to Google Assistant. The user study demonstrates the usefulness of Voicify in real-world scenarios.
- Subjects :
- Computer Science - Human-Computer Interaction
Subjects
Details
- Database :
- arXiv
- Journal :
- Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7, no. 1 (2023): 1-22
- Publication Type :
- Report
- Accession number :
- edsarx.2305.05198
- Document Type :
- Working Paper
- Full Text :
- https://doi.org/10.1145/3581998