Search

This paper presents Gismo, an extensible domain-specific modelling language for prototyping executable models of gestural interaction. Relying on an underlying customisable framework, domain-specific models can specify, simulate and execute the behaviour of how users interact with a software application through the use of different interaction controllers and gesture types (e.g., specific hand movements or other body gestures). Model transformation technology is used to define the domain-specific operational semantics of Gismo, as well as to verify domain-specific properties. ICO models are automatically generated from Gismo models, and are executed by an underlying framework that can communicate with the target software application. We illustrate the use of Gismo through a running example that models the gestural interaction of a graphical application using dynamic hand gestures to control an animated 3D character. We report on the usability of Gismo based on an evaluation with 12 participants.

Integrating new input devices and their associated interaction techniques into interactive applications has always been challenging and time-consuming, due to the learning curve and technical complexity involved. Modeling devices, interactions and applications helps reducing the accidental complexity. Visual modeling languages can hide an important part of the technical aspects involved in the development process, thus allowing a faster and less error-prone development process. However, even with the help of modeling a gap remains to be bridged in order to go from models to the actual implementation of the interactive application. In this paper we use ICO, a visual formalism based on high-level Petri nets, to develop a generic layered framework for specifying executable models of interaction using gestural input devices. By way of the CASE tool PETSHOP we demonstrate the framework's feasibility to handle the Kinect and gesture-based interaction techniques. We validate the approach through two case studies that illustrate how to use executable, reusable and ex-tensible ICO models to develop gesture-based gaming applications.