Search

Funding: Natural Sciences and Engineering Council of Canada (NSERC), NSERC’s Digital Surface Software Application Network (Surfnet), and the Graphics Animation & New Media (GRAND) NCE. Over the past decade, multi-touch surfaces have become commonplace, with many researchers and practitioners describing the benefits of their natural, physical-like interactions. We present a pair of studies that empirically investigates the psychophysical effects of direct interaction with both physical and virtual artefacts. We use the phenomenon of Kinesthetic Figural After Effects-a change in understanding of the physical size of an object after a period of exposure to an object of different size. Our studies show that, while this effect is robustly reproducible when using physical artefacts, this same effect does not manifest when manipulating virtual artefacts on a direct, multi-touch tabletop display. We contribute quantitative evidence suggesting a psychophysical difference in our response to physical vs. virtual objects, and discuss future research directions to explore measurable phenomena to evaluate the presence of physical-like changes from virtual on-screen objects. Postprint

In this paper, we present Presstures, an extension to current multi-touch operations that enriches common multi-finger gestures with pressure information. By using the initially applied pressure level for implicit mode switching, a gesture can be enhanced with different functionalities to enlarge the interaction space for multi-touch. To evaluate the feasibility of our concept, we conducted an experiment, which indicates good human sensorimotor skills for performing multi-touch gestures with a few number of pressure levels and without any additional feedback. Based on the experimental results, we discuss implications for the design of pressure-sensitive multi-touch gestures, and propose application scenarios that make optimal use of our concept.

Over the last several years there has been an explosion of powerful, affordable, multi-touch devices. This provides an outstanding opportunity for novel data visualization techniques that leverage new interaction methods and minimize their barriers to entry. In this paper we describe an approach for multivariate data visualization that uses physics-based affordances that are easy to intuit, constraints that are easy to apply and visualize, and a consistent view as data is manipulated in order to promote data exploration and interrogation. We provide a framework for exploring this problem space, and an example proof of concept system called Kinetica. We describe the results of a user study that suggest users of Kinetica were able to explore multiple dimensions of data at once, identify outliers, and discover trends with minimal training.

Fundamental performance results for crossing-based selec-tion tasks with direct touch input are presented. A close adaptation of Accot and Zhai's indirect stylus crossing ex-periment reveals similar trends for direct touch input: touch crossing task time is faster or equivalent to touch pointing; continuous selection of large orthogonal crossing targets is most effective; and continuous selection of small collinear targets is least effective. Unlike indirect stylus and mouse crossing, not every kind of direct touch pointing perfor-mance is modeled accurately with standard Fitts' law. Instead, Fitts' law, used previously for touch pointing with small targets, is used to more accurately model discrete touch crossing with a directionally constrained target. In addition, visual touch feedback is shown to have a strong effect on absolute accuracy. Our work empirically validates touch crossing as a practical and efficient selection technique, and motivates the exploration of novel forms of expressive multi-touch crossing.

We present a novel approach to graspable interfaces using Touchbugs, actuated physical objects for interacting with interactive surface computing applications. Touchbugs are active tangibles that are able to move across surfaces by employing vibrating motors and can communicate with camera based multi-touch surfaces using infrared LEDs. Touchbug's embedded inertial sensors and computational capabilities open a new interaction space by providing autonomous capabilities for tangibles that allow goal directed behavior.

The prevalence of multi-touch devices opens the space for rich interactions. However, the complexity for creating multi-touch interactions hinders this potential. In this paper, we present Gesture Studio, a tool for creating multi-touch interaction behaviors by combining the strength of two distinct but complementary approaches: programming by demonstration and declaration. We employ an intuitive video-authoring metaphor for developers to demonstrate touch gestures, compose complicated behaviors, test these behaviors in the tool and export them as source code that can be integrated into the developers' project.

In this paper, we present a novel multi-touch gesture-based authentication technique. We take advantage of the multi-touch surface to combine biometric techniques with gestural input. We defined a comprehensive set of five-finger touch gestures, based upon classifying movement characteristics of the center of the palm and fingertips, and tested them in a user study combining biometric data collection with usability questions. Using pattern recognition techniques, we built a classifier to recognize unique biometric gesture characteristics of an individual. We achieved a 90% accuracy rate with single gestures, and saw significant improvement when multiple gestures were performed in sequence. We found user ratings of a gestures desirable characteristics (ease, pleasure, excitement) correlated with a gestures actual biometric recognition rate - that is to say, user ratings aligned well with gestural security, in contrast to typical text-based passwords. Based on these results, we conclude that multi-touch gestures show great promise as an authentication mechanism.

ZeroTouch (ZT) is a unique optical sensing technique and architecture that allows precision sensing of hands, fingers, and other objects within a constrained 2-dimensional plane. ZeroTouch provides tracking at 80 Hz, and up to 30 concurrent touch points. Integration with LCDs is trivial. While designed for multi-touch sensing, ZT enables other new modalities, such as pen+touch and free-air interaction. In this paper, we contextualize ZT innovations with a review of other flat-panel sensing technologies. We present the modular sensing architecture behind ZT, and examine early diverse uses of ZT sensing.

Determining who is interacting with a multi-user interactive touch display is challenging. We describe a technique for associating multi-touch interactions to individual users and their accelerometer-equipped mobile devices. Real-time device accelerometer data and depth camera-based body tracking are compared to associate each phone with a particular user, while body tracking and touch contacts positions are compared to associate a touch contact with a specific user. It is then possible to associate touch contacts with devices, allowing for more seamless device-display multi-user interactions. We detail the technique and present a user study to validate and demonstrate a content exchange application using this approach.

This paper describes a field study of an interactive surface deployed in three family homes. The tabletop technology provides a central place where digital content, such as photos, can be easily archived, managed and viewed. The tabletop affords multi-touch input, allowing digital content to be sorted, triaged and interacted with using one or two-handed interactions. A physics-based simulation adds dynamics to digital content, providing users with rich ways of interacting that borrows from the real-world. The field study is one of the first of a surface computer within a domestic environment. Our goal is to uncover people's inter-actions, appropriations, perceptions and experiences with such technologies, exploring the potential barriers to use. Given these devices provide such a revolutionary shift in interaction, will people be able to engage with them in everyday life in the ways we intend? In answering this question, we hope to deepen our understanding of the design of such systems for home and consumer domains.

Multi-touch gestures have become popular on a wide range of touchscreen devices, but the programming of these gestures remains an art. It is time-consuming and error-prone for a developer to handle the complicated touch state transitions that result from multiple fingers and their simultaneous movements. In this paper, we present Gesture Coder, which by learning from a few examples given by the developer automatically generates code that recognizes multi-touch gestures, tracks their state changes and invokes corresponding application actions. Developers can easily test the generated code in Gesture Coder, refine it by adding more examples, and once they are satisfied with its performance integrate the code into their applications. We evaluated our learning algorithm exhaustively with various conditions over a large set of noisy data. Our results show that it is sufficient for rapid prototyping and can be improved with higher quality and more training data. We also evaluated Gesture Coder's usability through a within-subject study in which we asked participants to implement a set of multi-touch interactions with and without Gesture Coder. The results show overwhelmingly that Gesture Coder significantly lowers the threshold of programming multi-touch gestures.

While very large collaborative surfaces are already being widely employed to facilitate concurrent interactions with multiple users, they involve no personalization in the touch interactions. Augmenting them to identify the touch interactions with multiple smart-phones can enable interesting co-located communal applications with context-based personalized interactions and information exchange amongst users' portable devices and the shared wall display. This paper proposes a novel matching technique, called tilt correlation, which employs the built-in tilt sensor to identify smart-phones that make concurrent two-point contacts on a common multi-touch wall display. Experimental investigations suggest that the resultant error rate is relatively low; in addition, we also propose a quantitative measure, called the Bourne Identity Index to allow application designers to determine the reliability of each device identification.

Direct touch manipulations enable the user to interact with the on-screen content in a direct and easy manner closely mimicking the spatial manipulations in the physical world. However, they also suffer from well-known issues of precision, occlusion and an inability to isolate different degrees of freedom in spatial manipulations. We present a set of interactions, called Rock & Rails, that augment existing direct touch manipulations with shape-based gestures, thus providing on-demand gain control, occlusion avoidance, and separation of constraints in 2D manipulation tasks. Using shape gestures in combination with direct-manipulations allows us to do this without ambiguity in detection and without resorting to manipulation handles, which break the direct manipulation paradigm. Our set of interactions were evaluated by 8 expert graphic designers and were found to be easy to learn and master, as well as effective in accomplishing a precise graphical layout task.

We present G-nome Surfer 2.0, a tabletop interface for fostering inquiry-based learning of genomics. We conducted an experimental study with 48 participants that compared students' learning of genomic concepts using existing bioinformatics tools and using two alternative implementations of G-nome Surfer: a collaborative multi-mouse GUI and a tabletop interface. Our findings indicate that G-nome Surfer improves students' performance, reduces workload, and increases enjoyment. The comparison of tabletop and multi-mouse implementations further shows that the tabletop condition results in four educational benefits: 1) increasing physical participation, 2) encouraging reflection, 3) fostering effective collaboration, and 4) facilitating more intuitive interaction.

We present TUIC, a technology that enables tangible interaction on capacitive multi-touch devices, such as iPad, iPhone, and 3M's multi-touch displays, without requiring any hardware modifications. TUIC simulates finger touches on capacitive displays using passive materials and active modulation circuits embedded inside tangible objects, and can be used with multi-touch gestures simultaneously. TUIC consists of three approaches to sense and track objects: spatial, frequency, and hybrid (spatial plus frequency). The spatial approach, also known as 2D markers, uses geometric, multi-point touch patterns to encode object IDs. Spatial tags are straightforward to construct and are easily tracked when moved, but require sufficient spacing between the multiple touch points. The frequency approach uses modulation circuits to generate high-frequency touches to encode object IDs in the time domain. It requires fewer touch points and allows smaller tags to be built. The hybrid approach combines both spatial and frequency tags to construct small tags that can be reliably tracked when moved and rotated. We show three applications demonstrating the above approaches on iPads and 3M's multi-touch displays.

Manipulating 3D objects on a tabletop is inherently problematic. Tabletops lack a third degree of freedom and thus require novel solutions to support even the simplest 3D manipulations. Our solution is TZee - a passive tangible widget that enables natural interactions with 3D objects by exploiting the lighting properties of diffuse illumination (DI) multi-touch tabletops. TZee is assembled from stacked layers of acrylic glass to extend the tabletop's infrared light slightly above the surface without supplemental power. With TZee, users can intuitively scale, translate and rotate objects in all three dimensions, and also perform more sophisticated gestures, like "slicing" a volumetric object, that have not been possible with existing tabletop interaction schemes. TZee is built with affordable and accessible materials, and one tabletop surface can easily support multiple TZees. Moreover, since TZee is transparent, there are numerous possibilities to augment interactions with feedback, helpful hints, or other visual enhancements. We discuss several important design considerations and demonstrate the value of TZee with several applications.

Sand animation is a performance art technique in which an artist tells stories by creating animated images with sand. Inspired by this medium, we have developed a new multi-touch digital artistic medium named SandCanvas that simplifies the creation of sand animations. SandCanvas also goes beyond traditional sand animation with tools for mixing sand animation with video and replicating recorded free-form hand gestures. In this paper, we analyze common sand animation hand gestures, present SandCanvas's intuitive UI, and describe implementation challenges we encountered. We also present an evaluation with professional and novice artists that shows the importance and unique affordances of this new medium.

Diffusion curves are effective 2D vector-graphics primitives, for creating smoothly-shaded drawings with rich colors and unique styles. Conventional drawing systems for diffusion curves often require users to successively layout curve geometry and then specify colors, which is rather tedious for complex drawings. This paper proposes a novel multi-touch sketching interface for efficient design of 2D vector graphics with diffusion curves. In sharp contrast to previous interfaces, we develop a family of multi-touch gestures, allowing users to simultaneously sketch multiple diffusion curves and also to interactively edit and tune curve geometry and colors. Our experiments show that this not only brings novel painting experience to users but also provides a practical and effective tool for vector graphics design, useful for styles like silk painting, Disney cartoon, art poster, and photo-realistic effects. Lastly, we conduct a user study to explore the interface's intuitive and efficient drawing capability with both professional 2D artists and novice users.

For document visualization, folding techniques provide a focus-plus-context approach with fairly high legibility on flat sections. To enable richer interaction, we explore the design space of multi-touch document folding. We discuss several design considerations for simple modeless gesturing and compatibility with standard Drag and Pinch gestures. We categorize gesture models along the characteristics of Symmetric/Asymmetric and Serial/Parallel, which yields three gesture models. We built a prototype document workspace application that integrates folding and standard gestures, and a system for testing the gesture models. A user study was conducted to compare the three models and to analyze the factors of fold direction, target symmetry, and target tolerance in user performance when folding a document to a specific shape. Our results indicate that all three factors were significant for task times, and parallelism was greater for symmetric targets.

Despite the prominence of multi-touch technologies, there has been little work investigating its integration into the desktop environment. Bringing multi-touch into desktop computing would give users an additional input channel to leverage, enriching the current interaction paradigm dominated by a mouse and keyboard. We provide two main contributions in this domain. First, we describe the results from a study we performed, which systematically evaluates the various potential regions within the traditional desktop configuration that could become multi-touch enabled. The study sheds light on good or bad regions for multi-touch, and also the type of input most appropriate for each of these regions. Second, guided by the results from our study, we explore the design space of multi-touch-integrated desktop experiences. A set of new interaction techniques are coherently integrated into a desktop prototype, called Magic Desk, demonstrating potential uses for multi-touch enabled desktop configurations.

Precise alignment of graphical objects and the creation of accurate layouts are crucial activities in many applications, such as graphics design tools, presentation software or graph editors. Surface computing is very promising for these application domains but not fully explored yet. In this paper we contribute two tools which support layout tasks on interactive displays: interactive grids and multi-touch alignment guides. Both tools allow the precise positioning of graphical objects in a flexible and fluent way by multi-touch input. Direct bimanual interaction and physical metaphors are applied to arrange objects along straight lines and curves. A formative user evaluation showed promising results with regard to a productive and easy use of the tools.

We propose Radial-Stroke and Finger-Count Shortcuts, two techniques aimed at augmenting the menubar on multi-touch surfaces. We designed these multi-finger two-handed interaction techniques in an attempt to overcome the limitations of direct pointing on interactive surfaces, while maintaining compatibility with traditional interaction techniques. While Radial-Stroke Shortcuts exploit the well-known advantages of Radial Strokes, Finger-Count Shortcuts exploit multi-touch by simply counting the number of fingers of each hand in contact with the surface. We report the results of an experimental evaluation of our technique, focusing on expert-mode performance. Finger-Count Shortcuts outperformed Radial-Stroke Shortcuts in terms of both easiness of learning and performance speed.

We present an experimental comparison of multi-touch and tangible user interfaces for basic interface actions. Twelve participants completed manipulation and acquisition tasks on an interactive surface in each of three conditions: tangible user interface; multi-touch; and mouse and puck. We found that interface control objects in the tangible condition were easiest to acquire and, once acquired, were easier/more accurate to manipulate. Further qualitative analysis suggested that in the evaluated tasks tangibles offer greater adaptability of control and specifically highlighted a problem of exit error that can undermine fine-grained control in multi-touch interactions. We discuss the implications of these findings for interface design.

The introduction of tabletop interfaces has given rise to the need for the development of secure and usable authentication techniques that are appropriate for the co-located collaborative settings for which they have been designed. Most commonly, user authentication is based on something you know, but this is a particular problem for tabletop interfaces, as they are particularly vulnerable to shoulder surfing given their remit to foster co-located collaboration. In other words, tabletop users would typically authenticate in full view of a number of observers. In this paper, we introduce and evaluate a number of novel tabletop authentication schemes that exploit the features of multi-touch interaction in order to inhibit shoulder surfing. In our pilot work with users, and in our formal user-evaluation, one authentication scheme - Pressure-Grid - stood out, significantly enhancing shoulder surfing resistance when participants used it to enter both PINs and graphical passwords.

Enabling efficient exploration of large-scale virtual environments such as those simulating astrophysical environments is highly challenging. Astrophysical virtual worlds span exceptionally large spatial scales occupied mostly by empty space, and this makes it difficult for the user to comprehend the spatial context during exploratory navigation. Public exhibits, where novice users have little experience using complicated virtual navigation interfaces, pose additional challenges. To address these issues, we propose multi-touch techniques to deliver an effective interface to navigate the unique features of large-scale 3D environments such as astrophysical simulations. In this work, we carefully study conventional multi-touch methods and adapt them to the practical requirements of this application. A novel technique called the powers-of-ten ladder is introduced to support efficient movement across huge spatial scales using multi-touch interactions. We also investigate user experiences with various multi-touch finger gestures on our prototype digital planetarium.

We present Silicone iLluminated Active Peripherals (SLAP), a system of tangible, translucent widgets for use on multitouch tabletops. SLAP Widgets are cast from silicone or made of acrylic, and include sliders, knobs, keyboards, and buttons. They add tactile feedback to multi-touch tables, improving input accuracy. Using rear projection, SLAP Widgets can be relabeled dynamically, providing inexpensive, battery-free, and untethered augmentations. Furthermore, SLAP combines the flexibility of virtual objects with physical affordances. We evaluate how SLAP Widgets influence the user experience on tabletops compared to virtual controls. Empirical studies show that SLAPWidgets are easy to use and outperform virtual controls significantly in terms of accuracy and overall interaction time.

Current multi-touch interaction techniques typically only use the x-y coordinates of the human finger's contact with the screen. However, when fingers contact a touch-sensitive surface, they usually approach at an angle and cover a relatively large 2D area instead of a precise single point. In this paper, a Frustrated Total Internal Reflection (FTIR) based multi-touch device is used to collect the finger imprint data. We designed a series of experiments to explore human finger input properties and identified several useful properties such as contact area, contact shape and contact orientation which can be exploited to improve the performance of multi-touch selecting and pointing tasks. Based on the experimental results, we discuss some implications for the design of human finger input interfaces and propose several design prototypes which incorporate these implications. A set of raw data and several concrete recommendations which are useful for the research community are also presented.

Touchpad and touchscreen interaction using multiple fingers is emerging as a valuable form of high-degree-of-freedom input. While bimanual interaction has been extensively studied, touchpad interaction using multiple fingers of the same hand is not yet well understood. We describe two experiments on user perception and control of multi-touch interaction using one and two hands. The first experiment addresses how to maintain perceptual-motor compatibility in multi-touch interaction, while the second measures the separability of control of degrees-of-freedom in the hands and fingers. Results indicate that two-touch interaction using two hands is compatible with control of two points, while twotouch interaction using one hand is compatible with control of a position, orientation, and hand-span. A slight advantage is found for two hands in separating the control of two positions.

We present data from detailed observations of CityWall, a large multi-touch display installed in a central location in Helsinki, Finland. During eight days of installation, 1199 persons interacted with the system in various social configurations. Videos of these encounters were examined qualitatively as well as quantitatively based on human coding of events. The data convey phenomena that arise uniquely in public use: crowding, massively parallel interaction, teamwork, games, negotiations of transitions and handovers, conflict management, gestures and overt remarks to co-present people, and "marking" the display for others. We analyze how public availability is achieved through social learning and negotiation, why interaction becomes performative and, finally, how the display restructures the public space. The multi-touch feature, gesture-based interaction, and the physical display size contributed differentially to these uses. Our findings on the social organization of the use of public displays can be useful for designing such systems for urban environments.

The size of human fingers and the lack of sensing precision can make precise touch screen interactions difficult. We present a set of five techniques, called Dual Finger Selections, which leverage the recent development of multi-touch sensitive displays to help users select very small targets. These techniques facilitate pixel-accurate targeting by adjusting the control-display ratio with a secondary finger while the primary finger controls the movement of the cursor. We also contribute a "clicking" technique, called SimPress, which reduces motion errors during clicking and allows us to simulate a hover state on devices unable to sense proximity. We implemented our techniques on a multi-touch tabletop prototype that offers computer vision-based tracking. In our formal user study, we tested the performance of our three most promising techniques (Stretch, X-Menu, and Slider) against our baseline (Offset), on four target sizes and three input noise levels. All three chosen techniques outperformed the control technique in terms of error rate reduction and were preferred by our participants, with Stretch being the overall performance and preference winner.