Search

As Virtual Reality(VR), Augmented Reality(AR), Mixed Reality(MR) technology becomes more accessible, it is important to explore VR/AR/MR technologies that can be used for remote collaboration on physical tasks. Previous research has shown that gesture-based interaction is intuitive and expressive for remote collaboration, and using 3D CAD models can provide clear instructions for assembly tasks. In this paper, therefore, we describe a new MR remote collaboration system which combines the use of gesture and CAD models in a complementary manner. The prototype system enables a remote expert in VR to provide instructions based on 3D gesture and CAD models (3DGAM) for a local worker who uses AR to see these instructions. Using this interface, we conducted a formal user study to explore the effect of sharing 3D gesture and CAD models in an assembly training task. We found that the combination of 3D gesture and CAD models can improve remote collaboration on an assembly task with respect to the performance time and user experience. Finally, we provide some conclusions and directions for future research.

Research on Augmented Reality (AR)/Mixed Reality (MR) remote collaboration for physical tasks remains a compelling and dynamic area of study. AR systems have been developed which transmit virtual annotations between remote collaborators, but there has been little research on how haptic feedback can also be shared. In this paper, we present a Virtual Reality (VR)-Spatial Augmented Reality (SAR) remote collaborative system that provides haptic feedback with tangible interaction between a local worker and a remote expert helper. Using this system, we conducted a within-subject user study to compare two interfaces for remote collaboration between a local worker and expert helper, one with mid-air free drawing (MFD) and one with tangible physical drawing (TPD). The results showed that there were no significant differences with respect to performance time and operation errors. However, users felt that the TPD interface supporting passive haptic feedback could significantly improve the remote experts’ user experience in VR. Our research provides useful information on the way for gesture- and gaze-based multimodal interaction supporting haptic feedback in AR/MR remote collaboration on physical tasks. Refereed/Peer-reviewed

This paper investigates the effect of using augmented reality (AR) annotations and two different gaze visualizations, head pointer (HP) and eye gaze (EG), in an AR system for remote collaboration on physical tasks. First, we developed a spatial AR remote collaboration platform that supports sharing the remote expert’s HP or EG cues. Then the prototype system was evaluated with a user study comparing three conditions for sharing non-verbal cues: (1) a cursor pointer (CP), (2) HP and (3) EG with respect to task performance, workload assessment and user experience. We found that there was a clear difference between these three conditions in the performance time but no significant difference between the HP and EG conditions. When considering the perceived collaboration quality, the HP/EG interface was statistically significantly higher than the CP interface, but there was no significant difference for workload assessment between these three conditions. We used low-cost head tracking for the HP cue and found that this served as an effective referential pointer. This implies that in some circumstances, HP could be a good proxy for EG in remote collaboration. Head pointing is more accessible and cheaper to use than more expensive eye-tracking hardware and paves the way for multi-modal interaction based on HP and gesture in AR remote collaboration.

In this paper, we present a projector-based mixed reality (MR) remote collaborative system which enables remote users to collaboratively work on a physical task using gesture and head pointing (GHP). Using this platform, we studied the effects of GHP in a typical manufacturing use case. Our system supports natural and intuitive multimodal interaction based on GHP, and it can project the remote user's GHP into the local environment to enhance remote collaboration. Our prototype system was compared with an augmented reality (AR) condition (ANNOTATION), which is the most popular method currently for AR/MR remote collaboration. We found a significant difference between the ANNOTATION and GHP conditions in terms of performance. The GHP system significantly improved the collaborative experience (e.g., awareness of the user's attention),empathy (e.g., co-presence), and remote interaction. Moreover, we discuss the implications of this research and directions for future research. Refereed/Peer-reviewed

This paper provides a review of research into using Augmented Reality (AR) and Mixed Reality(MR) for remote collaboration on physical tasks. AR/MR-based remote collaboration on physical tasks has recently become more prominent in academic research and engineering applications. It has great potential in many fields, such as real-time remote medical consultation, education, training, maintenance, remote assistance in engineering, and other remote collaborative tasks. However, to the best of our knowledge there has not been any comprehensive review of research in AR/MR remote collaboration on physical tasks. Therefore, this paper presents a comprehensive survey of research between 2000 and 2018 in this domain. We collected 215 papers, more than 80% of which were published between 2010 and 2018, and all relevant works are discussed at length. Then we elaborate on the review from typical architectures, applications (e.g., industry, telemedicine, architecture, teleducation and others), and empathic computing. Next, we made an in-depth review of the papers from seven aspects: (1) collection and classification research, (2) using 3D scene reconstruction environments and live panorama, (3) periodicals and conducting research, (4) local and remote user interfaces, (5) features of user interfaces commonly used, (6) architecture and sharing non-verbal cues, (7) applications and toolkits. We find that most papers (160 articles, 74.4%) are published in conferences, using co-located collaboration to emulate remote collaboration is adopted by more than half (126, 58.6%) of the reviewed papers, the shared non-verbal cues can be mainly classified into five types (Virtual Replicas or Physical Proxy(VRP), AR Annotations or a Cursor Pointer(ARACP), avatar, gesture, and gaze), the local/remote interface is mainly divided into four categories (Head-Mounted Displays(HMD), Spatial Augmented Reality(SAR), Windows-Icon-Menu-Pointer(WIMP) and Hand-Held Displays(HHD)). From this, we can draw ten conclusions. Following this we report on issues for future works. The paper also provides an overall academic roadmap and useful insight into the state-of-the-art of AR/MR remote collaboration on physical tasks. This work will be useful for current and future researchers who are interested in collaborative AR/MR systems. Refereed/Peer-reviewed

We present an idea that uses a tracked graphics tablet attached a virtual or physical object to grasp in VR for direct 3D sketching on virtual objects, giving users the feeling of holding a physical object and drawing on it. We developed a prototype system and evaluated the effect of different tablet manipulation modes on user performance and experience when drawing strokes on virtual objects in VR. Results suggested that having a physical handle or object to grasp enhanced the participants' sense of realism, compared to conditions where there was no tablet or a simple tablet interface. Refereed/Peer-reviewed

In this paper, we describe a new Mixed Reality (MR) remote collaborative platform making use of 3D CAD models for training in the manufacturing industry. It enables a remote expert in Virtual Reality (VR) to train a local worker in a physical assembly task. For the local site, we use Spatial Augmented Reality (SAR) to enable the local worker see virtual cues without wearing any AR devices, leaving their user hands free to easily manipulate the physical parts. For the remote expert, we construct a 3D virtual environment using virtual replicas of the physical parts. We also report on the results of a usability study of the prototype. Refereed/Peer-reviewed

Current remote collaborative systems in manufacturing are mainly based on video-conferencing technology. Their primary aim is to transmit manufacturing process knowledge between remote experts and local workers. However, it does not provide the experts with the same hands-on experience as when synergistically working on site in person. The mixed reality (MR) and increasing networking performances have the capacity to enhance the experience and communication between collaborators in geographically distributed locations. In this paper, therefore, we propose a new gesture-based remote collaborative platform using MR technology that enables a remote expert to collaborate with local workers on physical tasks. Besides, we concentrate on collaborative remote assembly as an illustrative use case. The key advantage compared to other remote collaborative MR interfaces is that it projects the remote expert’s gestures into the real worksite to improve the performance, co-presence awareness, and user collaboration experience. We aim to study the effects of sharing the remote expert’s gestures in remote collaboration using a projector-based MR system in manufacturing. Furthermore, we show the capabilities of our framework on a prototype consisting of a VR HMD, Leap Motion, and a projector. The prototype system was evaluated with a pilot study comparing with the POINTER (adding AR annotations on the task space view through the mouse), which is the most popular method used to augment remote collaboration at present. The assessment adopts the following aspects: the performance, user’s satisfaction, and the user-perceived collaboration quality in terms of the interaction and cooperation. Our results demonstrate a clear difference between the POINTER and 2.5DHANDS interface in the performance time. Additionally, the 2.5DHANDS interface was statistically significantly higher than the POINTER interface in terms of the awareness of user’s attention, manipulation, self-confidence, and co-presence. Refereed/Peer-reviewed

This paper investigates how two different unique gaze visualizations (the head pointer(HP), eye gaze(EG)) affect table-size physical tasks in Mixed Reality (MR) remote collaboration. We developed a remote collaborative MR Platform which supports sharing of the remote expert's HP and EG. The prototype was evaluated with a user study comparing two conditions: sharing HP and EG with respect to their effectiveness in the performance and quality of cooperation. There was a statistically significant difference between two conditions on the performance time, and HP is a good proxy for EG in remote collaboration Refereed/Peer-reviewed

The Mixed Reality (MR) technology can be used to create unique collaborative experiences. In this paper, we propose a new remote collaboration platform using MR and eye-tracking that enables a remote helper to assist a local worker in an assembly task. We present results from research exploring the effect of sharing virtual gaze and annotations cues in an MR-based projector interface for remote collaboration. The key advantage compared to other remote collaborative MR interfaces is that it projects the remote expert's eye gaze into the real worksite to improve co-presence. The prototype system was evaluated with a pilot study comparing two conditions: POINTER and ET (eye-tracker cues). We observed that the task completion performance was better in the ET condition. And that sharing gaze significantly improved the awareness of each other's focus and co-presence Refereed/Peer-reviewed

In AR/MR remote collaboration, a remote expert often has to demonstrate tasks for a local worker. To make this easier, we have developed a new adaptive MR remote collaborative architecture which enables a remote expert to guide a local user in physical assembly and training tasks. A remote user can activate the instructions through simple and intuitive interaction, then clear instructions are shown in both AR (local) and VR (remote) views, enabling a local worker to operate the tool following the instructions. In the demonstration we use a hammer operation as a physical task, showing the benefits of the adaptive MR remote collaborative platform Refereed/Peer-reviewed

In this paper, we explore how Augmented Reality (AR) and anthropomorphism can be used to assign emotions to common physical objects based on their needs. We developed a novel emotional interaction model among personified physical objects so that they could react to other objects by changing virtual facial expressions. To explore the effect of such an emotional interface, we conducted a user study comparing three types of virtual cues shown on the real objects: (1) information only, (2) emotion only and (3) both information and emotional cues. A significant difference was found in task completion time and the quality of work when adding emotional cues to an informational AR-based guiding system. This implies that adding emotion feedback to informational cues may produce better task results than using informational cues alone Refereed/Peer-reviewed

AR instruction is a kind of virtual information presented on a human–computer interface. It allows users to view the geometric state, spatial relationship, operation method, and other information involved in the physical task, to form the spatial cognition of the current interaction process. At present, AR instructions cannot support high-precision procedural tasks. The reason is that the existing research work is to use visual elements to express the spatial relationship of physical tasks, without considering transforming the long-term accumulated potential experience of advanced users into a series of effective visual features and interaction modes, to promote the new users to quickly conceive the task intention. In this paper, a user-centered AR instruction(UcAI) is defined and tested for the first time in a procedural task. The control experiment and behavior analysis of 30 participants designing two tasks with different operation precision show that UcAI is more beneficial to improve the user's spatial cognitive ability than conventional AR instruction. Especially in the high-precision operation task, UcAI plays an important role. Our research results have a certain guiding significance for advanced AR instruction design, which extends AR technology to physical tasks with high cognitive complexity. Refereed/Peer-reviewed