Search

Mobile Augmented Reality (MAR) integrates computer-generated virtual objects with physical environments for mobile devices. MAR systems enable users to interact with MAR devices, such as smartphones and head-worn wearables, and perform seamless transitions from the physical world to a mixed world with digital entities. These MAR systems support user experiences using MAR devices to provide universal access to digital content. Over the past 20 years, several MAR systems have been developed, however, the studies and design of MAR frameworks have not yet been systematically reviewed from the perspective of user-centric design. This article presents the first effort of surveying existing MAR frameworks (count: 37) and further discusses the latest studies on MAR through a top-down approach: (1) MAR applications; (2) MAR visualisation techniques adaptive to user mobility and contexts; (3) systematic evaluation of MAR frameworks, including supported platforms and corresponding features such as tracking, feature extraction, and sensing capabilities; (4) and underlying machine learning approaches supporting intelligent operations within MAR systems. Finally, we summarise the development of emerging research fields and the current state-of-the-art and discuss the important open challenges and possible theoretical and technical directions. This survey aims to benefit both researchers and MAR system developers alike.

Mobile Augmented Reality (MAR) applications enable users to interact with physical environments through overlaying digital information on top of camera views. Detecting and classifying complex objects in the real world presents a critical challenge to enable immersive user experiences in MAR applications. Aiming to provide continuous MAR experiences, we address a key challenge of continuous object recognition, which requires accommodating an increasing number of recognition requests on different types of images in MAR systems and possible new types of images in emerging applications. Inspired by the latest advance in continual learning approaches in computer vision, this paper presents a novel MAR system to enhance its scalability with continual learning in realistic scenarios. Our experiments demonstrate that 1) the system enables efficiently recognising objects without requiring retraining from scratch; and 2) edge computing further reduces latency for continual object recognition.

Augmented Reality (AR) provides immersive user experiences by overlaying digital information on physical environments. Context-awareness is crucial for delivering relevant augmentations that best suit users’ requirements and their en-vironments. In this article, we combine context-aware reasoning with emerging AR applications to provide the most relevant information according to user and environment contexts. To support the best possible quality of experience, 5G edge computing enables the distribution of computation-intensive AR tasks to edge servers through 5G networks. We develop ConAR, a context-aware head-mounted display AR system that is deployed on the edge and cloud leveraging both environmental sensors and user profile context for navigation. ConAR is composed of a HoloLens application and a paired mobile client, which contains a context model for air quality forecasting, and rendering recommendations on holograms through a HoloLens 2 device. We evaluate our system performance by deploying our proposed air quality prediction algorithm on the edge and cloud while communicating to them using 5G and LTE connections. We measure network quality metrics and find the deployment on the edge with 5G connections significantly outperforms alternative solutions. Our results demonstrate that the 5G edge computing is suitable for supporting latency-sensitive analysis tasks for context-aware AR.

In a postpandemic world, remaining vigilant and maintaining social distancing are still crucial so societies can contain the virus and the public can avoid disproportionate health impacts. Augmented reality (AR) can visually assist users in understanding the distances in social distancing. However, integrating external sensing and analysis is required for social distancing beyond the users’ local environment. We present DistAR, an android-based application for social distancing leveraging AR and smart sensing using on-device analysis of optical images and environment crowdedness from smart campus data. Our prototype is one of the first efforts to combine AR and smart sensing technologies to create a real-time social distancing application.

This article contributes a research vision for using edge computing to deliver the computing infrastructure for emerging smart megacities, with use cases, key requirements, and reflections on the state of the art. We also address edge server placements, a key challenge for edge computing adoption.

Mobile Augmented Reality (MAR) mixes physical environments with user-interactive virtual annotations. Immersive MAR experiences are supported by computation-intensive tasks, which are typically offloaded to cloud or edge servers. Such offloading introduces additional network traffic and influences the motion-to-photon latency (a determinant of user-perceived quality of experience). Therefore, proper multimedia protocols are crucial to minimise transmission latency and ensure sufficient throughput to support MAR performance. Relatedly, 5G is a potential MAR supporting technology and is widely believed to be faster and more efficient than its predecessors. However, the suitability and performance of existing multimedia protocols for MAR in the 5G edge context have not been explored. In this work, we present a detailed evaluation of several popular multimedia protocols (HLS, MPEG-DASH, RTP, RTMP, RTMFP, and RTSP) and transport protocols (QUIC, UDP, and TCP) with a MAR system on a real-world 5G edge testbed. The evaluation results indicate that RTMP has the lowest median client-to-server packet latency on 5G and LTE for all image resolutions. In terms of individual image resolutions, from 144p to 480p over 5G and LTE, RTMP has the lowest median packet latency of 14.03±1.05ms. Whereas for jitter, HLS has the smallest median jitter across all image resolutions over LTE and 5G with medians of 2.62 ms and 1.41 ms, respectively. Our experimental results indicate that RTMP and HLS are the most suitable protocols for MAR.

Air pollution introduces a major challenge for societies, where it leads to the premature deaths of millions of people each year globally. Massive deployment of air quality sensing devices and data analysis for the resultant data will pave the way for the development of real-time intelligent applications and services, e.g., minimization of exposure to poor air quality either on an individual or city scale. 5G and edge computing supports dense deployments of sensors at high resolution with ubiquitous connectivity, high bandwidth, high-speed gigabit connections, and ultralow latency analysis. This article conceptualizes AI-powered scalable air quality monitoring and presents two systems of calibrating low-cost air quality sensors and the image processing of pictures captured by hyperspectral cameras to better detect air quality. We develop and deploy different AI algorithms in these two systems on a 5G edge testbed and perform a detailed analytics regarding to 1) the performance of AI algorithms and 2) the required communication and computation resources.

Physical world enhancements through virtually drawn annotations on mobile devices are a core component of mobile augmented reality (MAR) experiences. However, resource intensive computer vision tasks typically limit the full deployment of MAR pipelines on-device due to a lack of required computation resources. As an alternative, a distributed scheme between client devices and edge servers enables offloading computation intensive tasks to powerful machines. This introduces challenges, such as maintaining low communication latency and high bandwidth channels between client and edge server. Current LTE and WiFi infrastructure do not fulfil such requirements. 5G networks are an ideal candidate to provide seamless MAR experiences. This research investigates the enhancement of MAR with both 5G and edge computing and aims to address two particular challenges: 1) the creation of a new 5G-based edge computing infrastructure to support reliable MAR, and 2) investigating which particular transport protocols allow for optimised quality of experience (QoE) for MAR applications within 5G infrastructure.