Search

Recent advances in extended reality (XR) technologies have enabled new and increasingly realistic empathy tools and experiences. In XR, all interactions take place in different spatial contexts, all with different features, affordances, and constraints. We present a systematic literature survey of recent work on empathy in XR. As a result, we contribute a research roadmap with three future opportunities and six open questions in XR-enabled empathy research across both physical and virtual spaces. © 2023 Copyright held by the owner/author(s).

VR technology has brought new prospects and energy to art museums, prompting the advancement of new technologies that can tackle future challenges while catering to the demands of the metaverse era. Today’s digital archiving and guided tours, which rely on VR and more portable information retrieval, provide optimism for museum development. VR technology overcomes the limitations of traditional museums constrained by space and time as digital archiving gradually becomes a reality. This study highlights the benefits of metaverse-driven online archiving from the users’ perspective, followed by a review of recent examples of using VR for media art archiving. We emphasize the importance of further research into unexplored emerging technologies in media art archiving and propose that VR technologies in the metaverse are essential enablers for online archiving purposes. Our work aims to help art researchers, archivists, professionals, and artists to comprehend the challenges and future needs of archiving media art in the developing metaverse. Furthermore, we discuss present cases of employing VR for media art preservation after outlining the advantages of metaverse-driven online archiving from the users’ viewpoint. We stress the need for additional studies on new technologies in media art preservation. Moreover, metaverse VR technologies play a crucial role in achieving online archiving goals. This research contributes to a better understanding of the challenges and future requirements of archiving media art in the evolving metaverse for art researchers, archivists, and artists.

The Metaverse connects our physical reality with virtual worlds. Social VR platforms facilitate the creation of such virtual worlds, enabling activities such as interactive teaching, conferences, and community gatherings. These activities can be performed in mixed mode, with some participants physically present in the same location. In this paper, we evaluate the feasibility of such mixed-mode events by studying three leading social VR platforms. We uncover the network traffic patterns generated by these platforms, which affect the user experience when multiple users share the same network. We explore the traffic patterns to show that model loading creates a significant overhead and impacts user quality of experience. When the number of simultaneously connected users increases, some operations lead to network congestion that degrades or even interrupts service for most users. From these observations, we derive concrete action points to develop social VR platforms that enable mixed-mode events over the same network.

This chapter focuses on discussing the evolution of the Metaverse as well as the difficulties that have been encountered. It provides a concise overview of the evolution of cyberspace as well as the importance of technological enablers. Edge computing can reduce the user-experienced latency when compared with cloud offloading. The chapter emphasizes a number of essential challenges, both from a technical and an ecosystemic point of view, that are necessary for the construction and maintenance of the Metaverse. It looks at bottom-up approach to discuss the three most important aspects, which have been inherited from the viewpoints of ubiquitous computing in particular: the network and mobile edge, system interoperability, and user-centric immersive design, which are representing the network, the system, and human users. All the immersive technologies that will deliver entry points to Metaverse via multiple types of devices will have access to potentially private users' data. © 2024 The Institute of Electrical and Electronics Engineers, Inc. All rights reserved.

The emergence of the Metaverse enables the creation of alternative spaces at the intersection between digital and physical through the replication of physical events and objects within physical-digital twins. In this article, we apply such twins to connected vehicles within a Traffic Metaverse as an intermediate platform for the shared perception of the road environment. The platform builds a real-time digital copy of the road conditions to serve vehicular applications on a city scale through ubiquitous sensing, reliable and low-latency communications, artificial intelligence, and extended reality. This article focuses on collaboratively building virtual 3D maps of road networks that provide road users with a pervasive view of the road conditions to increase situational awareness. Through vehicle-to-everything (V2X) cooperative perception, such a Metaverse platform expands the driver's visibility beyond the vehicle's line of sight. The platform also enables virtual remote access to monitor and forecast traffic ahead of space and time. © 2017 IEEE.

While the academic community tries to define and experiment with the metaverse, businesses and institutions seek to build their representation in the metaverse. Many educational institutions build meta-campuses and move online classes into virtual environments beyond simple videoconferencing. This paper describes our experience building a university metaverse, highlighting the technical, human, and organisational challenges we have encountered through two major events. Considering the issue of real-time communication and scalability in the web-based metaverse, we also present an analysis of video streaming in virtual reality (VR) platforms: Meta's Workroom, Spatial, and Mozilla Hubs.

This proposal describes the design and demonstration of the Unlogical Instrument. This textile-based gesture-controlled sound interface invites the audience to generate sound by interacting with the textile using different gestures, such as poking, flicking, and patting. Unlogical is a statement about breaking the dogmatic approach of commonly used musical instruments. It brings resilience to the form of musical expression and interaction for the audience. It introduces an interactive mechanism mediated by intuitive human gestures and the curiosity to explore tangible material. Unlogical Instrument locates the focus on the textile. This artwork investigates the relationship between the gestures and the sound perception that originates from the textile through a material-driven approach. Inkjet printing is applied to transfer the original textile into a sensorial interface. The audience will listen to the sound while interacting with the textile surface, exploring and understanding this instrument through free-form playing.

With the advancement of virtual reality (VR) technology, virtual displays have become integral to how museums, galleries, and other tourist destinations present their collections to the public. However, the current lack of immersion in virtual reality displays limits the user's ability to experience and appreciate its aesthetics. This paper presents a case study of a creative approach taken by a tourist attraction venue in developing a physical network system that allows visitors to enhance VR's aesthetic aspects based on environmental parameters gathered by external sensors. Our system was collaboratively developed through interviews and sessions with twelve stakeholder groups interested in art and exhibitions. This paper demonstrates how our technological advancements in interaction, immersion and visual attractiveness surpass those of earlier virtual display generations. Through multimodal interaction, we aim to encourage innovation on the Web and create more visually appealing and engaging virtual displays. It is hoped that the greater online art community will gain fresh insight into how people interact with virtual worlds as a result of this work. © 2023 ACM.

Mobile augmented reality (MAR) applications are gaining popularity due to the wide adoption of mobile and especially wearable devices such as smartglasses. These devices often strike a compromise between mobility, energy efficiency, and performance. On the other hand, MAR applications rely on computationally intensive computer vision algorithms with extreme latency requirements. Cyber-foraging allows resource-constrained devices to leverage the computing power of nearby machines, and has often been proposed as a solution for increasing the computing capabilities of constrained devices. However, this process introduces new constraints in the application, especially in terms of latency and bandwidth. MAR applications are so demanding that current network infrastructures are barely ready for such traffic. Such resource-hungry applications may rapidly saturate future wireless networks such as 5G. As such, developing cyber-foraging solutions for MAR applications requires a delicate balance in network resource usage to ensure seamless user experience. In order to identify the opportunities for further improving the Quality of Experience (QoE) for mobile AR, we break down the end-to-end latency of the pipeline for typical MAR applications and pinpoint the dominating components in the critical path, from the physical transmission to the application. We then derive a set of actions to enhance the possibilities of cyber-foraging, taking into account the specificities of future networking technologies. © 2023, Springer Nature Switzerland AG.

Driven by the recent improvements in device and networks capabilities, Extended Reality (XR) is becoming more pervasive; industry and academia alike envision ambitious projects such as the metaverse. However, XR is still limited by the current architecture of mobile systems. This article makes the case for an XR-specific operating system (XROS). An XROS integrates hardware-support, computer vision algorithms, and XR-specific networking as the primitives supporting XR technology. These primitives represent the physical-digital world as a single shared resource among applications. Such an XROS allows for the development of coherent and system-wide interaction and display methods, systematic privacy preservation on sensor data, and performance improvement while simplifying application development. IEEE

The metaverse aims to blur the boundary between the physical world and digital content. To achieve this goal, the metaverse relies heavily on extended reality (XR), the Internet of Things, and communication technologies. Concurrently, connected vehicles and intelligent transportation systems (ITSs) are envisioned as the future paradigm of driving and becoming reality thanks to increasingly powerful onboard vehicular processing capacity and advanced vehicle-to-everything networking technologies. © 2005-2012 IEEE.

This paper presents "AI Nüshu," an emerging language system inspired by Nüshu (women's scripts), the unique language created and used exclusively by ancient Chinese women who were illiterate under a patriarchy society. Through an interactive art installation, two artificial intelligent (AI) agents continuously observe their environment and communicate with each other, developing a writing system that encodes Chinese. In this system, two AI agents observe the environment through cameras, record the unconscious behaviors of the audience, and generate summaries of their observations through visual recognition. Subsequently, the agent associates the corresponding original Nüshu poetry lines and generates new poetry text through a Language Model (LLM), representing its reflection. To develop their language, they continuously switch roles between the speaker and listener, constantly communicating their reflections, and encrypting a word in the poetry line with their self-created AI Nüshu character, allowing the other to guess and learn. Gradually, they reach a consensus on AI Nüshu, forming a unique "AI Nüshu Dictionary" for machines. This language, algorithmically combined into corresponding characters, has components derived from Nüshu, similar to Chinese characters and traditional textile patterns. Thus, like ancient women, the two agents gradually developed their Chinese writing system, corresponding one-to-one with Chinese characters. In contrast, humans, as the authority of the language system, became an object observed, interpreted, and inspired by machines to stimulate non-human language. This is the first media art project to interpret Nüshu from a computational linguistics perspective, infusing AI and art research with non-English natural language processing, Chinese cultural heritage, and a feminist viewpoint. This encourages the creation of more non-English, linguistically-oriented artworks for diverse cultures. We simulate communication in sisterhood through

WallStreetBets (WSB), a Reddit community, impacted stock markets during the 2021 GameStop Short Squeeze. We examine the content and user properties that influence engagement in WSB. Despite WSB's association with emojis and informal terms, engagement among community members depends on more than surface-level factors. Although emojis are commonly used, they are not as effective at fostering interactions among users. Community members engage more with posts that have longer and topic-specific text. Simply producing a high volume of posts is not enough to attract an audience. Consistent topical focus, reciprocal interactions, and previous authorship of catalyst posts influence engagement. WSB posts, regardless of length, generally remain relevant to the community's theme of stock trading. Our findings provide insights into WSB engagement patterns and can be useful for downstream research, such as financial predictive tasks using WSB data.

Interaction design for Augmented Reality (AR) is gaining attention from both academia and industry. This survey discusses 260 articles (68.8% of articles published between 2015-2019) to review the field of human interaction in connected cities with emphasis on augmented reality-driven interaction. We provide an overview of I luman-City Interaction and related technological approaches, followed by reviewing the latest trends of information visualization, constrained interfaces, and embodied interaction for AR headsets. We highlight under-explored issues in interface design and input techniques that warrant further research and conjecture that AR with complementary Conversational User Interfaces (CUIs) is a crucial enabler for ubiquitous interaction with immersive systems in smart cities. Our work helps researchers understand the current potential and future needs of AR in Human-City Interaction.

The General Data Protection Regulation (GDPR) presents directives to give data subjects control over their personal data. These directives impose data-collecting and processing organizations to take concrete actions for privacy preservation of users and non-users alike. Significant challenges arise when applying these directives to mobile and ubiquitous computing. Mobile and ubiquitous computing aim for computer use to be as transparent and seamless as possible. Inconspicuous devices continually sense their environment, often without the data subject's knowledge. This context significantly complicates the implementation of core GDPR directives, such as informing the user and collecting consent. In this paper, we challenge the mobile computing research community on how to address such issues in practical implementations that combine the philosophy of mobile and ubiquitous computing with often constraining privacy-regulations. © 2022 ACM.

The metaverse promises large-scale and persistent environments for users to share experiences at the intersection between physical and digital. Augmented reality (AR) is one of the primary technologies that supports blending digital content with physical environments. However, most AR applications are currently single-user and present significant constraints in terms of time and location. Scaling up the technology to large-sized environments and massively multi-user experiences thus represents one of the primary challenges to achieving such a vision. Due to these constraints, current proposals for the metaverse primarily consist of virtual worlds with little duality between physical and digital. This paper studies the feasibility of a large-scale and persistent AR experience shared among all visitors on our university campus. We define an integrated framework to enable the first AR campus metaverse by considering the nature of the various environments and the available sensing platforms. © 2022 IEEE.

In this paper, we highlight the use of Instagram for social activism, taking 2019 Hong Kong protests as a case study. Instagram focuses on image content and provides users with few features to share or repost, limiting information propagation. Nevertheless, users who are politically active offline also share their activism on Instagram. We first evaluate the effect of protests on social media activity for protesters and non-protesters over two significant protests. Protesters’ exposure to protest-related posts is much higher than non-protesters, and their network activity follows the protest schedule. They are also much more active on posts related to the protest that they participate in than the other protest. We then analyze the images posted by the users. Users predominantly use symbols related to protests and share personal thoughts on its primary actors. Users primarily share content to raise their network’s awareness, and the content choice is directly affected by Instagram’s intrinsic interaction modalities.

Connected vehicles, whether equipped with advanced driver-assistance systems or fully autonomous, require human driver supervision and are currently constrained to visual information in their line-of-sight. A cooperative perception system among vehicles increases their situational awareness by extending their perception range. Existing solutions focus on improving perspective transformation and fast information collection. However, such solutions fail to filter out large amounts of less relevant data and thus impose significant network and computation load. Moreover, presenting all this less relevant data can overwhelm the driver and thus actually hinder them. To address such issues, we present Augmented Informative Cooperative Perception (AICP), the first fast-filtering system which optimizes the informativeness of shared data at vehicles to improve the fused presentation. To this end, an informativeness maximization problem is presented for vehicles to select a subset of data to display to their drivers. Specifically, we propose (i) a dedicated system design with custom data structure and lightweight routing protocol for convenient data encapsulation, fast interpretation and transmission, and (ii) a comprehensive problem formulation and efficient fitness-based sorting algorithm to select the most valuable data to display at the application layer. We implement a proof-of-concept prototype of AICP with a bandwidth-hungry, latency-constrained real-life augmented reality application. The prototype adds only 12.6 milliseconds of latency to a current informativeness-unaware system. Next, we test the networking performance of AICP at scale and show that AICP effectively filters out less relevant packets and decreases the channel busy time.

The authors present the visual display installation as the basis for creating the artwork series 'Metanalysis'. The work explores the visual algorithmic representation of traditional Chinese ink painting and provides a video installation designed to showcase this visualization project. Exploring these visual representations is an attempt to understand and modernize expressions in the developing digital age visually. These visual displays are rendered in real-time and produced one after another in a continuous overlap and accumulation of black ink lines to reflect the specific random and malleable characteristics of the ink. They were eventually using mathematical algorithms to generate synthetic images that mimic the process of staining ink. The value of this paper lies in the fact that it proposes a new path for the expression of traditional Chinese ink painting and aims to bring the nonvitality classic art back to the stage through digital forms and to provide contemporary artists with new ideas to promote traditional culture. © 2022 IEEE.

Augmented Reality (AR) visits enhances the art exhibition experience by overlaying digital content. Although there has been significant interest in AR guides, few works leverage AR to bridge curatorial intent and audiences understanding. This paper focuses on integrating the curatorial intent within the AR overlays by developing the narrative layers established by the relationships between works. We develop a narrative system that identifies and links the primary art pieces of the exhibition within a digital story consistent with the curator's perspective. The system is applied to a physical exhibition composed of seven art pieces. We evaluate the impact of AR overlays through two user experiments, conducted on art professionals and general audience, respectively. Both groups considered that the AR tour system improved interactivity, self-reported learning, and user satisfaction significantly (> 4/5). Besides, visitors found the system easy to get to do what they want to (4.7/5), and would use it for future visits (4.6/5). This study raises essential design considerations towards designing integrated AR museum guides that combine the perspective of artists and curators towards a better visiting experience. © 2022 ACM.

Tai Chi is an essential concept of Chinese philosophy which refers to the universe's most primitive state of order. Tai Chi Chuan is a martial art and meditative practice that incorporates the Tai Chi philosophy. With the advent of the digital media age, this traditional martial art is falling into disuse, and most contemporary youths are losing interest. "Meditation in Motion" is an interactive media art installation inspired by the Tai Chi Chuan forms. It aims to convey the central concepts of balance, narrative, and universe of Tai Chi Chuan by breaking down its movements into overlapping circles, building a visual representation of the energy flows in the body.

The computational capabilities of recent mobile devices enable the processing of natural features for Augmented Reality (AR), but the scalability is still limited by the devices' computation power and available resources. In this paper, we propose EdgeXAR, a mobile AR framework that utilizes the advantages of edge computing through task offloading to support flexible camera-based AR interaction. We propose a hybrid tracking system for mobile devices that provides lightweight tracking with 6 Degrees of Freedom and hides the offloading latency from users' perception. A practical, reliable and unreliable communication mechanism is used to achieve fast response and consistency of crucial information. We also propose a multi-object image retrieval pipeline that executes fast and accurate image recognition tasks on the cloud and edge servers. Extensive experiments are carried out to evaluate the performance of EdgeXAR by building mobile AR apps upon it. Regarding the Quality of Experience (QoE), the mobile AR apps powered by EdgeXAR framework run on average at the speed of 30 frames per second with precise tracking of only 1-2 pixel errors and accurate image recognition of at least 97% accuracy. As compared to Vuforia, one of the leading commercial AR frameworks, EdgeXAR transmits 87% less data while providing a stable 30FPS performance and reducing the offloading latency by 50 to 70% depending on the transmission medium. Our work facilitates the large-scale deployment of AR as the next generation of ubiquitous interfaces.

Social media is often used to disseminate information during crises, including wars, natural disasters and pandemics. This paper discusses the challenges faced during crisis situations, which social media can both contribute to and ameliorate. We discuss the role that information polarisation plays in exacerbating problems. We then discuss how certain mal-actors exploit these divides. We conclude by detailing future avenues of work that can help mitigate these issues.

WallStreetBets (WSB) is a Reddit community that primarily discusses high-risk options and stock trading. In January 2021, it attracted worldwide attention as one of the epicentres of a significant short squeeze on US markets. Following this event, the number of users and their activity increased exponentially. In this paper, we study the changes caused in the WSB community by such an increase in activity. We perform a comparative analysis between long-term users and newcomers and examine their respective writing styles, topics, and susceptibility to community feedback. We report a significant difference in the post length and the number of emojis between the regular and new users joining WSB. Newer users’ activity also closely follows the affected companies’ stock prices. Finally, although community feedback affects the choices of topics for all users, new users are less prone to select their subsequent message topics based on past community feedback.

The COVID-19 pandemic led governments to rely on the versatility of social media to communicate with their citizens. This paper analyzes the Facebook communication of political leaders and health departments from 17 countries during the COVID-19 pandemic. We evaluate the citizen's response under the frameworks of media richness and user engagement. We note that governments and leaders communicate primarily through richer media (photos and videos), despite a negative correlation between media richness and user engagement. Plain-text messages posted by country leaders attract the most engagement, while their COVID-19 communication tends to generate lower engagement. On the other hand, health departments' pages experienced a sharp increase in engagement around COVID-19 communication as citizens sought information during the pandemic. Finally, topical analysis shows that Discussions and Co-Use of COVID-19 with other topics can boost citizens' engagement. Our comparative analysis shows practical implications for social media users and social media designers. Our findings can help governments and organizations design effective social media communication during crises.

Recent advances in information technology have revolutionized the automotive industry, paving the way for next-generation smart vehicular mobility. Specifically, vehicles, roadside units, and other road users can collaborate to deliver novel services and applications that leverage, for example, big vehicular data and machine learning. Relatedly, fifth-generation cellular networks (5G) are being developed and deployed for low-latency, high-reliability, and high bandwidth communications. While 5G adjacent technologies such as edge computing allow for data offloading and computation at the edge of the network thus ensuring even lower latency and context-awareness. Overall, these developments provide a rich ecosystem for the evolution of vehicular applications, communications, and computing. Therefore in this work, we aim at providing a comprehensive overview of the state of research on vehicular computing in the emerging age of 5G and big data. In particular, this paper highlights several vehicular applications, investigates their requirements, details the enabling communication technologies and computing paradigms, and studies data analytics pipelines and the integration of these enabling technologies in response to application requirements. © 2013 IEEE.

Urban events are the sets of real-world occurrences in urban space associated with specific topics. Discovering, understanding, and forecasting various urban events are essential to facilitate smart-city applications and benefit urban life. In recent years, mobile devices and sensors widely located in cities have collected large amounts of data produced in urban space, which form a large-scale, cross-domain, and multi-view data ecosystem. The collected data, termed urban big data, provide an unprecedented opportunity to discover and analyze urban events. In this thesis, we develop data-driven methodologies for urban events mining. We present four of our works on urban anomaly detection, urban location representation learning, road network traffic prediction, and urban events prediction, respectively. First, we introduce an urban dynamic decomposition method to detect urban anomalies. We propose a neural network model to estimate normal urban dynamics using spatiotemporal features and detect abnormal events via residual analysis. We validate the effectiveness of the proposed method with both synthetic and real-world event datasets. Regions are the primary spatial units for urban event analysis. The second work focuses on learning an embedding space from urban big data for urban regions. We propose to construct multi-view relations among urban regions and learn urban region embeddings through a multi-view joint learning model. Experiments on real-world datasets prove the effectiveness of the learned embeddings on downstream applications, including predicting long-term regional event statistics such as crime rates. Traffic prediction is a crucial task for congestion events diagnosis and control. In the third work, we propose a traffic prediction model called Adaptive Spatiotemporal Convolution Network, which models complicated spatiotemporal traffic dependencies adaptively with awareness of the real-time traffic conditions. Experiments on real-world traffic datasets fr

The modern urban landscape is becoming increasingly connected, from distributed vehicular safety systems to in-vehicle centralized entertainment. These technologies face network challenges related to the high mobility of vehicles and people. Such mobility raises new challenges. On-board detection of unsafe driving activities lacks a holistic view of the road situation while cloud-offloaded detection faces scalability and context-awareness issues. In terms of in-vehicle entertainment, the connection to remote gaming and VR servers encounters variable bandwidth, latency, and packet losses that affect the gaming and VR experience. This thesis presents four research contributions to provide reliable real-time content streaming for driving awareness (in the physical world), mobile cloud gaming (in the virtual world), and a reality-check of the Metaverse between physical and virtual worlds. The first contribution is CAD3, a distributed collaborative architecture for road-aware and driver-aware anomaly driving detection and real-time warning dissemination. CAD3 exploits the pervasive deployment of roadside units, and combines collaborative and context-aware computation with low-latency communication to detect unsafe driving behaviors and warn the drivers of nearby vehicles in real-time. The second contribution is Nebula, an end-to-end cloud gaming architecture to minimize the impact of network conditions on the user experience. Nebula relies on a heuristic algorithm and end-to-end distortion model to dynamically adapt the video bitrate and redundancy based on the measured network conditions. The third contribution is MERA, an edge-assisted learning-based end-to-end cloud gaming architecture to adapt the video bitrate to the network constraints. MERA relies on the transition, state-to-action mapping, then rewarding with a multi-objective reward function to maximize the user QoE. The fourth contribution is Metaversity, a reality check of social VR platforms. It uncovers the