Back to Search Start Over

Instant Reality: Gaze-Contingent Perceptual Optimization for 3D Virtual Reality Streaming.

Authors :
Chen S
Duinkharjav B
Sun X
Wei LY
Petrangeli S
Echevarria J
Silva C
Sun Q
Source :
IEEE transactions on visualization and computer graphics [IEEE Trans Vis Comput Graph] 2022 May; Vol. 28 (5), pp. 2157-2167. Date of Electronic Publication: 2022 Apr 08.
Publication Year :
2022

Abstract

Media streaming, with an edge-cloud setting, has been adopted for a variety of applications such as entertainment, visualization, and design. Unlike video/audio streaming where the content is usually consumed passively, virtual reality applications require 3D assets stored on the edge to facilitate frequent edge-side interactions such as object manipulation and viewpoint movement. Compared to audio and video streaming, 3D asset streaming often requires larger data sizes and yet lower latency to ensure sufficient rendering quality, resolution, and latency for perceptual comfort. Thus, streaming 3D assets faces remarkably additional than streaming audios/videos, and existing solutions often suffer from long loading time or limited quality. To address this challenge, we propose a perceptually-optimized progressive 3D streaming method for spatial quality and temporal consistency in immersive interactions. On the cloud-side, our main idea is to estimate perceptual importance in 2D image space based on user gaze behaviors, including where they are looking and how their eyes move. The estimated importance is then mapped to 3D object space for scheduling the streaming priorities for edge-side rendering. Since this computational pipeline could be heavy, we also develop a simple neural network to accelerate the cloud-side scheduling process. We evaluate our method via subjective studies and objective analysis under varying network conditions (from 3G to 5G) and edge devices (HMD and traditional displays), and demonstrate better visual quality and temporal consistency than alternative solutions.

Details

Language :
English
ISSN :
1941-0506
Volume :
28
Issue :
5
Database :
MEDLINE
Journal :
IEEE transactions on visualization and computer graphics
Publication Type :
Academic Journal
Accession number :
35148266
Full Text :
https://doi.org/10.1109/TVCG.2022.3150522