Search

With the rapid development of Internet and multimedia services in the past decade, a huge amount of user-generated and service provider-generated multimedia data become available. These data are heterogeneous and multi-modal in nature, imposing great challenges for processing and analyzing them. Multi-modal data consist of a mixture of various types of data from different modalities such as texts, images, videos, audios etc. In this article, we present a deep and comprehensive overview for multi-modal analysis in multimedia. We introduce two scientific research problems, data-driven correlational representation and knowledge-guided fusion for multimedia analysis. To address the two scientific problems, we investigate them from the following aspects: 1) multi-modal correlational representation: multi-modal fusion of data across different modalities, and 2) multi-modal data and knowledge fusion: multi-modal fusion of data with domain knowledge. More specifically, on data-driven correlational representation, we highlight three important categories of methods, such as multi-modal deep representation, multi-modal transfer learning, and multi-modal hashing. On knowledge-guided fusion, we discuss the approaches for fusing knowledge with data and four exemplar applications that require various kinds of domain knowledge, including multi-modal visual question answering, multi-modal video summarization, multi-modal visual pattern mining and multi-modal recommendation. Finally, we bring forward our insights and future research directions. [ABSTRACT FROM AUTHOR]

Recently, sponsored data started to gain wide-spread use in wireless networks. Existing approaches for regular data services when applying to video transmission in sponsored data faces tremendous challenges. In two-sided markets, internal and external competitions among service providers (SPs), content provider (CPs), and end-users (EUs) become more complex, because larger video-bandwidth significantly increases the cost. When the three parties benefit from sponsored data, pricing and transmission design become much more complicated and difficult in parameterized systems by profit decomposing models, because maximizing the total profits among the three parties is a NP-hard problem. In this paper, we address the problem of maximizing the total profits for sponsored video data. We propose a submodular-based optimization framework for wireless video transmission systems. First, we formulate the sponsored data models in wireless video transmission respective to SP, CPs and EUs sides. The proposed profit models are used in wireless broadcasting networks with a specific attention to sponsored video data. Second, we construct a layered video transmission model in two-sided market through incorporating submodularity. We approximate the NP-hard layered-video broadcasting problem by applying submodular theory. Third, we present a profit maximization framework that optimizes the number of sponsored users, pricing and bandwidth provision. The profit of SP is maximized as well as the profit of CPs is improved and the cost-effective QoE of EUs is guaranteed. To verify the efficiency, we perform the experiments for different pricing schemes. The simulation results demonstrate that the proposed method significantly outperforms existing approaches because the proposed method can achieve maximum profits of both SP and CPs in a wide range of transmission rates. [ABSTRACT FROM AUTHOR]

Owing to the rich emerging multimedia applications and services in the past decade, super large amount of multimedia data has been produced for the purpose of advanced research in multimedia. Furthermore, multimedia research has made great progress on image/video content analysis, multimedia search and recommendation, multimedia streaming, multimedia content delivery etc. At the same time, Artificial Intelligence (AI) has undergone a “new” wave of development since being officially regarded as an academic discipline in 1950s, which should give credits to the extreme success of deep learning. Thus, one question naturally arises: What happens when multimedia meets Artificial Intelligence? To answer this question, we introduce the concept of Multimedia Intelligence through investigating the mutual-influence between multimedia and Artificial Intelligence. We explore the mutual influences between multimedia and Artificial Intelligence from two aspects: i) multimedia drives Artificial Intelligence to experience a paradigm shift towards more explainability and ii) Artificial Intelligence in turn injects new ways of thinking for multimedia research. As such, these two aspects form a loop in which multimedia and Artificial Intelligence interactively enhance each other. In this paper, we discuss what and how efforts have been done in literature and share our insights on research directions that deserve further study to produce potentially profound impact on multimedia intelligence. [ABSTRACT FROM AUTHOR]

Multimedia now accounts for the largest share of all Internet traffic, highlighted by its volume, variety, multicast nature, and QoS constraints. Downstream toward consumers, multimedia traffic can traverse through middleboxes, undergoing additional data processing imposed by content providers and distributors. With the advent of network function virtualization (NFV), middleboxes are progressively embedded in off-the-shelf, general-purpose servers. Despite the benefits, NFV can incur an undue amount of energy consumption during high packet forwarding. In this article, the authors investigate how switching to state-of-the-art NFV products for multimedia content delivery can result in significant energy costs. They identify energy inefficiency in the NFV data plane, which can be exacerbated if not handled properly. They outline a power management framework that considers characteristics of multimedia traffic and exploits CPU frequency scaling to save energy. This article is part of a special issue on advancing multimedia distribution. [ABSTRACT FROM PUBLISHER]

Today’s Internet has witnessed an increase in the popularity of mobile video streaming, which is expected to exceed 3/4 of the global mobile data traffic by 2019. To satisfy the considerable amount of mobile video requests, video service providers have been pushing their content delivery infrastructure to edge networks—from regional content delivery network (CDN) servers to peer CDN servers (e.g., smartrouters in users’ homes)—to cache content and serve users with storage and network resources nearby. Among the edge network content caching paradigms, Wi-Fi access point caching and cellular base station caching have become two mainstream solutions. Thus, understanding the effectiveness and performance of these solutions for large-scale mobile video delivery is important. However, the characteristics and request patterns of mobile video streaming are unclear in practical wireless network. In this paper, we use real-world data sets containing 50 million trace items of nearly 2 million users viewing more than 0.3 million unique videos using mobile devices in a metropolis in China over two weeks, not only to understand the request patterns and user behaviors in mobile video streaming, but also to evaluate the effectiveness of Wi-Fi and cellular-based edge content caching solutions. To understand the performance of edge content caching for mobile video streaming, we first present temporal and spatial video request patterns, and we analyze their impacts on caching performance using frequency-domain and entropy analysis approaches. We then study the behaviors of mobile video users, including their mobility and geographical migration behaviors, which determine the request patterns. Using trace-driven experiments, we compare strategies for edge content caching, including least recently used (LRU) and least frequently used (LFU), in terms of supporting mobile video requests. We reveal that content, location, and mobility factors all affect edge content caching performance. Moreover, we design an efficient caching strategy based on the measurement insights and experimentally evaluate its performance. The results show that our design significantly improves the cache hit rate by up to 30% compared with LRU/LFU. [ABSTRACT FROM PUBLISHER]

Group recommendation plays a significant role in today's social media systems, where users form social groups to receive multimedia content together and interact with each other, instead of consuming the online content individually. Limitations of traditional group recommendation approaches are as follows. First, they usually infer group members’ preferences by their historical behaviors, failing to capture inactive users’ preferences from the sparse historical data. Second, relationships between group members are not studied by these approaches, which fail to capture the inherent personality of members in a group. To address these issues, we propose a social-aware group recommendation framework that jointly utilizes both social relationships and social behaviors to not only infer a group's preference, but also model the tolerance and altruism characteristics of group members. Based on the observation that the following relationship in the online social network reflects common interests of users, we propose a group preference model based on external experts of group members. Furthermore, we model users’ tolerance (willingness to receive content not preferred) and altruism (willingness to receive content preferred by friends). Finally, based on the group preference model, we design recommendation algorithms for users under different social contexts. Experimental results demonstrate the effectiveness of our approach, which significantly improves the recommendation accuracy against traditional approaches, especially in the cases of inactive group members. [ABSTRACT FROM PUBLISHER]

Batteries of modern mobile devices remain severely limited in capacity, which makes energy consumption a key concern for mobile applications, particularly for the computation-intensive video applications. Mobile devices can save energy by offloading computation tasks to the cloud, yet the energy gain must exceed the additional communication cost for cloud migration to be beneficial. The situation is further complicated by real-time video applications that have stringent delay and bandwidth constraints. In this paper, we closely examine the performance and energy efficiency of representative mobile cloud applications under dynamic wireless network channels and state-of-the-art mobile platforms. We identify the unique challenges of and opportunities for offloading real-time video applications and develop a generic model for energy-efficient computation offloading accordingly in this context. We propose a scheduling algorithm that makes adaptive offloading decisions in fine granularity in dynamic wireless network conditions and verify its effectiveness through trace-driven simulations. We further present case studies with advanced mobile platforms and practical applications to demonstrate the superiority of our solution and the substantial gain of our approach over baseline approaches. [ABSTRACT FROM PUBLISHER]

We have recently witnessed the convergence of online social network services and online video services: users import videos from content sharing sites, and propagate them along the social connections by re-sharing them. Such social behaviors have dramatically reshaped how videos are disseminated, and users are now actively engaged as part of the social ecosystem, rather than being passive consumers. Despite the increasingly abundant bandwidth and computation resources, the ever increasing data volume of user generated video content and the boundless coverage of socialized sharing have presented unprecedented challenges. This article first presents the challenges in social-aware video delivery. Then we present a principal framework for social-aware video delivery and a state-of-the-art overview of approaches. Moreover, we identify the unique characteristics of social-aware video access and social content propagation, and reveal the design of individual modules and their integration toward enhancing users' experience in the social network context. Finally, we discuss future research directions. [ABSTRACT FROM PUBLISHER]

The increasing popularity of online social networks (OSNs) has been transforming the dissemination pattern of social video contents. We can utilize the social information propagation pattern to improve the efficiency of social video distribution. In this paper, motivated by the social community classification, we present a social video replication and user request dispatching mechanism in the cloud content delivery network architecture to reduce the system operational cost, while guaranteeing the averaged service latency. Specifically, we first present a community classification method that clusters social users with social relationships, close geolocations, and similar video watching interests into various communities. Then, we conduct a large-scale measurement on a real OSN system to study the diversities of social video propagation and the effectiveness of our communities on smoothing the diversity. Finally, we propose the community-based video replication and request dispatching strategy and formulate it as a constrained optimization problem. Based on a stochastic optimization framework, we derive an online solution and rigorously prove the optimality. We evaluate our algorithm on a real trace under realistic settings and demonstrate that our algorithm can reduce the monetary cost by 30% against traditional approaches with the same service latency. [ABSTRACT FROM AUTHOR]

Instant social video sharing which combines the online social network and user-generated short video streaming services, has become popular in today’s Internet. Cloud-based hosting of such instant social video contents has become a norm to serve the increasing users with user-generated contents. A fundamental problem of cloud-based social video sharing service is that users are located globally, who cannot be served with good service quality with a single cloud provider. In this paper, we investigate the feasibility of dispersing instant social video contents to multiple cloud providers. The challenge is that inter-cloud social propagation is indispensable with such multi-cloud social video hosting, yet such inter-cloud traffic incurs substantial operational cost. We analyze and formulate the multi-cloud hosting of an instant social video system as an optimization problem. We conduct large-scale measurement studies to show the characteristics of instant social video deployment, and demonstrate the trade-off between satisfying users with their ideal cloud providers, and reducing the inter-cloud data propagation. Our measurement insights of the social propagation allow us to propose a heuristic algorithm with acceptable complexity to solve the optimization problem, by partitioning a propagation-weighted social graph in two phases: a preference-aware initial cloud provider selection and a propagation-aware re-hosting. Our simulation experiments driven by real-world social network traces show the superiority of our design. [ABSTRACT FROM PUBLISHER]

Video transcoding in an adaptive bitrate streaming (ABR) system is demanded to support video streaming over heterogenous devices and varying networks. However, it could incur a tremendous cost. Meanwhile, most viewers terminate viewing sessions within 20% of their durations; only a small fraction of each video is consumed. Built upon this user viewing pattern, we propose a Partial Transcoding Scheme for content management in media clouds. Particularly, each content is encoded into different bitrates and split into segments. Some of the segments are stored in cache, resulting in storage cost; others are transcoded online in the case of cache miss, resulting in computing cost. We aim to minimize the long-term overall cost by determining whether a segment should be cached or transcoded online. We formulate it as a constrained stochastic optimization problem. Leveraging Lyapunov optimization framework and Lagrangian relaxation, we design an online algorithm which can achieve the optimal solution within provable upper bounds. Experiments demonstrate that our proposed method can reduce 30% of operational cost, compared with the scheme of caching all the segments. [ABSTRACT FROM PUBLISHER]

Dynamic adaptive streaming has emerged as a popular approach for video services in today’s Internet. To date, the two important components in dynamic adaptive streaming, video transcoding that generates the adaptive bitrates of a video and video delivery that streams the videos to users, have been separately studied, resulting in a huge waste of computation and storage resource due to producing and caching different versions of videos regardless of their demands. We conduct extensive measurement studies of video sharing systems, including an IPTV service which streams regular, professionally made videos and an instant video clip sharing service which provides extremely short user-generated videos, as well as the availability of computation resource in conventional content delivery networks (CDNs). Based on the measurement insights, we propose an online joint transcoding and delivery approach for adaptive video streaming. We formulate optimization problems to enable high streaming quality for the users, and low computation and replication costs for the system. In particular, our strategy connects video transcoding and video delivery based on users’ preferences of CDN regions and regional preferences of video versions. We analyze hardness of these problems and design distributed solutions. Extensive trace-driven experiments further demonstrate the superiority of our design. [ABSTRACT FROM PUBLISHER]

There is an unprecedented trend that content providers (CPs) are building their own content delivery networks (CDNs) to provide a variety of content services to their users. By exploiting powerful CP-level information in content distribution, these CP-built CDNs open up a whole new design space and are changing the content delivery landscape. In this paper, we adopt a measurement-based approach to understanding why, how, and how much CP-level intelligences can help content delivery. We first present a measurement study of the CDN built by Tencent, a largest content provider based in China. We observe new characteristics and trends in content delivery which pose great challenges to the conventional content delivery paradigm and motivate the proposal of CPCDN, a CDN powered by CP-aware information. We then reveal the benefits obtained by exploiting two indispensable CP-level intelligences, namely context intelligence and user intelligence, in content delivery. Inspired by the insights learnt from the measurement studies, we systematically explore the design space of CPCDN and present the novel architecture and algorithms to address the new content delivery challenges that have arisen. Our results not only demonstrate the potential of CPCDN in pushing content delivery performance to the next level, but also identify new research problems calling for further investigation. [ABSTRACT FROM PUBLISHER]

Video delivery in television white space (TVWS) is a challenging task due to the variation of the spectrum availability in TVWS and the stringent bandwidth requirements of video streaming. Most existing frameworks employ an opportunistic spectrum access style which result in bandwidth inefficiency since only one spectrum hole in TVWS can be used. In this paper, we propose a bandwidth efficient TVWS video delivery framework. First, a multisector structure is employed to improve the bandwidth efficiency. Second, a novel spectrum sensing scheme and an efficient spectrum selection algorithm are proposed to obtain the optimal spectrum holes based on this structure. Third, a joint optimization approach is proposed to create an optimal map between the sectors and the spectrum, and to adapt the video rate commensurate with dynamic channel conditions. The simulation results demonstrate that the proposed framework can obtain significantly higher bandwidth efficiency and superior video performance than traditional approaches. [ABSTRACT FROM AUTHOR]