Search

Multimedia data analysis is at the heart of multimedia research, where the patterns, information, and knowledge are mined and recognized from the multimedia data for various applications. We have witnessed significant performance improvements in multimedia data analysis driven by the recent advances in artificial intelligence and deep learning. While most of the existing methods focus on analyzing unimodal data, many novel multimodal multimedia data analysis techniques have also been developed and achieved better performance than those using unimodal data. Multimodal multimedia data has proven to be effective to improve the performance and reliability of data analysis systems and applications. It is also important to match and retrieve data with similar contents across modalities. However, due to the highly complex nature of multimodal multimedia data, further research in multimodal multimedia data analysis is needed to address the challenges including missing modalities, lack of benchmark datasets, and so on.

Multimedia is one of the key drivers improving virtual reality and augmented reality (VR/AR), which are promising to reform human–computer interaction in the future with lower-cost and all-in-one headsets containing powerful hardware. Advances in multimedia research on video compression and human–computer interfaces have further enhanced the immersion and efficiency of experiences on the platform. However, many VR/AR experiences are still very difficult to build using traditional engineering methods and many available behavioral and biometric data have not been well explored. Further research in the multimedia community is needed to enhance the usefulness of these systems, with potential in affective learning, resource generation, and developer tools.

Multimedia has played an indispensable role in the success of various real-world applications, from smart healthcare to intelligent surveillance systems. In this new era of technology, one of these essential and useful applications is autonomous driving or self-driving cars. The idea of driverless vehicles was introduced many years ago when autopilot systems were designed for airplanes. However, over the past few years, there have been incredible advances in autonomous driving, thanks to new technologies in multimedia and artificial intelligence. In particular, multimedia research using a broad range of techniques such as machine learning, computer vision, audio processing, simulation systems, robotics, and big data has significantly enhanced and expedited the autonomous driving technology over time. In addition to the main applications of self-driving cars such as autonomous delivery vehicles, taxis, and freight trucks, the autonomous driving technology can provide many opportunities in military, constructions, healthcare, and emergency management. Despite the effectiveness and efficiency that self-driving cars bring to the future of transportation, many challenges remain that need to be solved before fully-autonomous driving becomes a reality.

The field of multimedia covers a broad range of research and technologies that aim to develop solutions for individual disciplines as well as multidisciplinary domains. From the beginning, multimedia research has employed Artificial Intelligence (AI) techniques to address various challenges in this area. Specifically, AI has been extensively used for multimedia retrieval, management, and analysis. Over the past few years, multimedia has continued to utilize AI to bring revolutionary advancement in numerous applications and domains, mainly thanks to the recent progress in machine learning algorithms and computing powers. However, many challenges remain, which calls for the need to extend Multimedia AI to tackle these problems and to lead to great opportunities in the near future.

By achieving breakthrough results on domains such as speech recognition, natural language processing, and computer vision, it is no surprise that deep neural networks are receiving a lot of attention these days. Specifically in the field of multimedia data analysis, there is a tremendous amount of multimedia big data that is being generated every day. Deep learning has the potential to overcome the issue of multimedia data having massive and heterogeneous characteristics that make it a challenge to store and analyze the data. This can be accomplished by allowing computers to easily and automatically extract features from unstructured data without the need to rely on human intervention. Although recent multimedia deep learning methods have achieved some remarkable results, deep learning challenges such as interpretability and generalization still make it difficult to be fit for critical decision-making tasks from fields such as medicine and defense.

Coronavirus Disease 2019 (COVID-19) has been affecting most of the countries and impacting almost every aspect of people's lives. More than one hundred million confirmed cases and two million deaths have been reported due to COVID-19 as of February 2021. While our society suffers an unanticipated epidemic, researchers and engineers have developed various technologies to manage this global emergency. Specifically, multimedia tools, techniques, and applications have been developed and played essential roles in facilitating the recovery, resilience, and management of COVID-19, including pandemic status monitoring and impact prediction, enhancing public awareness and telehealth, etc. However, there are many challenges that require further investigation and research to better manage COVID-19 and prepare for future pandemics.

In today's digital world, with the exponential growth of data, new approaches to aggregate and analyze data are bringing considerable benefits to many fields, such as healthcare, Internet of Things, social media, business, and public policy. Data science (DS) is considered as an interdisciplinary field that covers how data is prepared, analyzed, interpreted, modeled, and presented. It is a combination of data analytics, machine learning, math, and statistics, as well as domain and business knowledge. One of the main goals of DS is to leverage Big Data technologies with an adept analysis to obtain as much information as possible from the data and facilitate the decision-making process. Many research areas such as medicine and astrophysics have heavily utilized DS, usually focusing on structured scientific data. Using DS, the scientist can obtain a better understanding of the data and conduct a more precise analysis. In addition, DS has become a crucial foundation for artificial intelligence based on the right mix of machine learning and domain knowledge and continued to impact all aspects of life, through the discovery of new knowledge and hidden meaning within the data.

The global cost of natural disasters in 2017 increased by almost twice from the ones in 2016. In particular, disastrous events such as wildfires, flood, hurricanes, and earthquakes caused the economic loss of $306 billion in 2017, which is much higher than the average 10-year loss of $190 billion. Due to these significant devastations and losses, there is an urgent need for the development of disaster information management systems.

Presents the introductory editorial for this issue of the publication. Also reports on the impact factor associated with this publication.

A multimedia semantic retrieval system based on hidden coherent feature groups (HCFGs) can support multimedia semantic retrieval on mobile applications. The system can capture the correlation between features and partition the original feature set into HCFGs, which have strong intragroup correlation while maintaining low intercorrelation. The authors present a novel, multimodel fusion scheme to effectively fuse the multimodel results and generate the final ranked retrieval results. In addition, to incorporate user interaction for effective retrieval, the proposed system also features a user feedback mechanism that helps refine the retrieval results.

I am humbled by this great honor to be the new editor in chief (EIC) of IEEE MultiMedia. With this opportunity, I would like to share my thoughts and visions on how to move IEEE MultiMedia forward.

This special issue is a collaboration between the 2014 IEEE International Symposium on Multimedia (ISM 2014) and IEEE MultiMedia. For over a decade, ISM has been an internationally renowned forum for researchers and practitioners to develop solutions and exchange ideas in emerging multimedia research and applications. The articles in this issue are extended versions of the top ISM 2014 papers on multimedia research.

The motivation for organizing the First IEEE International Conference on Multimedia Big Data (BigMM 2015) was the proliferation of multimedia data and ever-growing requests for multimedia applications, which have made multimedia the "biggest big data" and an important source of insights and information. This conference report provides a brief overview of the keynotes, presentations, panels, summit, grand challenge, and workshops.

The rapid increase in the amount of digital data has posed significant challenges to retrieval systems, which are expected to effectively and efficiently filter the data and provide relevant results. Filtering is a technique that automatically selects features or data instances to represent the data, reduce storage costs, prune redundancy, decrease computation costs, enhance model learning performance, and dynamically deliver the media. On the other hand, ranking is a critical step to render as many relevant results as possible on the basis of some loss functions or similarity measures. This article presents a novel retrieval framework that consists of weighted subspace-based filtering and ranking components to address these challenges. One critical issue in multimedia retrieval is the difficulty of comparing semantics between low-level features (such as color and shape) and high-level concepts (such as sky and ocean). Because multiple correspondence analysis (MCA) is powerful in exploring relationships among high-dimensional categorical variables, it is attractive to apply MCA to explore relationships between feature categories and concept classes. However, existing correlation based algorithms are weak in capturing semantics through a numeric measure such as conditional entropy to guide the search of features. Most existing subspace-based algorithms, while considering semantics, mainly focus on the representations of features in new spaces or relationships among high-level concepts. Therefore, our filtering algorithm transfers the semantic correlation captured in the subspace to the feature weights to distinguish features and imbalanced data instances. Relevance of retrieval results is also crucial in multimedia, where effectiveness is measured by the ranking process. Our proposed algorithm is designed to overcome the limitation of the most commonly used ranking methods, which treat all data instances equally within one group during the training step. Our algorithm ranks data instances using dissimilarity values in the subspace toward both positive and nega tive one-class models from the learning phase. The idea behind this is that even those data instances in the same relevant group may have implicit different significances. For exam ple, the data instances closer to the relevant group's center could indicate more relevance than those that are further from the center. Meanwhile, a low-dimensional representation of the data instances could compactly characterize the structure of the data instance groups, making it possible to achieve an easy separation of relevant and irrelevant groups and effectively rank the retrieved results. To demonstrate the effectiveness of the proposed framework, we evaluated 30 high-level concepts and data sets from Trecvid 2008 and 2009.

Recent advances in pervasive computing and the proliferation of multimedia-capable devices have stimulated the development of intelligent and pervasive multimedia applications. This special issue provides excellent coverage of this area, including interactive multimedia education, quality control and personalization of multimedia services, peer-to-peer multimedia streaming, mobile TV, and VoIP systems.