Search

Building a retrieval system with lifelogging data is more complicated than with ordinary data due to the redundancies, blurriness, massive amount of data, various sources of information accompanying lifelogging data, and especially the ad-hoc nature of queries. The Lifelog Search Challenge (LSC) is a benchmarking challenge that encourages researchers and developers to push the boundaries in lifelog retrieval. For LSC'22, we develop FIRST 3.0, a novel and flexible system that leverages expressive cross-domain embeddings to enhance the searching process. Our system aims to adaptively capture the semantics of an image at different levels of detail. We also propose to augment our system with an external search engine to help our system with initial visual examples for unfamiliar concepts. Finally, we organize image data in hierarchical clusters based on their visual similarity and location to assist users in data exploration. Experiments show that our system is both fast and effective in handling various retrieval scenarios.

Cross-data analytics and retrieval have gained significant improvement recently. People can now extract more data insights precisely and quickly towards having many excellent applications serving human lives. Since people create multimedia and other types of data that reflect the diverse perspectives of human lives, these data are just pieces of the puzzle of the world's pictures. Hence, it is necessary to assembly all these pieces towards having a better solution for human-centered problems. Hence, the workshop welcomes those who work with multimedia and others and come from diverse research domains and disciplines to work on intelligent cross-data analytics and retrieval to bring a smart, sustainable society to human beings. The research domain can vary from well-being, disaster prevention and mitigation, mobility to food computing, to name a few.

In this paper, we present the interactive lifelog retrieval engine developed for the LSC'21 comparative benchmarking challenge. The LifeSeeker 3.0 interactive lifelog retrieval engine is an enhanced version of our previous system participating in LSC'20 - LifeSeeker 2.0. The system is developed by both Dublin City University and the Ho Chi Minh City University of Science. The implementation of LifeSeeker 3.0 focuses on searching and filtering by text query using a weighted Bag-of-Words model with visual concept augmentation and three weighted vocabularies. The visual similarity search is improved using a bag of local convolutional features; while improving the previous version's performance, enhancing query processing time, result displaying, and browsing support.

With a huge collection of photos and video clips, it is essential to provide an efficient and easy-to-use system for users to retrieve moments of interest with a wide variation of query types. This motivates us to develop and upgrade our flexible interactive retrieval system for visual lifelog exploration. In this paper, we briefly introduce version 2 of our system with the following main features. Our system supports multiple modalities for interaction and query processing, including visual query by meta-data, text query and visual information matching based on a joint embedding model, scene clustering based on visual and location information, flexible temporal event navigation, and query expansion with visual examples. With the flexibility in system architecture, we expect our system can easily integrate new modules to enhance its functionalities.

It is essential to develop efficient methods to detect abnormal events, such as car-crashes or stalled vehicles, from surveillance cameras to provide in-time help. This motivates us to propose a novel method to detect traffic accidents in traffic videos. To tackle the problem where anomalies only occupy a small amount of data, we propose a semi-supervised method using Generative Adversarial Network trained on regular sequences to predict future frames. Our key idea is to model the ordinary world with a generative model, then compare a predicted frame with the real next frame to determine if an abnormal event occurs. We also propose a new idea of encoding motion descriptors and scaled intensity loss function to optimize GAN for fast-moving objects. Experiments on the Traffic Anomaly Detection dataset of AI City Challenge 2019 show that our method achieves the top 3 results with F1 score 0.9412 and RMSE 4.8088, and S3 score 0.9261. Our method can be applied to different related applications of anomaly and outlier detection in videos.

Inspired by recent advances in computer vision and deep learning, we propose new enhancements to tackle problems appearing in endoscopic image analysis, especially abnormality finding and anatomical landmark detection. In details, a combination of Residual Neural Network and Faster R-CNN are jointly applied in order to take all of their advantages and improve the overall performance. Nevertheless, novel data augmentation is designed and adapted to corresponding domains. Our approaches prove their competitive results in term of not only the accuracy but also the inference time in Medico: The 2018 Multimedia for Medicine Task and The Biomedia ACM MM Grand Challenge 2019. These results show the great potential of the collaborating between deep learning models and data augmentation in medical image analysis applications. Especially, more than 4900 bounding boxes localizing the symptom of some classes from KVASIR dataset that we annotated and used in this project are shared online for future research.

Lifelogging is a new trend of research in which we collect and analyze people's daily activities to gain insight into people daily basis and enhance people's life quality. This leads to many research in retrieval systems to be developed for lifelogging analysis. One of the most important factors of a successful lifelogging retrieval system is to enhance the user's ability to perform any queries, even arbitrary ones. This motivates our proposal of a retrieval system in which we focus on the two main properties. The first property is the ability to accurately annotate the lifelog dataset with metadata that is meaningful for the retrieving process. The second property focuses on a powerful user interface that supports a novice user to perform the retrieval efficiently.

Many categories of the illusion of self-motion have been widely studied with the potential support of virtual reality. However, the effects of directly and indirectly powering the movement on the possibility of perceiving changes in moving speed and their relationship with sensory feedback on users' speed change perception have not been investigated before. In this paper, we present the results of our user study on the difference in perceiving changes in moving speed between two different movement techniques: "pedaling" and "throttling". We also explore the effects of different velocity gains, accelerations and speeds of airflow, and their interactions with the movement techniques on users' perception of speed changes in addition to user performance and perception. We built a bike simulator that supports both of the movement techniques and provides sensory feedback. In general, "pedaling" gave users more possibility to perceive changes in moving velocity than "throttling".

Lifelogging data provides useful insight understanding about our lives during daily activities. Thus, it is essential to develop a system to assist users to retrieve events or memories from lifelogging data from ad-hoc text queries. In this paper, we first propose a method to process lifelogging data by grouping images into visual shots and clusters, then extract semantic concepts on scene category and attributes, entities, and actions. We then develop a query system that supports 4 main types of query conditions: temporal, spatial, entity and action, and extra data criteria. Our system is expected to efficiently assist users to search for past moments in daily logs.

News classification is among essential needs for people to organize, better understand, and utilize information from the Internet. This motivates the authors to propose a novel method to classify news from social media. First, we propose to vectorize an article with TD2V, our pre-trained Twitter-based universal document representation following Doc2Vec approach. We then define Modified Distance to better measure the semantic distance between two document vectors. Finally, we apply retrieval and automatic query expansion to get the most relevant labeled documents in a corpus to determine the category for a new article. As our TD2V is created from 297 million words in 420,351 news articles from more than one million tweets in Twitters from 2010 to 2017, it can be used as one of the efficient pre-trained models for English document representation in various applications. Experiments on datasets from different online sources show that our method achieves the classification accuracy better than existing methods, specifically 98.4±0.3% (BBC dataset), 98.9±0.7% (BBC Sport dataset), 94.1±0.2% (Amazon4 dataset), and 78.6% (20NewsGroup dataset). Furthermore, in the classification training process, we just encode all articles in the training set with TD2V, not to train a dedicated classification model for each of these datasets.

People frequently post their photos on social network sites (e.g. Facebook, Instagram) as a way to share memorable moments, emotions, or locations visited. While sharing photos with the same subjects as past photos could lead to user reminiscence and potentially create valuable benefits, existing products still cannot support this adequately. Based on a survey on user habits in sharing and revisiting photos on social network sites, we propose NowAndThen, a photo recommendation concept and tool that assists reminiscence when sharing photos on social network sites. By combining visual features of the photos and associated tags, a prototype we developed can recommend old photos that have common subjects with the user's current photos of interest. Our study with the prototype shows that this approach can help users positively revive past memories and connections with their friends. In addition, our results include various design insights and implications for future reminiscence-support systems.

Video instance search or also well known as object retrieval is a fundamental task in computer vision field and has a lot of applications. Most state-of-the-art systems are based on the Bag-of-Words model (BOW) for representing video frames and target object. When searching on nearly planar and rich-textured objects such as buildings and book cover, BOW argue to be a suitable model with very high performance. However, when searching on harder but more popular objects such as a specific person, BOW model still keep a lower performance. In this paper, we consider a new type of query which covers most popular topics: searching a person in a specific location. Inspired by recent successes of deep learning techniques, we propose new framework which leverage the powerful of both BOW model and deep feature in instance search. In particular, we use a linear kernel classifier instead of using L2 distance to compute similarity between two deep features. For further improvement, scene tracking are employed to deal with the cases face of query person is not detected. To evaluate the proposed methods, we conduct experiments over a standard benchmark dataset (TRECVID Instance Search 2016) with more than 300 GB in storage and 464 hours in duration. The results show that, our proposed methods significant improve the baseline system.