Search

International audience; Human Action Recognition is one of the key tasks in video understanding. Deep Convolutional Neural Networks (CNN) are often used for this purpose. Although they usually perform impressively, their decision interpretation remains challenging. We propose a novel visual CNN features understanding technique. Its objective is to find salient features that played a key role in decision making of the network. The technique only uses the features from the last convolutional layer before the fully connected layers of a trained model and builds an importance map of features. The map is propagated to the original frame thus highlighting the regions in them that contribute to the final decision. The method is fast as it does not require gradient computation as many state-of-the-art methods do. Proposed technique is applied to the Twin Spatio-Temporal 3D Convolutional Neural Network (TSTCNN), designed for Table Tennis Actions recognition. Features visualization is performed at the RGB and Optical flow branches of the network. Obtained results are compared to other visualization techniques both in terms of human understanding and similarity metrics. The metrics show that generated maps are similar to those obtained with known Grad-CAM method, e.g. Pearson Correlation Coefficient between the maps generated of RGB data for Grad-CAM and our method is 0.7 ± 0.05 and 0.72 ± 0.06 on Optical Flow data.

Many universities adopt educational systems where the teacher lecture is video recorded and the video lecture is made available to students with minimum post-processing effort. These cost-effective solutions suffer from the limited amount of annotations associated with the video content, which strongly limits the usability of the service when students need to retrieve specific portions of video, e.g., to revise unclear aspects covered in the past lectures. This paper presents, as a real case study, the system developed and implemented in our university for video lecture annotation and indexing. The original video recordings, which last around 1.5 hour, are first partitioned into smaller segments and then annotated by mapping their content with the entities in a multilingual knowledge base. To this purpose, the proposed approach analyzes both the transcription of the teacher's speech and the text appearing in the video (e.g., the slide content, the note written on the whiteboard) by means of an ad hoc Named Entity Recognition and Disambiguation (NERD) step. NERD relies on a supervised classification approach tailored to the domain under analysis. More specifically, to identify the most salient entities of the knowledge base matching the video content it considers not only text similarity measures but also the semantic pertinence of the candidate entities to the main subject of the video lectures. The performance of the proposed system was validated on a ground truth against the techniques available in the general entity annotation system GERBIL. The preliminary results demonstrate the effectiveness of the proposed approach.

Videos are used often for communicating ideas, concepts, experience, and situations, because of the significant advances made in video communication technology. The social media platforms enhanced the video usage expeditiously. At, present, recognition of a video is done, using the metadata like video title, video descriptions, and video thumbnails. There are situations like video searcher requires only a video clip on a specific topic from a long video. This paper proposes a novel methodology for the analysis of video content and using video storytelling and indexing techniques for the retrieval of the intended video clip from a long duration video. Video storytelling technique is used for video content analysis and to produce a description of the video. The video description thus created is used for preparation of an index using wormhole algorithm, guarantying the search of a keyword of definite length L, within the minimum worst-case time. This video index can be used by video searching algorithm to retrieve the relevant part of the video by virtue of the frequency of the word in the keyword search of the video index. Instead of downloading and transferring a whole video, the user can download or transfer the specifically necessary video clip. The network constraints associated with the transfer of videos are considerably addressed.

Egocentric videos are becoming popular since the possibility to observe the scene flow from the user’s point of view (First Person Vision). Among the different applications of egocentric vision is the daily living monitoring of a user wearing the camera. We propose a system able to automatically organize egocentric videos acquired by the user over different days. Through an unsupervised temporal segmentation, each egocentric video is divided in chapters by considering the visual content. The obtained video segments related to the different days are hence connected according to the scene context in which the user acts. Experiments on a challenging egocentric video dataset demonstrate the effectiveness of the proposed approach that outperforms with a good margin the state of the art in accuracy and computational time.

The wide availability of video lectures within digital archives is causing significant problems to video providers. Indeed, students search for very specific contents and the few minutes of video lecture that cover the required topic have to be found in an archive composed of thousands of hours of videos. Therefore, the indexing of video material is becoming a critical process. Unfortunately, most of the systems return a list of full-length video lectures, and force students to use VCR-like functions in order to find what they are looking for. This approach is not adequate. Furthermore, the indexing process never considers the quality of the lecture (e.g., “interesting”, “meaningless”, etc.). A recent approach proposed to improve the indexing process by using social tagging. To be effective, this approach requires having a considerable number of tags. In this paper, we propose TagLecture, a Game With A Purpose that introduces game elements into the indexing process in order to involve students in the indexing process of video lectures. Through an experimental investigation we highlight that the gamification of the video indexing process is well accepted by students: more than 80% of participants liked the idea of using quality-based tags to classify video lectures, and more than half of them liked the idea of a game where students challenge each other to tag pieces of video lectures.

International audience; This paper introduces a complete framework for temporal video segmentation. First, a computationally efficient shot extraction method is introduced, which adopts the normalized graph partition approach, enriched with a non-linear, multiresolution filtering of the similarity vectors involved. The shot boundary detection technique proposed yields high precision (90%) and recall (95%) rates, for all types of transitions, both abrupt and gradual. Next, for each detected shot we construct a static storyboard, by introducing a leap keyframe extraction method. The video abstraction algorithm is 23% faster than existing, state of the art techniques, for similar performances. Finally, we propose a shot grouping strategy that iteratively clusters visually similar shots, under a set of temporal constraints. Two different types of visual features are here exploited: HSV color histograms and interest points. In both cases, the precision and recall rates present average performances of 86%.

Recognition of human activities in restricted settings such as airports, parking lots and banks is of significant interest in security and automated surveillance systems. In such settings, data is usually in the form of surveillance videos with wide variation in quality and granularity. Interpretation and identification of human activities requires an activity model that a) is rich enough to handle complex multi-agent interactions, b) is robust to uncertainty in low-level processing and c) can handle ambiguities in the unfolding of activities. We present a computational framework for human activity representation based on Petri nets. We propose an extension-Probabilistic Petri Nets (PPN)-and show how this model is well suited to address each of the above requirements in a wide variety of settings. We then focus on answering two types of questions: (i) what are the minimal sub-videos in which a given activity is identified with a probability above a certain threshold and (ii) for a given video, which activity from a given set occurred with the highest probability? We provide the PPN-MPS algorithm for the first problem, as well as two different algorithms (naive PPN-MPA and PPN-MPA) to solve the second. Our experimental results on a dataset consisting of bank surveillance videos and an unconstrained TSA tarmac surveillance dataset show that our algorithms are both fast and provide high quality results.

The use of emotions has recently been considered to improve the indexing of video contents and two different approaches are usually followed: computation of objective emotions through low-level video features analysis and computation of subjective emotions through analysis of the viewers' physical signals. In this paper, we propose a different approach and we present ViMood, a novel mechanism designed to improve the indexing of video material by integrating objective and subjective emotions. ViMood indexes every video scene with emotion(s) obtained through a combination of low-level feature analysis and on-the-fly viewer's emotion annotation. The goal is to allow viewers to browse video material using either general information (e.g., title, director) or specific emotions (e.g., “joy”, “sadness”, “surprise”). Results obtained in the evaluation process showed that participants were very interested in the hybrid approach, as it fixes some of the problems of the objective and subjective approaches.

Recognizing human action in complex video sequences has always been challenging for researchers due to articulated movements, occlusion, background clutter, and illumination variation. Human action recognition has wide range of applications in surveillance, human computer interaction, video indexing and video annotation. In this paper, a discrete cosine transform based features have been exploited for action recognition. First, motion history image is computed for a sequence of images and then blocked-based truncated discrete cosine transform is computed for motion history image. Finally, K-Nearest Neighbor (K-NN) classifier is used for classification. This technique exhibits promising results for KTH and Weizmann dataset. Moreover, the proposed model appears to be computationally efficient and immune to illumination variations; however, this model is prone to viewpoint variations

This work addresses the development of a unified approach to content-based indexing and retrieval of digital videos from television archives. The proposed approach has been designed to deal with arbitrary television genres, making it suitable for various applications. To achieve this goal, the main steps of a content-based video retrieval system are addressed in this work, namely: video segmentation, key-frame extraction, content-based video indexing and the video retrieval operation itself. Video segmentation is addressed as a typical TV broadcast structuring problem, which consists in automatically determining the boundaries of each broadcasted program (like movies, news, among others) and inter-program (for instance, commercials). Specifically, to segment the videos, Electronic Program Guide (EPG) metadata is combined with the detection of two special cues, namely, audio cuts (silence) and dark monochrome frames. On the other hand, a color histogram-based approach performs key-frame extraction. Video indexing and retrieval are accomplished by using hashing and k-d tree methods, while visual signatures containing color, shape and texture information are estimated for the key-frames, by using image and frequency domain techniques. Experimental results with the dataset of a multimedia information system especially developed for managing television broadcast archives demonstrate that our approach works efficiently, retrieving videos in 0.16 seconds on average and achieving recall, precision and F1 measure values, as high as 0.76, 0.97 and 0.86 respectively.

International audience; Audio-Visual People Diarization (AVPD) is an original framework that simultaneously improves audio, video, and audiovisual diarization results. Following a literature review of people diarization for both audio and video content and their limitations, which includes our own contributions, we describe a proposed method for associating both audio and video information by using co-occurrence matrices and present experiments which were conducted on a corpus containing TV news, TV debates, and movies. Results show the effectiveness of the overall diarization system and confirm the gains audio information can bring to video indexing and vice versa.

In this work we present M-VIVIE, a system for video sequence indexing based on the identity of appearing subjects, which exploits a multithread architecture for fast processing. The system is composed of more conceptual component modules, each performing a specific kind of processing. Each module can possibly be substituted with a different one performing the same task. Special attention is devoted to classification and clustering activities. M-VIVIE presents a number of peculiarities, that distinguish it from existing systems with similar goals. Among these, the account for the concomitant appearance of two identities in the same clip, and the use of such information in the process of identity mapping. M-VIVIE was tested on different kinds of real video clips to assess both its efficacy and efficiency.

In this chapter the authors tackle the analysis and characterization of the artistic animated movies in view of constituting an automatic content-based retrieval system. First, they deal with temporal segmentation, and propose cut, fade and dissolve detection methods adapted to the constraints of this domain. Further, they discuss a fuzzy linguistic approach for automatic symbolic/semantic content annotation in terms of color techniques and action content and we test its potential in automatic video classification. The browsing issue is dealt by providing methods for both, static and dynamic video abstraction. For a quick browse of the movie’s visual content the authors create a storyboard-like summary, while for a “sneak peak” of the movie’s exciting action content they propose a trailer-like video skim. Finally, the authors discuss the architecture of a prototype client-server 3D virtual environment for interactive video retrieval. Several experimental results are presented.

International audience; This work aims at helping multimedia content understanding by deriving bene t from textual clues embedded in digital videos. For this, we developed a complete video Optical Character Recognition system (OCR), speci cally adapted to detect and recognize embedded texts in videos. Based on a neural approach, this new method outperforms related work, especially in terms of robustness to style and size variabilities, to background complexity and to low resolution of the image. A language model that drives several steps of the video OCR is also introduced in order to remove ambiguities due to a local letter by letter recognition and to reduce segmentation errors. This approach has been evaluated on a database of French TV news videos and achieves an outstanding character recognition rate of 95%, corresponding to 78% of words correctly recognized, which enables its incorporation into an automatic video indexing and retrieval system.

VIVIE is a system for video sequence indexing. Video frames are annotated according to the identities of appearing subjects. Different interacting modules perform different processing steps, and each can be possibly substituted with a different one performing the same task using a different method. Classification and clustering are the most challenging activities. Differently from most existing similar systems, VIVIE accounts for the concomitant appearance of two identities in the same clip, and exploits such information for identity mapping. VIVIE was tested on 7 video clips and on a subset of the SCFace database to assess its performances.

International audience; This paper presents a method for indexing activities of daily living in videos obtained from wearable cameras. In the context of dementia diagnosis by doctors, the videos are recorded at patients' houses and later visualized by the medical practitioners. The videos may last up to two hours, therefore a tool for an efficient navigation in terms of activities of interest is crucial for the doctors. The specific recording mode provides video data which are really difficult, being a single sequence shot where strong motion and sharp lighting changes often appear. Our work introduces an automatic motion based segmentation of the video and a video structuring approach in terms of activities by a hierarchical two-level Hidden Markov Model. We define our description space over motion and visual characteristics of video and audio channels. Experiments on real data obtained from the recording at home of several patients show the difficulty of the task and the promising results of our approach.

This paper presents a method for indexing human ac- tivities in videos captured from a wearable camera being worn by patients, for studies of progression of the dementia diseases. Our method aims to produce indexes to facilitate the navigation throughout the individual video recordings, which could help doctors search for early signs of the dis- ease in the activities of daily living. The recorded videos have strong motion and sharp lighting changes, inducing noise for the analysis. The proposed approach is based on a two steps analysis. First, we propose a new approach to segment this type of video, based on apparent motion. Each segment is characterized by two original motion de- scriptors, as well as color, and audio descriptors. Second, a Hidden-Markov Model formulation is used to merge the multimodal audio and video features, and classify the test segments. Experiments show the good properties of the ap- proach on real data., Comment: 2011 9th International Workshop on Content-Based Multimedia Indexing (CBMI), Madrid : Spain (2011)

This paper presents a novel unsupervised method for identifying the semantic structure in long semi-structured video streams. We identify ‘chains’, local clusters of repeated features from both the video stream and audio transcripts. Each chain serves as an indicator that the temporal interval it demarcates is part of the same semantic event. By layering all the chains over each other, dense regions emerge from the overlapping chains, from which we can identify the semantic structure of the video. We analyze two clustering strategies that accomplish this task. ispartof: pages:103-108 ispartof: Proceedings of the 9th international workshop on content-based multimedia indexing pages:103-108 ispartof: 9th international workshop on content-based multimedia indexing location:Madrid, Spain date:13 Jun - 15 Jun 2011 status: published

International audience; Our research focuses on analysing human activities according to a known behaviorist scenario, in case of noisy and high dimensional collected data. The data come from the monitoring of patients with dementia diseases by wearable cameras. We define a structural model of video recordings based on a Hidden Markov Model. New spatio-temporal features, color features and localization features are proposed as observations. First results in recognition of activities are promising.

International audience; With the fastly developing standards for video compression, the availability of the video content at a large scale in the compressed form has become an everyday reality. Compressed video streams whatever is the compression standard contain primary information which without full decompression of the content can serve for its indexing and retrieval. This primary features optimized for the compression purposes are very often noisy and subject to compression artefacts. Hence their re-use for analysis and indexing purposes remains a challenge. In this paper we present the methodology which starting from the early low - resolution standards of MPEG family up to the new high-qualityhigh definition video compression approaches aims at answering the same question: how to re-use noisy primary features for fast and efficient video indexing.

8 pages; International audience; This paper presents four spatio-temporal wavelet decompositions for characterizing dynamic textures. The main goal of this work is to compare the influence of spatial and temporal variables in the wavelet decomposition scheme. Its novelty is to establish a comparison between the only existing method and three other spatio-temporal decompositions. The four decomposition schemes are presented and successfully applied on a large dynamic texture database. Construction of feature descriptors are tackled as well their relevance, and performances of the methods are discussed. Finally, future prospects are exposed.

International audience; Exploration of video surveillance material for healthcare becomes a reality in medical research. In this paper we propose a video monitoring system with wearable cameras for early diagnostics of Dementia. A video acquisition set-up is designed and the methods are developed for indexing the recorded video. The noisiness of audio-visual material and its particularity yield challenging problems for automatic indexing of this content.

International audience; The benchmarking of various methods of video analysis and indexing has become a research problem per se. While a lot of effort nowadays is devoted to organization of evaluation campaigns both on international, such as TREC Video and national levels, an in-deep analysis of the performance of the methods still remains an open issue. Several aspects in the evaluation of methods have to be addressed: how to design a benchmarking corpus in order to cover a sufficiently wide range of applications, what are the tasks to address, what are the most relevant evaluation metrics. The Argos evaluation campaign supported by the French Techno-Vision program aimed at developing esources for a benchmarking of video content analysis and indexing methods. The paper describes the type of the evaluated tasks, the way the content set has been produced, metrics and tools developed for the evaluations and some of the results obtained at the end of the first phase. Perspectives based on current works will be given to conclude this paper.

International audience; Automatic semantic classification of image databases is very useful for users searching and browsing, but it is at the same time a very challenging research problem as well. Local features based image classification is one of the key issues to bridge the semantic gap in order to detect concepts. This paper proposes a framework for incorporating contextual information into the concept detection process. The proposed method combines local and global classifiers with stacking, using SVM.We studied the impact of topologic and semantic contexts in concept detection performance and proposed solutions to handle the large amount of dimensions involved in classified data. We conducted experiments on TRECVID�04 subset with 48104 images and 5 concepts. We found that the use of context yields a significant improvement both for the topologic and semantic contexts.

The high availability of video streams is making necessary mechanisms for indexing such contents in the Web world. In this paper we focus on news programs and we propose a mechanism that integrates low and high level video features to provide a high level semantic description. A color/luminance analysis is coupled with audio analysis to provide a better identification of all the video segments that compose the video stream. Each video segment is subject to speech detection and is described through MPEG7 so that the resulting metadata description can be used to index the video stream. An experimental evaluation shows the benefits of integrating audio and video analysis.

近年，映像技術・機器の発達によって増加し続ける映像の量に対応するため，映像の要点を抜き出した要約を生成する技術が求められている．本研究では，対象をTVサッカー映像に限定した要約生成手法の確立を目指し，そのための要素技術として，映像の意味を理解し，映像をメタデータ化する処理（Indexing），メタデータ化された映像を要約するルールの設定（Index重みの設定）処理を実装した．結果として，メタデータ化については高い認識率を，ルール設定についてはある程度の結果を得ることができた．, 修士論文

Many algorithms for temporal video partitioning rely on the analysis of uncompressed video features. Since the information relevant to the partitioning process can be extracted directly from the MPEG compressed stream, higher efficiency can be achieved utilizing information from the MPEG compressed domain. This paper introduces a real-time algorithm for scene change detection that analyses the statistics of the macroblock features extracted directly from the MPEG stream. A method for extraction of the continuous frame difference that transforms the 3D video stream into a 1D curve is presented. This transform is then further employed to extract temporal units within the analysed video sequence. Results of computer simulations are reported.

Automatically generated summaries of TV content are indispensable for content selection and navigation in CE-devices. We show two different types of summaries: Short video trailers and visual overviews consisting of representative frames. The demo does not only show the feasibility of the proposed algorithms, but also shows how different types of generated summaries can be used in future CE-devices.

Visual database systems require efficient indexing to enable fast access to the images and video. In addition, the large memory capacity and channel bandwidth requirements for the storage and transmission of visual data necessitate the use of compression techniques. Future multimedia applications are likely to increasingly store and transmit the visual information in compressed form. Hence indexing the visual content in compressed domain is expected to result in significant savings in computational complexity. Vector quantization (VQ) is an efficient technique for low bit rate image and video compression. In addition, the lower complexity of the decoder makes VQ attractive for low power systems and applications which require fast decoding. Most importantly, VQ is naturally an indexing technique, where a block of pixels is compactly represented using an index (label) which corresponds to a codebook. In this thesis, we propose the novel concept of using VQ for joint compression and indexing of images and video. The images/image frames are compressed using VQ and the labels and codewords are employed in indexing the visual content. First, we present a review of image/video compression and indexing. We then propose two techniques in the VQ compressed domain for image indexing. In the first technique, the histogram of codewords weighted by the number of labels is used as feature vector for indexing. In the second technique, the histogram of the labels, which are used to represent an image, is used as an index. We also propose a new technique based on adaptive wavelet VQ, which provides an improvement in coding and retrieval performance. Here, the images are decomposed using wavelet transform followed by VQ of the transform coefficients. A usage map of codewords is generated for each image and is stored along with the image. In the retrieval process, the usage map of the query image (VQ encoded) is compared with the corresponding usage maps of the target images in the database. Since video has both spatial and temporal dimensions, a straightforward extension of the image indexing techniques for video indexing is inefficient. We propose to employ both the spatial and temporal features for efficient indexing of video clips. The video sequence is partitioned into shots using the label maps of the individual frames and the camera operations and motion within each shot are then determined by further processing the label maps. Each shot is then represented using a spatio-temporal index. The spatial index represents the content of the key frame (image) of a shot, while the temporal index represents the motion and camera operations within the shot. Detailed simulations have been carried out using a large database of images and video sequences. Simulation results demonstrate the excellent retrieval performance of the proposed techniques at a significantly reduced computational complexity.