Search

In this paper, we address the problem of high performance and computationally efficient content-based video retrieval in large-scale datasets. Current methods typically propose either: (i) fine-grained approaches employing spatio-temporal representations and similarity calculations, achieving high performance at a high computational cost or (ii) coarse-grained approaches representing/indexing videos as global vectors, where the spatio-temporal structure is lost, providing low performance but also having low computational cost. In this work, we propose a Knowledge Distillation framework, called Distill-and-Select (DnS), that starting from a well-performing fine-grained Teacher Network learns: (a) Student Networks at different retrieval performance and computational efficiency trade-offs and (b) a Selector Network that at test time rapidly directs samples to the appropriate student to maintain both high retrieval performance and high computational efficiency. We train several students with different architectures and arrive at different trade-offs of performance and efficiency, i.e., speed and storage requirements, including fine-grained students that store/index videos using binary representations. Importantly, the proposed scheme allows Knowledge Distillation in large, unlabelled datasets—this leads to good students. We evaluate DnS on five public datasets on three different video retrieval tasks and demonstrate (a) that our students achieve state-of-the-art performance in several cases and (b) that the DnS framework provides an excellent trade-off between retrieval performance, computational speed, and storage space. In specific configurations, the proposed method achieves similar mAP with the teacher but is 20 times faster and requires 240 times less storage space. The collected dataset and implementation are publicly available: https://github.com/mever-team/distill-and-select.

The continual technological advancement of handheld devices and personal computers over past few decades has reshaped the world's communication system by enabling the humans and robots to capture and share images and videos in digitized form at large. Practically, annotation-based video indexing and retrieval systems are widely being used to maintain the ongoing growth of multimedia content. These systems grant multimedia content retrieval using textual annotations, but are limited to predefined annotation/keywords. The online multimedia content libraries require manual annotation of video while uploading, which is a hectic and time-consuming assignment that sometimes even does not align with the visual content. This limits the searching capacity, as user may be unable to retrieve video because of incomplete video description at the time of annotation. Therefore, it strongly requires an efficient and sophisticated video indexing and retrieval system. To accomplish it, content-based video indexing is an optimal solution by detecting text appearing in videos. This dissertation demonstrates a new text detection system based on advance deep learning approach to bridge the gap by building an automatic and efficient content-based video indexing and retrieval system for Turkish videos. The text appearing in videos provides useful information that can be exploited for developing automatic video indexing and retrieval system. Therefore, this study integrates heuristic and deep learning-based approaches that utilizes CNN for automatic text detection and extraction. To train the proposed CNN-based model, a new dataset is generated by collecting videos from various Turkish channels related to News, financial and business, sports and cartoon channels. The dataset is fed to proposed model that first generates features maps and then classifies the image as textual or non-textual class. Extensive trails and experiments are carried out with different structural combination of convolutional layers, thus ended up with a best model out of three proposed models that can accurately detect the text. Next, the extracted text is fed to publicly available Tesseract OCR for recognition, which is then indexed in database along with video information such as file storage location. Lastly, a web-based user interface is provided for querying purposes. For each user query, the proposed system retrieved the most relevant videos based on its textual content appearing inside. Besides displaying the retrieved videos in provided user interface, the system also informs the user about the appearance time of queried words inside each retrieved video so that user can directly jump to the point of interest by using sleek bar. All basic functionalities are provided to play, pause, maximize, minimize, and download the retrieved video with additional controls for volume and sleek bar. Moreover, various conventional machine-learning algorithms such as SVM and LR, and few state-of-the-art image classification models (including VGG16, ResNet50 and DenseNet121) are also implemented and trained with identical datasets. The proposed models outperformed the prior state-of-the-art deep learning frameworks and machine learning classifiers. Son zamanlarda el cihazlarının ve kişisel bilgisayarların teknolojik gelişimi, insanların ve robotların görüntüler ve videoları yakalayıp büyük ölçüde dijitalleştirilmiş biçimde paylaşılmasını sağlayarak dünyanın iletişim sistemini yeninden şekillendirmiştir. Pratikte, açıklama tabanlı video indeksleme ve bilgi erişim sistemleri günümüzde büyüyen multimedya içeriklerinin sürdürebilirliğini devam ettirmek amacıyla kullanılmaktadırlar. İlgili sistemler, metin notlarını kullanarak multimedya içeriğinin elde edilmesini sağlamaktadırlar, ancak ek açıklamaları tanımlamada sınırlıdırlar. Çevrimiçi multimedya içerikli kütüphanelerde, video yükleme işleminde videonun açıklaması manuel olarak gerçekleşmesi gerekmektedir. Bu durum, zaman gerektiren bir işlem olduğu gibi bazı durumlarda video açıklaması görsel içerik ile uyuşmamaktadır. Aynı zamanda, videolarda oluşacak eksik açıklamalar nedeniyle kullanıcıların arama yapma kapasitelerini sınırlamaktadır. Bu nedenle, verimli ve sofistike bir video indeksleme ve erişim sistemi gereklidir. Bu problemi çözüme kavuşturmak için, videolarda mevcut olan metinleri tespit ederek içerik tabanlı video indeksleme sistemi geliştirmek en uygun bir çözümdür. Bu tez çalışmasında, Türkçe videolar için otomatik ve verimli içerik tabanlı video indeksleme ve bilgi erişim sistemi oluşturmak amaçlı derin öğrenmeye dayalı yeni bir metin algılama sistemi geliştirilmektedir. Videolarda görünen metin, otomatik video indeksleme ve bilgi erişim sistemini geliştirmek için kullanılabilecek faydalı bilgiler sağlamaktadır. Dolayısıyla, bu çalışma otomatik metin algılama ve çıkarma işlemlerini gerçekleştirmek için Evrişimsel Sinir Ağlarından yararlanarak, sezgisel ve Derin Öğrenmeye dayalı yaklaşımları bütünleştirmektedir. Önerilen Evrişimsel Sinir Ağı tabanlı modeli eğitmek için, Haberler, finans ve iş, spor ve çizgi ile ilgili çeşitli Türk televizyon kanallarından videolar toplanarak yeni bir veri kümesi oluşturulmuştur. Oluşturulan veri seti, ilk aşamada önerilen modele özellik haritalarının elde edilmesi amacıyla beslenmektedir, devamında önerilen model görüntüyü metinsel veya metinsel olmayan sınıf olarak sınıflandırmaktadır. Evrişimsel katmanların farklı yapısal kombinasyonları ile kapsamlı deneyler yapıldıktan sonra önerilen üç modelden metni en doğru bir şekilde algılayabilen model elde edildi. Devamında, çıkarılan (elde edilen) metnin tanınması için Tesseract OCR'a beslenir, ve dosya depolama konumu gibi video bilgileri ile birlikte veri tabanında indekslenir. Son olarak, sorgulama amacıyla web tabanlı bir kullanıcı arayüzü geliştirilir. Her kullanıcı sorgusu için, önerilen sistem görüntü içinde görünen metin içeriğine göre en alakalı videoları kullanıcıya sunmaktadır. Buna ek olarak, sistem kullanıcın sorguladığı kelimeleri videonun hangi süreleri arasında geçtiğine dair bilgi vermektedir, dolayısıyla kullanıcı arama çubuğunu kullanarak doğrudan ilgi alanına gidebilir. Sunulan videoyu oynatmak, duraklatmak, büyütmek, küçültmek ve indirmek için ek ses ve kontrol çubuğu kontrolleriyle birlikte tüm temel işlevler sağlanmıştır. Ayrıca, Destek Vektör Makineleri ve Lojistik Regresyon gibi çeşitli geleneksel makine öğrenimi algoritmaları ve birkaç son teknoloji görüntü sınıflandırma modeli (VGG16, ResNet50 ve DenseNet121) aynı veri setiyle uygulanır ve eğitilir. Önerilen modeller, son teknoloji derin öğrenme modeleri ve makine öğrenimi sınıflandırıcılarından daha iyi performans göstermiştir.

International audience; Human Action Recognition is one of the key tasks in video understanding. Deep Convolutional Neural Networks (CNN) are often used for this purpose. Although they usually perform impressively, their decision interpretation remains challenging. We propose a novel visual CNN features understanding technique. Its objective is to find salient features that played a key role in decision making of the network. The technique only uses the features from the last convolutional layer before the fully connected layers of a trained model and builds an importance map of features. The map is propagated to the original frame thus highlighting the regions in them that contribute to the final decision. The method is fast as it does not require gradient computation as many state-of-the-art methods do. Proposed technique is applied to the Twin Spatio-Temporal 3D Convolutional Neural Network (TSTCNN), designed for Table Tennis Actions recognition. Features visualization is performed at the RGB and Optical flow branches of the network. Obtained results are compared to other visualization techniques both in terms of human understanding and similarity metrics. The metrics show that generated maps are similar to those obtained with known Grad-CAM method, e.g. Pearson Correlation Coefficient between the maps generated of RGB data for Grad-CAM and our method is 0.7 ± 0.05 and 0.72 ± 0.06 on Optical Flow data.

Action recognition in videos is one of the key problems in visual data interpretation. Despite intensive research, differencing and recognizing similar actions remains a challenge. This thesis deals with fine-grained classification of sport gestures from videos, with an application to table tennis.In this manuscript, we propose a method based on deep learning for automatically segmenting and classifying table tennis strokes in videos. Our aim is to design a smart system for students and teachers for analyzing their performances. By profiling the players, a teacher can therefore tailor the training sessions more efficiently in order to improve their skills. Players can also have an instant feedback on their performances.For developing such a system with fine-grained classification, a very specific dataset is needed to supervise the learning process. To that aim, we built the “TTStroke-21” dataset, which is composed of 20 stroke classes plus a rejection class. The TTStroke-21 dataset comprises video clips of recorded table tennis exercises performed by students at the sport faculty of the University of Bordeaux - STAPS. These recorded sessions were annotated by professional players or teachers using a crowdsourced annotation platform. The annotations consist in a description of the handedness of the player and information for each stroke performed (starting and ending frames, class of the stroke). Fine-grained action recognition has some notable differences with coarse-grained action recognition. In general, datasets used for coarse-grained action recognition, the background context often provides discriminative information that methods can use to classify the action, rather than focusing on the action itself. In fine-grained classification, where the inter-class similarity is high, discriminative visual features are harder to extract and the motion plays a key role for characterizing an action.In this thesis, we introduce a Twin Spatio-Temporal Convolutional Neural Network. This deep learning network takes as inputs an RGB image sequence and its computed Optical Flow. The RGB image sequence allows our model to capture appearance features while the optical flow captures motion features. Those two streams are processed in parallel using 3D convolutions, and fused at the last stage of the network. Spatio-temporal features extracted in the network allow efficient classification of video clips from TTStroke-21. Our method gets an average classification performance of 87.3% with a best run of 93.2% accuracy on the test set. When applied on joint detection and classification task, the proposed method reaches an accuracy of 82.6%.A systematic study of the influence of each stream and fusion types on classification accuracy has been performed, giving clues on how to obtain the best performances. A comparison of different optical flow methods and the role of their normalization on the classification score is also done. The extracted features are also analyzed by back-tracing strong features from the last convolutional layer to understand the decision path of the trained model. Finally, we introduce an attention mechanism to help the model focusing on particular characteristic features and also to speed up the training process. For comparison purposes, we provide performances of other methods on TTStroke-21 and test our model on other datasets. We notice that models performing well on coarse-grained action datasets do not always perform well on our fine-grained action dataset.The research presented in this manuscript was validated with publications in one international journal, five international conference papers, two international workshop papers and a reconductible task in MediaEval workshop in which participants can apply their action recognition methods to TTStroke-21. Two additional international workshop papers are in process along with one book chapter., La reconnaissance des actions à partir de vidéos est l'un des principaux problèmes de vision par ordinateur. Malgré des recherches intensives, la différenciation et la reconnaissance d'actions similaires restent un défi. Cette thèse porte sur la classification des gestes sportifs à partir de vidéos, avec comme cadre applicatif le tennis de table. Nous proposons une méthode d’apprentissage profond pour segmenter et classifier automatiquement les différents coup de Tennis de Table. Notre objectif est de concevoir un système intelligent permettant d'analyser les performances des élèves pongistes, et de donner la possibilité à l’entraîneur d'adapter ses séances d'entraînement pour améliorer leurs performances.Dans ce but, nous avons élaboré la base de données “TTStroke-21”, constituée de clips vidéo d'exercices de tennis de table, enregistrés par les étudiants de la faculté de sport de l'Université de Bordeaux – STAPS. Cette base de données a ensuite été annotée par des professionnels du domaine à l'aide d'une plateforme crowdsourcing. Les annotations consistent en une description des coups effectués (début, fin et type de coup). Au total, 20 différents coups de tennis de table sont considérés plus une classe de rejet. La reconnaissance des actions similaires présente des différences avec la reconnaissance d’actions classique. En effet, dans les bases de données classiques, le contexte de l’arrière plan fournit souvent des informations discriminantes que les méthodes peuvent utiliser pour classer l'action plutôt que de se concentrer sur l'action elle-même. Dans notre cas, la similarité entre classes est élevée, les caractéristiques visuelles discriminantes sont donc plus difficiles à extraire et le mouvement joue un rôle clef dans la caractérisation de l’action.Dans cette thèse, nous introduisons un réseau de neurones spatio-temporel convolutif avec une architecture Jumelle. Ce réseau d'apprentissage profond prend comme entrées une séquence d'images RVB et son flot optique estimé. Les données RVB permettent à notre modèle de capturer les caractéristiques d'apparence tandis que le flot optique capture les caractéristiques de mouvement. Ces deux flux sont traités en parallèle à l'aide de convolutions 3D, et sont fusionnés à la dernière étape du réseau. Les caractéristiques spatio-temporelles extraites dans le réseau permettent une classification efficace des clips vidéo de TTStroke-21. Notre méthode obtient une performance de classification de 93.2% sur l'ensemble des données tests. Appliquée à la tâche jointe de détection et de classification, notre méthode atteint une précision de 82.6%.Nous étudions les performances en fonction des types de données utilisés en entrée et la manière de les fusionner. Différents estimateurs de flot optique ainsi que leur normalisation sont testés afin d’améliorer la précision. Les caractéristiques de chaque branche de notre architecture sont également analysées afin de comprendre le chemin de décision de notre modèle. Enfin, nous introduisons un mécanisme d'attention pour aider le modèle à se concentrer sur des caractéristiques discriminantes et aussi pour accélérer le processus d’entraînement. Nous comparons notre modèle avec d'autres méthodes sur TTStroke-21 et le testons sur d'autres ensembles de données. Nous constatons que les modèles fonctionnant bien sur des bases de données d’actions classiques ne fonctionnent pas toujours aussi bien sur notre base de données d'actions similaires.Les travaux présentés dans cette thèse ont été validés par des publications dans une revue internationale, cinq papiers de conférences internationales, deux papiers d’un workshop international et une tâche reconductible dans le workshop MediaEval où les participants peuvent appliquer leurs méthodes de reconnaissance d'actions à notre base de données TTStroke-21. Deux autres papiers de workshop internationaux sont en cours de préparation, ainsi qu'un chapitre de livre.

Many universities adopt educational systems where the teacher lecture is video recorded and the video lecture is made available to students with minimum post-processing effort. These cost-effective solutions suffer from the limited amount of annotations associated with the video content, which strongly limits the usability of the service when students need to retrieve specific portions of video, e.g., to revise unclear aspects covered in the past lectures. This paper presents, as a real case study, the system developed and implemented in our university for video lecture annotation and indexing. The original video recordings, which last around 1.5 hour, are first partitioned into smaller segments and then annotated by mapping their content with the entities in a multilingual knowledge base. To this purpose, the proposed approach analyzes both the transcription of the teacher's speech and the text appearing in the video (e.g., the slide content, the note written on the whiteboard) by means of an ad hoc Named Entity Recognition and Disambiguation (NERD) step. NERD relies on a supervised classification approach tailored to the domain under analysis. More specifically, to identify the most salient entities of the knowledge base matching the video content it considers not only text similarity measures but also the semantic pertinence of the candidate entities to the main subject of the video lectures. The performance of the proposed system was validated on a ground truth against the techniques available in the general entity annotation system GERBIL. The preliminary results demonstrate the effectiveness of the proposed approach.

Videos are used often for communicating ideas, concepts, experience, and situations, because of the significant advances made in video communication technology. The social media platforms enhanced the video usage expeditiously. At, present, recognition of a video is done, using the metadata like video title, video descriptions, and video thumbnails. There are situations like video searcher requires only a video clip on a specific topic from a long video. This paper proposes a novel methodology for the analysis of video content and using video storytelling and indexing techniques for the retrieval of the intended video clip from a long duration video. Video storytelling technique is used for video content analysis and to produce a description of the video. The video description thus created is used for preparation of an index using wormhole algorithm, guarantying the search of a keyword of definite length L, within the minimum worst-case time. This video index can be used by video searching algorithm to retrieve the relevant part of the video by virtue of the frequency of the word in the keyword search of the video index. Instead of downloading and transferring a whole video, the user can download or transfer the specifically necessary video clip. The network constraints associated with the transfer of videos are considerably addressed.

Egocentric videos are becoming popular since the possibility to observe the scene flow from the user’s point of view (First Person Vision). Among the different applications of egocentric vision is the daily living monitoring of a user wearing the camera. We propose a system able to automatically organize egocentric videos acquired by the user over different days. Through an unsupervised temporal segmentation, each egocentric video is divided in chapters by considering the visual content. The obtained video segments related to the different days are hence connected according to the scene context in which the user acts. Experiments on a challenging egocentric video dataset demonstrate the effectiveness of the proposed approach that outperforms with a good margin the state of the art in accuracy and computational time.

The wide availability of video lectures within digital archives is causing significant problems to video providers. Indeed, students search for very specific contents and the few minutes of video lecture that cover the required topic have to be found in an archive composed of thousands of hours of videos. Therefore, the indexing of video material is becoming a critical process. Unfortunately, most of the systems return a list of full-length video lectures, and force students to use VCR-like functions in order to find what they are looking for. This approach is not adequate. Furthermore, the indexing process never considers the quality of the lecture (e.g., “interesting”, “meaningless”, etc.). A recent approach proposed to improve the indexing process by using social tagging. To be effective, this approach requires having a considerable number of tags. In this paper, we propose TagLecture, a Game With A Purpose that introduces game elements into the indexing process in order to involve students in the indexing process of video lectures. Through an experimental investigation we highlight that the gamification of the video indexing process is well accepted by students: more than 80% of participants liked the idea of using quality-based tags to classify video lectures, and more than half of them liked the idea of a game where students challenge each other to tag pieces of video lectures.

International audience; This paper introduces a complete framework for temporal video segmentation. First, a computationally efficient shot extraction method is introduced, which adopts the normalized graph partition approach, enriched with a non-linear, multiresolution filtering of the similarity vectors involved. The shot boundary detection technique proposed yields high precision (90%) and recall (95%) rates, for all types of transitions, both abrupt and gradual. Next, for each detected shot we construct a static storyboard, by introducing a leap keyframe extraction method. The video abstraction algorithm is 23% faster than existing, state of the art techniques, for similar performances. Finally, we propose a shot grouping strategy that iteratively clusters visually similar shots, under a set of temporal constraints. Two different types of visual features are here exploited: HSV color histograms and interest points. In both cases, the precision and recall rates present average performances of 86%.

Recognition of human activities in restricted settings such as airports, parking lots and banks is of significant interest in security and automated surveillance systems. In such settings, data is usually in the form of surveillance videos with wide variation in quality and granularity. Interpretation and identification of human activities requires an activity model that a) is rich enough to handle complex multi-agent interactions, b) is robust to uncertainty in low-level processing and c) can handle ambiguities in the unfolding of activities. We present a computational framework for human activity representation based on Petri nets. We propose an extension-Probabilistic Petri Nets (PPN)-and show how this model is well suited to address each of the above requirements in a wide variety of settings. We then focus on answering two types of questions: (i) what are the minimal sub-videos in which a given activity is identified with a probability above a certain threshold and (ii) for a given video, which activity from a given set occurred with the highest probability? We provide the PPN-MPS algorithm for the first problem, as well as two different algorithms (naive PPN-MPA and PPN-MPA) to solve the second. Our experimental results on a dataset consisting of bank surveillance videos and an unconstrained TSA tarmac surveillance dataset show that our algorithms are both fast and provide high quality results.

Exposição sobre processos e métodos utilizados para a indexação e recuperação textual da informação semântica em vídeo, tendo como base a identificação e classificação do seu conteúdo visual e sonoro.Presentation of methods and processes applied to classification and retrieval of semantic information of video programs, through identification of sound and visual content.

International audience; We propose in this paper an original method of camera motion classification based on Transferable Belief Model (TBM). It consists in locating in a video the motions of translation and zoom, and the absence of camera motion (i.e static camera). The classification process is based on a rule-based system that is divided into three stages. From a parametric motion model, the first stage consists in combining data to obtain frame-level belief masses on camera motions. To ensure the temporal coherence of motions, a filtering of belief masses according to TBM is achieved. The second stage carries out a separation between static and dynamic frames. In the third stage, a temporal integration allows the motion to be studied on a set of frames and to preserve only those with significant magnitude and duration. Then, a more detailed description of each motion is given. Experimental results obtained show the effectiveness of the method.

The goal of the project is to adapt the LIRE image retrieval system to videos. This project explores the expansion of Lucene Image Retrieval Engine (LIRE), an open-source Content-Based Image Retrieval (CBIR) system, for video retrieval on large scale video datasets. The fast growth of the need to store huge amounts of video in servers requires efficient, scalable search and indexing engines capable to assist users in their management and retrieval. In our tool, queries are formulated by visual examples allowing users to find the videos and the moment of time when the query image is matched with. The video dataset used on this scenario comprise over 1,000 hours of different news broadcast channels. This thesis presents an extension and adaptation of Lire and its plugin for Solr, an open-source enterprise search platform from the Apache Lucene project, for video retrieval based on visual features, as well as a web-interface for users from different devices. Este proyecto investiga el potencial de LIRE, una librería de código abierto para consulta de imágenes mediante ejemplo (CBIR), como herramienta de consulta e indexación para grandes bases de datos de vídeos. El rápido crecimiento de las necesidades de almacenar enormes cantidades de vídeos en diversos servidores requiere un motor de búsqueda e indexación escalable y eficiente, así como una herramienta que proporcione a los usuarios métodos para la gestión y consulta de contenido. En nuestra herramienta, las consultas se formulan mediante ejemplos visuales, permitiendo a los usuarios encontrar los vídeos y los segundos a los que pertenece un fotograma en concreto. La recopilación de vídeos utilizados en este escenario comprende más de 1000 horas de vídeo recopilados de diferentes canales de noticias internacionales. En esta tesis se presenta una extensión y adaptación de Lire y su plugin para Solr, una plataforma de búsqueda empresarial de código abierto del proyecto Apache Lucene, para la consulta e indexación de vídeos en servidores basado en características visuales, así como una interfaz web accesible para usuarios de diferentes dispositivos. Aquest projecte investiga el potencial de LIRE, una llibreria de codi obert per a consulta d'imatges mitjançant exemple (CBIR), com a eina de consulta i indexació per a grans bases de dades de vídeos. El ràpid creixement de les necessitats d'emmagatzemar enormes quantitats de vídeos en diversos servidors requereix un motor de cerca i indexació escalable i eficient, així com una eina que proporcioni als usuaris mètodes per a la gestió i consulta de contingut. En la nostra eina, les consultes es formulen mitjançant exemples visuals, permetent als usuaris trobar els vídeos i els segons als quals pertany un fotograma en concret. La recopilació de vídeos utilitzats en aquest escenari comprèn més de 1000 hores de vídeo recopilats de diferents canals de notícies internacionals. En aquesta tesi es presenta una extensió i adaptació de Lire i el seu plugin per Solr, una plataforma de cerca empresarial de codi obert del projecte Apache Lucene, per a la consulta i indexació de vídeos en servidors basat en característiques visuals, així com una interfície web accessible per a usuaris de diferents dispositius.

The use of emotions has recently been considered to improve the indexing of video contents and two different approaches are usually followed: computation of objective emotions through low-level video features analysis and computation of subjective emotions through analysis of the viewers' physical signals. In this paper, we propose a different approach and we present ViMood, a novel mechanism designed to improve the indexing of video material by integrating objective and subjective emotions. ViMood indexes every video scene with emotion(s) obtained through a combination of low-level feature analysis and on-the-fly viewer's emotion annotation. The goal is to allow viewers to browse video material using either general information (e.g., title, director) or specific emotions (e.g., “joy”, “sadness”, “surprise”). Results obtained in the evaluation process showed that participants were very interested in the hybrid approach, as it fixes some of the problems of the objective and subjective approaches.

Recognizing human action in complex video sequences has always been challenging for researchers due to articulated movements, occlusion, background clutter, and illumination variation. Human action recognition has wide range of applications in surveillance, human computer interaction, video indexing and video annotation. In this paper, a discrete cosine transform based features have been exploited for action recognition. First, motion history image is computed for a sequence of images and then blocked-based truncated discrete cosine transform is computed for motion history image. Finally, K-Nearest Neighbor (K-NN) classifier is used for classification. This technique exhibits promising results for KTH and Weizmann dataset. Moreover, the proposed model appears to be computationally efficient and immune to illumination variations; however, this model is prone to viewpoint variations

This paper presents a two-step algorithm for detecting video shot boundaries in a wavelet transform domain. It captures gradual shoto transitions such as dissolves and fades as well as abrupt shot transitions such as cuts. At the first step, it computes a distance between video intervals to find an isolated interval. Then, at the second step, it computes a distance between frames to specify the exact location of a shot boundary. Its effectiveness is evaluated in terms of recall and precision.

Text data present in multimedia contain useful information for automatic annotation, indexing. Extracted information used for recognition of the overlay or scene text from a given video or image. The Extracted text can be used for retrieving the videos and images. In this paper, firstly, we are discussed the different techniques for text extraction from images and videos. Secondly, we are reviewed the techniques for indexing and retrieval of image and videos by using extracted text., 12 pages

This work addresses the development of a unified approach to content-based indexing and retrieval of digital videos from television archives. The proposed approach has been designed to deal with arbitrary television genres, making it suitable for various applications. To achieve this goal, the main steps of a content-based video retrieval system are addressed in this work, namely: video segmentation, key-frame extraction, content-based video indexing and the video retrieval operation itself. Video segmentation is addressed as a typical TV broadcast structuring problem, which consists in automatically determining the boundaries of each broadcasted program (like movies, news, among others) and inter-program (for instance, commercials). Specifically, to segment the videos, Electronic Program Guide (EPG) metadata is combined with the detection of two special cues, namely, audio cuts (silence) and dark monochrome frames. On the other hand, a color histogram-based approach performs key-frame extraction. Video indexing and retrieval are accomplished by using hashing and k-d tree methods, while visual signatures containing color, shape and texture information are estimated for the key-frames, by using image and frequency domain techniques. Experimental results with the dataset of a multimedia information system especially developed for managing television broadcast archives demonstrate that our approach works efficiently, retrieving videos in 0.16 seconds on average and achieving recall, precision and F1 measure values, as high as 0.76, 0.97 and 0.86 respectively.

Son yıllarda, otomatik video kopya bulmaya olan ihtiyaç yeni teknolojik gelişmelerle birlikte artmıştır. Genellikle, geliştirilen sistemin büyük veritabanlarına uygulanabilmesi için birkaç zorunlu gereksinimi karşılaması gerekmektedir. Gereksinimler yüksek doğruluk bulma, düşük karşılaştırma karmaşıklığı ve düşük hafıza ihtiyacıdır. Bu amaçla, bu tez kapsamında, öznitelik çıkarma, nicemleme tabanlı indeksleme ve geometric doğrulama olmak üzere üç ana bölümden oluşan bir içerik tabanlı kopya bulma sistemi önerilmektedir. Öznitelik çıkarma bölümünde, referans ve sorgu videolardan, benzerlik hesaplamalarında kullanılmak için yerel uzamsal ve uzamsal-zamansal öznitelikler çıkartılmaktadır. Uzamsal uzayda, ölçek değişimsiz öznitelik dönüşümü (SIFT), Karşıt SIFT, Çevirme Değişimsiz SIFT ve Hızlandırılmış Dayanıklı Dönüşüm (SURF) tanımlayıcıları, uzamsal-zamansal uzayda, Dönüşüm Değişimleri Histogramı (HoG) ve Hareket Sınır Histogramı (MBH) tanımlayıcıları hesaplanmıştır. İkinci bölümde, yerel öznitelikler arasında hızlı karşılaştırma yapabilmek için, yerel tanımlayıcılar, üç yeni önerilen Kelime Çantası, Hamming Yerleştirme ve Çarpım Nicemlemesi indekleme metodlarıyla indekslere nicemlenmektedir. Son adımda, öznitelik indekslerinin eşleşmesi sırasında doğru eşleşmeyen noktalar olabileceği için, her iki uzamsal ve uzamsal-zamansal öznitelikler için, geometrik modeli uygularak sonuçları iyileştiren bir geometri son basamağından faydalanılmaktadır. Ayrıca, yerel ilgi noktaları bilgisini kodlayan yoğun bir geometrik imza hesaplanmaktadır. Test sonuçları, iyi bilinen TRECVID 2009 içerik tabanlı kopya bulma veritabanında sunulmaktadır. Sonuçlar göstermektedirki, Çevirme değişimsiz SIFT, Hamming yerleştirme, iyileştirilmiş zayıf geometrik tutarlılık ve görsel grup ikili imza kombinasyonu en iyi toplam sonucu vermektedir. In recent years, need in automatic video copy detection has been increased rapidly with the recent technical developments. In general, a developed system should provide a few requirements to conduct over large database including high detection accuracy, low comparison time and low memory usage. For that purpose, within the scope of the thesis, we propose a content-based video copy detection system that consists of three crucial stages namely feature extraction, quantization-based indexing and geometric verification. In feature extraction stage, local spatial and spatio-temporal features are extracted from reference and query videos to be used for similarity score calculation. In spatial domain, Scale Invariant Feature Transform (SIFT), Opponent SIFT, Flip Invariant SIFT (F-SIFT) and Speed Up Robust Transform (SURF) descriptors, in spatio-temporal domain, Histogram of Orientated Gradient (HoG) and Motion Boundary Histogram (MBH) descriptors are utilized. In the second stage, in order to make efficient comparison among local features, the local features are quantized into indices with three state-of-the-art indexing schemes Bag-of-word, Hamming Embedding and Product Quantization. In the final stage, since there would be outliers during matching content indices, a geometric post-processing stage is utilized for both spatial and spatio-temporal features that impose an overall geometric model to refine the accuracy. Additionally, a compact geometric signature that encodes the local relation of interest points with binary signature is computed. The experimental results are presented on the well-known TRECVID 2009 content-based video copy detection dataset. The experiments show that combination of Flip Invariant SIFT, Hamming embedding, enhanced weak geometric consistency and visual group binary signature yields the best overall result. 106

Visual material comprising images and videos is growing ever so rapidly over the internet and in our personal collections. This necessitates automatic understanding of the visual content which calls for the conception of intelligent methods to correctly index, search and retrieve images and videos. This thesis aims at improving the automatic detection of concepts in the internet videos by exploring all the available information and putting the most beneficial out of it to good use. Our contributions address various levels of the concept detection framework and can be divided into three main parts. The first part improves the Bag of Words (BOW) video representation model by proposing a novel BOW construction mechanism using concept labels and by including a refinement to the BOW signature based on the distribution of its elements. We then devise methods to incorporate knowledge from similar and dissimilar entities to build improved recognition models in the second part. Here we look at the potential information that the concepts share and build models for meta-concepts from which concept specific results are derived. This improves recognition for concepts lacking labeled examples. Lastly we contrive certain semi-supervised learning methods to get the best of the substantial amount of unlabeled data. We propose techniques to improve the semi-supervised cotraining algorithm with optimal view selection., Les documents visuels comprenant des images et des vidéos sont en croissance rapide sur Internet et dans nos collections personnelles. Cela nécessite une analyse automatique du contenu visuel qui fait appel à la conception de méthodes intelligentes pour correctement indexer, rechercher et récupérer des images et des vidéos. Cette thèse vise à améliorer la détection automatique des concepts dans les vidéos sur Internet. Nos contributions portent sur des différents niveaux dans le cadre de détection de concept et peuvent être divisés en trois parties principales. La première partie se focalise sur l’amélioration du modèle de représentation des vidéos « Bag-of-Words (BOW) » en proposant un nouveau mécanisme de construction qui utilise des étiquettes de concepts et une autre technique qui ajoute un raffinement à la signature BOW basée sur la distribution de ses éléments. Nous élaborons ensuite des méthodes pour intégrer des entités semblables et dissemblables pour construire des modèles de reconnaissance améliorés dans la deuxième partie. A ce stade-là, nous observons l’information potentielle que les concepts partagent et construisons des modèles pour les méta-concepts dont sont dérivés les résultats spécifiques de concepts. Cela améliore la reconnaissance des concepts qui ont peu d’exemples annotés. Enfin, nous concevons certaines méthodes d'apprentissage semi-supervisé pour bénéficier de la quantité importante de données non étiquetées. Nous proposons des techniques pour améliorer l'algorithme de cotraining avec une sélection optimale des classifieurs utilisés.

International audience; The research context of this article is the recognition and description of dynamic textures. In image processing, the wavelet transform has been successfully used for characterizing static textures. To our best knowledge, only two works are using spatio-temporal multiscale decomposition based on tensor product for dynamic texture recognition. One contribution of this article is to analyse and compare the ability of the 2D+T curvelet transform, a geometric multiscale decomposition, for characterizing dynamic textures in image sequences. Two approaches using the 2D+T curvelet transform are presented and compared using three new large databases. A second contribution is the construction of these three publicly available benchmarks of increasing complexity. Existing benchmarks are either too small, not available or not always constructed using a reference database.\\ Feature vectors used for recognition are described as well as their relevance, and performances of the different methods are discussed. Finally, future prospects are exposed.

This report presents a small and concise market study of the challenges and opportunities that faces the realization of an open digital environment enabling individual and collective actors in the digital humanities to produce, valorize and share their cultural and scientific heritage in form of digital audiovisual archives. This study has been realized in the framework of the European FP-7 project CONVERGENCE (n. FP7-257123; 2010 - 2013).

International audience; Audio-Visual People Diarization (AVPD) is an original framework that simultaneously improves audio, video, and audiovisual diarization results. Following a literature review of people diarization for both audio and video content and their limitations, which includes our own contributions, we describe a proposed method for associating both audio and video information by using co-occurrence matrices and present experiments which were conducted on a corpus containing TV news, TV debates, and movies. Results show the effectiveness of the overall diarization system and confirm the gains audio information can bring to video indexing and vice versa.

This report presents a small and concise market study of the challenges and opportunities that faces the realization of an open digital environment enabling individual and collective actors in the digital humanities to produce, valorize and share their cultural and scientific heritage in form of digital audiovisual archives. This study has been realized in the framework of the European FP-7 project CONVERGENCE (n. FP7-257123; 2010 - 2013).

In this work we present M-VIVIE, a system for video sequence indexing based on the identity of appearing subjects, which exploits a multithread architecture for fast processing. The system is composed of more conceptual component modules, each performing a specific kind of processing. Each module can possibly be substituted with a different one performing the same task. Special attention is devoted to classification and clustering activities. M-VIVIE presents a number of peculiarities, that distinguish it from existing systems with similar goals. Among these, the account for the concomitant appearance of two identities in the same clip, and the use of such information in the process of identity mapping. M-VIVIE was tested on different kinds of real video clips to assess both its efficacy and efficiency.

The objective of this thesis is to extract the informative temporal segments from video sequences, more particularly in aerial video. Manual interpretation of such videos for information gathering faces an ever growing volume of available data. We have thus considered an algorithmic assistance based on different modalities of indexation in order to locate "segments of interest" and avoid a complete visualization of the video. We have chosen two methods in particular and have respectively developed them in each part of this thesis. Part 1 describes how viewing conditions can be used as a method of indexation. The assessment of image quality enables to filter out the temporal segments for which the quality is low and which can thus not be exploited. The classification of global image motion, which is directly linked to camera motion, leads to a method of indexation for video sequences. Indeed, it emphasizes possible segments of interest or, conversely, difficult segments for which motion is very fast or oscillating. Part 2 focuses on the dynamic content of video sequences, especially the presence of moving objects. We first present a local (in time) approach. This approach refines the results obtained after a first classification by supervised learning by using contextual information, spatial then semantic information. We have then investigated several methods for moving object detection which are global in time. Such approaches aim to enforce the temporal consistency of the detected objects and to reduce false detections.; Les travaux réalisés dans le cadre de cette thèse ont pour objectif d’extraire les différents segments informatifs au sein de séquences vidéo, plus particulièrement aériennes. L’interprétation manuelle de telles vidéos dans une optique de renseignement se heurte en effet au volume des données disponibles. Une assistance algorithmique fondée sur diverses modalités d’indexation est donc envisagée, dans l’objectif de repérer les "segments d’intérêt" et éviter un parcours intégral de la vidéo. Deux approches particulières ont été retenues et respectivement développées au sein de chaque partie. La partie 1 propose une utilisation des conditions de prise de vue (CPDV) comme modalités d’indexation. Une évaluation de la qualité image permet ainsi de filtrer les segments temporels de mauvaise qualité et donc inexploitables. La classification du mouvement image apparent directement lié au mouvement caméra, fournit une indexation de séquences vidéo en soulignant notamment les segments potentiels d’intérêt ou au contraire les segments difficiles présentant un mouvement très rapide ou oscillant. La partie 2 explore le contenu dynamique de la séquence vidéo, plus précisément la présence d’objets en mouvement. Une première approche locale en temps est présentée. Elle filtre les résultats d’une première classification par apprentissage supervisé en exploitant les informations de contexte, spatial puis sémantique. Différentes approches globales en temps sont par la suite explorées. De telles approches permettent de garantir la cohérence temporelle des résultats et réduire les fausses alarmes.

The research of this PhD thesis is fulfilled in the context of wearable video monitoring of patients with aged dementia. The idea is to provide a new tool to medical practitioners for the early diagnosis of elderly dementia such as the Alzheimer disease. More precisely, Instrumental Activities of Daily Living (IADL) have to be indexed in videos recorded with a wearable recording device. Such videos present specific characteristics i.e. strong motion or strong lighting changes. Furthermore, the tackled recognition task is of a very strong semantics. In this difficult context, the first step of analysis is to define an equivalent to the notion of "shots" in edited videos. We therefore developed a method for partitioning continuous video streams into viewpoints according to the observed motion in the image plane. For the recognition of IADLs we developed a solution based on the formalism of Hidden Markov Models (HMM). A hierarchical HMM with two levels modeling semantic activities or intermediate states has been introduced. A complex set of features (dynamic, static, low-level, mid-level) was proposed and the most effective description spaces were identified experimentally. In the mid-level features for activities recognition we focused on the semantic objects the person manipulates in the camera view. We proposed a new concept for object/image description using local features (SURF) and the underlying semi-local connected graphs. We introduced a nested approach for graphs construction when the same scene can be described by levels of graphs with increasing number of nodes. We build these graphs with Delaunay triangulation on SURF points thus preserving good properties of local features i.e. the invariance with regard to affine transformation of image plane: rotation, translation and zoom. We use the graph features in the Bag-of-Visual-Words framework. The problem of distance or dissimilarity definition between graphs for clustering or recognition is obviously arisen. We propose a dissimilarity measure based on the Context Dependent Kernel of H. Sahbi and show its relation with the classical entry-wise norm when comparing trivial graphs (SURF points). The experiments are conducted on the first corpus in the world of wearable videos of IADL for HMM based activities recognition, and on publicly available academic datasets such as SIVAL and Caltech-101 for object recognition.; Le travail de recherche de cette thèse de doctorat s'inscrit dans le cadre du suivi médical des patients atteints de démences liées à l'âge à l'aide des caméras videos portées par les patients. L'idée est de fournir aux médecins un nouvel outil pour le diagnostic précoce de démences liées à l'âge telles que la maladie d'Alzheimer. Plus précisément, les Activités Instrumentales du Quotidien (IADL : Instrumental Activities of Daily Living en anglais) doivent être indexées automatiquement dans les vidéos enregistrées par un dispositif d'enregistrement portable. Ces vidéos présentent des caractéristiques spécifiques comme de forts mouvements ou de forts changements de luminosité. De plus, la tâche de reconnaissance visée est d'un très haut niveau sémantique. Dans ce contexte difficile, la première étape d'analyse est la définition d'un équivalent à la notion de " plan " dans les contenus vidéos édités. Nous avons ainsi développé une méthode pour le partitionnement d'une vidéo tournée en continu en termes de " points de vue " à partir du mouvement apparent. Pour la reconnaissance des IADL, nous avons développé une solution selon le formalisme des Modèles de Markov Cachés (MMC). Un MMC hiérarchique à deux niveaux a été introduit, modélisant les activités sémantiques ou des états intermédiaires. Un ensemble complexe de descripteurs (dynamiques, statiques, de bas niveau et de niveau intermédiaire) a été exploité et les espaces de description joints optimaux ont été identifiés expérimentalement. Dans le cadre de descripteurs de niveau intermédiaire pour la reconnaissance d'activités nous nous sommes particulièrement intéressés aux objets sémantiques que la personne manipule dans le champ de la caméra. Nous avons proposé un nouveau concept pour la description d'objets ou d'images faisant usage des descripteurs locaux (SURF) et de la structure topologique sous-jacente de graphes locaux. Une approche imbriquée pour la construction des graphes où la même scène peut être décrite par plusieurs niveaux de graphes avec un nombre de nœuds croissant a été introduite. Nous construisons ces graphes par une triangulation de Delaunay sur des points SURF, préservant ainsi les bonnes propriétés des descripteurs locaux c'est-à-dire leur invariance vis-à-vis de transformations affines dans le plan image telles qu'une rotation, une translation ou un changement d'échelle. Nous utilisons ces graphes descripteurs dans le cadre de l'approche Sacs-de-Mots-Visuels. Le problème de définition d'une distance, ou dissimilarité, entre les graphes pour la classification non supervisée et la reconnaissance est nécessairement soulevé. Nous proposons une mesure de dissimilarité par le Noyau Dépendant du Contexte (Context-Dependent Kernel : CDK) proposé par H. Sahbi et montrons sa relation avec la norme classique L2 lors de la comparaison de graphes triviaux (les points SURF). Pour la reconnaissance d'activités par MMC, les expériences sont conduites sur le premier corpus au monde de vidéos avec caméra portée destiné à l'observation des d'IADL et sur des bases de données publiques comme SIVAL et Caltech-101 pour la reconnaissance d'objets.

One important class of online videos is that of news broadcasts. Most news organisations provide near-immediate access to topical news broadcasts over the Internet, through RSS streams or podcasts. Until lately, technology has not made it possible for a user to automatically go to the smaller parts, within a longer broadcast, that might interest them. Recent advances in both speech recognition systems and natural language processing have led to a number of robust tools that allow us to provide users with quicker, more focussed access to relevant segments of one or more news broadcast videos. Here we present our new interface for browsing or searching news broadcasts (video/audio) that exploits these new language processing tools to (i) provide immediate access to topical passages within news broadcasts, (ii) browse news broadcasts by events as well as by people, places and organisations, (iii) perform cross lingual search of news broadcasts, (iv) search for news through a map interface, (v) browse news by trending topics, and (vi) see automatically-generated textual clues for news segments, before listening. Our publicly searchable demonstrator currently indexes daily broadcast news content from 50 sources in English, French, Chinese, Arabic, Spanish, Dutch and Russian., Comment: NEM Summit, Torino : Italy (2011)

In this chapter the authors tackle the analysis and characterization of the artistic animated movies in view of constituting an automatic content-based retrieval system. First, they deal with temporal segmentation, and propose cut, fade and dissolve detection methods adapted to the constraints of this domain. Further, they discuss a fuzzy linguistic approach for automatic symbolic/semantic content annotation in terms of color techniques and action content and we test its potential in automatic video classification. The browsing issue is dealt by providing methods for both, static and dynamic video abstraction. For a quick browse of the movie’s visual content the authors create a storyboard-like summary, while for a “sneak peak” of the movie’s exciting action content they propose a trailer-like video skim. Finally, the authors discuss the architecture of a prototype client-server 3D virtual environment for interactive video retrieval. Several experimental results are presented.

International audience; This work aims at helping multimedia content understanding by deriving bene t from textual clues embedded in digital videos. For this, we developed a complete video Optical Character Recognition system (OCR), speci cally adapted to detect and recognize embedded texts in videos. Based on a neural approach, this new method outperforms related work, especially in terms of robustness to style and size variabilities, to background complexity and to low resolution of the image. A language model that drives several steps of the video OCR is also introduced in order to remove ambiguities due to a local letter by letter recognition and to reduce segmentation errors. This approach has been evaluated on a database of French TV news videos and achieves an outstanding character recognition rate of 95%, corresponding to 78% of words correctly recognized, which enables its incorporation into an automatic video indexing and retrieval system.

VIVIE is a system for video sequence indexing. Video frames are annotated according to the identities of appearing subjects. Different interacting modules perform different processing steps, and each can be possibly substituted with a different one performing the same task using a different method. Classification and clustering are the most challenging activities. Differently from most existing similar systems, VIVIE accounts for the concomitant appearance of two identities in the same clip, and exploits such information for identity mapping. VIVIE was tested on 7 video clips and on a subset of the SCFace database to assess its performances.

International audience; This paper presents a method for indexing activities of daily living in videos obtained from wearable cameras. In the context of dementia diagnosis by doctors, the videos are recorded at patients' houses and later visualized by the medical practitioners. The videos may last up to two hours, therefore a tool for an efficient navigation in terms of activities of interest is crucial for the doctors. The specific recording mode provides video data which are really difficult, being a single sequence shot where strong motion and sharp lighting changes often appear. Our work introduces an automatic motion based segmentation of the video and a video structuring approach in terms of activities by a hierarchical two-level Hidden Markov Model. We define our description space over motion and visual characteristics of video and audio channels. Experiments on real data obtained from the recording at home of several patients show the difficulty of the task and the promising results of our approach.

The objective of this thesis is to extract the informative temporal segments from video sequences, more particularly in aerial video. Manual interpretation of such videos for information gathering faces an ever growing volume of available data. We have thus considered an algorithmic assistance based on different modalities of indexation in order to locate "segments of interest" and avoid a complete visualization of the video. We have chosen two methods in particular and have respectively developed them in each part of this thesis. Part 1 describes how viewing conditions can be used as a method of indexation. The assessment of image quality enables to filter out the temporal segments for which the quality is low and which can thus not be exploited. The classification of global image motion, which is directly linked to camera motion, leads to a method of indexation for video sequences. Indeed, it emphasizes possible segments of interest or, conversely, difficult segments for which motion is very fast or oscillating. Part 2 focuses on the dynamic content of video sequences, especially the presence of moving objects. We first present a local (in time) approach. This approach refines the results obtained after a first classification by supervised learning by using contextual information, spatial then semantic information. We have then investigated several methods for moving object detection which are global in time. Such approaches aim to enforce the temporal consistency of the detected objects and to reduce false detections., Les travaux réalisés dans le cadre de cette thèse ont pour objectif d’extraire les différents segments informatifs au sein de séquences vidéo, plus particulièrement aériennes. L’interprétation manuelle de telles vidéos dans une optique de renseignement se heurte en effet au volume des données disponibles. Une assistance algorithmique fondée sur diverses modalités d’indexation est donc envisagée, dans l’objectif de repérer les "segments d’intérêt" et éviter un parcours intégral de la vidéo. Deux approches particulières ont été retenues et respectivement développées au sein de chaque partie. La partie 1 propose une utilisation des conditions de prise de vue (CPDV) comme modalités d’indexation. Une évaluation de la qualité image permet ainsi de filtrer les segments temporels de mauvaise qualité et donc inexploitables. La classification du mouvement image apparent directement lié au mouvement caméra, fournit une indexation de séquences vidéo en soulignant notamment les segments potentiels d’intérêt ou au contraire les segments difficiles présentant un mouvement très rapide ou oscillant. La partie 2 explore le contenu dynamique de la séquence vidéo, plus précisément la présence d’objets en mouvement. Une première approche locale en temps est présentée. Elle filtre les résultats d’une première classification par apprentissage supervisé en exploitant les informations de contexte, spatial puis sémantique. Différentes approches globales en temps sont par la suite explorées. De telles approches permettent de garantir la cohérence temporelle des résultats et réduire les fausses alarmes.

This paper presents a method for indexing human ac- tivities in videos captured from a wearable camera being worn by patients, for studies of progression of the dementia diseases. Our method aims to produce indexes to facilitate the navigation throughout the individual video recordings, which could help doctors search for early signs of the dis- ease in the activities of daily living. The recorded videos have strong motion and sharp lighting changes, inducing noise for the analysis. The proposed approach is based on a two steps analysis. First, we propose a new approach to segment this type of video, based on apparent motion. Each segment is characterized by two original motion de- scriptors, as well as color, and audio descriptors. Second, a Hidden-Markov Model formulation is used to merge the multimodal audio and video features, and classify the test segments. Experiments show the good properties of the ap- proach on real data., Comment: 2011 9th International Workshop on Content-Based Multimedia Indexing (CBMI), Madrid : Spain (2011)

This paper presents a novel unsupervised method for identifying the semantic structure in long semi-structured video streams. We identify ‘chains’, local clusters of repeated features from both the video stream and audio transcripts. Each chain serves as an indicator that the temporal interval it demarcates is part of the same semantic event. By layering all the chains over each other, dense regions emerge from the overlapping chains, from which we can identify the semantic structure of the video. We analyze two clustering strategies that accomplish this task. ispartof: pages:103-108 ispartof: Proceedings of the 9th international workshop on content-based multimedia indexing pages:103-108 ispartof: 9th international workshop on content-based multimedia indexing location:Madrid, Spain date:13 Jun - 15 Jun 2011 status: published

The research of this PhD thesis is fulfilled in the context of wearable video monitoring of patients with aged dementia. The idea is to provide a new tool to medical practitioners for the early diagnosis of elderly dementia such as the Alzheimer disease. More precisely, Instrumental Activities of Daily Living (IADL) have to be indexed in videos recorded with a wearable recording device. Such videos present specific characteristics i.e. strong motion or strong lighting changes. Furthermore, the tackled recognition task is of a very strong semantics. In this difficult context, the first step of analysis is to define an equivalent to the notion of "shots" in edited videos. We therefore developed a method for partitioning continuous video streams into viewpoints according to the observed motion in the image plane. For the recognition of IADLs we developed a solution based on the formalism of Hidden Markov Models (HMM). A hierarchical HMM with two levels modeling semantic activities or intermediate states has been introduced. A complex set of features (dynamic, static, low-level, mid-level) was proposed and the most effective description spaces were identified experimentally. In the mid-level features for activities recognition we focused on the semantic objects the person manipulates in the camera view. We proposed a new concept for object/image description using local features (SURF) and the underlying semi-local connected graphs. We introduced a nested approach for graphs construction when the same scene can be described by levels of graphs with increasing number of nodes. We build these graphs with Delaunay triangulation on SURF points thus preserving good properties of local features i.e. the invariance with regard to affine transformation of image plane: rotation, translation and zoom. We use the graph features in the Bag-of-Visual-Words framework. The problem of distance or dissimilarity definition between graphs for clustering or recognition is obviously arisen. We propose a dissimilarity measure based on the Context Dependent Kernel of H. Sahbi and show its relation with the classical entry-wise norm when comparing trivial graphs (SURF points). The experiments are conducted on the first corpus in the world of wearable videos of IADL for HMM based activities recognition, and on publicly available academic datasets such as SIVAL and Caltech-101 for object recognition., Le travail de recherche de cette thèse de doctorat s'inscrit dans le cadre du suivi médical des patients atteints de démences liées à l'âge à l'aide des caméras videos portées par les patients. L'idée est de fournir aux médecins un nouvel outil pour le diagnostic précoce de démences liées à l'âge telles que la maladie d'Alzheimer. Plus précisément, les Activités Instrumentales du Quotidien (IADL : Instrumental Activities of Daily Living en anglais) doivent être indexées automatiquement dans les vidéos enregistrées par un dispositif d'enregistrement portable. Ces vidéos présentent des caractéristiques spécifiques comme de forts mouvements ou de forts changements de luminosité. De plus, la tâche de reconnaissance visée est d'un très haut niveau sémantique. Dans ce contexte difficile, la première étape d'analyse est la définition d'un équivalent à la notion de " plan " dans les contenus vidéos édités. Nous avons ainsi développé une méthode pour le partitionnement d'une vidéo tournée en continu en termes de " points de vue " à partir du mouvement apparent. Pour la reconnaissance des IADL, nous avons développé une solution selon le formalisme des Modèles de Markov Cachés (MMC). Un MMC hiérarchique à deux niveaux a été introduit, modélisant les activités sémantiques ou des états intermédiaires. Un ensemble complexe de descripteurs (dynamiques, statiques, de bas niveau et de niveau intermédiaire) a été exploité et les espaces de description joints optimaux ont été identifiés expérimentalement. Dans le cadre de descripteurs de niveau intermédiaire pour la reconnaissance d'activités nous nous sommes particulièrement intéressés aux objets sémantiques que la personne manipule dans le champ de la caméra. Nous avons proposé un nouveau concept pour la description d'objets ou d'images faisant usage des descripteurs locaux (SURF) et de la structure topologique sous-jacente de graphes locaux. Une approche imbriquée pour la construction des graphes où la même scène peut être décrite par plusieurs niveaux de graphes avec un nombre de nœuds croissant a été introduite. Nous construisons ces graphes par une triangulation de Delaunay sur des points SURF, préservant ainsi les bonnes propriétés des descripteurs locaux c'est-à-dire leur invariance vis-à-vis de transformations affines dans le plan image telles qu'une rotation, une translation ou un changement d'échelle. Nous utilisons ces graphes descripteurs dans le cadre de l'approche Sacs-de-Mots-Visuels. Le problème de définition d'une distance, ou dissimilarité, entre les graphes pour la classification non supervisée et la reconnaissance est nécessairement soulevé. Nous proposons une mesure de dissimilarité par le Noyau Dépendant du Contexte (Context-Dependent Kernel : CDK) proposé par H. Sahbi et montrons sa relation avec la norme classique L2 lors de la comparaison de graphes triviaux (les points SURF). Pour la reconnaissance d'activités par MMC, les expériences sont conduites sur le premier corpus au monde de vidéos avec caméra portée destiné à l'observation des d'IADL et sur des bases de données publiques comme SIVAL et Caltech-101 pour la reconnaissance d'objets.

This report is focused on the study and the characterization of Dynamic Textures (DTs), with the aim of video indexing in large databases.This research topic being new and emerging, we propose a taxonomy, a definition of DTs and a state of the art. The most representative DT class is described by a model that considers DTs as the superposition of several wavefronts and local oscillating phenomena.The design of spatio-temporal analysis tools adapted to DT is our main contribution. We first show that the 2D+T curvelet transform is relevant for representing wavefronts. In order to analyse and better understand the DTs, we propose in a second step to adapt the Morphological Component Analysis approach using new thresholding strategies. These methods are tested on several applications: decomposition of DTs, spatio-temporal segmentation, global motion estimation of a DT, ...We have shown that Morphological Component Analysis and multi-scale approaches enable significant results for content-based retrieval applications and dynamic texture indexing on the DynTex database. This thesis constitutes a first step towards automatic indexing of DTs in image sequences and opens the way for many new developments in this topic. Moreover, the proposed approaches are generic and could be applied in a broader context, for instance the processing of 3D data.; Nous nous intéresserons dans cette thèse à l'étude et la caractérisation des Textures Dynamiques (TDs), avec comme application visée l'indexation dans de grandes bases de vidéos.Ce thème de recherche étant émergent, nous proposons une définition des TDs, une taxonomie de celles-ci, ainsi qu'un état de l'art. La classe de TD la plus représentative est décrite par un modèle formel qui considère les TDs comme la superposition d'ondes porteuses et de phénomènes locaux.La construction d'outils d'analyse spatio-temporelle adaptés aux TDs est notre principale contribution. D'une part, nous montrons que la transformée en curvelets 2D+T est pertinente pour la représentation de l'onde porteuse. D'autre part, dans un objectif de décomposition des séquences vidéos, nous proposons d'utiliser l'approche par Analyse en Composantes Morphologiques. Notre contribution consiste en l'apport et l'étude de nouvelles stratégies de seuillage. Ces méthodes sont testées sur plusieurs applications: segmentation spatio-temporelle, décomposition de TDs, estimation du mouvement global d'une TD, ...Nous avons de plus montré que l'Analyse en Composantes Morphologiques et les approches multi-échelles donnent des résultats significatifs pour la recherche par le contenu et l'indexation de Textures Dynamiques de la base de données DynTex.Cette thèse constitue ainsi un premier pas vers l'indexation automatique de textures dynamiques dans des séquences d'images, et ouvre la voie à de nombreux développements sur ce sujet nouveau. Enfin, le caractère générique des approches proposées permet d'envisager leurs applications dans un cadre plus large mettant en jeu par exemple des données 3D.

International audience; This paper introduces a novel on-line video browsing and retrieval platform, so-called OVIDIUS (On-line VIDeo Indexing Universal System). In contrast with traditional and commercial main stream video retrieval platforms, where video content is treated in a more or less monolithic manner (i.e., with global descriptions associated with the whole document), the proposed approach makes it possible to browse and access video content in a finer, per-segment basis. The hierarchical metadata structure exploits the MPEG-7 approach for structural description of video content. The MPEG-7 description schemes have been here enriched with both semantic and content-based metadata. The developed approach shows all its pertinence within a multi-terminal context and in particular for video access from mobile devices. The platform has been recently (February, 2010) validated within the framework of the Médi@TIC French national project

International audience; Our research focuses on analysing human activities according to a known behaviorist scenario, in case of noisy and high dimensional collected data. The data come from the monitoring of patients with dementia diseases by wearable cameras. We define a structural model of video recordings based on a Hidden Markov Model. New spatio-temporal features, color features and localization features are proposed as observations. First results in recognition of activities are promising.

This paper investigates the problem of managing very large digital television archives. This problem is called television structuring (Or TV broadcast macro-segmentation) and is defined as the process of identifying the structure of a television stream as watchers perceive it: a succession of programs. This is the very first step in order to manage a television collection. In this report, a complete solution for television structuring is proposed, which makes use of simple yet efficient methods in order to deal with huge datasets. Methods from commercial detection are generalized to be able to distinguish regular programs from non-programs. It is shown how television program guides can be used to label the identified programs. It is finally shown how an update procedure can improve the segmentation results over time. Results are provided on 3 weeks of French television.; Ce rapport de recherche s'intéresse à la structuration de larges volumes d'archives de télévision. Par structuration, nous entendons l'identification des programmes de télévision, leur début, leur fin, dans le flux, et donc le découpage de ce flux en une succession de programmes. Ceci est la toute première étape dans un processus d'indexation d'un flux de télévision, afin de le rendre facilement navigable et requêtable. Nous présentons une solution somplète basée sur des méthodes simples afin de pouvoir traiter de très importantes quantités de données. Nous généralisons des méthodes provenant de la détection de publicités télévisées afin de distinguer les programmes et les inter-programmes. Il est également montré comment les guides de programmes peuvent être utilisés afin d'étiqueter les programmes identifiés. Nous proposons finalement une procédure de mise à jour, qui permet d'obtenir des résultats constants au cours du temps. Des résultats sur trois semaines de télévision française permettent de vérifier l'efficacité des méthodes.

International audience; With the fastly developing standards for video compression, the availability of the video content at a large scale in the compressed form has become an everyday reality. Compressed video streams whatever is the compression standard contain primary information which without full decompression of the content can serve for its indexing and retrieval. This primary features optimized for the compression purposes are very often noisy and subject to compression artefacts. Hence their re-use for analysis and indexing purposes remains a challenge. In this paper we present the methodology which starting from the early low - resolution standards of MPEG family up to the new high-qualityhigh definition video compression approaches aims at answering the same question: how to re-use noisy primary features for fast and efficient video indexing.

This thesis is about to explore the use of tools to support semantics of data in the field of multimedia. The first contribution concerns the generation of high-level descriptions. We propose a description language that allows high-level definition of events and objects from low-level features. The second contribution is the exploration of certain types of uncertainty reasoning in the context of multimedia semantics. We propose a semantic language (based on fuzzy conceptual graphs) for descriptions of videos and define mechanisms underlying reasoning. The third contribution relates to the semantic indexing and retrieval in multimedia databases. We propose a query language from deductive databases for the expression of spatiotemporal and semantic queries, Cette thèse consiste à explorer l'usage d'outils de support de la sémantique des données dans le domaine du multimédia. La première contribution concerne la génération de descriptions de haut-niveau. Nous proposons un langage de description de haut-niveau qui permet la définition d'événements et d'objets à partir de caractéristiques de bas-niveau. La deuxième contribution concerne l'exploration de certains types de raisonnement dans le contexte de la multimédia sémantique. Nous proposons un langage sémantique (fondé sur les graphes conceptuels flous) pour la description des vidéos et définissons des mécanismes de raisonnement sous-jacents. La troisième contribution porte sur l'indexation et la recherche sémantique dans les bases de données multimédia. Nous proposons un langage de requêtes issu des bases de données déductives pour l'expression de requêtes spatiotemporelles et sémantiques

This thesis attempts to realize three advances in the domain of semantic multimedia, which can be defined as the application of semantic (Web) techniques to multimedia resources. The first contribution concerns the generation of high-level (semantic) descriptions: indeed as the extraction of low-level multimedia features was largely studied and solved in the image analysis community, the automatic generation of high-level descriptions from low level features is still an open problem. In this thesis, a high level description language for defining events and objects from low-level features is proposed as an attempt to narrow this "semantic gap". The second contribution concerns the reasoning for semantic multimedia. Indeed as semantic web languages have been conceived for the description of all kinds of resources, they may lack special features for tackling and reasoning on multimedia resources, that contain especially spatial and temporal information, and whose interpretation often contains a part of uncertainty. The second contribution of this thesis addresses this problem by proposing a semantic language for videos based on fuzzy conceptual graphs, and by providing the corresponding reasoning procedures. The third contribution concerns multimedia databases and the indexing and retrieval techniques for semantic multimedia. Indeed a query language for semantic multimedia should allow a user to express spatiotemporal constraints as well as the semantic constraints, and a query engine should be able to process theses constraints together. The third contribution in this thesis consists in proposing a datalog-like query language for expressing spatiotemporal and semantic queries, and a constraint solving and reasoning procedure for answering to these queries, Cette thèse a pour but de réaliser trois avancés sur le domaine de la multimédia sémantique, qui peut être définie comme l'application des techniques du sémantique web aux ressources multimédia. La première contribution concerne la génération de descriptions haut-niveau (sémantiques); en effet, alors que l'extraction de caractéristiques de bas niveau a été largement étudiée par la communauté du traitement d'images, la génération automatique de descriptions de haut-niveau est encore un problème ouvert. Dans cette thèse, un langage de descriptions de haut-niveau, qui permet la définition d'événements et d'objets à partir de caractéristiques de bas-niveau, est proposé comme une tentative pour résoudre le problème du "semantic gap". La deuxième contribution concerne le résonnement dans la multimédia sémantique. En effet, bien que les langages du sémantique web aient été conçus pour la description de tous les types de ressources, ces langages ne sont pas aussi performent quand il s'agit de raisonner sur la multimédia sémantique. Celle-ci contient des informations spatiotemporelles dont l'interprétation nécessite souvent la résolution de l'incertitude. La deuxième contribution de cette thèse traite ce problème en proposant un langage sémantique pour la description des vidéos basé sur les graphs conceptuels flous, mais aussi en proposant les procédures de raisonnement correspondantes. La troisième contribution porte sur l'indexation et la recherche sémantique dans les bases de données multimédia. En effet un langage de requêtes pour la multimédia sémantique doit permettre à l'utilisateur d'exprimer facilement les contraintes spatiotemporelles aussi bien que les contraintes sémantiques. Par ailleurs, le moteur de requêtes doit être capable de résoudre l'ensemble de ses requêtes. La troisième contribution de cette thèse propose un langage de requête issu des bases de données déductives pour l'expression de requêtes spatiotemporelles et sémantiques, mais aussi les méthodes de raisonnement nécessaires pour répondre à ces requêtes

International audience; In the video indexing framework, we have developed a user assistance system in order to define concept models (i.e semantic index) according to features automatically extracted from the video. Because the manual indexing is a long and tedious task, we propose to focus the user attention on pre-selected prototypes that a priori correspond to the searched concepts. The proposed system is decomposed in three steps. In the first one, some basic spatio-temporal blocks are extracted from the video, a particular block being associated to a particular property of one feature. In the second step, a Question/Answer system allows the user to define links between basic blocks in order to define concept block models. And finally, some concept blocks are extracted and proposed as prototypes of the concepts. In this paper, we present the two first steps, particularly the block structure, illustrated by an example of video indexing that corresponds to the concept running in athletic videos.

Video indeksleme ve erişimi sistemleri büyük çaptaki video verilerine hızlı, doğal ve kolay bir şekilde ulaşılabilmesini amaçlar. Son zamanlarda video arşivlerinin çok hızlı büyümesiyle bu sistemlerin önemi daha da artmıştır. Bu tez, video çözümleme, indeksleme ve erişimi konularında yeni yöntemler öneren BilVideo-7 sistemini sunmaktadır.BilVideo-7, karmaşık çok kipli video sorgularını aynı anda destekleyen, dağıtık mimariye sahip, MPEG-7 uyumlu bir video indeksleme ve erişimi sistemidir. Video veri modeli bir MPEG-7 pro ? li üzerine bina edilmiş olup, videolar bu pro ? le uygun olarak çekimlere, anahtar karelere, durağan ve hareketli bölgelere ayrılmaktadır. Videoların bu veri modeline uygun XML gösterimleri, BilVideo-7'nin MPEG-7 uyumlu video öznitelik çıkarma ve etiketleme yazılımı yardımıyla elde edilip XML veritabanında saklanmaktadır. Kullanıcılar, görsel sorgulama arayüzünü kullanarak metin, renk, doku, biçim, konum, hareket ve uzamsal-zamansal sorguları kolay bir şekilde yapabilmektedir. Kompozit sorgu arayüzü, kullanıcıların, istenilen sayıda video parçasını ve betimleyicisini bir araya getirip aralarındaki uzamsal-zamansal ilişkileri belirleyerek, oldukça karmaşık, çok kipli sorguları kolayca formüle edebilmesini sağlamaktadır. Sorgular, çok izlekli bir sorgu işleme sunucusu tarafından işlenmekte; istemcilerden gelen sorgular önce alt sorgulara ayrılmakta ve herbir sorgu, kendi sorgu tipine ait biz izlek tarafından işlenmektedir. Daha sonra, alt sorgu sonuçları birleştirilerek nihai sorgu sonucu elde edilip istemciye geri gönderilmektedir. Sistemin bir bütün olarak özgünlüğü, MPEG-7 uyumlu bir ortamda, detaylı bir video veri modeli, çok sayıda betimleyici ve çok kipli sorgu işleme özelliği ile güçlü bir video indeksleme ve sorgulama sistemi olmasıdır. Video indexing and retrieval aims to provide fast, natural and intuitive access to large video collections. This is getting more and more important as the amount of video data increases at a stunning rate. This thesis introduces the BilVideo-7 system to address the issues related to video parsing, indexing and retrieval.BilVideo-7 is a distributed and MPEG-7 compatible video indexing and retrieval system that supports complex multimodal queries in a uni ? ed framework. The video data model is based on an MPEG-7 pro ? le which is designed to represent the videos by decomposing them into Shots, Keyframes, Still Regions and Moving Regions. The MPEG-7 compatible XML representations of videos according to this pro ? le are obtained by the MPEG-7 compatible video feature extraction and annotation tool of BilVideo-7, and stored in a native XML database. Users can formulate text, color, texture, shape, location, motion and spatio-temporal queries on an intuitive, easy-to-use visual query interface, whose composite query interface can be used to formulate very complex queries containing any type and number of video segments with their descriptors and specifying the spatio-temporal relations between them. The multi-threaded query processing server parses incoming queries into subqueries and executes each subquery in a separate thread. Then, it fuses subquery results in a bottom-up manner to obtain the ? nal query result and sends the result to the originating client. The whole system is unique in that it provides very powerful querying capabilities with a wide range of descriptors and multimodal query processing in an MPEG-7 compatible interoperable environment. 116

This report is focused on the study and the characterization of Dynamic Textures (DTs), with the aim of video indexing in large databases.This research topic being new and emerging, we propose a taxonomy, a definition of DTs and a state of the art. The most representative DT class is described by a model that considers DTs as the superposition of several wavefronts and local oscillating phenomena.The design of spatio-temporal analysis tools adapted to DT is our main contribution. We first show that the 2D+T curvelet transform is relevant for representing wavefronts. In order to analyse and better understand the DTs, we propose in a second step to adapt the Morphological Component Analysis approach using new thresholding strategies. These methods are tested on several applications: decomposition of DTs, spatio-temporal segmentation, global motion estimation of a DT, ...We have shown that Morphological Component Analysis and multi-scale approaches enable significant results for content-based retrieval applications and dynamic texture indexing on the DynTex database. This thesis constitutes a first step towards automatic indexing of DTs in image sequences and opens the way for many new developments in this topic. Moreover, the proposed approaches are generic and could be applied in a broader context, for instance the processing of 3D data., Nous nous intéresserons dans cette thèse à l'étude et la caractérisation des Textures Dynamiques (TDs), avec comme application visée l'indexation dans de grandes bases de vidéos.Ce thème de recherche étant émergent, nous proposons une définition des TDs, une taxonomie de celles-ci, ainsi qu'un état de l'art. La classe de TD la plus représentative est décrite par un modèle formel qui considère les TDs comme la superposition d'ondes porteuses et de phénomènes locaux.La construction d'outils d'analyse spatio-temporelle adaptés aux TDs est notre principale contribution. D'une part, nous montrons que la transformée en curvelets 2D+T est pertinente pour la représentation de l'onde porteuse. D'autre part, dans un objectif de décomposition des séquences vidéos, nous proposons d'utiliser l'approche par Analyse en Composantes Morphologiques. Notre contribution consiste en l'apport et l'étude de nouvelles stratégies de seuillage. Ces méthodes sont testées sur plusieurs applications: segmentation spatio-temporelle, décomposition de TDs, estimation du mouvement global d'une TD, ...Nous avons de plus montré que l'Analyse en Composantes Morphologiques et les approches multi-échelles donnent des résultats significatifs pour la recherche par le contenu et l'indexation de Textures Dynamiques de la base de données DynTex.Cette thèse constitue ainsi un premier pas vers l'indexation automatique de textures dynamiques dans des séquences d'images, et ouvre la voie à de nombreux développements sur ce sujet nouveau. Enfin, le caractère générique des approches proposées permet d'envisager leurs applications dans un cadre plus large mettant en jeu par exemple des données 3D.

Tez (Yüksek Lisans) -- İstanbul Teknik Üniversitesi, Fen Bilimleri Enstitüsü, 2010, Thesis (M.Sc.) -- İstanbul Technical University, Institute of Science and Technology, 2010, Bu tez kapsamında, pek çok alanda kullanılan video parmak izi sistemlerinin özellikle videoda kopya sezme alanında kullanımı üzerinde durulmuştur. Kopya sezme problemlerinde bir parmak izi sisteminin, aynı video içeriğinin kopyasını bozulmaya uğramış olsa dahi sezebilmesi istenir. Bu yüzden video parmak izi sistemleri video verisine uygulanabilecek global bozulmalara (sıkıştırma, çerçeve hızını değiştirme, gürültü, gama bozulması vb.) ve geometrik dönüşümlere (aynalama, kaydırma, kırpma, yeniden boyutlandırma gibi) karşı dayanıklı olmalıdır. Pratikte kopya sezme alanında kullanılan arama-getirme sistemleri, kendilerine verilen bir videonun, daha önceden parmak izi çıkarma uygulaması tarafından veritabanına kaydedilmiş videolar arasında olup olmadığını anlamaya çalışır. Bu çalışmada biri anahtar-çerçeve, diğeri içerik tabanlı olmak üzere iki video parmak izi çıkarma yöntemi önerilmiştir. Önerilen yöntemlerin her ikisinde de video çerçevelerinden negatif olmayan matris ayrıştırma (NOMA) yöntemi ile elde edilen baz ve kodlama matrisleri kullanılarak elde edilen video parmakizi vektörlerinin kullanımı önerilmektedir. Performans testleri TRECVID 2002, 2009 ve 2010 veritabanlarından alınan video klipleri üzerinden gerçeklenmiştir ve başarım doğruluk, hatırlama ve bunların harmonik ortalaması olan kopya tespit başarım oranı (DCR) cinsinden raporlanmıştır., Modelling and retrieving video content is a difficult problem due to increasing amounts of media in applications such as broadcast monitoring, P2P network filtering or copy detection and constraints on media content representations. Video fingerprinting systems have been proposed to deal with such problems and used in wide range of applications. Copy detection features of video fingerprinting systems are the main concern of this work. In copy detection applications, a fingerprinting system should be able to match an input query video clip with its original content which has been previously introduced to the system. Thus, video fingerprinting system should be robust aginst possible modifications on input data. These modifications are generally mentioned under two different classes as global distortions and geometric transformations. Video re-encoding, frame rate conversion, changes in brightness, contrast and gamma, introduction of additive noise and blur can be given as examples of global distortions while flipping, shifting, cropping and resizing attacks are the examles of geometric transformations. In this work two video fingerprinting approaches are proposed. Both approaches use non-negative matrix factorization (NMF) as feature extraction tool but differs especially in selection of the input data. In the first approach, extracted fingerprints only correspond to the selected key-frames of the input video clip, while in the content based second method fingerprints are extracted for groups of successive frames. Proposed systems are tested on TRECVID 2002, 2009 and 2010 video databases and performance of the system is evaluated by using precision and recall rates as well as their harmonical average DCR (detection cost rate)., Yüksek Lisans, M.Sc.

I review my research activities in Video Indexing and Action Recognition and sketch a research agenda for bringing those two lines of research together to address the difficult problem of recognizing actions in movies. I first present a series of older projects in Video Indexing, starting with the DIVAN project at INA and the MPEG expert group (1998-2000), and continuing at INRIA under the VIBES project (2001-2004). This research falls under the general approach of "computational media aesthetics", where we attempt to recognize film events based on our knowledge of filming styles and conventions (cinematography and editing). This is illustrated with two applications - the automatic segmentation of TV news into topics; and the automatic indexing of movies with their scripts. I then present my more recent research in Action Recognition with the MOVI group at INRIA (2005-2008). Building upon the GRIMAGE infrastructure, I present experiments in (1) learning and recognizing a small repertoire of full body gestures in 3D using "motion history volumes"; (2) segmenting a raw stream of 3D image sequences into recognizable "primitive actions" ; and (3) using statistical models learned in 3D for recovering primitive actions and relative camera positions from a single 2D video recording of similar actions.; Je présente mes activités de recherche en indexation video et en reconnaissance d'actions, et je propose un programme de recherche permettant d'aborder ensemble ces deux questions au cours des prochaines années. Je décris d'abord une série de travaux réalisés dans le cadre du groupe MPEG et des projets DIVAN à l'INA (1998-2000), puis VIBES à l'INRIA (2001-2004), et qui visent à aborder l'indexation video à travers la reconnaissance des styles et conventions de la prise de vues et du montage. Cette première partie est illustrée par deux applications - le découpage d'un journal télévisé en sujets, et l'indexation d'un film de cinéma par son script. Je présente ensuite des travaux réalisés à l'INRIA en 2005-2008 au sein de l'équipe MOVI. Je montre comment nous avons utilisé l'infrastructure GRIMAGE pour (1) apprendre des modèles statistiques 3D d'un petit répertoire d'actions humaines permettant de les reconnaitre lorsqu'elles sont exécutées par d'autres acteurs, sous d'autres points de vue ; (2) découper une séquence d'images 3D en actions primitives reconnaissables; et (3) reconnaître ces mêmes actions selon le point de vue d'une seule caméra. Enfin, je propose quelques pistes pour étendre les résultats précédents afin d'aborder simultanément les deux problèmes de la reconnaissance des actions et des styles de mise en scène dans les films. Je présente les avantages et les difficultés d'une approche unifiée de ces deux problèmes, ainsi que des applications possibles dans les domaines de la fiction interactive, du jeu vidéo et du machinima.

I review my research activities in Video Indexing and Action Recognition and sketch a research agenda for bringing those two lines of research together to address the difficult problem of recognizing actions in movies. I first present a series of older projects in Video Indexing, starting with the DIVAN project at INA and the MPEG expert group (1998-2000), and continuing at INRIA under the VIBES project (2001-2004). This research falls under the general approach of "computational media aesthetics", where we attempt to recognize film events based on our knowledge of filming styles and conventions (cinematography and editing). This is illustrated with two applications - the automatic segmentation of TV news into topics; and the automatic indexing of movies with their scripts. I then present my more recent research in Action Recognition with the MOVI group at INRIA (2005-2008). Building upon the GRIMAGE infrastructure, I present experiments in (1) learning and recognizing a small repertoire of full body gestures in 3D using "motion history volumes"; (2) segmenting a raw stream of 3D image sequences into recognizable "primitive actions" ; and (3) using statistical models learned in 3D for recovering primitive actions and relative camera positions from a single 2D video recording of similar actions.; Je présente mes activités de recherche en indexation video et en reconnaissance d'actions, et je propose un programme de recherche permettant d'aborder ensemble ces deux questions au cours des prochaines années. Je décris d'abord une série de travaux réalisés dans le cadre du groupe MPEG et des projets DIVAN à l'INA (1998-2000), puis VIBES à l'INRIA (2001-2004), et qui visent à aborder l'indexation video à travers la reconnaissance des styles et conventions de la prise de vues et du montage. Cette première partie est illustrée par deux applications - le découpage d'un journal télévisé en sujets, et l'indexation d'un film de cinéma par son script. Je présente ensuite des travaux réalisés à l'INRIA en 2005-2008 au sein de l'équipe MOVI. Je montre comment nous avons utilisé l'infrastructure GRIMAGE pour (1) apprendre des modèles statistiques 3D d'un petit répertoire d'actions humaines permettant de les reconnaitre lorsqu'elles sont exécutées par d'autres acteurs, sous d'autres points de vue ; (2) découper une séquence d'images 3D en actions primitives reconnaissables; et (3) reconnaître ces mêmes actions selon le point de vue d'une seule caméra. Enfin, je propose quelques pistes pour étendre les résultats précédents afin d'aborder simultanément les deux problèmes de la reconnaissance des actions et des styles de mise en scène dans les films. Je présente les avantages et les difficultés d'une approche unifiée de ces deux problèmes, ainsi que des applications possibles dans les domaines de la fiction interactive, du jeu vidéo et du machinima.

8 pages; International audience; This paper presents four spatio-temporal wavelet decompositions for characterizing dynamic textures. The main goal of this work is to compare the influence of spatial and temporal variables in the wavelet decomposition scheme. Its novelty is to establish a comparison between the only existing method and three other spatio-temporal decompositions. The four decomposition schemes are presented and successfully applied on a large dynamic texture database. Construction of feature descriptors are tackled as well their relevance, and performances of the methods are discussed. Finally, future prospects are exposed.