Search

The goal of Content-Based Retrieval (CBR) is to provide quick access to relevant content stored in multimedia digital libraries that contain enormous video data. Most video CBR systems retrieve shots, or a collection of shots, based on user input. Thus, the tools for retrieving segments of a video program are not explored fully, though they form a meaningful utility for a CBR user. Parsing video programs into program segments is useful in retrieval of individual segments and video summarization. Many video classes show structure in them that can be effectively modeled using Finite-State Automata (FSA). In this paper, we present a FSA-based system that extracts contextual structure from news video database. Each video segment such as newscaster sequence, weather sequence, etc., becomes a node in FSA. The transition is fired from one node to another node, based on arc conditions, which can be easily obtained by employing statistical methods on classified data. Modeling with FSA avoids the use of complex rule-based system. Experimental results presented with FSA approach for more than 8 hours of video data show an accuracy of 88% in recognizing the components of news video. [ABSTRACT FROM AUTHOR]

We introduce a behavior-based similarity measure that tells us whether two different space-time intensity patterns of two different video segments could have resulted from a similar underlying motion field. This is done directly from the intensity information, without explicitly computing the underlying motions. Such a measure allows us to detect similarity between video segments of differently dressed people performing the same type of activity. It requires no foreground/background segmentation, no prior learning of activities, and no motion estimation or tracking. Using this behavior-based similarity measure, we extend the notion of two-dimensional image correlation into the three-dimensional space-time volume and thus allowing to correlate dynamic behaviors and actions. Small space-time video segments (small video clips) are "correlated" against the entire video sequences in all three dimensions (x, y, and t). Peak correlation values correspond to video locations with similar dynamic behaviors. Our approach can detect very complex behaviors in video sequences (for example, ballet movements, pool dives, and running water), even when multiple complex activities occur simultaneously within the field of view of the camera. We further show its robustness to small changes in scale and orientation of the correlated behavior. [ABSTRACT FROM AUTHOR]

Abstract: We propose in this paper an original method of camera motion classification based on Transferable Belief Model (TBM). It consists in locating in a video the motions of translation and zoom, and the absence of camera motion (i.e static camera). The classification process is based on a rule-based system that is divided into three stages. From a parametric motion model, the first stage consists in combining data to obtain frame-level belief masses on camera motions. To ensure the temporal coherence of motions, a filtering of belief masses according to TBM is achieved. The second stage carries out a separation between static and dynamic frames. In the third stage, a temporal integration allows the motion to be studied on a set of frames and to preserve only those with significant magnitude and duration. Then, a more detailed description of each motion is given. Experimental results obtained show the effectiveness of the method. [Copyright &y& Elsevier]

For video indexing, the problem of video cut detection remains largely an open problem because of the wide nature of transitions that occur in a digital video. This paper describes a shot boundary detection technique, which is an amalgamation of few statistical methods and measures, and robustly detects camera breaks in a full-motion video clip. The proposed algorithm incorporates a weighted histogram, an error-propagation technique for increased robustness, and a curve-fitting technique to extract partitions from the similarity curve for avoiding heuristically chosen threshold value. The algorithm has been validated on many video clips and is shown to give improved results, including for videos with rapid scanning, changes in illumination, fade-ins and fade-outs, and with special effects like dissolve and filters. [ABSTRACT FROM AUTHOR]

Copyright of Ciência da Informação is the property of Instituto Brasileiro de Informacao em Ciencia e Tecnologia (IBICT) and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Abstract: The benchmarking of various methods of video analysis and indexing has become a research problem per se. While a lot of effort nowadays is devoted to organization of evaluation campaigns both on international, such as TREC Video and national levels, an in-depth analysis of the performance of the methods still remains an open issue. Several aspects in the evaluation of methods have to be addressed: how to design a benchmarking corpus in order to cover a sufficiently wide range of applications, what are the tasks to address, what are the most relevant evaluation metrics. The Argos evaluation campaign supported by the French Techno-Vision program aimed at developing resources for a benchmarking of video content analysis and indexing methods. The paper describes the type of tasks evaluated, the way the content set has been produced, metrics and tools developed for the evaluations and some of the results obtained at the end of the first phase. Perspectives based on current works will be given to conclude this paper. [Copyright &y& Elsevier]

Abstract: In this paper, we compare active learning strategies for indexing concepts in video shots. Active learning is simulated using subsets of a fully annotated data set instead of actually calling for user intervention. Training is done using the collaborative annotation of 39 concepts of the TRECVID 2005 campaign. Performance is measured on the 20 concepts selected for the TRECVID 2006 concept detection task. The simulation allows exploring the effect of several parameters: the strategy, the annotated fraction of the data set, the size of the data set, the number of iterations and the relative difficulty of concepts. Three strategies were compared. The first two, respectively, select the most probable and the most uncertain samples. The third one is a random choice. For easy concepts, the “most probable” strategy is the best one when less than 15% of the data set is annotated and the “most uncertain” strategy is the best one when 15% or more of the data set is annotated. The “most probable” and “most uncertain” strategies are roughly equivalent for moderately difficult and difficult concepts. In all cases, the maximum performance is reached when 12–15% of the whole data set is annotated. This result is, however, dependent upon the step size and the training set size. One-fortieth of the training set size is a good value for the step size. The size of the subset of the training set that has to be annotated in order to reach the maximum achievable performance varies with the square root of the training set size. The “most probable” strategy is more “recall oriented” and the “most uncertain” strategy is more “precision oriented”. [Copyright &y& Elsevier]

The performance of shot boundary detection algorithms using low level features can hardly fulfill the requirement of automatic sports video analysis, which is a promising module in Personal Video Recorders. Two kinds of middle level features are proposed in this paper to effectively enhance the shot boundary detection. One kind of the features is extracted from the projection of the Dominant Color mask; the other kind is extracted from the reliable block-based Motion Vectors. These novel features, together with the Region Color Histograms feature, are integrated into a unified model which employs Support Vector Machines to detect both cuts and gradual transitions. Experiments on diversified soccer video sequences demonstrate that our scheme outperforms the existing algorithms and is quite competent for the targeted applications. [ABSTRACT FROM AUTHOR]

In this paper we are proposing an improved cut detection algorithm adapted to the animation movies domain. A cut is the direct concatenation of two different shots and produces an important visual discontinuity in the video stream. As color is one major feature of the animation movies (each movie has its own particular color palette) the proposed approach performs by thresholding the distance values obtained between frame color histograms. To overcome the difficulty raised by the peculiarity of the animation movies several improvements have been adopted. The frames are divided intro quadrants to reduce the influence of the entering/exiting objects. The second order derivative is used to reduce the influence of the fast object/camera motions. For the frame classification, an automatic threshold estimation is proposed. Moreover, to reduce the false detections, we are proposing an algorithm to detect a particular color effect, named "short color changes" (i.e. lightnings, explosions, flashes etc.). The proposed cut detection algorithm achieved better results compared to the conventional histogram-based and motion-discontinuity based approaches as shown by tests conducted on several animation movies. [ABSTRACT FROM AUTHOR]

Abstract: Video in digital format is now commonplace and widespread in both professional use, and in domestic consumer products from camcorders to mobile phones. Video content is growing in volume and while we can capture, compress, store, transmit and display video with great facility, editing videos and manipulating them based on their content is still a non-trivial activity. In this paper, we give a brief review of the state of the art of video analysis, indexing and retrieval and we point to research directions which we think are promising and could make searching and browsing of video archives based on video content, as easy as searching and browsing (text) web pages. We conclude the paper with a list of grand challenges for researchers working in the area. [Copyright &y& Elsevier]

The film grammar used in movie-making industry dictates that most story units start with an establishment shot, which has the intention of bringing the audience to a new environment. Identification of establishment shots can aid in scene segmentation and video summarization. This paper describes a retrieval system that is capable of detecting establishment shots using camera motion features. Experimental results on several hours of real video data show the potential of our approach. [ABSTRACT FROM AUTHOR]

This paper aims at repeat clip mining and knowledge discovery from video data. A unified approach is proposed to detect both unknown video repeats and known video clips of arbitrary length. Two detectors in a cascade structure are employed to achieve fast and accurate detection, and a reinforcement learning approach is adopted to efficiently maximize detection accuracy. In this approach very short video repeats (<1 s) and long ones can be detected by a single process, while overall accuracy remains high. Since video segmentation is essential for repeat detection, performance analysis is also conducted for several segmentation methods. Furthermore we propose a method to analyze video syntactical structure based on short video repeats detection. Experimental results on news videos demonstrate that identifying short video repeats is an effective way for video structure discovery and syntactical segmentation [ABSTRACT FROM PUBLISHER]

Abstract: This paper describes a fully automatic content-based approach for browsing and retrieval of MPEG-2 compressed video. The first step of the approach is the detection of shot boundaries based on motion vectors available from the compressed video stream. The next step involves the construction of a scene tree from the shots obtained earlier. The scene tree is shown to capture some semantic information as well as to provide a construct for hierarchical browsing of compressed videos. Finally, we build a new model for video similarity based on global as well as local motion associated with each node in the scene tree. To this end, we propose new approaches to camera motion and object motion estimation. The experimental results demonstrate that the integration of the above techniques results in an efficient framework for browsing and searching large video databases. [Copyright &y& Elsevier]

Real-world action recognition applications require the development of systems which are fast, can handle a large variety of actions without a priori knowledge of the type of actions, need a minimal number of parameters, and necessitate as short as possible learning stage, in this paper, we suggest such an approach. We regard dynamic activities as long-term temporal objects, which are characterized by spatio-temporal features at multiple temporal scales. Based on this, we design a simple statistical distance measure between video sequences which captures the similarities in their behavioral content. This measure is nonparametric and can thus handle a wide range of complex dynamic actions having a behavior-based distance measure between sequences, we use it for a variety of tasks, including: video indexing, temporal segmentation, and action-based video clustering. These tasks are performed without prior knowledge of the types of actions, their models, or their temporal extents. [ABSTRACT FROM AUTHOR]

This paper targets hard-cut detection for archive film, i.e., mainly black-and-white videos from the beginning of the last century, which is a particularly difficult task due to heavy visual degradations encountered in the sequences. A robust hard-cut detection system based on modified phase correlation is presented. Phase-correlation-based hard-cut detection is carried out using spatially sub-sampled video frames, and a candidate hard-cut is indicated in the case of low correlation. A double thresholding approach consisting of a global threshold used in conjunction with an adaptive local threshold is used to detect candidate hard-cuts. For uniformly colored video frames the phase correlation is extremely sensitive to noise and visual defects. Mean and variance based simple heuristic false removal at uniformly colored video frames is used at the final stage to prevent false detections in such cases. The paper provides a through theoretical analysis to show the usefulness of spatial sub-sampling. Furthermore through experimental results are presented for visual defects encountered in archive film material, to present the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

Motion characterization plays a critical role in video indexing. An effective way of characterizing camera motion facilitates the video representation, indexing and retrieval tasks. This paper describes a novel nonparametric motion representation to achieve an effective and robust recognition of parts of the video in which camera is static, or panning, or tilting, or zooming, etc. This representation employs the mean shift filtering and the vector histograms to produce a compact description of a motion field. The basic idea is to perform spatio-temporal mode-seeking in the motion feature space and use the histograms-based spatial distributions of dominant motion modes to represent a motion field. Unlike most existing approaches, which focus on the estimation of a parametric motion model from a dense optical flow field (OFF) or a block matching-based motion vector field (MVF), the proposed method combines the motion representation and machine learning techniques (e.g., support vector machines) to perform camera motion analysis from the classification point of view. The main motivation lies in the impossibility of uniformly securing a proper parametric assumption in a wide range of video scenarios. The diverse camera shot sizes and frequent occurrences of bad OFF/MVF necessitates a learning mechanism, which can not only capture the domain-independent parametric constraints, but also acquire the domain-dependent knowledge to tolerate the influence of bad OFF/MVF. In order to improve performance, we can use this learning-based method to train enhanced classifiers aiming at a certain context (i.e., shot size, neighbor OFF/MVFs, and video genre). Other visual cues (e.g., dominant color) can also be incorporated for further motion analysis. Our main aim is to use a generic feature space analysis method to explore a flexible OFF/MVF representation in a nonparametric technique, which could be fed into a learning framework to robustly capture the global motion by incorporating the context information. Results on videos with various types of content (23 191 MVFs culled from MPEG-7 dataset, and 20 000 MVFs culled from broadcast tennis, soccer, and basketball videos) are reported to validate the proposed approach. [ABSTRACT FROM PUBLISHER]

Abstract: This paper describes a fully automatic content-based approach for browsing and retrieval of MPEG-2 compressed video. The first step of the approach is the detection of shot boundaries based on motion vectors available from the compressed video stream. The next step involves the construction of a scene tree from the shots obtained earlier. The scene tree is shown to capture some semantic information as well as provide a construct for hierarchical browsing of compressed videos. Finally, we build a new model for video similarity based on global as well as local motion associated with each node in the scene tree. To this end, we propose new approaches to camera motion and object motion estimation. The experimental results demonstrate that the integration of the above techniques results in an efficient framework for browsing and searching large video databases. [Copyright &y& Elsevier]

A recent focus in web based learning systems has been the development of reusable learning materials that can be delivered as personalized courses depending of a number of factors such as the user's background, his/her learning preferences, current knowledge based on previous assessments, or previous browsing patterns. The student is often confronted with complex information mining tasks in which the semantics of individual sources require a deeper modelling than is offered by current learning systems. Most authored content exist in the form of videos, audio, slides, text, and simulations. In the absence of suitable annotations, the conversion of such materials for on-line distribution, presentation, and personalization has proven to be difficult. Based on our experiences with Open Courseware (OCW) and Singapore-MIT Alliance (SMA) video database, this paper presents a personalized delivery system that uses a domain ontology and pedagogical models to compose course materials in response to a user's query. We also present several important E-learning applications emerging from the framework. [ABSTRACT FROM AUTHOR]

Gradual shot change detection is one of the most important research issues in the field of video indexing/retrieval. Among the numerous types of gradual transitions, the dissolve-type gradual transition is considered the most common one, but it is also the most difficult one to detect. In most of the existing dissolve detection algorithms, the false/miss detection problem caused by motion is very serious. In this paper, we present a novel dissolve-type transition detection algorithm that can correctly distinguish dissolves from disturbance caused by motion. We carefully model a dissolve based on its nature and then use the model to filter out possible confusion caused by the effect of motion. Experimental results show that the proposed algorithm is indeed powerful. [ABSTRACT FROM PUBLISHER]

Robust signal hashing defines a feature vector that characterizes the signal, independently of "nonsignificant" distortions of its content. When dealing with images, the considered distortions are typically due to compression or small geometrical manipulations. In other words, robustness means that images that are visually indistinguishable should produce equal or similar hash values. To discriminate image contents, a hash function should produce distinct outputs for different images. Our paper first proposes a robust hashing algorithm for still images. It is based on radial projection of the image pixels and is denoted the Radial hASHing (RASH) algorithm. Experimental results provided on the USC-SIPI dataset reveal that the proposed RASH feature vector is more robust and provides much stronger discrimination than a conventional histogram-based feature vector. The RASH vector appears to be a good candidate to build indexing algorithms, copy-detection systems, or content-based authentication mechanisms. To take benefit from the RASH vector capabilities, video content is summarized into key frames, each of them characterizing a video shot and described by its RASH vector. The resulting video hashing system works in real time and supports most distortions due to common spatial and temporal video distortions. [ABSTRACT FROM PUBLISHER]

We present a system for automatically extracting the region of interest (ROI) and controlling virtual cameras' control based on panoramic video. It targets applications such as classroom lectures and video conferencing. For capturing panoramic video, we use the FlyCam system that produces high resolution, wide-angle video by stitching video images from multiple stationary cameras. To generate conventional video, a region of interest can be cropped from the panoramic video. We propose methods for ROI detection, tracking, and virtual camera control that work in both the uncompressed and compressed domains. The ROI is located from motion and color information in the uncompressed domain and macroblock information in the compressed domain, and tracked using a Kalman filter. This results in virtual camera control that simulates human controlled video recording. The system has no physical camera motion and the virtual camera parameters are readily available for video indexing. [ABSTRACT FROM PUBLISHER]

Due to the information redundancy of video, automatically extracting essential video content is one of key techniques for accessing and managing large video library. In this paper, we present a generic framework of a user attention model, which estimates the attentions viewers may pay to video contents. As human attention is an effective and efficient mechanism for information prioritizing and filtering, user attention model provides an effective approach to video indexing based on importance ranking. In particular, we define viewer attention through multiple sensory perceptions, i.e. visual and aural stimulus as well as partly semantic understanding. Also, a set of modeling methods for visual and aural attentions are proposed. As one of important applications of user attention model, a feasible solution of video summarization, without fully semantic understanding of video content as well as complex heuristic rules, is implemented to demonstrate the effectiveness, robustness, and generality of the user attention model. The promising results from the user study on video summarization indicate that the user attention model is an alternative way to video understanding. [ABSTRACT FROM PUBLISHER]

In this paper, we propose an optimal key frame representation scheme based on global statistics for video shot retrieval Each pixel in this optimal key frame is constructed by considering the probability of occurrence of those pixels at the corresponding pixel position among the frames in a video shot. Therefore, this constructed key frame is called temporally maximum occurrence frame (TMOF), which is an optimal representation of all th frames in a video shot. The retrieval performance of this representation scheme is further improved by considering the k pixel values with the largest probabilities of occurrence and the highest peaks of the probability distribution of occurrence at each pixel position for a video shot. The corresponding schemes an called k-TMOF and k-pTMOF, respectively. These key frame representation schemes are compared to other histogram-based techniques for video shot representation and retrieval. In the experiments, three video sequences in the MPEG-7 content set were used to evaluate the performances of the different key frame representation schemes. Experimental results show that our proposed representations outperform the alpha-trimmed average histogram for video retrieval. [ABSTRACT FROM AUTHOR]

We propose the time interval multimedia event (TIME) framework as a robust approach for classification of semantic events in multimodal video documents. The representation used in TIME extends the Allen temporal interval relations and allows for proper inclusion of context and synchronization of the heterogeneous information sources involved in multimodal video analysis. To demonstrate the viability of our approach, it was evaluated on the domains of soccer and news broadcasts. For automatic classification of semantic events, we compare three different machine learning techniques, i.c. C4.5 decision tree, maximum entropy, and support vector machine. The results show that semantic video indexing results significantly benefit from using the TIME framework. [ABSTRACT FROM PUBLISHER]

Abstract: The content-based multimedia indexing is today considered as being a key technology for numerous applications. A standard, called MPEG-7, has been elaborated. It provides a framework of standardized tools that can be used to describe and efficiently manage multimedia content. While many researches have concentrated on generating the descriptors the environments and applications, there has been little effort devoted to real time descriptors extraction. The real time constraint is very important for several applications such as real time annotation of MPEG-7 video streams. Video-parsing methods dealing with uncompressed video are characterized by their computational intensive aspect. In addition, complex methods or methods combining several algorithms are more robust and lead to better quality, but they are known by their greediness in computational resources. So, when dealing with real-time treatment, we have to use efficient technological solutions to succeed the implementation of these techniques. The used solutions must satisfy the different application requirements such as calculation power, flexibility, consumption, etc. In this paper we propose a hardware implementation of the compact color descriptor (CCD), which is a color descriptor in the MPEG-7 standard. It can be used for content filtering; image matching and browsing application. Several implementation solutions of the different CCD extraction steps are given. [Copyright &y& Elsevier]

A first step required to allow video indexing and retrieval of visual data is to perform a temporal segmentation, that is, to find the location of camera-shot transitions, which can be either abrupt (i.e., cuts) or gradual (e.g., fades, dissolves, wipes). After a critical review of most approaches seeking to solve this problem, we propose a unified detection model (both for abrupt and all types of gradual transitions), as well as an implementation whose results improve upon those of all the inspected reports. The innovation of the approach presented here is centered on mapping the space of inter-frame distances onto a new space of decision better suited to achieving a sequence-independent thresholding. This mapping aims to consider frame ordering information within the thresholding process; it is based on the parametric modeling of the patterns that transitions generate on the distances' output. As opposed to most reviewed works, our results are detailed over a large and representative sample of more than 1500 cuts and 250 gradual transitions, which make up a significant part (200 min) of the MPEG-7 testing material; this ensures a high degree of confidence in the validity of our approach. [ABSTRACT FROM PUBLISHER]

The dramatic growth in the amount of multimedia data in recent years has caused techniques for analyzing those data to become increasingly important. To make use of information contained in multimedia data to the fullest extent, attention has been focused on techniques for processing multiple types of media in an integrated manner. Among the various kinds of videos that exist, the authors selected cooking videos, which are closely related to everyday life, to propose a technique for associating the videos with preparation steps described in related text-based materials referred to simply as textbooks. With educational videos having related textbooks such as cooking program videos, the textbooks are easier to consult than the videos but the videos contain useful visual information that cannot be represented in the textbooks. Therefore, integration of the video and textbook is expected to result in high-order video structuring and indexing. In this paper, the authors first analyze the video structure and define the video blocks that are to be the video units used for association. They also analyze the preparation steps described in the closed captions and textbooks and propose and define an association technique based on keyword extraction. In addition, they show through evaluation experiments that association in terms of video block units or, in other words, video indexing, can be performed with high precision by using the proposed technique. They also show that limiting the subject matter and skillfully incorporating relatively simple elemental techniques enables a practical level of precision to be obtained. © 2005 Wiley Periodicals, Inc. Syst Comp Jpn, 36(2): 51–62, 2005; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.20131 [ABSTRACT FROM AUTHOR]

This paper looks into a new direction in video content analysis - the representation and modeling of affective video content . The affective content of a given video clip can be defined as the intensity and type of feeling or emotion (both are referred to as affect) that are expected to arise in the user while watching that clip. The availability of methodologies for automatically extracting this type of video content will extend the current scope of possibilities for video indexing and retrieval. For instance, we will be able to search for the funniest or the most thrilling parts of a movie, or the most exciting events of a sport program. Furthermore, as the user may want to select a movie not only based on its genre, cast, director and story content, but also on its prevailing mood, the affective content analysis is also likely to contribute to enhancing the quality of personalizing the video delivery to the user. We propose in this paper a computational framework for affective video content representation and modeling. This framework is based on the dimensional approach to affect that is known from the field of psychophysiology. According to this approach, the affective video content can be represented as a set of points in the two-dimensional (2-D) emotion space that is characterized by the dimensions of arousal (intensity of affect) and valence (type of affect). We map the affective video content onto the 2-D emotion space by using the models that link the arousal and valence dimensions to low-level features extracted from video data. This results in the arousal and valence time curves that, either considered separately or combined into the so-called affect curve, are introduced as reliable representations of expected transitions from one feeling to another along a video, as perceived by a viewer. [ABSTRACT FROM PUBLISHER]

How many times did you miss a conference talk in a parallel track and wish you had a second chance to see it? Or you just wanted to see a few talks from a recent conference you did not attend? Video proceedings, which contain videos of all the conference talks, would be of great value in these cases. With recent progress in digital video, streaming technology, large storage, Internet and especially video indexing and retrieval technology, video proceedings finally become a reality. The key challenges are efficient production with minimal labor and easy, intuitive, and rapid user access to talks and thought-for snippet of information. This paper describes an application that allows a nearly automatic, real time creation of video proceedings. All the talks are captured in video, and are automatically indexed by speech recognition and video analysis tools. Free text search in speech and efficient multi-view video browsing are combined with the conference table of contents and speakers biography to make fully searchable and browsable video proceedings. The paper covers the work flow, processing steps, and technical details of the video segmentation, visualization, and user study results. The system was used to produce video proceedings for three local conferences. [ABSTRACT FROM AUTHOR]

In this paper, we present a general framework and new effective algorithms to detect the syntactic structures that are at a level higher than shots. In sports video, such high-level structures are often characterized by the specific views (e.g., pitching or serve) and the subsequent temporal transition patterns within each temporal structural segment. We have developed robust statistical models for detecting the domain-specific views with real-time performance and high accuracy. The models combine domain-independent global color filtering method and domain-specific constraints on the spatio-temporal properties of the segmented regions (e.g., locations, shapes, and motion of the objects). The real-time performance was accomplished by using efficient compressed-domain processing at the front end and computational expensive object-level processing on filtered candidates only. High-level events (e.g., strokes, net plays, and baseline plays) are also detected after the view recognition. Results of such structure and event detection allow for efficient browsing and summarization of long sports video programs. [ABSTRACT FROM AUTHOR]

As amounts of publicly available video data grow, the need to automatically infer semantics from raw video data becomes significant. In this paper, we address the use of three different techniques that support that task, namely, spatio-temporal rule-based method, hidden Markov models, and dynamic Bayesian networks. First, the application of these techniques for detection and recognition of diverse events is briefly described using two case studies (Tennis and Formula 1). We explain the relationships and differences of the three approaches, as well as benefits of their integrated use. Then the focus is moved to the main point of the paper, which is the integration of the aforementioned techniques within a database management system, which provides efficient, flexible, scalable, and domain independent content-based video retrieval. We identify and consider the most important issues when extending a traditional database management system with content-based video retrieval functionality, namely issues concerning video data models, dynamic feature extraction, and extensions of different layers of database architecture. The advantages of the integrated system are demonstrated on examples from our two case studies. [ABSTRACT FROM AUTHOR]

We present a psychophysical and analytical framework for comparing the performance of motion activity measures for video segments, with respect to a subjective ground truth. We first obtain a ground truth for the motion activity by conducting a psychophysical experiment. Then we present several low-complexity motion activity descriptors computed from compressed domain block motion vectors. In the first analysis, we quantize the descriptors and show that they perform well against the ground truth. The MPEG-7 motion activity descriptor is also among the best performers. In the second analysis, we examine the specific cases where each descriptor fails, using a novel pair-wise comparison method. The analytical measures overestimate or underestimate the intensity of motion activity under strong camera motion or extreme camera angles. We finally discuss the experimental methodology and analysis methods we used, and possible alternatives. We review the applications of motion activity and how our results relate to them. [ABSTRACT FROM AUTHOR]

A video retrieval system consists of two major subsystems for indexing and querying, respectively. In this paper, we propose a framework for retrieving video sequences using modular operations on temporal similarity. In the indexing process, the contents (objects) of a video sequence are used to build temporal triples representing temporal relationships between objects. For every temporal triple generated from different video sequences, a prime number is assigned to it by the proposed prime number assignment (PNA) algorithm. For each video sequence, a standing value is generated via its set of prime numbers. In the query process, successive modular operations (SMO) on the standing values are used to retrieve video sequences. Based on the PNA and the SMO, we built an experimental video retrieval system. From the simulation results, we notice that a query can be done in a linear time. [Copyright &y& Elsevier]

Human motion analysis is a recent topic of interest among the computer vision and video processing community. Research in this area is motivated by its wide range of applications such as surveillance and monitoring systems. In this paper we describe a system for recognition of various human actions from compressed video based on motion history information. We introduce the notion of quantifying the motion involved, through what we call Motion Flow History (MFH). The encoded motion information readily available in the compressed MPEG stream is used to construct the coarse Motion History Image (MHI) and the corresponding MFH. The features extracted from the static MHI and MFH compactly characterize the spatio-temporal and motion vector information of the action. Since the features are extracted from the partially decoded sparse motion data, the computational load is minimized to a great extent. The extracted features are used to train the KNN, Neural network, SVM and the Bayes classifiers for recognizing a set of seven human actions. The performance of each feature set with respect to various classifiers are analyzed. [Copyright &y& Elsevier]

We present an approach for compact video summaries that allows fast and direct access to video data. The video is segmented into shots and, in appropriate video genres, into scenes, using previously proposed methods. A new concept that supports an hierarchical representation of video is presented, and is based on physical setting and camera locations. We use mosaics to represent shots and then scenes. We use a novel method for mosaic comparison which is robust against changes in viewpoint and illumination. In contrast to approaches to video indexing which rely on a frame-based representation, our efficient mosaic-based representation allows fast clustering of scenes into physical settings, a new conceptual form grounded in the recognition of real-world backgrounds. We employ a technique for choosing representative mosaics for each physical setting, for a more compact representation and faster comparison between settings. This compact representation and comparison method runs in real time and allows fast and accurate summaries and comparison of scenes across different videos, and serves as a basis for indexing video libraries. We demonstrate our work using situation comedies (sitcoms), where each half-hour episode is well structured by rules governing background use. Consequently, browsing, indexing, and comparison across videos by physical setting is very fast. Further, we show that physical settings lead to a higher-level contextual identification of the main plots in each video. We demonstrate these contributions with a browsing tool whose top-level single page displays the settings of several episodes. In sports videos where settings are not as well defined, our approach allows classifying shots for characteristic event detection. [Copyright &y& Elsevier]

TV advertising commercials are a critical marketing tool for many companies. Their interspersion within regular broadcast television programming can be entertaining, informing, annoying or a sales gold mine depending on one's viewpoint. As a result, there are two major reasons for being able to detect commercial segments within television broadcasts. These two applications' goals - at least indirectly - are at odds with each other. One application seeks to identify and track when specific commercials are broadcast. Advertisers, in particular, like to verify that their contracts with broadcasters are fulfilled as promised. The other group wants to detect commercials for the purpose of eliminating them from their recordings. This group consists of viewers who want to watch their recorded television shows without the annoyance of commercials. Video database maintainers would also appreciate the ability to automatically edit out commercials in stored shows and thereby decrease storage requirements. Of course, advertisers are strongly opposed to such devices because that defeats the purpose of the commercials. The article discusses several algorithms that have been experimentally used to detect commercials, as well as devices that are currently available for this purpose. [ABSTRACT FROM PUBLISHER]

An adaptive mechanism for video partitioning in the transform domain is proposed. Different quantitative measures for motion complexity and activity levels in a scene are defined, based on which a video scene can be consistently categorised into identifiable classes. Further, the video quality, as measured by the mean square error (MSE) is related to certain parameters used in video partitioning. Adaptability is realized by tailoring the parameters of the video partitioning algorithm to the specific characteristics of the video scene, as embodied in the video scene class and the video quality. Experimental results are included. [ABSTRACT FROM PUBLISHER]

In this paper we propose a highlight detection method and an indexing method for broadcast sports video using the collaborative processing of text, audio, and image. In the proposed method, the appearance pattern of words in the closed caption text stream is analyzed, and candidate intervals for highlights are detected. Next, these intervals are checked based on their audio levels, and those which seem to be erroneous are rejected. Finally, the resulting highlight intervals are segmented into shots, and the shots are indexed by identifying the highlight shots based on audio levels and dominant color information. In the results of using this method on a real football broadcast, highlight intervals were effectively detected with a recall rate of 77% and a precision rate of 84%. Moreover, for the intervals in which highlights were correctly detected, shot indexing was performed accurately 75% for only the first candidate, and 97% for up to the second candidate. We verified experimentally that efficient processing can be achieved by stepwise analysis of the text, audio, and image. © 2003 Wiley Periodicals, Inc. Syst Comp Jpn, 34(12): 22–31, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10493 [ABSTRACT FROM AUTHOR]

One key technology of intelligent transportation systems is the use of advanced sensor systems for on-line surveillance to gather detailed information on traffic conditions. Traffic video analysis can provide a wide range of useful information to traffic planners. In this context, the object-level indexing of video data can enable vehicle classification, traffic flow analysis, incident detection and analysis at intersections, vehicle tracking for traffic operations, and update of design warrants. In this paper, a learning-based automatic framework is proposed to support the multimedia data indexing and querying of spatio-temporal relationships of vehicle objects in a traffic video sequence. The spatio-temporal relationships of vehicle objects are captured via the proposed unsupervised image/video segmentation method and object tracking algorithm, and modeled using a multimedia augmented transition network model and multimedia input strings. An efficient and effective background learning and subtraction technique is employed to eliminate the complex background details in the traffic video frames. It substantially enhances the efficiency of the segmentation process and the accuracy of the segmentation results to enable more accurate video indexing and annotation. The paper uses four real-life traffic video sequences from several road intersections under different weather conditions in the study experiments. The results show that the proposed framework is effective in automating data collection and access for complex traffic situations. [ABSTRACT FROM PUBLISHER]

In this paper, we attempt to provide a comprehensive and high-level review of audio-visual features that can be extracted from the standard compressed domains, such as MPEG-1 and MPEG-2. The paper is motivated by the myriad of active research works in extraction and application of compressed-domain features in various fields, such as indexing, filtering, and manipulation. Compressed-domain approaches avoid expensive computation and memory requirements involved in decoding and/or re-encoding. Selected features are categorized into four groups—spatial visual (e.g., color, texture, edge, shape), motion (e.g., motion field, trajectory), audio (e.g., energy, spectral features, pitch), and coding (e.g., bit rate, frame/block type). For each feature, we briefly discuss the extraction methods, computational complexity, potential effectiveness in applications, and possible limitations caused by compress-domain approaches. Finally, we also discuss the possibilities of extracting some important MPEG-7 visual and audio descriptors directly from the compressed domain. [Copyright &y& Elsevier]

The increased availability and usage of multimedia information have created a critical need for efficient multimedia processing algorithms. These algorithms must offer capabilities related to browsing, indexing, and retrieval of relevant data. A crucial step in multimedia processing is that of reliable video segmentation into visually coherent video shots through scene change detection. Video segmentation enables subsequent processing operations on video shots, such as video indexing, semantic representation, or tracking of selected video information. Since video sequences generally contain both abrupt and gradual scene changes, video segmentation algorithms must be able to detect a large variety of changes. While existing algorithms perform relatively well for detecting abrupt transitions (video cuts), reliable detection of gradual changes is much more difficult. A novel one-pass, real-time approach to video scene change detection based on statistical sequential analysis and operating on a compressed multimedia bitstream is proposed. Our approach models video sequences as stochastic processes, with scene changes being reflected by changes in the characteristics (parameters) of the process. Statistical sequential analysis is used to provide an unified framework for the detection of both abrupt and gradual scene changes. [ABSTRACT FROM PUBLISHER]

Video representation through key frames has been addressed frequently as an efficient way of preserving the whole temporal information of sequence with a considerably smaller amount of data. Such compact video representation is suitable for the purpose of video browsing in limited storage or transmission bandwidth environments. In this case, the controllability of the total key frame number (i.e. key frame rate) depending on the storage or bandwidth capacity is an important requirement for the key frame selection method. In this paper, we present a sequential key frame selection method when the number of key frames is given as a constraint. It first selects the pre-determined number of initial key frames and time-intervals. Then, it adjusts the positions of key frames and time-intervals by iteration, which reduces the distortion step by step. Experimental results demonstrate the improved performance of our algorithm over the existing approaches. [Copyright &y& Elsevier]

There are a growing number of applications, which extensively use the visual media. A key requirement in those applications is efficient access to the stored visual information for the purposes of indexing, fast retrieval, and scene analysis. To support this vision, many scene change detection algorithms have been developed for both sudden and gradual scene change in uncompressed domain. This paper presents a comparison of shot boundary detection and classification techniques and their variations including histogram, pixel differences, likelihood ratio, and motion vector. In addition, the pros and cons of each approach with many clarifying comments are presented. [ABSTRACT FROM AUTHOR]

Current information and communication technologies provide the infrastructure to transport bits anywhere, but do not indicate how to easily and precisely access and/or route information at the semantic level. To facilitate intelligent access to the rich multimedia data over the Internet, we develop an on-line knowledge- and rule-based video classification system that supports automatic “indexing” and “filtering” based on the semantic concept hierarchy. This paper investigates the use of video and audio content analysis, feature extraction and clustering techniques for further video semantic concept classification. A supervised rule-based video classification system is proposed using video automatic segmentation, annotation and summarization techniques for seamless information browsing and updating. In the proposed system, a real-time scene-change detection proxy performs an initial video-structuring process by splitting a video clip into scenes. Motional, visual and audio features are extracted in real-time for every detected scene by using on-line feature-extraction proxies. Higher semantics are then derived through a joint use of low-level features along with classification rules in the knowledge base. Classification rules are derived through a supervised learning process that relies on some representative samples from each semantic category. An indexing and filtering process can now be built using the semantic concept hierarchy to personalize multimedia data based on users’ interests. In real-time filtering, multiple video streams are blocked, combined, or sent to certain channels depending on whether or not the video streams are matched with the user''s profile. We have extensively experimented and evaluated the classification and filtering techniques using basketball sports video data. In particular, in our experiment, the basketball video structure is examined and categorized into different classes according to distinct motional, visual and audio characteristics features by a rule-based classifier. The concept hierarchy describing the motional/visual/audio feature descriptors and their statistical relationships are reported in this paper along with detailed experimental results using on-line sports videos. [Copyright &y& Elsevier]

Shot boundary detection, or scene change detection, is a technique used in the initial step of video indexing. With the recent introduction of object-based shot boundary detection technique, the deficiency of traditional detection techniques in handling long transitions has been overcome. However, such a technique merely detects the existence of a transition without giving accurate positions of the transition start and end for long transitions. Shot boundary refinement fills this gap by pinpointing the exact location of the shot boundary regardless of the transition types. This paper presents the theories behind the transition and classifies existing transition types. A novel shot boundary refinement algorithm is then constructed which works independently of the contents and transition length, and is accurate under most transition types. The results of testing using different indicators and different transition types show that the boundaries of long transitions are detected with minimal errors. The success of shot boundary refinement brings us one step closer in developing an accurate, fully automatic indexing system. [ABSTRACT FROM PUBLISHER]

Various types of information can be mixed in a continuous video stream without any clear boundary. For this reason, there seems to be no simple solution to support content-based queries in a video database. The meaning of a video scene can be interpreted by multiple levels of abstraction and its description can be varied among different users. Therefore, it is important for a user to be able to describe a scene flexibly while the description given by different users should be maintained consistently. This paper proposes an effective way to represent the different types of video information in the conventional relational database model. Flexibly defined attributes and their values are organized as tree-structured dictionaries while the description of video data is stored in a fixed database schema. In order to assist a user, two browsing methods are introduced. The dictionary browser simplifies the annotation process as well as the querying process of a user while the result browser can help a user analyze the results of a query in terms of various combinations of query conditions. [ABSTRACT FROM AUTHOR]

This article presents applications of entropic spanning graphs to imaging and feature clustering applications. Entropic spanning graphs span a set of feature vectors in such a way that the normalized spanning length of the graph converges to the entropy of the feature distribution as the number of random feature vectors increases. This property makes these graphs naturally suited to applications where entropy and information divergence are used as discriminants: texture classification, feature clustering, image indexing, and image registration. Among other areas, these problems arise in geographical information systems, digital libraries, medical information processing, video indexing, multisensor fusion, and content-based retrieval. [ABSTRACT FROM PUBLISHER]

In this paper, we propose event-based video indexing, which is a kind of indexing by its semantical contents. Because video data is composed of multimodal information streams such as visual, auditory, and textual [closed caption (CC)] streams, we introduce a strategy of intermodal collaboration, i.e., collaborative processing taking account of the semantical dependency between these streams. Its aim is to improve the reliability and efficiency in contents analysis of video. Focusing here on temporal correspondence between visual and CC streams, the proposed method attempts to seek for time spans in which events are likely to take place through extraction of keywords from the CC stream and then to index shots in the visual stream. The experimental results for broadcasted sports video of American football games indicate that intermodal collaboration is effective for video indexing by the events such as touchdown (TD) and field goal (FG) [ABSTRACT FROM PUBLISHER]

In this paper, we introduce the Interactive Systems Laboratories multimedia data indexing and retrieval system ‘View4You’. The main components of the system, namely the segmenter, the speech recognizer and the information retrieval engine, are described in detail. In the View4You system, public television newscasts are recorded on a daily basis. The newscasts are automatically segmented and an index is created for each of the segments by means of automatic speech recognition. The user can query the system in natural language. The system returns a list of segments which is sorted by relevance with respect to the user query. By selecting a segment, the user can watch the corresponding part of the news show on his or her computer screen. Several end to end evaluations on real world data, using questions from naive users, are described. By substituting each of the components of the system with a perfect (manually simulated) one, the effect of the components' imperfection on the end to end result can be determined. We show that the information retrieval component has the largest impact on the system performance, followed by the segmentation. The quality of the speech recognizer, as long as its error rate is below approximately 25%, is shown to have only a relatively small importance. [ABSTRACT FROM AUTHOR]

We present an original approach for motion-based video retrieval involving partial query. More precisely, we propose a unified statistical framework allowing us to simultaneously extract entities of interest in video shots and supply the associated content-based characterization, which can be used to satisfy partial queries. It relies on the analysis of motion activity in video sequences based on a non-parametric probabilistic modeling of motion information. Areas comprising relevant types of motion activity are extracted from a Markovian region-level labeling applied to the adjacency graph of an initial block-based partition of the image. As a consequence, given a set of videos, we are able to construct a structured base of samples of entities of interest represented by their associated statistical models of motion activity. The retrieval operations is then formulated as a Bayesian inference issue using the MAP criterion. We report different results of extraction of entities of interest in video sequences and examples of retrieval operations performed on a base composed of one hundred video samples. [ABSTRACT FROM AUTHOR]