Search

As natural images usually contain multiple objects, multi-label image classification is more applicable "in the wild" than single-label classification. However, exhaustively annotating images with every object of interest is costly and time-consuming. We aim to train multi-label classifiers from single-label annotations only. We show that adding a consistency loss, ensuring that the predictions of the network are consistent over consecutive training epochs, is a simple yet effective method to train multi-label classifiers in a weakly supervised setting. We further extend this approach spatially, by ensuring consistency of the spatial feature maps produced over consecutive training epochs, maintaining per-class running-average heatmaps for each training image. We show that this spatial consistency loss further improves the multi-label mAP of the classifiers. In addition, we show that this method overcomes shortcomings of the "crop" data-augmentation by recovering correct supervision signal even when most of the single ground truth object is cropped out of the input image by the data augmentation. We demonstrate gains of the consistency and spatial consistency losses over the binary cross-entropy baseline, and over competing methods, on MS-COCO and Pascal VOC. We also demonstrate improved multi-label classification mAP on ImageNet-1K using the ReaL multi-label validation set., Comment: 24 pages, 9 figures

ispartof: pages:269-278 ispartof: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) pages:269-278 ispartof: IEEE/CVF winter conference on applications of computer vision - WACV 2023 location:Waikoloa, Hawaii, USA date:3 Jan - 7 Jan 2023 status: published

Spatially dynamic neural networks adjust network execution based on the input data, saving computations by skipping non-important image regions. Yet, GPU implementations fail to achieve speedups from these spatially dynamic execution patterns for most neural network architectures. This paper investigates hardware constraints preventing such speedup and proposes and compares novel processor architectures and dataflows enabling latency improvements due to the dynamic execution with minimal loss of utilization. The presented architectures flexibly support spatial execution of a broad range of networks. For the derived architectures, the spatial unrolling for each layer type is optimized and validated making use of the ZigZag design space exploration framework where appropriate. This allows to benchmark and compare the hardware architectures on NNs for classification and human pose estimation, increasing throughput up to $\times 1.9$ and $\times 2.3$ compared to their static executions, respectively. This is the same order of magnitude as other dynamic execution methods, while being complementary to those.

Anticipating the motion of dynamic objects is critical for making intelligent decisions navigating through an environment while avoiding collisions. In this work, we propose a CNN model that estimates 3D motion of objects using sequences of monocular images. We show that we can train this model without using any manual annotations by using Iterative Closest Points (ICP) to align pointclouds of an object at different points in time. We compare our unsupervised approach to a model that was trained using ground truth supervision, on the KITTI tracking dataset. We further improve our model by training our model on a larger dataset, which would otherwise not be possible due to the lack of ground truth data. We also compare our approach with a 3D object detector that estimates motion using a simple tracking scheme.

Learning from non-stationary data streams and overcoming catastrophic forgetting still poses a serious challenge for machine learning research. Rather than aiming to improve state-of-the-art, in this work we provide insight into the limits and merits of rehearsal, one of continual learning's most established methods. We hypothesize that models trained sequentially with rehearsal tend to stay in the same low-loss region after a task has finished, but are at risk of overfitting on its sample memory, hence harming generalization. We provide both conceptual and strong empirical evidence on three benchmarks for both behaviors, bringing novel insights into the dynamics of rehearsal and continual learning in general. Finally, we interpret important continual learning works in the light of our findings, allowing for a deeper understanding of their successes., Preprint, code publicly available

Mixture models are well-established learning approaches that, in computer vision, have mostly been applied to inverse or ill-defined problems. However, they are general-purpose divide-and-conquer techniques, splitting the input space into relatively homogeneous subsets in a data-driven manner. Not only ill-defined but also well-defined complex problems should benefit from them. To this end, we devise a framework for spatial regression using mixture density networks. We realize the framework for object detection and human pose estimation. For both tasks, a mixture model yields higher accuracy and divides the input space into interpretable modes. For object detection, mixture components focus on object scale, with the distribution of components closely following that of ground truth the object scale. This practically alleviates the need for multi-scale testing, providing a superior speed-accuracy trade-off. For human pose estimation, a mixture model divides the data based on viewpoint and uncertainty -- namely, front and back views, with back view imposing higher uncertainty. We conduct experiments on the MS COCO dataset and do not face any mode collapse.

This work investigates the task of unsupervised model personalization, adapted to continually evolving, unlabeled local user images. We consider the practical scenario where a high capacity server interacts with a myriad of resource-limited edge devices, imposing strong requirements on scalability and local data privacy. We aim to address this challenge within the continual learning paradigm and provide a novel Dual User-Adaptation framework (DUA) to explore the problem. This framework flexibly disentangles user-adaptation into model personalization on the server and local data regularization on the user device, with desirable properties regarding scalability and privacy constraints. First, on the server, we introduce incremental learning of task-specific expert models, subsequently aggregated using a concealed unsupervised user prior. Aggregation avoids retraining, whereas the user prior conceals sensitive raw user data, and grants unsupervised adaptation. Second, local user-adaptation incorporates a domain adaptation point of view, adapting regularizing batch normalization parameters to the user data. We explore various empirical user configurations with different priors in categories and a tenfold of transforms for MIT Indoor Scene recognition, and classify numbers in a combined MNIST and SVHN setup. Extensive experiments yield promising results for data-driven local adaptation and elicit user priors for server adaptation to depend on the model rather than user data. Hence, although user-adaptation remains a challenging open problem, the DUA framework formalizes a principled foundation for personalizing both on server and user device, while maintaining privacy and scalability., CVPR 2020

In this work we propose an unsupervised training method that finetunes a single image depth estimation CNN towards a new environment. The network, which has been pretrained on stereo data, only requires monocular input for finetuning. Unlike other unsupervised methods, it produces depth estimations with absolute scale – a feature that is essential for most practical applications, yet has mostly been overlooked in the literature. First, we show how our method allows adapting a network trained on one dataset (Cityscapes) to another (KITTI). Next, by splitting KITTI in subsets, we show the sensitivity of pretrained models to a domain shift. We then demonstrate that, by finetuning the model using our method, it is possible to improve the performance on the target subset, without using stereo or any form of groundtruth depth and with preservation of the correct absolute scale.

This paper focuses on the problem of unsupervised image-to-image translation. More specifically, we aim at finding a translation network such that objects and shapes that only appear in the source domain are translated to objects and shapes only appearing in the target domain, while style color features present in the source domain remain the same. To achieve this, we use a domain-specific variational autoencoder and represent each image in its latent space representation. In a second step, we learn a translation between latent spaces of different domains using generative adversarial networks. We evaluate this framework on multiple datasets and verify the effect of multiple perceptual losses. Experiments on the MNIST and SVHN datasets show the effectiveness of the proposed translation method.

Training deep neural network policies end-to-end for real-world applications so far requires big demonstration datasets in the real world or big sets consisting of a large variety of realistic and closely related 3D CAD models. These real or virtual data should, moreover, have very similar characteristics to the conditions expected at test time. These stringent requirements and the time consuming data collection processes that they entail, are currently the most important impediment that keeps deep reinforcement learning from being deployed in real-world applications. Therefore, in this work we advocate an alternative approach, where instead of avoiding any domain shift by carefully selecting the training data, the goal is to learn a policy that can cope with it. To this end, we propose the DoShiCo challenge: to train a model in very basic synthetic environments, far from realistic, in a way that it can be applied in more realistic environments as well as take the control decisions on real-world data. In particular, we focus on the task of collision avoidance for drones. We created a set of simulated environments that can be used as benchmark and implemented a baseline method, exploiting depth prediction as an auxiliary task to help overcome the domain shift. Even though the policy is trained in very basic environments, it can learn to fly without collisions in a very different realistic simulated environment. Of course several benchmarks for reinforcement learning already exist - but they never include a large domain shift. On the other hand, several benchmarks in computer vision focus on the domain shift, but they take the form of a static datasets instead of simulated environments. In this work we claim that it is crucial to take the two challenges together in one benchmark., Comment: Published at SIMPAR 2018. Please visit the paper webpage for more information, a movie and code for reproducing results: https://kkelchte.github.io/doshico

© 2018 IEEE. The promise of self-driving cars promotes several advantages, e.g. they have the ability to outperform human drivers while being safer. Here we take a deeper look into some aspects from algorithms aimed at making this promise a reality. More specifically, we analyze an end-to-end neural network to predict a car's steering actions on a highway based on images taken from a single car-mounted camera. We focus our analysis on several aspects which could have a significant impact on the performance of the system. These aspects are: the input data format, the temporal dependencies between consecutive inputs, and the origin of the data. We show that, for the task at hand, regression networks outperform their classifier counterparts. In addition, there seems to be a small difference between networks that use coloured images and ones that use grayscale images as input. For the second aspect, by feeding the network three concatenated images, we get a significant decrease of 30% in mean squared error. For the third aspect, by using simulation data we are able to train networks that have a performance comparable to networks trained on real-life datasets. We also qualitatively demonstrate that the standard metrics that are used to evaluate networks do not necessarily accurately reflect a system's driving behaviour. We show that a promising confusion matrix may result in poor driving behaviour while a very ill-looking confusion matrix may result in good driving behaviour. Heylen J., Iven S., De Brabandere B., Oramas Mogrovejo J.A., Van Gool L., Tuytelaars T., ''From pixels to actions: Learning to drive a car with deep neural networks'', IEEE winter conference on applications of computer vision - WACV 2018, March 12-14, 2018, Lake Tahoe, Nevada, USA (accepted). ispartof: pages:606-615 ispartof: IEEE Winter Conference on Applications in Computer Vision vol:2018-January pages:606-615 ispartof: IEEE winter conference on applications of computer vision - WACV 2018 location:Lake Tahoe, Nevada, USA date:12 Mar - 14 Mar 2018 status: published

© 2018 IEEE. Online action detection is a challenging problem: A system needs to decide what action is happening at the current frame, based on previous frames only. Fortunately in real-life, human actions are not independent from one another: There are strong (long-term) dependencies between them. An online action detection method should be able to capture these dependencies, to enable a more accurate early detection. At first sight, an LSTM seems very suitable for this problem. It is able to model both short-term and long-term patterns. It takes its input one frame at the time, updates its internal state and gives as output the current class probabilities. In practice, however, the detection results obtained with LSTMs are still quite low. In this work, we start from the hypothesis that it may be too difficult for an LSTM to learn both the interpretation of the input and the temporal patterns at the same time. We propose a two-stream feedback network, where one stream processes the input and the other models the temporal relations. We show improved detection accuracy on an artificial toy dataset and on the Breakfast Dataset [21] and the TVSeries Dataset [7], reallife datasets with inherent temporal dependencies between the actions. De Geest R., Tuytelaars T., ''Modeling temporal structure with LSTM for online action detection'', IEEE winter conference on applications of computer vision - WACV 2018, March 12-14, 2018, Lake Tahoe, Nevada, USA (accepted. ispartof: pages:1549-1557 ispartof: WACV 2018 vol:2018-January pages:1549-1557 ispartof: IEEE winter conference on applications of computer vision - WACV 2018 location:Lake Tahoe, Nevada, USA date:12 Mar - 14 Mar 2018 status: published

© 2017 IEEE. We propose a novel mid-level representation for action/activity recognition on RGB videos. We model the evolution of improved dense trajectory features not only for the entire video sequence, but also on subparts of the video. Subparts are obtained using a spectral divisive clustering that yields an unordered binary tree decomposing the entire cloud of trajectories of a sequence. We then compute videodarwin on video subparts, exploiting more finegrained temporal information and reducing the sensitivity of the standard time varying mean strategy of videodarwin. After decomposition, we model the evolution of features through both frames of subparts and descending/ascending paths in tree branches. We refer to these mid-level representations as node-darwintree and branch-darwintree respectively. For the final classification, we construct a kernel representation for both mid-level and holistic videodarwin representations. Our approach achieves better performance than standard videodarwin and defines the current state-of-the-art on UCF-Sports and Highfive action recognition datasets. Clapes A., Tuytelaars T., Escalera S., ''Darwintrees for action recognition'', ICCV 2017 ChaLearn Looking at People Workshop, pp. 3169-3178, October 29, 2017, Venice, Italy. ispartof: pages:3169-3178 ispartof: ICCV 2017 workshops vol:2018-January pages:3169-3178 ispartof: ICCV 2017 ChaLearn Looking at People Workshop location:Venice, Italy date:29 Oct - 29 Oct 2017 status: published

This paper demonstrates the use of a single forward facing camera for obstacle avoidance on a quadrotor. We train a CNN for estimating depth from a single image. The depth map is then fed to a behaviour arbitration based control algorithm that steers the quadrotor away from obstacles. We conduct experiments with simulated and real drones in a variety of environments.

Undoing the image formation process and therefore decomposing appearance into its intrinsic properties is a challenging task due to the under-constraint nature of this inverse problem. While significant progress has been made on inferring shape, materials and illumination from images only, progress in an unconstrained setting is still limited. We propose a convolutional neural architecture to estimate reflectance maps of specular materials in natural lighting conditions. We achieve this in an end-to-end learning formulation that directly predicts a reflectance map from the image itself. We show how to improve estimates by facilitating additional supervision in an indirect scheme that first predicts surface orientation and afterwards predicts the reflectance map by a learning-based sparse data interpolation. In order to analyze performance on this difficult task, we propose a new challenge of Specular MAterials on SHapes with complex IllumiNation (SMASHINg) using both synthetic and real images. Furthermore, we show the application of our method to a range of image-based editing tasks on real images., project page: http://homes.esat.kuleuven.be/~krematas/DRM/

© 2015 IEEE. We present a supervised learning to rank algorithm that effectively orders images by exploiting the structure in image sequences. Most often in the supervised learning to rank literature, ranking is approached either by analysing pairs of images or by optimizing a list-wise surrogate loss function on full sequences. In this work we propose MidRank, which learns from moderately sized sub-sequences instead. These sub-sequences contain useful structural ranking information that leads to better learnability during training and better generalization during testing. By exploiting sub-sequences, the proposed MidRank improves ranking accuracy considerably on an extensive array of image ranking applications and datasets. Fernando B., Gavves S., Muselet D., Tuytelaars T., ''Learning to rank based on subsequences'', Proceedings 15th international conference on computer vision - ICCV 2015, pp. 2785-2793, December 11-18, 2015, Santiago, Chile. ispartof: pages:2785-2793 ispartof: Proceedings ICCV 2015 vol:2015 International Conference on Computer Vision, ICCV 2015 pages:2785-2793 ispartof: International conference on computer vision - ICCV 2015 location:Santiago, Chile date:11 Dec - 18 Dec 2015 status: published

In this paper we evaluate the quality of the activation layers of a convolutional neural network (CNN) for the gen- eration of object proposals. We generate hypotheses in a sliding-window fashion over different activation layers and show that the final convolutional layers can find the object of interest with high recall but poor localization due to the coarseness of the feature maps. Instead, the first layers of the network can better localize the object of interest but with a reduced recall. Based on this observation we design a method for proposing object locations that is based on CNN features and that combines the best of both worlds. We build an inverse cascade that, going from the final to the initial convolutional layers of the CNN, selects the most promising object locations and refines their boxes in a coarse-to-fine manner. The method is efficient, because i) it uses the same features extracted for detection, ii) it aggregates features using integral images, and iii) it avoids a dense evaluation of the proposals due to the inverse coarse-to-fine cascade. The method is also accurate; it outperforms most of the previously proposed object proposals approaches and when plugged into a CNN-based detector produces state-of-the- art detection performance., ICCV 2015

© 2015 IEEE. A common issue in deformable object detection is finding a good way to position the parts. This issue is even more outspoken when considering detection and pose estimation for 3D objects, where parts should be placed in a three-dimensional space. Some methods extract the 3D shape of the object from 3D CAD models. This limits their applicability to categories for which such models are available. Others represent the object with a predefined and simple shape (e.g. a cuboid). This extends the applicability of the model, but in many cases the pre-defined shape is too simple to properly represent the object in 3D. In this paper we propose a new method for the detection and pose estimation of 3D objects, that does not use any 3D CAD model or other 3D information. Starting from a simple and general 3D shape, we learn in a weakly supervised manner the 3D part locations that best fit the training data. As this method builds on a iterative estimation of the part locations, we introduce several speedups to make the method fast enough for practical experiments. We evaluate our model for the detection and pose estimation of faces and cars. Our method obtains results comparable with the state of the art, it is faster than most of the other approaches and does not need any additional 3D information. Pedersoli M., Tuytelaars T., ''Learning where to position parts in 3D'', Proceedings 15th international conference on computer vision - ICCV 2015, pp. 136-144, December 11-18, 2015, Santiago, Chile. ispartof: pages:136-144 ispartof: Proceedings ICCV 2015 vol:2015 International Conference on Computer Vision, ICCV 2015 pages:136-144 ispartof: International conference on computer vision - ICCV 2015 location:Santiago, Chile date:11 Dec - 18 Dec 2015 status: published

Over the years, hand gesture recognition has been mostly addressed considering hand trajectories in isolation. However, in most sign languages, hand gestures are defined on a particular context (body region). We propose a pipeline which models hand movements in the context of other parts of the body captured in the 3D space using the Kinect sensor. In addition, we perform sign recognition based on the different hand postures that occur during a sign. Our experiments show that considering different body parts brings improved performance when compared with methods which only consider global hand trajectories. Finally, we demonstrate that the combination of hand postures features with hand gestures features helps to improve the prediction of a given sign. Martínez-Camarena M., Oramas Mogrovejo J.A., Tuytelaars T., ''Towards sign language recognition based on body parts relations'', IEEE international conference on image processing - ICIP 2015, 5 pp., September 27-30, 2015, Quebec, Canada. ispartof: pages:2454-2458 ispartof: Proceedings ICIP 2015 vol:2015-December pages:2454-2458 ispartof: IEEE International Conference on Image Processing - ICIP 2015 location:Quebec, Canada date:27 Sep - 30 Sep 2015 status: published

More than thirty percent of persons over 65 years fall at least once a year and are often not able to get up again. The lack of timely aid after such a fall incident can lead to severe complications. This timely aid can however be assured by a camera-based fall detection system triggering an alarm when a fall occurs. Most algorithms described in literature use the biggest object detected using background subtraction to extract the fall features. In this paper we compare the performance of our state-of-the-art fall detection algorithm when using only background subtraction, when using a particle filter to track the person and a hybrid method in which the particle filter is only used to enhance the background subtraction and not for the feature extraction. We tested this using our simulation data set containing reenactments of real-life falls. This comparison shows that this hybrid method significantly increases the sensitivity and robustness of the fall detection algorithm resulting in a sensitivity of 76.1% and a PPV of 41.2%. Debard G., Baldewijns G., Goedemé T., Tuytelaars T., Vanrumste B., ''Camera-based fall detection using a particle filter'', Proceedings 37th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBS) - EMBC 2015, pp. 6947-6950, August 25-29, 2015, Milan, Italy. ispartof: pages:6947-6950 ispartof: Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society vol:37 pages:6947-6950 ispartof: International Conference of the IEEE Engineering in Medicine and Biology Society location:Milan, Italy date:25 Aug - 29 Aug 2015 status: published

© 2015 IEEE. Weakly supervised object detection, is a challenging task, where the training procedure involves learning at the same time both, the model appearance and the object location in each image. The classical approach to solve this problem is to consider the location of the object of interest in each image as a latent variable and minimize the loss generated by such latent variable during learning. However, as learning appearance and localization are two interconnected tasks, the optimization is not convex and the procedure can easily get stuck in a poor local minimum, i.e. the algorithm 'misses' the object in some images. In this paper, we help the optimization to get close to the global minimum by enforcing a 'soft' similarity between each possible location in the image and a reduced set of 'exemplars', or clusters, learned with a convex formulation in the training images. The help is effective because it comes from a different and smooth source of information that is not directly connected with the main task. Results show that our method improves a strong baseline based on convolutional neural network features by more than 4 points without any additional features or extra computation at testing time but only adding a small increment of the training time due to the convex clustering. Bilen H., Pedersoli M., Tuytelaars T., ''Weakly supervised object detection with convex clustering'', 28th IEEE conference on computer vision and pattern recognition - CVPR 2015, pp. 1081-1089, June 7-12, 2015, Boston, Massachusetts, USA. ispartof: pages:1081-1089 ispartof: Proceedings CVPR 2015 vol:07-12-June-2015 pages:1081-1089 ispartof: IEEE conference on computer vision and pattern recognition - CVPR 2015 location:Boston, Massachusetts, USA date:7 Jun - 12 Jun 2015 status: published

© 2015 IEEE. In this work we focus on the problem of image caption generation. We propose an extension of the long short term memory (LSTM) model, which we coin gLSTM for short. In particular, we add semantic information extracted from the image as extra input to each unit of the LSTM block, with the aim of guiding the model towards solutions that are more tightly coupled to the image content. Additionally, we explore different length normalization strategies for beam search to avoid bias towards short sentences. On various benchmark datasets such as Flickr8K, Flickr30K and MS COCO, we obtain results that are on par with or better than the current state-of-the-art. Jia X., Gavves S., Fernando B., Tuytelaars T., ''Guiding the long-short term memory model for image caption generation'', Proceedings 15th international conference on computer vision - ICCV 2015, pp. 2407-2415, December 11-18, 2015, Santiago, Chile. ispartof: pages:2407-2415 ispartof: Proceedings ICCV 2015 vol:2015 International Conference on Computer Vision, ICCV 2015 pages:2407-2415 ispartof: IEEE International Conference on Computer Vision location:Santiago, Chile date:11 Dec - 18 Dec 2015 status: published

© 2015 IEEE. In this paper we present a method to capture video-wide temporal information for action recognition. We postulate that a function capable of ordering the frames of a video temporally (based on the appearance) captures well the evolution of the appearance within the video. We learn such ranking functions per video via a ranking machine and use the parameters of these as a new video representation. The proposed method is easy to interpret and implement, fast to compute and effective in recognizing a wide variety of actions. We perform a large number of evaluations on datasets for generic action recognition (Hollywood2 and HMDB51), fine-grained actions (MPII- cooking activities) and gestures (Chalearn). Results show that the proposed method brings an absolute improvement of 7-10%, while being compatible with and complementary to further improvements in appearance and local motion based methods. Fernando B., Gavves E., Oramas Mogrovejo J., Ghodrati A., Tuytelaars T., ''Modeling video evolution for action recognition'', 28th IEEE conference on computer vision and pattern recognition - CVPR 2015, pp. 5378-5387, June 7-12, 2015, Boston, Massachusetts, USA. ispartof: pages:5378-5387 ispartof: Proceedings CVPR 2015 vol:07-12-June-2015 pages:5378-5387 ispartof: IEEE conference on computer vision and pattern recognition - CVPR 2015 location:Boston, Massachusetts, USA date:7 Jun - 12 Jun 2015 status: published

© 2015 IEEE. In this paper, we treat the problem of continuous pose estimation for object categories as a regression problem on the basis of only 2D training information. While regression is a natural framework for continuous problems, regression methods so far achieved inferior results with respect to 3D-based and 2D-based classification-and-refinement approaches. This may be attributed to their weakness to high intra-class variability as well as to noisy matching procedures and lack of geometrical constraints. We propose to apply regression to Fisher-encoded vectors computed from large cells by learning an array of Fisher regressors. Fisher encoding makes our algorithm flexible to variations in class appearance, while the array structure permits to indirectly introduce spatial context information in the approach. We formulate our problem as a MAP inference problem, where the likelihood function is composed of a generative term based on the prediction error generated by the ensemble of Fisher regressors as well as a discriminative term based on SVM classifiers. We test our algorithm on three publicly available datasets that envisage several difficulties, such as high intra-class variability, truncations, occlusions, and motion blur, obtaining state-of-the-art results. Fenzi M., Leal-Taixe L., Ostermann J., Tuytelaars T., ''Continuous pose estimation with a spatial ensemble of fisher regressors'', Proceedings 15th international conference on computer vision - ICCV 2015, pp. 1035-1043, December 11-18, 2015, Santiago, Chile. ispartof: pages:1035-1043 ispartof: Proceedings ICCV 2015 vol:2015 International Conference on Computer Vision, ICCV 2015 pages:1035-1043 ispartof: International conference on computer vision - ICCV 2015 location:Santiago, Chile date:11 Dec - 18 Dec 2015 status: published

© 2015 IEEE. As the amount of visual data increases, so does the need for summarization tools that can be used to explore large image collections and to quickly get familiar with their content. In this paper, we propose dataset fingerprints, a new and powerful method based on data mining that extracts meaningful patterns from a set of images. The discovered patterns are compositions of discriminative midlevel features that co-occur in several images. Compared to earlier work, ours stands out because i) it's fully unsupervised, ii) discovered patterns cover large parts of the images, often corresponding to full objects or meaningful parts thereof, and iii) different patterns are connected based on co-occurrence, allowing a user to 'browse' the images from one pattern to the next and to group patterns in a semantically meaningful manner. Rematas K., Fernando B., Dellaert F., Tuytelaars T., ''Dataset fingerprints: exploring image collections through data mining'', 28th IEEE conference on computer vision and pattern recognition - CVPR 2015, pp. 4867-4875, June 7-12, 2015, Boston, Massachusetts, USA. ispartof: pages:4867-4875 ispartof: Proceedings CVPR 2015 vol:07-12-June-2015 pages:4867-4875 ispartof: IEEE conference on computer vision and pattern recognition - CVPR 2015 location:Boston, Massachusetts, USA date:7 Jun - 12 Jun 2015 status: published

© 2014 IEEE. In classification of objects substantial work has gone into improving the low level representation of an image by considering various aspects such as different features, a number of feature pooling and coding techniques and considering different kernels. Unlike these works, in this paper, we propose to enhance the semantic representation of an image. We aim to learn the most important visual components of an image and how they interact in order to classify the objects correctly. To achieve our objective, we propose a new latent SVM model for category level object classification. Starting from image-level annotations, we jointly learn the object class and its context in terms of spatial location (where) and appearance (what). Furthermore, to regularize the complexity of the model we learn the spatial and co-occurrence relations between adjacent regions, such that unlikely configurations are penalized. Experimental results demonstrate that the proposed method can consistently enhance results on the challenging Pascal VOC dataset in terms of classification and weakly supervised detection. We also show how semantic representation can be exploited for finding similar content. Bilen H., Pedersoli M., Namboodiri V., Tuytelaars T., Van Gool L., ''Object classification with adaptive regions'', 27th IEEE conference on computer vision and pattern recognition - CVPR 2014, pp. 3662-3669, June 23-28, 2014, Columbus, Ohio, USA. ispartof: pages:3662-3669 ispartof: Proceedings CVPR 2014 pages:3662-3669 ispartof: IEEE conference on computer vision and pattern recognition - CVPR 2014 location:Ohio, USA date:23 Jun - 28 Jun 2014 status: published

Recently a lot of progress has been made in the field of video segmentation. The question then arises whether and how these results can be exploited for this other video processing challenge, action recognition. In this paper we show that a good segmentation is actually very important for recognition. We propose and evaluate several ways to integrate and combine the two tasks: i) recognition using a standard, bottom-up segmentation, ii) using a top-down segmentation geared towards actions, iii) using a segmentation based on inter-video similarities (co-segmentation), and iv) tight integration of recognition and segmentation via iterative learning. Our results clearly show that, on the one hand, the two tasks are interdependent and therefore an iterative optimization of the two makes sense and gives better results. On the other hand, comparable results can also be obtained with two separate steps but mapping the feature-space with a non-linear kernel.

© 2014 IEEE. We propose a technique to use the structural informa- tion extracted from a set of 3D models of an object class to improve novel-view synthesis for images showing unknown instances of this class. These novel views can be used to 'amplify' training image collections that typically contain only a low number of views or lack certain classes of views entirely (e. g. top views). We extract the correlation of position, normal, re- flectance and appearance from computer-generated images of a few exemplars and use this information to infer new appearance for new instances. We show that our approach can improve performance of state-of-the-art detectors using real-world training data. Additional applications include guided versions of inpainting, 2D-to-3D conversion, super- resolution and non-local smoothing. Rematas K., Ritschel T., Fritz M., Tuytelaars T., ''Image-based synthesis and re-synthesis of viewpoints guided by 3D models'', 27th IEEE conference on computer vision and pattern recognition - CVPR 2014, pp. 3898-3905, June 23-28, 2014, Columbus, Ohio, USA. ispartof: pages:3898-3905 ispartof: Proceedings CVPR 2014 pages:3898-3905 ispartof: IEEE conference on computer vision and pattern recognition - CVPR 2014 location:Columbus, Ohio, USA date:23 Jun - 28 Jun 2014 status: published

© 2014 IEEE. Estimating the age of historical photographs is a challenging task for human beings. Only recently this task has been addressed in computational image analysis perspective. The characteristics of the device used to acquire each photograph are discriminative features for this task. We aim at extracting such characteristics from a historical color photographs. The acquisition device mainly effects two properties of the colors: the distribution of their derivatives and the angles drawn by three consecutive pixels in the RGB space. We propose two color features that take advantage of these observations. We show that these two color descriptors (namely color derivatives and color angles) attain the state-of-the-art in the context of image dating. Fernando B., Muselet D., Khan R., Tuytelaars T., ''Color features for dating historical color image'', IEEE international conference on image processing - ICIP 2014, 5 pp., October 27-30, 2014, Paris, France. ispartof: pages:2589-2593 ispartof: Proceedings ICIP 2014 pages:2589-2593 ispartof: IEEE international conference on image processing - ICIP 2014 location:Paris, France date:27 Oct - 30 Oct 2014 status: published

In this paper we present a method for action classification in videos using trajectory features. The novelty of our approach is in formulating the problem of simultaneous detection and localization as a probabilistic chains model. In our formulation, chains are sets of regions in the video that are connected based on their joint probabilities. We describe our approach for connecting subvolumes in the video into chains, and using them as spatio-temporal detectors for actions. Our approach allows the detection and localization of multiple actions occurring simultaneously or at different locations in a single video. We test the performance of our method on two challenging action recognition datasets, and compare to state of the art methods. © 2014 IEEE. Sharir G., Tuytelaars T., ''Action in chains: A chains model for action localization and classification'', IEEE winter conference on applications of computer vision - WACV 2014, 8 pp., March 24-26, 2014, Steamboat Springs, CO, USA. ispartof: pages:610-617 ispartof: Proceedings WACV 2014 pages:610-617 ispartof: IEEE winter conference on applications of computer vision - WACV 2014 location:Steamboat Springs, CO, USA date:24 Mar - 26 Mar 2014 status: published

In this paper we propose a new method for detecting multiple specific 3D objects in real time. We start from the template-based approach based on the LINE2D/LINEMOD representation introduced recently by Hinterstoisser et al., yet extend it in two ways. First, we propose to learn the templates in a discriminative fashion. We show that this can be done online during the collection of the example images, in just a few milliseconds, and has a big impact on the accuracy of the detector. Second, we propose a scheme based on cascades that speeds up detection. Since detection of an object is fast, new objects can be added with very low cost, making our approach scale well. In our experiments, we easily handle 10-30 3D objects at frame rates above 10fps using a single CPU core. We outperform the state-of-the-art both in terms of speed as well as in terms of accuracy, as validated on 3 different datasets. This holds both when using monocular color images (with LINE2D) and when using RGBD images (with LINEMOD). Moreover, we propose a challenging new dataset made of 12 objects, for future competing methods on monocular color images. © 2013 IEEE. Rios-Cabrera R., Tuytelaars T., ''Discriminatively trained templates for 3D object detection: A real time scalable approach'', Proceedings 14th international conference on computer vision - ICCV 2013, pp. 2048-2055, December 3-6, 2013, Sydney, Australia. ispartof: pages:2048-2055 ispartof: Proceedings ICCV 2013 pages:2048-2055 ispartof: International conference on computer vision - ICCV 2013 location:Sydney, Australia date:3 Dec - 6 Dec 2013 status: published

In this paper we present a new method for object retrieval starting from multiple query images. The use of multiple queries allows for a more expressive formulation of the query object including, e.g., different viewpoints and/or viewing conditions. This, in turn, leads to more diverse and more accurate retrieval results. When no query images are available to the user, they can easily be retrieved from the internet using a standard image search engine. In particular, we propose a new method based on pattern mining. Using the minimal description length principle, we derive the most suitable set of patterns to describe the query object, with patterns corresponding to local feature configurations. This results in a powerful object-specific mid-level image representation. The archive can then be searched efficiently for similar images based on this representation, using a combination of two inverted file systems. Since the patterns already encode local spatial information, good results on several standard image retrieval datasets are obtained even without costly re-ranking based on geometric verification. © 2013 IEEE. Fernando B., Tuytelaars T., ''Mining multiple queries for image retrieval: On-the-fly learning of an object-specific mid-level representation'', Proceedings 14th international conference on computer vision - ICCV 2013, pp. 2544-2551, December 3-6, 2013, Sydney, Australia. ispartof: pages:2544-2551 ispartof: Proceedings ICCV 2013 pages:2544-2551 ispartof: International conference on computer vision - ICCV 2013 location:Sydney, Australia date:3 Dec - 6 Dec 2013 status: published

The advent of inexpensive RGB-D cameras brings new opportunities to capture a 3D environment. This paper presents a method to create a modular setup for generating a large 3D point cloud, with attention to the study of interference, the influence of a USB extension cable, and the calibration procedure. The study of interference includes the influence of the distance between the cameras, the orientation of the cameras, and the illumination. Furthermore, this paper proposes a number of evaluation metrics for similar setups.

Object detection, and in particular pedestrian detection, is a challenging task, due to the wide variety of appearances. The application domain is extremely broad, ranging from e.g. surveillance to automotive safety systems. Many practical applications however often rely on stringent real-time processing speeds combined with high accuracy needs. These demands are contradictory, and usually a compromise needs to be made. In this paper we present a pedestrian detection framework which is extremely fast (500 detections per second) while still maintaining excellent accuracy results. We achieve these results by combining our fast pedestrian detection algorithm (implemented as a hybrid CPU and GPU combination) with the exploitation of scene constraints (using a warping window approach and temporal information), which yields state-of-the-art detection accuracy. We present profound evaluation results of our algorithm concerning both speed and accuracy on the challenging Caltech dataset. Furthermore we present evaluation results on a very specific application showing the full potential of this warping window approach: detection of pedestrians in a truck's blind spot zone. De Smedt F., Van Beeck K., Tuytelaars T., Goedemé T., ''Pedestrian detection at warp speed: exceeding 500 detections per second'', 9th IEEE embedded vision workshop - EVW2013, held in conjunction with IEEE CVPR 2013, pp. 622-628, June 24, 2013, Portland, Oregon, USA. ispartof: pages:622-628 ispartof: 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops CVPRW 2013 pages:622-628 ispartof: IEEE embedded vision workshop - EVW2013, held in conjunction with IEEE CVPR 2013 location:Portland, Oregon date:24 Jun - 24 Jun 2013 status: published

In this paper, we introduce a new domain adaptation (DA) algorithm where the source and target domains are represented by subspaces described by eigenvectors. In this context, our method seeks a domain adaptation solution by learning a mapping function which aligns the source subspace with the target one. We show that the solution of the corresponding optimization problem can be obtained in a simple closed form, leading to an extremely fast algorithm. We use a theoretical result to tune the unique hyper parameter corresponding to the size of the subspaces. We run our method on various datasets and show that, despite its intrinsic simplicity, it outperforms state of the art DA methods. © 2013 IEEE. Fernando B., Habrard A., Sebban M., Tuytelaars T., ''Unsupervised visual domain adaptation using subspace alignment'', Proceedings 14th international conference on computer vision - ICCV 2013, pp. 2960-2967, December 3-6, 2013, Sydney, Australia. ispartof: pages:2960-2967 ispartof: Proceedings ICCV 2013 pages:2960-2967 ispartof: International conference on computer vision - ICCV 2013 location:Sydney, Australia date:3 Dec - 6 Dec 2013 status: published

Real-world scenes involve many objects that interact with each other in complex semantic patterns. For example, a bar scene can be naturally described as having a variable number of chairs of similar size, close to each other and aligned horizontally. This high-level interpretation of a scene relies on semantically meaningful entities and is most generally described using relational representations or (hyper-) graphs. Popular in early work on syntactic and structural pattern recognition, relational representations are rarely used in computer vision due to their pure symbolic nature. Yet, today recent successes in combining them with statistical learning principles motivates us to reinvestigate their use. In this paper we show that relational techniques can also improve scene classification. More specifically, we employ a new relational language for learning with kernels, called kLog. With this language we define higher-order spatial relations among semantic objects. When applied to a particular image, they characterize a particular object arrangement and provide discriminative cues for the scene category. The kernel allows us to tractably learn from such complex features. Thus, our contribution is a principled and interpretable approach to learn from symbolic relations how to classify scenes in a statistical framework. We obtain results comparable to state-of-the-art methods on 15 Scenes and a subset of the MIT indoor dataset. Antanas L., Hoffmann M., Frasconi P., Tuytelaars T., De Raedt L., ''A relational kernel-based approach to scene classification'', IEEE workshop on the applications of computer vision - WACV 2013, pp. 133-139, January 17-18, 2013, Clearwater Beach, Florida, USA. ispartof: pages:133-139 ispartof: Proceedings of the 13th IEEE Workshop on Applications of Computer Vision pages:133-139 ispartof: IEEE Workshop on Applications of Computer Vision (WACV) location:Clearwater Beach, FL, USA USA date:15 Jan - 17 Jan 2013 status: published

In this paper, we extend a recently proposed method for generic object detection in images, category-independent object proposals, to the case of video. Given a video, the output of our algorithm is a set of video segments that are likely to contain an object. This can be useful, e.g., as a first step in a video object detection system. Given the sheer amount of pixels in a video, a straightforward extension of the 2D methods to a 3D (spatiotemporal) volume is not feasible. Instead, we start by extracting object proposals in each frame separately. These are linked across frames into object hypotheses, which are then used as higher-order potentials in a graph-based video segmentation framework. Running multiple segmentations and ranking the segments based on the likelihood that they correspond to an object, yields our final set of video object proposals.

Huge amounts of audiovisual material have been digitized recently, resulting in a great source of information relevant both from a cultural and historical point of view. However, in spite of millions of man hours spent on manual annotation and recent advances in (semi-)automatic metadata generation, accessing these archives and retrieving relevant information from them remains a difficult task. Up to recently, the main paradigm to open up archives by automatic tools for audiovisual analysis has been a concept-based indexing and retrieval oriented approach. However, this approach has its limitations, in that it does not scale well, it requires strong supervision, and does not really match well to the user's needs. In this paper, we discuss some upcoming alternative approaches that try to overcome or circumvent some of these issues. This includes i) the use of knowledge modeling to bridge the semantic gap; ii) on-the-fly learning of new, user-defined concepts; and iii) weakly supervised methods that learn from associated text data. We also discuss what we consider important open issues at this time that deserve more attention from the research community.

Traditional bag-of-features approaches often vector-quantise the features into a visual codebook. This process inevitably causes loss of information. Recently codebook-free methods that avoid the vector-quantisation step have become more popular. Used in conjunction with nearest-neighbour approaches these methods have shown remarkable classification performance. In this paper we show how to exploit the concept of nearest neighbour based classification for object detection. Our codebook-free exemplar model combines the classification power of nearest neighbour methods with a detection concept based on exemplar models. We demonstrate the performance of our proposed system on a real-world dataset of images of motorbikes.

Naive Bayes Nearest Neighbor (NBNN) has recently been proposed as a powerful, non-parametric approach for object classification, that manages to achieve remarkably good results thanks to the avoidance of a vector quantization step and the use of image-to-class comparisons, yielding good generalization. In this paper, we introduce a kernelized version of NBNN. This way, we can learn the classifier in a discriminative setting. Moreover, it then becomes straightforward to combine it with other kernels. In particular, we show that our NBNN kernel is complementary to standard bag-of-features based kernels, focussing on local generalization as opposed to global image composition. By combining them, we achieve state-of-the-art results on Caltech101 and 15 Scenes datasets. As a side contribution, we also investigate how to speed up the NBNN computations.

Deformable Part Models (DPMs) as introduced by Felzenszwalb et al. have shown remarkably good results for category-level object detection. In this paper, we explore whether they are also well suited for the related problem of category-level object pose estimation. To this end, we extend the original DPM so as to improve its accuracy in object category pose estimation and design novel and more effective learning strategies. We benchmark the methods using various publicly available data sets. Provided that the training data is sufficiently balanced and clean, our method outperforms the state-of-the-art.

This paper presents an integrated solution for the problem of detecting, tracking and identifying vehicles in a tunnel surveillance application, taking into account practical constraints including realtime operation, poor imaging conditions, and a decentralized architecture. Vehicles are followed through the tunnel by a network of non-overlapping cameras. They are detected and tracked in each camera and then identified, i.e. matched to any of the vehicles detected in the previous camera(s). To limit the computational load, we propose to reuse the same set of Haar-features for each of these steps. For the detection, we use an Adaboost cascade. Here we introduce a composite confidence score, integrating information from all stage of the cascades. A subset of the features used for detection is then selected, optimizing for the identification problem. This results in a compact binary ‘vehicle fingerprint’, requiring very limited bandwidth. Finally, we show that the same set of features can also be used for tracking. This haar features based ‘tracking-by-identification’ yields surprisingly good results on standard datasets, without the need to update the model online. Rios-Cabrera R., Tuytelaars T., Van Gool L., ''Efficient multi-camera detection, tracking, and identification using a shared set of haar-features'', IEEE computer society conference on computer vision and pattern recognition - CVPR2011, pp. 65-71, June 21-23, 2011, Colorado Springs, CO, USA. ispartof: pages:65-71 ispartof: Proceedings IEEE computer society conference on computer vision and pattern recognition - CVPR2011 pages:65-71 ispartof: IEEE computer society conference on computer vision and pattern recognition - CVPR2011 location:Colorado Springs, CO, USA date:21 Jun - 23 Jun 2011 status: published

We propose a real-time multi-camera tracking approach to follow vehicles in a tunnel surveillance environment with multiple non-overlapping cameras. In such system, vehicles have to be tracked in each camera and passed correctly from one camera to another through the tunnel. This task becomes extremely difficult when intra-camera errors are accumulated. Most typical issues to solve in tunnel scenes are due to low image quality, poor illumination and lighting from the vehicles. Vehicle detection is performed using Adaboost detector, speeded up by separating different cascades for cars and trucks improving general accuracy of detection. A Kalman Filter with two observations, given by the vehicle detector and an averaged optical flow vector, is used for single-camera tracking. Information from collected tracks is used for feeding the inter-camera matching algorithm, which measures the correlation of Radon transform-like projections between the vehicle images. Our main contribution is a novel method to reduce the false positive rate induced by the detection stage. We impose recall over precision in the detection correctness, and identify false positives patterns which are then included subsequently in a high-level decision making step. Results are presented for the case of 3 cameras placed consecutively in an inter-city tunnel. We demonstrate the increased tracking performance of our method compared to existing Bayesian filtering techniques for vehicle tracking in tunnel surveillance. © 2011 IEEE. Niño Castañeda J., Rios Cabrera R., Jelaca V., Frias A., Pizurica A., Tuytelaars T., Philips W., ''Non-overlapping multi-camera detection and tracking of vehicles in tunnel surveillance'', Proceedings DICTA The international conference on digital image computing: techniques and applications - DICTA 2011, pp. 591-596, December 6-8, 2011, Noosa, Queensland, Australia. ispartof: pages:591-596 ispartof: Proceedings DICTA pages:591-596 ispartof: The international conference on digital image computing: techniques and applications - DICTA 2011 location:Noosa, Queensland, Australia date:6 Dec - 8 Dec 2011 status: published

Labeling persons appearing in video frames with names detected from the video transcript helps improving the video content identification and search task. We develop a face naming method that learns from labeled and unlabeled examples using iterative label propagation in a graph of connected faces or name-face pairs. The advantage of this method is that it can use very few labeled data points and incorporate the unlabeled data points during the learning process. Anchor detection and metric learning for face classification techniques are incorporated into the label propagation process to help boosting the face naming performance. On BBC News videos, the label propagation algorithm yields better results than a Support Vector Machine classifier trained on the same labeled data Pham P.T., Moens M.-F., Tuytelaars T., ''Naming persons in news video with label propagation'', Workshop on visual content identification and search - VCIDS’10 in conjunction with IEEE international conference on multimedia and expo - ICME 2010, July 23, 2010, Singapore. ispartof: pages:1528-1533 ispartof: Proceedings of the international workshop on visual content identification and search (VCIDS 2010) at the IEEE international conference on multimedia & expo (ICME 2010) pages:1528-1533 ispartof: International workshop on visual content identification and search (VCIDS 2010) at the IEEE international conference on multimedia & expo (ICME 2010) location:Singapore date:19 Jul - 23 Jul 2010 status: published

Recently there has been considerable interest in topic models based on the bag-of-features representation of images. The strong independence assumption inherent in the bag-of-features representation is not realistic however: patches often overlap and share underlying image structures. Moreover, important information with respect to relative scales of the features is completely ignored, for the sake of scale invariance. Considering both spatial and scale-based constraints one can derive spatially constrained natural feature hierarchies within images. We explore the use of topic models that build such spatially constrained scale-induced hierarchies of the features in an unsupervised fashion. Our model uses standard topic models as a starting point. We then incorporate information about the hierarchical and spatial relations of the features into the model. We illustrate the hierarchical nature of the resulting models using datasets of natural images, including the MSRC2 dataset as well as a challenging set of images of trees collected from the Internet.

Thanks to recent progress in category-level object recognition, we have now come to a point where these techniques have gained sufficient maturity and accuracy to succesfully feed back their output to other processes. This is what we refer to as cognitive feedback. In this paper, we study one particular form of cognitive feedback, where the ability to recognize objects of a given category is exploited to infer meta-data such as depth cues, 3D points, or object decomposition in images of previously unseen object instances. Our approach builds on the Implicit Shape Model of Leibe and Schiele, and extends it to transfer annotations from training images to test images. Experimental results validate the viability of our approach. Thomas A., Ferrari V., Leibe B., Tuytelaars T., Van Gool L., ''Depth-from-recognition : inferring meta-data by cognitive feedback'', Proceedings ICCV '07 workshop on 3D representation for recognition - 3dRR-07, 8 pp., October 20, 2007, Rio de Janeiro, Brazil. ispartof: pages:23-+ ispartof: Proceedings ICCV '07 workshop on 3D representation for recognition - 3dRR-07 pages:23-+ ispartof: ICCV '07 workshop on 3D representation for recognition - 3dRR-07 location:Rio de Janeiro, Brazil date:20 Oct - 20 Oct 2007 status: published

In this paper we present a technique to linearly recover 2D structure and motion in man made environments from three uncalibrated omnidirectional views. We use vertical lines from the scene which are projected as radial lines in the images and are automatically matched. The algorithm is based on a 1D radial trifocal tensor which encodes the relations of the three views and the projected lines. We include experiments with real images, which demonstrate the good performance of the method and its application to robotic tasks, such as robot localization based in a database of reference images or to obtain the initial values of robot and landmarks localization in SLAM algorithms. Sagüés C., Murillo A.C., Guerrero J.J., Goedemé T., Tuytelaars T., Van Gool L., ''Localization with omnidirectional images using the radial trifocal tensor'', Proceedings IEEE international conference on robotics and automation - ICRA 2006, pp. 551-556, May 15-19, 2006, Orlando, Florida, USA. ispartof: pages:551-556 ispartof: Proceedings IEEE international conference on robotics and automation - ICRA 2006 vol:2006 pages:551-556 ispartof: IEEE international conference on robotics and automation - ICRA 2006 location:Orlando, Florida, USA date:15 May - 19 May 2006 status: published

Upcoming fast vision techniques for finding image correspondences enable reliable real-time visual homing, i.e. the guidance of a mobile robot from a arbitrary start pose towards a goal pose defined by an image taken there. Two approaches emerge in the field that differ in the fact that the structure of the scene is estimated or not. In this paper, we compare these two approaches for the general case and especially for our application, being automatic wheelchair navigation. Goedemé T., Vanhooydonck D., Demeester E., Nuttin M., Tuytelaars T., Van Gool L., ''Is structure needed for omnidirectional visual homing ?'', 6th IEEE International Symposium on Computational Intelligence in Robotics and Automation - CIRA 2005, June 27-30, 2005, Espoo, Finland. ispartof: pages:303-308 ispartof: Proceedings 6th IEEE International Symposium on Computational Intelligence in Robotics and Automation - CIRA 2005 pages:303-308 ispartof: IEEE International Symposium on Computational Intelligence in Robotics and Automation - CIRA 2005 location:Espoo, Finland date:27 Jun - 30 Jun 2005 status: published