Search

The Dunhuang Caves are the home to the largest Buddhist art sites in the world and are listed as a UNESCO World Heritage Site. Over time, the murals have been damaged by both humans and nature. In this article, we present an immersive virtual reality system for exploring spatial cultural heritage, which utilizes the digitized data from the Dunhuang Research Academy to represent the virtual environment of the cave. In this system, the interaction techniques that allow users to flexibly experience any of the artifacts or displays contribute to their understanding of the cultural heritage. Additionally, we evaluated the system by conducting a user study to examine the extent of user acquaintance after the entire experience. Our result has shown what participants learn from the spatial context and augmented information in the VR. This can be used as design considerations for developing other spatial heritages.

This paper proposes an ensemble of invariant features (EIFs), which can properly handle the variations of color difference and human poses/viewpoints for matching pedestrian images observed in different cameras with nonoverlapping field of views. Our proposed method is a direct reidentification (re-id) method, which requires no prior domain learning based on prelabeled corresponding training data. The novel features consist of the holistic and region-based features. The holistic features are extracted by using a publicly available pretrained deep convolutional neural network used in generic object classification. In contrast, the region-based features are extracted based on our proposed two-way Gaussian mixture model fitting, which overcomes the self-occlusion and pose variations. To make a better generalization during recognizing identities without additional learning, the ensemble scheme aggregates all the feature distances using the similarity normalization. The proposed framework achieves robustness against partial occlusion, pose, and viewpoint changes. Moreover, the evaluation results show that our method outperforms the state-of-the-art direct re-id methods on the challenging benchmark viewpoint invariant pedestrian recognition and 3D people surveillance data sets.

This paper presents an algorithm for ego-positioning by using a low-cost monocular camera for systems based on the Internet-of-Vehicles. To reduce the computational and memory requirements, as well as the communication load, we tackle the model compression task as a weighted k-cover problem for better preserving the critical structures. For real-world vision-based positioning applications, we consider the issue of large scene changes and introduce a model update algorithm to address this problem. A large positioning data set containing data collected for more than a month, 106 sessions, and 14275 images is constructed. Extensive experimental results show that submeter accuracy can be achieved by the proposed ego-positioning algorithm, which outperforms existing vision-based approaches.

This paper presents an effective approach for detecting abandoned luggage in surveillance videos. We combine short- and long-term background models to extract foreground objects, where each pixel in an input image is classified as a 2-bit code. Subsequently, we introduce a framework to identify static foreground regions based on the temporal transition of code patterns, and to determine whether the candidate regions contain abandoned objects by analyzing the back-traced trajectories of luggage owners. The experimental results obtained based on video images from 2006 Performance Evaluation of Tracking and Surveillance and 2007 Advanced Video and Signal-based Surveillance databases show that the proposed approach is effective for detecting abandoned luggage, and that it outperforms previous methods.

In this paper, we propose a novel algorithm for high-definition displays to enlarge low-resolution images while maintaining perceptual constancy (i.e., the same field-of-view, perceptual blur radius, and the retinal image size in viewer's eyes). We model the relationship between a viewer and a display by considering two main aspects of visual perception, i.e., scaling factor and perceptual blur radius. As long as we enlarge an image while adjust its image blur levels on the display, we can maintain viewer's perceptual constancy. We show that the scaling factor should be set in proportion to the viewing distance and the blur levels on the display should be adjusted according to the focal length of a viewer. Toward this, we first refer to edge directions to interpolate a low-resolution image with the increasing of viewing distance and the scaling factor. After images are interpolated, we utilize a local contrast to estimate the spatially varying image blur levels of the interpolated image. We then further adjust the image blur levels using a parametric deblurring method, which combines L1 as well as L2 reconstruction errors, and Tikhonov with total variation regularization terms. By taking these factors into account, high-resolution images adaptive to viewing distance on a display can be generated. Experimental results on both natural image metric and user subjective studies across image scales demonstrate that the proposed super-resolution algorithm for high-definition displays performs favorably against the state-of-the-art methods.

Glint features have important roles in gaze-tracking systems. However, when the operation range of a gaze-tracking system is enlarged, the performance of glint-feature-based (GFB) approaches will be degraded mainly due to the curvature variation problem at around the edge of the cornea. Although the pupil contour feature may provide complementary information to help estimating the eye gaze, existing methods do not properly handle the cornea refraction problem, leading to inaccurate results. This paper describes a contour-feature-based (CFB) 3-D gaze-tracking method that is compatible to cornea refraction. We also show that both the GFB and CFB approaches can be formulated in a unified framework and, thus, they can be easily integrated. Furthermore, it is shown that the proposed CFB method and the GFB method should be integrated because the two methods provide complementary information that helps to leverage the strength of both features, providing robustness and flexibility to the system. Computer simulations and real experiments show the effectiveness of the proposed approach for gaze tracking.

We propose a facial trait code (FTC) to encode human facial images, and apply it to face recognition. Extracted from an exhaustive set of local patches cropped from a large stack of faces, the facial traits and the associated trait patterns can accurately capture the appearance of a given face. The extraction has two phases. The first phase is composed of clustering and boosting upon a training set of faces with neutral expression, even illumination, and frontal pose. The second phase focuses on the extraction of the facial trait patterns from the set of faces with variations in expression, illumination, and poses. To apply the FTC to face recognition, two types of codewords, hard and probabilistic, with different metrics for characterizing the facial trait patterns are proposed. The hard codeword offers a concise representation of a face, while the probabilistic codeword enables matching with better accuracy. Our experiments compare the proposed FTC to other algorithms on several public datasets, all showing promising results.

Most face recognition scenarios assume that frontal faces or mug shots are available for enrollment to the database, faces of other poses are collected in the probe set. Given a face from the probe set, one needs to determine whether a match in the database exists. This is under the assumption that in forensic applications, most suspects have their mug shots available in the database, and face recognition aims at recognizing the suspects when their faces of various poses are captured by a surveillance camera. This paper considers a different scenario: given a face with multiple poses available, which may or may not include a mug shot, develop a method to recognize the face with poses different from those captured. That is, given two disjoint sets of poses of a face, one for enrollment and the other for recognition, this paper reports a method best for handling such cases. The proposed method includes feature extraction and classification. For feature extraction, we first cluster the poses of each subject's face in the enrollment set into a few pose classes and then decompose the appearance of the face in each pose class using Embedded Hidden Markov Model, which allows us to define a set of subject-specific and pose-priented (SSPO) facial components for each subject. For classification, an Adaboost weighting scheme is used to fuse the component classifiers with SSPO component features. The proposed method is proven to outperform other approaches, including a component-based classifier with local facial features cropped manually, in an extensive performance evaluation study.

We propose a viewpoint-independent object-detection algorithm that detects objects in videos based on their 2-D and 3-D information. Object-specific quasi-3-D templates are proposed and applied to match objects' 2-D contours and to calculate their 3-D sizes. A quasi-3-D template is the contour and the 3-D bounding cube of an object viewed from a certain panning and tilting angle. Pedestrian templates amounting to 2660 and 1995 vehicle templates encompassing 19 tilting and 35 panning angles are used in this study. To detect objects, we first match the 2-D contours of object candidates with known objects' contours, and some object templates with large 2-D contour-matching scores are identified. In this step, we exploit some prior knowledge on the viewpoint on which the object is viewed to speed up the template matching, and the viewpoint likelihood for each contour-matched template is also assigned. Then, we calculate the 3-D widths, heights, and lengths of the contour-matched candidates, as well as the corresponding 3-D-size-matching scores. The overall matching score is obtained by combining the aforementioned likelihood and scores. The major contributions of this paper are to explore the joint use of 2-D and 3-D features in object detection. It shows that, by considering 2-D contours and 3-D sizes, one can achieve promising object detection rates. The proposed algorithms were evaluated on both pedestrian and vehicle sequences. It yielded significantly better detection results than the best results reported in PETS 2009, showing that our algorithm outperformed the state-of-the-art pedestrian-detection algorithms.

Large-scale and high-resolution monitoring systems are ideal for many visual surveillance applications. However, existing approaches have insufficient resolution and low frame rate per second, or have high complexity and cost. We take inspiration from the human visual system and propose a multi-resolution design, e-Fovea, which provides peripheral vision with a steerable fovea that is in higher resolution. In this paper, we firstly present two user studies, with a total of 36 participants, to compare e-Fovea to two existing multi-resolution visual monitoring designs. The user study results show that for visual monitoring tasks, our e-Fovea design with steerable focus is significantly faster than existing approaches and preferred by users. We then present our design and implementation of e-Fovea, which combines both multi-resolution camera input and multi-resolution steerable projector output. Finally, we present our deployment of e-Fovea in three installations to demonstrate its feasibility.

In this paper, we propose the Interest Meter (IM), a system making the computer conscious of user's reactions to measure user's interest and thus use it to conduct video summarization. The IM takes account of users' spontaneous reactions when they view videos. To estimate user's viewing interest, quantitative interest measures are devised based on the perspectives of attention and emotion. For estimating attention states, variations of user's eye movement, blink, and head motion are considered. For estimating emotion states, facial expression is recognized as positive or neural emotion. By combining characteristics of attention and emotion by a fuzzy fusion scheme, we transform users' viewing behaviors into quantitative interest scores, determine interesting parts of videos, and finally concatenate them as video summaries. Experimental results show that the proposed concept “editing by viewing” works well and may provide a promising direction to consider the human factor in video summarization.

Watershed segmentation has recently become a popular tool for image segmentation. There are two approaches to implementing watershed segmentation: immersion approach and toboggan simulation. Conceptually, the immersion approach can be viewed as an approach that starts from low altitude to high altitude and the toboggan approach as an approach that starts from high altitude to low altitude. The former seemed to be more popular recently (e.g., Vincent and Soille), but the latter had its own supporters (e.g., Mortensen and Barrett). It was not clear whether the two approaches could lead to exactly the same segmentation result and which approach was more efficient. In this paper, we present two "order-invariant" algorithms for watershed segmentation, one based on the immersion approach and the other on the toboggan approach. By introducing a special RIDGE label to achieve the property of order-invariance, we find that the two conceptually opposite approaches can indeed obtain the same segmentation result. When running on a Pentium-III PC, both of our algorithms require only less than 1/30 s for a 256 /spl times/ 256 image and 1/5 s for a 512 /spl times/ 512 image, on average. What is more surprising is that the toboggan algorithm, which is less well known in the computer vision community, turns out to run faster than the immersion algorithm for almost all the test images we have used, especially when the image is large, say, 512 /spl times/ 512 or larger. This paper also gives some explanation as to why the toboggan algorithm can be more efficient in most cases.

Block matching is a widely used method for stereo vision, visual tracking, and video compression. Many fast algorithms for block matching have been proposed in the past, but most of them do not guarantee that the match found is the globally optimal match in a search range. This paper presents a new fast algorithm based on the winner-update strategy which utilizes an ascending lower bound list of the matching error to determine the temporary winner. Two lower bound lists derived by using partial distance and by using Minkowski's inequality are described. The basic idea of the winner-update strategy is to avoid, at each search position, the costly computation of the matching error when there exists a lower bound larger than the global minimum matching error. The proposed algorithm can significantly speed up the computation of the block matching because: 1) computational cost of the lower bound we use is less than that of the matching error itself; 2) an element in the ascending lower bound list will be calculated only when its preceding element has already been smaller than the minimum matching error computed so far; 3) for many search positions, only the first several lower bounds in the list need to be calculated. Our experiments have shown that, when applying to motion vector estimation for several widely-used test videos, 92% to 98% of operations can be saved while still guaranteeing the global optimality. Moreover, the proposed algorithm can be easily modified either to meet the limited time requirement or to provide an ordered list of best candidate matches. Our source codes of the proposed algorithm are available at http://smart.iis.sinica.edu.tw/html/winup.html.

We use vertically invariant morphological filters for time-varying or adaptive signal processing. The morphological filters adopted in this paper are vertically invariant openings and closings. Vertically invariant openings and closings have intuitive geometric interpretations and can provide different filtering scales with respect to different spatial positions. Hence, they are suitable for adaptive signal filtering. To adaptively assign structuring elements of the vertically invariant openings or closings, we develop the progressive umbra-filling (PUF) procedure. Experimental results have shown that our approach can successfully eliminate noise without oversmoothing the important features of a signal.

In this paper, we propose a new method, the RANSAC-based DARCES method (data-aligned rigidity-constrained exhaustive search based on random sample consensus), which can solve the partially overlapping 3D registration problem without any initial estimation. For the noiseless case, the basic algorithm of our method can guarantee that the solution it finds is the true one, and its time complexity can be shown to be relatively low. An extra characteristic is that our method can be used even for the case that there are no local features in the 3D data sets.

This paper proposes a new technique for generating multiviewpoint video from a two-view stereo video sequence. The two-view stereo video can be easily obtained by using inexpensive two-view stereo video capture devices. For each image pair in the two-view stereoscopic video signals, our system first estimates the corresponding points of each pixel based on an epipolar constraint. A smoothing algorithm is used to smooth the estimation result, and then generate the disparity maps for each image pair. Then the proposed system can generate multiple perspective stereo video by interpolating or extrapolating the original views based on the generated disparity maps. Compared to the traditional method of capturing the multiple perspective video directly, our method can significantly reduce both the difficulties of video capturing and processing as well as the amount of video data.

In this paper, we show how an active binocular head, the IIS head, can be easily calibrated with very high accuracy. Our calibration method can also be applied to many other binocular heads. In addition to the proposal and demonstration of a four-stage calibration process, there are three major contributions in this paper. First, we propose a motorized-focus lens (MFL) camera model which assumes constant nominal extrinsic parameters. The advantage of having constant extrinsic parameters is to having a simple head/eye relation. Second, a calibration method for the MFL camera model is proposed in this paper, which separates estimation of the image center and effective focal length from estimation of the camera orientation and position. This separation has been proved to be crucial; otherwise, estimates of camera parameters would be very noise-sensitive. Thirdly, we show that, once the parameters of the MFL camera model is calibrated, a nonlinear recursive least-square estimator can be used to refine all the 35 kinematic parameters. Real experiments have shown that the proposed method can achieve accuracy of one pixel prediction error and 0.2 pixel epipolar error, even when all the joints, including the left and right focus motors, are moved simultaneously. This accuracy is good enough for many 3D vision applications, such as navigation, object tracking and reconstruction.

This paper proposes a new closed-form solution for identifying the kinematic parameters of an active binocular head having four revolute joints and two prismatic joints by using three-dimensional (3-D) point (position) measurements of a calibration point. Since this binocular head is composed of off-the-shelf components, its kinematic parameters are unknown. Therefore, we can not directly apply those existing nonlinear optimization methods. Even if we want to use the nonlinear optimization methods, a closed-form solution can be first applied to obtain accurate enough initial values. Hence, this paper considers only methods that provide closed-form solutions, i.e., those requiring no initial estimates. Notice that most existing closed-form solutions require pose (i.e., both position and orientation) measurements. However, as far as we know, there is no inexpensive technique which can provide accurate pose measurements. Therefore, existing closed-form solutions based on pose measurements can not give us the required accuracy. As a result, we have developed a new method that does not require orientation measurements and can use only the position measurements of a calibration point to obtain highly accurate estimates of kinematic parameters using closed-form solutions. The proposed method is based on the complete and parametrically continuous (CPC) kinematic model, and can be applied to any kind of kinematic parameter identification problems with or without multiple end-effecters, providing that the links are rigid, the joints are either revolute or prismatic and no closed-loop kinematic chain is included.

The statistical behavior of a special type of morphological operation, the space-varying opening, is analyzed and investigated. The major distinction between the space-invariant morphological operation and the space-varying one is that the structuring element corresponding to each position in the space-invariant operation is the same, but the structuring element corresponding to each position in the space-varying operation may be different. A regular statistical analysis method, which was originally designed for the analysis of the space-invariant opening, is applied and generalized to the analysis of the space-varying one. For the filtering of a noisy step edge, some comparative analytic results are obtained revealing that edge-preserving ability can be improved by the use of the space-varying opening.

The paper by Zhuang and Roth (ibid. vol.9, p.174-85 (1993)) presents a linear solution to kinematic parameter identification of robot manipulators. With their method, the orientation parameters for all revolute joints have to be solved first before solving for the translation parameters for all revolute joints and the orientation parameters for all prismatic joints simultaneously. Our major modification here is to decompose the kinematic parameters estimation problem into many subproblems of a single joint axis such that the complexity is reduced and easier implementation is derived. In addition, this gives a unified solution for any serial manipulator with an arbitrary combination of prismatic and revolute joints. >