Search

This paper presents a relation-centric algorithm for solving arithmetic word problems (AWPs) by synergizing a syntax-semantics extractor for extracting explicit relations, and a neural network miner for mining implicit relations. This is the first algorithm that has a specific component to acquire implicit knowledge items for solving AWPs. This paper proposes a three-phase scheme to decompose the challenging task of designing an algorithm for solving AWPs into three smaller tasks. The first phase proposes a state-action paradigm; the second phase instantiates the paradigm into a relation-centric approach; and the third phase implements a relation-centric algorithm for solving AWPs. There are two main steps in the proposed algorithm: problem understanding and symbolic solver. By adopting the relation-centric approach, problem understanding becomes a task of relation acquisition. For conducting the task of relation acquisition, a relaxed syntax-semantics method first extracts a group of explicit relation candidates. In parallel, a neural network miner acquires implicit relation candidates. The miner computes the vectors encoded by BERT to determine which implicit relations should be added. Thus, problem understanding can acquire both explicit relations and implicit relations, which addresses the challenge of building a problem understanding method that can acquire all the knowledge items to find the solution. In the subsequent step of symbolic solver, a fusion procedure forms a distilled set of relations from all the candidates by discarding unnecessary relations. Experimentation on nine benchmark datasets validates the superiority of the proposed algorithm that outperforms the state-of-the-art algorithms.

This article presents an algorithm for reading both single and multiple digital video clocks by using a context-aware pixel periodicity method and a deep learning technique. Reading digital video clocks in real time is a very challenging problem. The first challenge is the clock digit localization. The existing pixel periodicity is not applicable to localizing multiple second-digit places. This article proposes a context-aware pixel periodicity method to identify the second-pixels of each clock. The second challenge is clock-digit recognition. For this task, the algorithms based a domain knowledge and deep learning technique is proposed to recognize clock digits. The proposed algorithm is better than the existing best one in two aspects. The first one is that it can read not only single digit video clock but also multiple digit video clocks. The other is that it requires a short length of a video clip. The experimental results show that the proposed algorithm can achieve 100% of accuracy in both localization and recognition for both single and multiple clocks.

This paper addresses the challenging problem of developing the automatic algorithm for solving direct current circuit problem. Leveraging on the innovated methods it proposes a high-performance relation based algorithm, called RaDCC. The challenges of the problem lie in relation acquisition and relation inference presentation after adopting the newly-established relation principle of solving problems. A high-performance procedure is developed for the challenging task of relation acquisition by leveraging on three innovated methods. Three methods are an enhanced schematics understanding method that can understand complicated structures of schematics, an extended syntax-semantics model method and a unit-theorem inference method to acquire schematic relations, explicit text relations and implicit text relations, respectively. To address another challenging problem of readable solution generation an action-schema presentation method is proposed to convert relation inference actions into relation inference presentations. The experimental results show that the proposed algorithm is high-performance since it achieves an accuracy of over 83.2% for solving problems from textbooks and 70.6% for solving problems from examination papers on a dataset that contains 1012 direct current circuit problems collected from the authority sources, much higher than the performance of the baseline algorithm.

Robust speaker identification of Internet of Things (IoT) becomes an active research problem in the field of biometrics, and it has wide applications in commercial fields and public security. In order to improve the robustness of speaker identification of IoT, we propose a novel Stacked Sparse Denoising Auto-encoder (SSDAE) for robust speaker identification in this paper. Firstly, the i-vector features are estimated based on the universal background model and total variability space model. For the second stage, sparsity constraints and corrupting operations are fused together into the general auto-encoder, and then the single sparse denoising auto-encoder is constructed. Then, the SSDAE is built with multi-hidden sparse denoising auto-encoder layers by layer wisely stacking operation. In the final layer of the designed deep neural network, the Softmax layer is included for classification purpose. The experimental results have shown that the proposed robust speaker identification method achieves better performance in robust speaker identification task than the state-of-the-art approaches for different kinds and levels of noisy inputs.

This paper designs a structure of 3D convolutional neural network to detect the global exam events in invigilation videos. Exam events in invigilation videos are defined according to the human activity performed at a certain phase in the entire exam process. Unlike general event detection which involves different scenes, global event detection focuses on differentiating different collective activities in the exam room ambiance. The challenges lie in the great intra-class variations within the same type of events due to various camera angles and different exam room ambiances, as well as inter-class similarities which are challengeable. This paper adopts the 3D convolutional neural network based on its ability in extracting spatio-temporal features and its effectiveness in detecting video events. Experiment results show the designed 3D convolutional neural network achieves an accuracy of its capability of 93.94% in detecting the global exam events, which demonstrates the effectiveness of our model.

Zero Velocity Update (ZUPT) has played a key role in Pedestrian Dead Reckoning (PDR) with inertial measurement units (IMU). However, it is both crucial and difficult to determine ZUPT conditions given complex and varying motion types such as walking, fast walking or running, and different walking habits of distinct people, which have direct and significant impact on the tracking accuracy. In this research we proposed a model based on deep neural networks to determine moments when the ZUPT should be conducted. The proposed model ensures nearly identical performance regardless of different motion types. It has been demonstrated by extensive experiments conducted in three different scenarios that our model can work equally well with different pedestrians and walking patterns, enabling the wide use of PDR in real-world applications.

Many types of data exhibit characteristic of rotational symmetry. Chemical Organic Ring Structure(ORS) Symbol is such a case. In this paper, we focus on offline handwritten chemical ORS Symbols recognition using convolutional neural networks(CNNs), from application point of view, in order to relax the inconvenience and ineffectiveness of the traditional clickand-drag style of interaction when input chemical notations into electronic devices; from scientific point of view, to explore the capacity of rotation invariance of CNNs using data augmentation. We propose a VGGNet-based classifier for offline handwritten chemical ORS Symbols. To evaluate it, a new dataset of 3600 samples are collected of which 90% is for training while 10% is for test. The recognition accuracy is 84.3% with VGGNet-16 and 92.4% with VGGNet-19.

In this paper, we propose an end-to-end trainable system for recognizing handwritten chemical formulae. This system recognize once a time a chemical formula, instead of one chemical symbol or a whole chemical equation, which is in line with people's writing habits, at the same time could help to develop methods for the complicated chemical equations recognition. The proposed system adopts the CNN+RNN+CTC framework, which is one of state of the art methods in imagebased sequence labelling tasks. We extend the capability of the CNN+RNN+CTC framework to interpret 2D spatial relationships (such as 'subscript' existing in chemical formula) by introducing additional labels to represent them. The system evaluated on a self-collected data set of 12,224 samples, achieves the recognition rate of 94.98% at the chemical formula level.

This paper presents a novel approach for automatically extracting implicit quantity relations (IQRs) from Chinese algebra word problems (AWPs). Our algorithm first classifies AWPs into categories and analyzes each sentence in the problem to identify the variables and values, then maps them to equations according to the category, and enables its trivial as shown in Figure 1. Our algorithm is able to discover 15 kinds of IQRs in AWPs with an average accuracy of 88% when the training sample accounts for more than 3% of the total. Our data and code are publicly available.11Our data and code is available at: https://pan.baidu.com/s/1g1OI6EZTHCunrvXSzBuTew,Extraction code: zt1y

In order to analyze facial expression in human-computer interaction, real time face-tracking has become significant research problem. Traditional face tracking methods have achieved good results in some constrained environments (such as good illumination, no background interference, etc.) However, these methods require to design manual facial features depending on researcher’s experience. In addition, lacking ability for generalization problems is worthy of study. The robustness of face tracking in complex scenes is challenging due to fast moving, multi-scale changes, rotation and occlusion, illumination changes, etc. In view of the above considerations, this paper proposes an improved method based on Siamese-Net to optimize for face tracking tasks. Our work mainly includes four aspects. First, the first two convolutional layers of deeper VGG-16 are used to extract feature. So we call our method Siamese-VGG. Second, we report experiment on face tracking using a pre-trained VGG-Face model which is trained by 2.6M images for face recognition and then fine-tuning to acceleration convergence. Third, in this research the same size crops are input to two branches in the framework and then the inner smaller template feature maps are extracted during training. The proposed method in this paper reduce offset losses by this way. Finally, L2 regularization add to the loss function to improve the generalization ability of the model. The experiment results show better robustness and generalization performance over the original algorithm. In complex scenes, the proposed improved method have achieved the great improvements by almost 11% on average overlap. But, the frame rate of improved method is still 18.5fps on the Nvidia GTX1070Ti GPU.The improved method proposed in this paper is more practical in terms of speed and accuracy.

In a pure visual odometry, a pose transformation matrix between adjacent two frames is estimated by an algorithm based on pixel variation between images. However, pure monocular visual odometers cannot obtain absolute scales; in addition, relying solely on recursive calculations will inevitably lead to cumulative errors. For pure inertial solution calculations, the low-precision IMU will diverge very quickly in a short time. We derive the IMU prediction form based on Lie group and Lie algebra, and apply it to VIO. Based on the idea of graph optimization in pure visual SLAM, the IMU relative measurement information between frame and frame is converted into constraint node. The side of the pose is involved in the optimization framework. Using the IMU preintegration theory, these IMU’s are processed relative to the measurement so that it is decoupled from the absolute pose (or only requires linear operations to correct), which greatly increases the speed of optimization. In addition, this optimized architecture also makes the unacceptable gravity of the accelerometer measurement an advantageous condition - the presence of gravity will make the entire system observable to the absolute attitude.

This paper presents a double channel 3D convolution neural network to classify the exam scenes of invigilation videos. The first channel is based on the C3D convolution neural network, which is the status-of-arts method of the video scene classification. The structure of this channel is redesigned for classifying the exam-room scenes of invigilation videos. Another channel is based on the two-stream convolution neural network using the optical flow graph sequence as its input. This channel uses the data from the optical flow of video to improve the performance of the video scene classification. The formed double channel 3D convolution neural network has appropriate size of convolution kernel and pooling kernel design. Experiments show that the proposed neural network can classify the exam-room scenes of invigilation videos faster and more accurately than the existing methods.

This paper presents an algorithm for reading two digital video clocks of broadcast basketball videos by using context-aware pixel periodicity method and deep learning technique. Reading two digital video clocks is a very challenge problem as a difficulty special case of reading video text. The first challenge task is the clock digit localization. The existing pixel periodicity is a very good method for localizing second-digit place if a single clock is in video, but it is not applicable to localizing two second-digit places. This paper proposes a context-aware pixel periodicity method to identify the second-pixels of each clock based on the domain knowledge of digital video clocks. The second challenge task is clock-digit recognition. For this task, the CNN-based procedures are proposed to recognize clock digits. The experimental results show that the proposed algorithm can achieve 100% of accuracy in both localization and recognition for two clocks at very low computational cost by using very short input clips.

The structure line is an important clue for understanding indoor space. Unfortunately, in the context of indoor scene, very little structural data is available due to occlusion. To address this issue, this paper proposes to extract structural lines and analyze their relations to obtain the structure lines, which are continuous, anti-occlusion, able to reconstruct indoor space. In this paper, Plucker line is used to express the lines in 3D indoor scene. And then it studies the properties and the benefits of using Plucker line. For the first time, a new strategy is suggested to maintain the integrity of indoor space, which uses the extracted high confidence lines to infer the properties of other lines. Experimental results show that our methods have many advantages in obtaining a complete 3D interior structure line, which resists most occlusion in near ground.

This paper presents a framework to automatically replant timestamps for panorama video surveillance. The system mainly comprises six modules: 1) timestamp localization; 2) image registration; 3) timestamp pixel recovery; 4) color correction; 5) timestamp recognition; and 6) panorama video generation. The proposed framework first develops a method to localize the timestamp by employing the method of locating digital video clocks. Then, a fast and novel method is developed to recover the pixels covered by timestamps. The proposed method has fast speed because a fast image translation estimation procedure is introduced based on the PTZ camera motion estimation and global histogram-based image matching. Moreover, a hybrid color correction method by combining region matching with the gamma correction and the linear correction is presented. To plant timestamps accurately, a timestamp recognition procedure is developed by adopting the time recognition method for digit video clocks. Experimental results show that the proposed approach can accurately remove timestamps in real time, the recovered frames are visually consistent with the real scene, and the performance of color correction for panorama video is visually acceptable.

Positive samples of facial actions are much fewer than negative samples in natural conditions. The highly imbalanced class-distributions may cause very slow rate of convergence of error when using neural networks for facial action recognition. Traditional methods tackle this class-imbalance problem by changing data distributions, which is challenging for preventing the loss of useful information. In this paper we tackle this problem by using very deep (>10 layers) architectures to increase the chance that network training has acceptable rate of convergence using highly imbalanced data sets. Experimental results on EmotioNet Challenge data set show that the error rates of very deep covolutional networks converge to 40% after 90 epochs while shallower networks only converge to 60%. The results also show that very deep network outperforms shallower network by 0.2 on accuracy score. The proposed neural networks won the first place of the first track in the automatic detection of action units (AUs) of EmotioNet Challenge.

This paper presents a framework for solving math problems stated in a natural language (NL) and applies the framework to develop algorithms for solving explicit arithmetic word problems and proving plane geometry theorems. We focus on problem understanding, that is, the transformation of a NL description of a math problem to a formal representation. We view this as a relation extraction problem, and adopt a greedy algorithm to extract the mathematical relations using a syntax-semantics model, which is a set of patterns describing how a syntactic pattern is mapped to its formal semantics. Our method yields a human readable solution that shows how the mathematical relations are extracted one at a time. We apply our framework to solve arithmetic word problems and prove plane geometry theorems. For arithmetic word problems, the extracted relations are transformed into a system of equations, and the equations are then solved to produce the solution. For plane geometry theorems, these extracted relations are input to an inference system to generate the proof. We evaluate our approach on a set of arithmetic word problems stated in Chinese, and two sets of plane geometry theorems stated in Chinese and English. Our algorithms achieve high accuracies on these datasets and they also show some desirable properties such as brevity of algorithm description and legibility of algorithm actions.

This paper presents an algorithm for proving plane geometry theorems stated by text and diagram in a complementary way. The problem of proving plane geometry theorems involves two challenging subtasks, being theorem understanding and theorem proving. This paper proposes to consider theorem understanding as a problem of extracting relations from text and diagram. A syntax–semantics (S2) model method is proposed to extract the geometric relations from theorem text, and a diagram mining method is proposed to extract geometry relations from diagram. Then, a procedure is developed to obtain a set of relations that is consistent with the given theorem with high confidence. Finally, theorem proving is conducted by using the existing proving methods which take the extracted geometric relations as input. The experimental results show that the proposed theorem proving algorithm can prove 86% of plane geometry theorems in the test dataset of 200 theorems, which is all the theorems in the popular textbook. The proposed algorithm outperforms the existing algorithms mainly because it can extract relations not only from text but also from diagram.

This paper presents an end-to-end algorithm for solving circuit problems in secondary physics. A key challenge in solving circuit problems is to automatically understand circuit problems over the modals of both text and schematic. Existing methods have a limited capacity in problem understanding due to the they cannot deal with the numerous expressions of problems in natural language and the various circuit diagrams. In fact that this paper, a batch of methods is proposed to work against the challenge of solving circuit problems. The problem understanding is modeled as a problem of relation extraction and a scheme is proposed to extract relations from both text and schematic. A syntax–semantics model is adopted to extract explicit relations from text, whereas a unit-theorem-based method is proposed to extract implicit relations. And a mesh search method is proposed to extract relations from schematic. Based on the result of problem understanding, an algorithm is proposed to produce the solutions of circuit problems, in which the solutions are presented in a readable way. The experimental results demonstrate the effectiveness of the proposed algorithm in solving circuit problems. To the best of our knowledge, this paper is the first literature which reports the quantitative results in understanding and solving circuit problems.

Automatically understanding natural language problems is a long-standing challenging research problem in automatic solving. This paper models the understanding of geometry problems as a problem of relation extraction, instead of as the problem of semantic understanding of natural language. Then it further proposes a supervised machine learning method to extract geometric relations, targeting to produce a group of relations to represent the given geometry problem. This method identifies the actual geometric relations from the relation candidates using a classifier trained from the labelled examples. The formalized geometric relations can then be transformed into the target system-native representations for manipulation in various tasks. Experiments conducted on the test problem dataset show that the proposed method can extract geometric relations at high F1 scores. The comparisons also demonstrate that the proposed method can achieve good performance against the baseline methods. Integrating the automatic understanding method with different geometry systems will greatly enhance the efficiency and intelligence in geometry tutoring.

This paper presents the techniques used in our contribution to Emotion Recognition in the Wild 2016’s video based sub-challenge. The purpose of the sub-challenge is to classify the six basic emotions (angry, sad, happy, surprise, fear & disgust) and neutral. Compared to earlier years’ movie based datasets, this year’s test dataset introduced reality TV videos containing more spontaneous emotion. Our proposed solution is the fusion of facial expression recognition and audio emotion recognition subsystems at score level. For facial emotion recognition, starting from a network pre-trained on ImageNet training data, a deep Convolutional Neural Network is fine-tuned on FER2013 training data for feature extraction. The classifiers, i.e., kernel SVM, logistic regression and partial least squares are studied for comparison. An optimal fusion of classifiers learned from different kernels is carried out at the score level to improve system performance. For audio emotion recognition, a deep Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) is trained directly using the challenge dataset. Experimental results show that both subsystems individually and as a whole can achieve state-of-the art performance. The overall accuracy of the proposed approach on the challenge test dataset is 53.9%, which is better than the challenge baseline of 40.47% .

The extraction of implicit quantity relations is one of critical steps of building the machine solver for solving arithmetic word problems. Extracting implicit quantity relations is a challenge research problem as partial problem of natural language understanding, which has no mature solution method. To address this challenge, the existing work in the extraction of quantity relations mainly relies on creating problem solving framework and extracting system of linear equations. However, these existing methods can solve the relatively simple problems due to they have no methods to extract implicit quality relations. This paper proposes a novel method for extracting implicit quantity relations, which is achieved by the process of Chinese phrase parse, classification and instantiation method of required general implicit quantity relations with semantic models. A set of experiments were conducted to classify the arithmetic word problems, and an example was taken to illustrate the proposed method of identifying general implicit quantity relations. The experimental results demonstrate that the proposed method is effective in extracting implicit quantity relations.

This paper presents an original algorithm to automatically acquire accurate camera calibration from broadcast tennis video (BTV) as well as demonstrates two of its many applications. Accurate camera calibration from BTV is challenging because the frame-data of BTV is often heavily distorted and full of errors, resulting in wildly fluctuating camera parameters. To meet this challenge, we propose a frame grouping technique, which is based on the observation that many frames in BTV possess the same camera viewpoint. Leveraging on this fact, our algorithm groups frames according to the camera viewpoints. We then perform a group-wise data analysis to obtain a more stable estimate of the camera parameters. Recognizing the fact that some of these parameters do vary somewhat even if they have similar camera viewpoint, we further employ a Hough-like search to tune such parameters, minimizing the reprojection disparity. This two-tiered process gains stability in the estimates of the camera parameters, and yet ensures good match between the model and the reprojected camera view via the tuning step. To demonstrate the utility of such stable calibration, we apply the camera matrix acquired to two applications: (a) 3D virtual content insertion; and (b) tennis-ball detection and tracking. The experimental results show that our algorithm is able to acquire accurate camera matrix and the two applications have very good performances.

This paper addresses the problem of solving arithmetic word problems that are directly stated in Chinese without any implicit quantity relations. This problem relates with the multiple challenging problems in artificial intelligent area. This paper proposes multiple new methods to overcome the challenges of the problem and forms a four-step algorithm. The first step is to parse the input math problem into phrases with property annotation. The second one is to use a pool of the semantic models to convert a math problem into a set of high-order equations. The third one is a procedure to solve the set of equations. The last step is to convert the machine solution into a humanoid solution. The experimental results on 140 arithmetic word problems from the text books show that the proposed method has a very good potential.

This paper presents a novel algorithm for finding suspension boundaries (starting and end time points of event) of broadcast basketball videos, which employs the superimposed features extracted from the game clock and the shot clock to infer the suspension boundary. The main challenge of suspension boundary detection in the proposed way is to read (localize and recognize) the game clock and the shot clock. The difficulties of reading the two clocks from basketball videos lie in that the two clocks are independent from each other and are paused and reset frequently. This paper proposes two new techniques against the difficulties. The first new technique is a new set of functions of instantiating the second-periodicity method to identify the pixels inside second-regions without knowing the transit frames of two clocks. The second technique is a procedure of finding the proper digit-sequence from the second place of the shot clock. This technique solves the difficulty caused by the frequent pauses and the resets of two clocks. With the information extracted from the game clock and the shot clock this paper infers the start and the end time points of suspension clips. Experimental results show that the proposed algorithm can find much more accurate boundary points of suspensions at a much lower computing cost than the existing algorithms can.

This paper presents a fast algorithm to locate and remove overlaid video timestamps for panorama video surveillance. The timestamp removal for panorama video surveillance is a challenge research problem because it allows a very short processing time and it requires to recovery the covered objects. And the existing general object localization methods cannot localize the timestamp reliably and the existing object removal methods cannot remove timestamps with high visual scene consistence. For the timestamp localization, this paper customizes the existing method of localizing digital video clocks. For timestamp removal this paper proposes a fast and novel procedure. Its fast speed is because it employs a fast image registration method mainly by fusing the PTZ camera motion estimation and global histogram-based image matching. Then it recoveries the pixels covered by timestamps by using a temporal-neighbor pixel recovery method. The extensive experimental results show that the proposed algorithm can remove a timestamp within 2 seconds and the recovered frames are visually consistent with the real scene.

Developing problem-solving humanoid machines has been being a hot research topic in the circle of developing educational technology due to many valuable and demanding learning services for K-12 education can developed based on such machines. The ultimate goal of our research team is to develop a machine to solve all kinds of problems of all main subjects of K-12 is a huge system, but this paper presents only the preliminary research and development in developing such a machine. First, we give the system design with the function description of main modules. Second, we identify the main research problems to develop such a machine following the process that the machine solves a given problem. Third, we present an approach towards an engine of problem solving. Finally, we discuss the education services that we can develop based on such machines.

Timestamp replanting is required when we want to remove timestamps in individual videos and to plant a timestamp into their merged panorama video. This paper presents a preliminary automatic timestamp replanting algorithm for producing panorama surveillance video. Timestamp replanting is a challenge problem because localization, removal, and recognition of timestamp are three difficulty tasks. This paper develops methods to attack the difficulties to finish the tasks. First, it presents a novel localization procedure which first localizes second-digit by using a pixel secondly-periodicity method. And then it localizes timestamp via extracting all digits of timestamp. Second, it adopts a homography-based method to conduct timestamp removal. Third, it presents a digit-sequence recognition method to recognize second-digit and online template matching to recognize the other digits. Experimental results show that the algorithm can accurately localize timestamp in a very low computing cost and that the performances of replanting are visually acceptable.

This paper presents an algorithm for removing the timestamp of a video surveillance before it is merged to form a panorama surveillance video. Timestamp localization and timestamp pixel recovery are two challenging tasks for developing timestamp removal algorithm. This paper presents a novel localization procedure which first localizes the second-digit by using a pixel secondly-periodicity method. And then it develops a procedure for timestamp removal. Experimental results show that the algorithm can accurately localize timestamp in a very low computing cost and that the performance of timestamp removal is visually acceptable.

This paper presents a novel and robust system for recognizing the time of the digital video clock by using the domain knowledge. This system comprises a set of the functions for the time recognition so that the user can conveniently use these functions to recognize the time of digital video clocks or to execute some steps of recognizing time. These functions are novel and robust because they use the novel methods derived from the domain knowledge of the digital video clock. These methods are region second periodicity, global maximum model, digit location model, digit-sequence recognition, and on-the-fly SVM. Experimental results show that both the functions and the system can achieve very high recognition accuracy.

This paper proposes an efficient system which integrates multiple vision models for robust multiperson detection and tracking for mobile service and social robots in public environments. The core technique is a novel maximum likelihood (ML)-based algorithm which combines the multimodel detections in mean-shift tracking. First, a likelihood probability which integrates detections and similarity to local appearance is defined. Then, an expectation-maximization (EM)-like mean-shift algorithm is derived under the ML framework. In each iteration, the E-step estimates the associations to the detections, and the M-step locates the new position according to the ML criterion. To be robust to the complex crowded scenarios for multiperson tracking, an improved sequential strategy to perform the mean-shift tracking is proposed. Under this strategy, human objects are tracked sequentially according to their priority order. To balance the efficiency and robustness for real-time performance, at each stage, the first two objects from the list of the priority order are tested, and the one with the higher score is selected. The proposed method has been successfully implemented on real-world service and social robots. The vision system integrates stereo-based and histograms-of-oriented-gradients-based human detections, occlusion reasoning, and sequential mean-shift tracking. Various examples to show the advantages and robustness of the proposed system for multiperson tracking from mobile robots are presented. Quantitative evaluations on the performance of multiperson tracking are also performed. Experimental results indicate that significant improvements have been achieved by using the proposed method.

In this paper, a novel vision system is proposed to estimate attention of people from rich visual clues for social robot to perform natural interactions with multiple participants in public environments. The vision detection and recognition modules include multi-person detection and tracking, upper-body pose recognition, face and gaze detection, lip motion analysis for speaking recognition, and facial expression recognition. A computational approach is proposed to generate a quantitative estimation of human attention. The vision system is implemented on a robotic receptionist "EVE" and encouraging results have been obtained.

This paper presents a rotation-invariant expression recognition algorithm that uses both localized eyes and local binary pattern (LBP) feature for SVM. This is a complete algorithm from the image input of cameras to the output of the facial expression recognition. It first localizes the eyes and then uses the localized eyes to rotate the face into upright face to achieve the rotation-invariant. It also uses the localized eyes to calculate the face box, whereas most of the existing algorithms use the face box acquired by the face detection algorithms. The experimental results show that the face boxes calculated from eye locations can improve the performance of expression recognition.

There were many eye localization algorithms depending on face detection in the literature. Differently this paper presents a novel eye detection and localization system not depending on face detection for natural human and robot interaction using both stereo and visual cameras. To build a robust system we use stereo and visual cameras in synergy. The stereo camera is used to localize the head of the person to replace face detection. Then our eye identification algorithm detects and localizes two eyes inside head box. In eye detection step, our algorithm uses a HOG-moment (Histogram Of Gradient) feature to detect two eyes inside the head box. In eye localization step, we employ an iterative procedure to search the best location for eye pair. The experimental results show that the proposed eye detection and localization algorithm, not depending on face detection, has a similar robustness as the existing eye localization algorithms.

In this paper, we propose an algorithm to recognize the time of video clock in broadcast soccer videos. A naive OCR approach to the problem will be confounded by standard image segmentation issues such as merged digits and low-resolution. In contrast, our methods achieve robustness by exploiting the domain knowledge governing clock digit transitions. The time periodicity information enables the reliable recognition of overlaid digit video clock and also facilitates self-consistency checks. A robust time recognition algorithm was presented in our previous papers. However, that algorithm requires the precise ROI (Region Of Interest) as its input. Besides inheriting the existing techniques this paper proposes two new ones to overcome the impreciseness of ROIs. The first one is the extended-ROI digit transit periodicity detection. It can correctly detect the second transit even though the known ROI is not precise. The second one is a Hough-like search for precise second ROI in digit-sequence similarity measure. Experimental results on 70 soccer video clips confirm that our methods can achieve the accurate results.

This paper develops a HOG-based multistage approach for object detection and object pose recognition for service robots. This approach makes use of the merits of both multi-class and bi-class HOG-based detectors to form a three-stage algorithm at low computing cost. In the first stage, the multi-class classifier with coarse features is employed to estimate the orientation of a potential target object in the image; in the second stage, a bi-class detector corresponding to the detected orientation with intermediate level features is used to filter out most of false positives; and in the third stage, a bi-class detector corresponding to the detected orientation using fine features is used to achieve accurate detection with low rate of false positives. The training of multi-class and bi-class SVMs with their respective features in different levels is described. Experiments in real-world environments have shown that the proposed method is much more accurate than the detection method as it uses only multi-class detector. The proposed method is also much more efficient than the detection method as it uses a bi-class detector for each possible orientation. The approach works well on the scenarios where the SIFT-based detector may fail. The method can achieve real-time object detection, localization, and pose recognition on a P4 2.4GHz PC.

Head pose estimation is one of key steps for human and robot interaction because head pose is highly related with the attention of human to robot. Although head pose estimation has progressed in the past few decades, existing algorithms for visual imagery are not fast enough for smooth human and robot interaction. This paper proposes an algorithm to estimate head pose of a previously unseen head from thermal images, motivated by the fact that thermal imagery provides us the possibility to develop light-weight computing algorithm for estimating head pose. The proposed algorithm has multiple merits. Firstly, it is a fully-automatic algorithm. That does not require any manual initialization. Second, its operation is very simple. It recognizes the head pose from a single image. Third, it is a light-weight computing algorithm. This is mainly because it has a fast head localization procedure. The proposed algorithm can estimate head pose at 110 fps. Last, it is a very robust algorithm in head localization. The performance in head pose estimation is promising.

This paper presents a novel approach of object detection and localization for service robots, which combines color-based and gradient-based detectors and automatically adapts the color model according to the variation of lighting conditions. Exploiting complementary visual features, the fusion of color-based and gradient-based detectors can achieve both robust detection and accurate localization. In real world environment, the color-based detection according to an offline-learned general model may fail. From a new linear color variation model proposed in this paper, our approach can generate a specific model for the target object in the image and achieve self-adaptation of color detector for robust detection and accurate localization. The experiments show that the proposed method can significantly increase the detection rate for target object in various real world environments.

In this paper, we propose a novel Adaboost template to recognize human upper body poses from disparity images for natural human robot interaction (HRI). First, the upper body poses of standing persons are classified into seven categories of views. For each category, a mean template, variance template, and percentage template are generated. Then, the template region is divided into positive and negative regions, corresponding to the region of bodies and surrounding open space. A weak classifier is designed for each pixel in the template. A new EM-like Adaboost learning algorithm is designed to learn the Adaboost template. Different from existing Adaboost classifiers, we show that the Adaboost template can be used not only for recognition but also for adaptive top-down segmentation. By using Adaboost template, only a few positive samples for each category are required for learning. Comparison with conventional template matching techniques has been made. Experimental results show that significant improvements can be achieved in both cases. The method has been deployed in a social robot to estimate human attentions to the robot in real-time human robot interaction.

A novel stereo vision system for real-time human detection and tracking on a mobile service robot is presented in this paper. The system integrates the individually enhanced stereo-based human detection, HOG-based human detection, color-based tracking, and motion estimation for the robust detection and tracking of humans with large appearance and scale variations in real-world environments. A new framework of maximum likelihood based multi-model fusion is proposed to fuse these four human detection and tracking models according to the detection-track associations in 3D space, which is robust to the possible missed detections, false detections, and duplicated responses from the individual models. Multi-person tracking is implemented in a sequential near-to-far way, which well alleviates the difficulties caused by human-over-human occlusions. Extensive experimental results demonstrate the robustness of the proposed system under real-world scenarios with large variations in lighting conditions, cluttered backgrounds, human clothes and postures, and complex occlusion situations. Significant improvements in human detection and tracking have been achieved. The system has been deployed on six robot butlers to serve drinks, and showed encouraging performance in open ceremony events.

Personalized multimedia alert service of fall event is an important service for ageing in place thanks to it can enhance the life safety of the elderly and then relieve the continuous worrying of the caregivers. This paper presents a personalized alert service of fall event for aging in place based the proposed fall detection algorithm. Our novel algorithm detects falls in sliding windows of video recorded by single fixed camera. The novelty of this algorithm is multiple. First, it acquires 3D information of heads from 2D image locations by looking up a 2D--3D table. Second, it mainly works on sliding windows. In each sliding window, it detects the existences of primitive actions and calculates the measures. Last, it uses a trained SVM to distinguish fall windows from non-fall windows. The inputs to the SVM are the vector consisting of the existences of primitive actions and the calculated measures from sliding windows. The lookup table and measure calculations are achieved with the aid of camera calibration. Experiments show that the proposed algorithm performs well in fall detection. Based on the fall detection we design a personalized multimedia alert service of fall event.

Lift operation is one of critical functions for mobile service robot to move across levels in buildings. Lift operation poses the problem of liftbutton detection and recognition for computer vision. This paper presents a framework for vision-based liftbutton detection and recognition. This problem is challenging due to reflection and complex background. To achieve the robustness in lift operation, we adopt multiple techniques to combat the challenges. First, we propose a multiple partial models method to increase the robustness of button panel detection. Second, we do button recognition by combining structural inference, Hough transform, and multi-symbol recognition techniques. Lastly, we use ultrasonic distance measure devices to aid the vision of the robot. The experimental results show that our framework can achieve the promising results in recognizing both the internal and external buttons of lift.

Physiotherapay exercise is a vital medical treatment process as it helps bringing a normal life back to patients. However due to the requirements of time spent and interactions with medical specialists such as doctors, therapists, and nurses, patients have to deal with some difficulties such as time and cost of travelling, waiting for availability of specialists and ineffective personal exercises. These difficulties could cause worse on those patients living in the areas distant from medical centres or lacking of medical staffs and experts. Therefore the concept of tele-physiotherapy was created to improve the quality of physiotherapy services. The objective of tele-physiotherapy is to allow patients and medical experts carrying on their sessions through telecommunication networks as if they are in the same place. Computer vision technologies, then, can be useful and helpful in monitoring process of system because the quality assessments of physiotherapy exercise are mainly based on vision. This article is discussing the feasibility of applying computer vision technolgies in tele-physiotherapy while showing an example of using computer vision in fall detection. The study shows that computer vision has some potentials in enhancing and improving the telephysiotherapy system but the study of some considerations is needed before implementing.

Fall detection has been an active research problem as fall detection technology is critical for the ageing-at-home of the elderly and it can enhance life safety of the elderly and boost their confidence of ageing-at-home by immediately alerting fall occurrence to care givers. This paper presents an algorithm of fall detection for the ageing-at-home of the elderly. This algorithm detects fall events by identifying (human) shape state change pattern reflecting a fall incident from video recorded by a single fixed camera. The novelty of the algorithm is multiple. First, it detects fall occurrence by identifying the state change pattern. Second, it uses the camera projection matrix in its computing. Thus, it eliminates camera setting-related learning. Lastly, it adds constraints to state change pattern to reduce false alarms. Experiments show that the proposed algorithm has a promising performance.

In this paper, we describe the implemented service robot, called FusionBot. The goal of this research is to explore and demonstrate the utility of an interactive service robot in a smart home environment, thereby improving the quality of human life. The robot has four main features: 1) speech recognition, 2) object recognition, 3) object grabbing and fetching and 4) communication with a smart coffee machine. Its software architecture employs a multimodal dialogue system that integrates different components, including spoken dialog system, vision understanding, navigation and smart device gateway. In the experiments conducted during the TechFest 2008 event, the FusionBot successfully demonstrated that it could autonomously serve coffee to visitors on their request. Preliminary survey results indicate that the robot has potential to not only aid in the general robotics but also contribute towards the long term goal of intelligent service robotics in smart home environment.

This paper describes a novel robotic butler, developed by a multi-disciplinary team of researchers. The robotic butler is capable of detecting and tracking human, recognize hand gestures, serving beverages and performs dialog conversation with guest about their interests and their preferences; and providing specific information on the facilities at Fusionopolis building and various technologies used by the robot. The robot employs an event driven dialogue management system (DMS) architecture, speech recognition, ultra wideband, vision understanding and radio frequency identification. All these components and agents that are integrated in the DMS architecture are modular and can be re-used by other applications. In this paper, we will first describe the design concept and the architecture of the robotic butler. Secondly, we will describe in detail the workings of the speech and vision technology as this paper mainly focuses on human-robot interaction aspects of the social robot. Lastly, this paper will highlight some key challenges that were faced during the implementation of speech and vision technology into the robot.

Fall detection for elderly and patient has been an active research topic due to that the healthcare industry has a big demand for products and technology of fall detection. This paper gives a survey of fall detection for elderly and patient, focusing on identifying approaches and principles of the existing fall detection methods. To properly build the classification tree of the methods of fall detection we first study the characteristics of fall. Then according to what sensors and how sensors are used we first divide the methods of fall detection into three approaches: wearable device, ambience device, and camera-based. Further we divide each approach into two to three classes according to the used principles. For each class of algorithm we analyze their merits and demerits. We also give comments on how we can improve some algorithms.

In this paper, foreground detection is performed by scene interpretation. A natural scene in different illumination conditions is characterized by scene context which contains spatial and appearance representations. The spatial representation is obtained in two steps. First, the large homogenous regions in each sample image are extracted using local and global dominant color histograms (DCH). Then, the latent semantic regions of the scene are generated by combining the coincident regions in the segmented images. The appearance representation is learned by the probabilistic latent semantic analysis (PLSA) model with local DCH visual words. The scene context is then applied to interpret incoming images from the scene. For a new image, its global appearance is first recognized and then the pixels are labelled under the constraint of the scene appearance. The proposed method has been tested on various scenes under different weather conditions and very promising results have been obtained.