Search

The demand for smart surveillance systems has been driven by the ubiquity of cameras in modern society. Among the crucial tasks in such systems, person re-identification (re-ID) and multiple object tracking (MOT) are paramount. Despite the common photographic challenges they share, these tasks serve distinct objectives, complicating their integration into a unified system. To be specific, most existing work lacks a comprehensive study on effectively integrating re-ID models with object trackers to achieve optimal MOT performance. A decrease in MOT performance may occur without proper calibration for the integration of both components despite using an enhanced re-ID model for the tracker. To address these issues, we propose a straightforward and effective solution that integrates an improved re-ID model into a MOT framework, the BoT-SORT tracker, ensuring enhanced MOT performance on the well-known benchmarks MOT17 and MOT20 with fewer parameters for tuning. Recognizing the sub-optimal performance of existing re-ID models with their original loss functions, we introduce a novel loss function that incorporates a polynomial cross-entropy component to enhance training efficiency on closed-world datasets. As a result, the re-ID model trained with the proposed method achieves state-of-the-art performance on Market1501 and DukeMTMC datasets, and is subsequently applied to a BoT-SORT tracker with a post-processing re-ranking module for MOT. Experimental results show that the proposed method achieves 81.2% and 77.8% MOTA scores on MOT17 and MOT20 datasets, respectively, outperforming the state-of-the-art MOT methods.

In brain-computer interface (BCI) work, how correctly identifying various features and their corresponding actions from complex Electroencephalography (EEG) signals is a challenging technology. However, most current methods do not consider EEG feature information in spatial, temporal and spectral domains, and the structure of these models cannot effectively extract discriminative features, resulting in limited classification performance. To address this issue, we propose a novel text motor-imagery EEG discrimination method, namely wavelet-based temporal-spectral-attention correlation coefficient (WTS-CC), to simultaneously consider the features and their weighting in spatial, EEG-channel, temporal and spectral domains in this study. The initial Temporal Feature Extraction (iTFE) module extracts the initial important temporal features of MI EEG signals. The Deep EEG-Channel-attention (DEC) module is then proposed to automatically adjust the weight of each EEG channel according to its importance, thereby effectively enhancing more important EEG channels and suppressing less important EEG channels. Next, the Wavelet-based Temporal-Spectral-attention (WTS) module is proposed to obtain more significant discriminative features between different MI tasks by weighting features on two-dimensional time-frequency maps. Finally, a simple discrimination module is used for MI EEG discrimination. The experimental results indicate that the proposed text WTS-CC method can achieve promising discrimination performance that outperforms the state-of-the-art methods in terms of classification accuracy, Kappa coefficient, F1 score, and AUC on three public datasets.

This paper attempts to use a delta robot’s structure and reliable coordinates to develop a self-learning Chinese calligraphy-writing system that requires precise control. Ideally, to achieve human-like behavior, a delta robot can learn stroke trajectories autonomously and present the stroke beauty of calligraphy characters. Unfortunately, state-of-the-art approaches have not yet considered the presentation of stroke beauty resulting from angles of rotation and tilt of the brush. This paper presents an integrated system consisting of a stroke processing module, a hypothesis generation net (HGN) learning model with self-learning capability, a delta robot, and an image capture module. Our approach utilizes both the stroke trajectories from the stroke processing module and angles information from the HGN learning model to automatically produce five degrees of freedom action instructions. Based on the instructions, the delta robot completes calligraphy writing. Then, the image capture module provides feedback to the writing system for error calculation and coordinate correction. We utilize the mean absolute percentage error to verify the performance of the writing results. A correction algorithm and linear regression were used to improve the error correction results (less than 2% error). After several cycles, the written results approached the target sample finally. Consequently, the written results produced by the delta robot prove that our proposed system with learning ability can write Chinese calligraphy aesthetically.

In this paper, we propose a robotic cognitive system which can learn itself to do a specific assignment by accumulating experiences through bottom-up thinking to make decision by itself via top-down thinking according to the experiences. That is, the cognitive system has a self-learning ability which can accumulate its experiences to make itself smarter. In essence, the cognitive system possesses a perception model, a memory model, and a hypothesis model. The perception model converts image information into perception codes. The memory model stores experiences in the past and present to provide to the perception model and the hypothesis model. The hypothesis model, which generates the next decision according to the experiences from the memory model, is the most important part of the proposed cognitive system. To validate the performance of the proposed system, we utilize Chinese calligraphy writing tasks by a virtual robot through simulation to evaluate the abilities of the cognitive system. In order to generate the coordinates of the writing brush, we made the virtual robot practice to learn Chinese calligraphy through bottom-up thinking to construct the writing patterns. The illustrative examples in this paper show that the virtual robot can learn to write Chinese calligraphy by top-down thinking according to its own experiences.

Lidar sensors are commonly equipped on a mobile mapping system (MMS) to establish point clouds for HD map creation. However, the point clouds themselves do not contain object attributes. Therefore, human operators have to manually obtain objects’ position to assign attributes for further HD map conversion, inevitably resulting in time-consuming processes and significant labor costs. To solve the above problems, in this paper, we present an MMS equipped with non-survey grade Lidar, commercial grade camera, and entry level GNSS/INS, which incorporates ground control points (GCPs) with a Normal Distribution Transform Simultaneously Localization and Mapping (NDT SLAM) refinement and fluctuation adjustment to secure both absolute position accuracy and relative position accuracy of the reconstructed point cloud. Meanwhile, a deep neural network for image detection is employed to obtain the bounding box of traffic signs from each image frame. By applying the translation and rotation transformation between Lidar points and camera pixels, intersection of the detected object in the image and Lidar scan points can be found. By accumulating extracted Lidar points of the traffic sign in several detection frames, we can then obtain an accurate 3D geodetic coordinate of the traffic signs. Experimental results show that point clouds can be reconstructed with an average 3D RMSE of only 8.6cm, and center geodetic coordinates of traffic signs can be further extracted in sub-meter accuracy to significantly reduce labor work in HD map creation.

The accurate estimation of three-dimensional (3D) object pose is important in a wide range of applications, such as robotics and augmented reality. The key to estimate object poses is matching feature points in the captured image with predefined ones of the 3D model of the object. Existing learning-based pose estimation systems utilize a voting strategy to estimate the feature points in a vector space for improving the accuracy of the estimated pose. However, the loss function of such approaches only takes account of the direction of the vector, resulting in an error-prone localization of feature points. Therefore, this paper considers a projection loss function dealing with the error of the vector field and incorporates a refinement network to revise the predicted pose to obtain a good final output. Experimental results show that the proposed methods outperform the state-of-the-art methods in ADD(-S) metric on the LINEMOD and Occlusion LINEMOD datasets. Moreover, the proposed method can be applied to real-world practical scenarios in real time to simultaneously estimate the poses of multiple objects.

Haze reduces the contrast of an image and causes the loss in colors, which has a negative effect on the subsequent object detection; therefore, single image dehazing is a challenging visual task. In addition, defects exist in previous existing dehazing approaches: Pixel-based dehazing approaches are likely to result in insufficient information to estimate the transmission, whereas patch-based ones are prone to generate shadows. They both also tend to induce color deviations. Therefore, this study proposes a novel method based on multi-scale wavelet and non-local dehazing. A hazy image is first decomposed into a low-frequency and three high-frequency sub-images by wavelet transform. Non-local dehazing and wavelet denoising are then employed on the low-frequency and high-frequency sub-images to remove the haze and noise, respectively. Finally, a haze-free image is obtained from the reconstruction of sub-images. The proposed method focuses on the dehazing and denoising on the low-frequency and high-frequency images respectively, through which the details on the image can be well preserved. Experimental results indicate that the proposed method is superior to the state-of-the-art approaches in both quantitative and qualitative evaluation on the synthetic and real-world image datasets.

Although pedestrian detection technology is constantly improving, pedestrian detection remains challenging because of the uncertainty and diversity of pedestrians in different scales and of occluded pedestrian modes. This study followed the common framework of single-shot object detection and proposed a divide-and-rule method to solve the aforementioned problems. The proposed model introduced a segmentation function that can split pedestrians who do not overlap in one image into two subimages. By using a network architecture, multiresolution adaptive fusion was performed on the output of all images and subimages to generate the final detection result. This study conducted an extensive evaluation of several challenging pedestrian detection data sets and finally proved the effectiveness of the proposed model. In particular, the proposed model achieved the most advanced performance on data sets from Visual Object Classes 2012 (VOC 2012), the French Institute for Research in Computer Science and Automation, and the Swiss Federal Institute of Technology in Zurich and obtained the most competitive results in a triple-width VOC 2012 experiment carefully designed by the present study.

The spectrum distribution of the light source between scenes is usually uniform in all scenes lit by a given light source per the color constancy assumption. Nonuniform spectrum distribution results in pictures with different color temperatures in different lighting conditions. With the conventional automated white balance method, the results under different degrees of color cast are inconsistent. To resolve this, a new automated white balance algorithm is proposed to achieve uniformity in images with color cast caused by color temperature variations. The proposed method features the selection of near-white points through automated regression and the dynamic adjustment of the color components’ weights. Finally, white balance is implemented according to the selected near-white points with various percentages. The proposed method can effectively select the appropriate white reference point of the image scene under different color temperatures, so that the adjusted results of the method under different color temperatures are close. The first major contribution of this study is the minimization of differences of continual color temperature images with different color temperatures and the achievement of favorable visual effects. The second contribution is the superior outcomes yielded by the proposed method in the visualization and quantification of data compared with those from the existing methods. The experimental results demonstrate that this method yields favorable results in the improvement of experiments with continual different color temperatures. The images at various color temperatures are adjusted to approach real-world color temperatures. In addition, this method provides superior color constancy preservation to other methods.

In recent years, a tremendous amount of effort has been devoted to modeling the cognition of human brain, particularly hypothesis generation process. Most research of the hypothesis generation model is probability-based. However, computation of human brains is still neuron-based instead of calculating the probability. As an attempt to solve this problem in this paper, we propose a novel neuron-based hypothesis generation model, called hypothesis generation net, to model human cognition, including how to make decisions and how to do actions. Basically, the proposed hypothesis generation model consists of two parts, i.e., a hypothesis model and an evaluation model. When these two models interact, the system is able to generate hypotheses to solve complex tasks based on historical experiences. To validate the feasibility of the proposed hypothesis generation model, we show a virtual robot with its cognition system can learn how to write Chinese calligraphy in a simulation environment, where an image-to-action translation via a cognitive framework is proposed to learn the pattern of Chinese characters. Based on the proposed deep thinking and learning mechanism, the virtual robot is able to write Chinese calligraphy well, which is a difficult task requiring extremely complicated motions, through thinking and practicing according to a human writing sample.

In this paper, a novel parallel indirect visual odometry (VO) system is proposed based on a newly designed map management method, key-frame selection, and a camera pose correction model, where the speeded-up robust features (SURF) algorithm is used to extract features from an image, and a linear exhaustive search (LES) algorithm is introduced to match features. To minimize computation time, a key-frame selection mechanism is proposed to distinguish key frames among the input images. Moreover, map management is proposed to filter out unstable landmarks and add features for a reliable estimation of the relative camera pose. In addition, estimation accuracy is improved by the camera pose correction model. To enhance computational efficiency of the VO system, the proposed approach is implemented on a TX2 embedded system with multiple graphics processing units (GPUs), taking advantage of a heterogeneous parallel computing architecture. To validate the performances of the proposed system, several experiments are conducted using an ASUS Xtion 3D camera and a laptop. Average errors of pose estimations are compared with those via the conventional VO to show the effectiveness of the proposed VO system. Thanks to the proposed approach, a real-time and reliable VO system can be established with low cost, low power consumption, high processing efficiency, and portability. The experimental results show that based on heterogeneous computing, the required computation time of the overall system with GPUs is approximately 80 times faster than that with only a CPU, when dealing with 80 features in the environment. To the best of our knowledge, this is the first released paper that implements an indirect VO on a CPU/GPU heterogeneous computing platform.

In this study, we propose a novel method incorporating faster region-based convolutional neural network and active shape model to automatically recognize, segment, and track the left ventricle in cardiac ultrasound image sequences, respectively. Ultrasound images typically contain noise and artifacts. The conventional filters cannot preserve the edges of image contours, and thus blurry images are often obtained. In this study, we propose an improved adaptive anisotropic diffusion filter to effectively reduce noise and reinforce image contours. In addition, because of the shape and appearance of the left ventricle vary considerably between adjacent images, conventional methods cannot automatically identify the position of the left ventricle or accurately segment them. A novel method that combines the faster region-based convolutional neural network with the active shape model is proposed to automatically recognize, segment, and track the left ventricle in cardiac ultrasound image sequences. Compared with four state-of-the-art approaches, the method proposed in this study can be applied to accurately segment and track the left ventricle in cardiac ultrasound image sequences. The proposed method produces the most satisfactory results in terms of visual presentation and segmentation quality based on four criteria.

This paper proposes a novel hardware design method of scale-invariant feature transform (SIFT) algorithm for implementation on field-programmable gate array (FPGA). To reduce the computing costs, Gaussian kernels are calculated offline for use in Gaussian filters. To eliminate low-contrast points, the inverse of a Hessian matrix is required for hardware implementation, which results in poor performance because dividers are needed. To solve this problem, this paper presents a new mathematical derivation model to implement the low-contrast detection, avoiding the use of any dividers. For the implementation of the normalization module, a large number of dividers are required by traditional methods, which adversely affects the computational efficiency. This paper presents a new architecture using only one divider to implement the normalization function in hardware. Thanks to the parallel processing architecture proposed to design the image pyramid, SIFT detection, and SIFT descriptor, the computational efficiency of the SIFT algorithm is significantly improved. As a result of the proposed design method, the requirement of logic elements in the FPGA hardware is greatly reduced and system frequency is significantly increased. Experimental results show that the proposed hardware architecture outperforms existing techniques in terms of resource usage and computational efficiency for real-time image processing.

Pedestrian detection has become an important subject in applications such as automobile autopilot assistance and pedestrian tracking in surveillance systems. A number of powerful object detectors have been used in pedestrian detection to solve problems created by pedestrians’ posture, stature, clothing, viewing angle, shielding, and illumination; however, it is still a challenge to detect small-size and low-resolution (LR) images in pedestrian detection. In order to capture more scenes, surveillance cameras are usually set high above the ground, causing image sizes to be too small or the resolution too low. When these blurred or noisy images are imported into the object detector, the extraction of pedestrian feature information will lead to a relatively low detection rate. To solve this problem, this study proposes a novel stationary wavelet dilated residual super-resolution (SWDR-SR) network to greatly enhance the edge information of SR images and thereby improve pedestrian detection. The LR image is decomposed with stationary wavelet transform (SWT) into low- and high-frequency sub-images, which are then processed to restore the structure of low-frequency content and the details of high-frequency information so that the reconstructed SR image can also retain edge details, which is conducive to pedestrian detection. In addition, this study proposes a novel low-to-high frequency connection (L2HFC) to better retain edge details so as to further enhance pedestrian detection. Finally, you only look once version 4 (YOLOv4) is used for pedestrian detection on the SR images to validate the performance of the proposed method. Using seven common datasets, the experimental results indicate that the proposed SWDR-SR method is effective at detecting small-size and LR images. [ABSTRACT FROM AUTHOR]