Search

The number of pollen-bearing honey bees serves as a vital indicator for assessing colony balance and health. Despite its significance, prevailing detection techniques still rely heavily on manual observation and annotation, leading to time-consuming processes that cannot sustain long-term, continuous monitoring efforts. To facilitate automatic beehive monitoring, this study introduces an efficient method for pollen-bearing bee detection. Initially, we furnish a comprehensive dataset, dubbed VnPollenBee, meticulously annotated for pollen-bearing honey bee detection and classification. The dataset comprises 60,826 annotated boxes that delineate both pollen-bearing and non-pollen-bearing bees in 2051 images captured at the entrances of beehives under various environmental conditions. To the best of our knowledge, this represents the first dedicated dataset for pollen-bearing bee detection. The VnPollenBee dataset is publicly accessible to the research community at https://comvis-hust.github.io/datasets/pollenbee.html. Subsequently, we propose the incorporation of diverse techniques into two baseline models, namely YOLOv5 and Faster RCNN, to effectively address the imbalance that arises during the detection of pollen-bearing bees due to their number being typically much lower than the total number of bees present at hive entrances. The experimental results demonstrate that our proposed method outperforms the baseline models on the VnPollenBee dataset, yielding Precision, Recall, and F1 score of 99%, 93%, and 95%, respectively. Specifically, the improvements obtained are 3% and 2% in Recall and F1 score when using YOLOv5, and 3%, 2%, and 2% in Precision, Recall, and F1 score when using Faster RCNN. These findings confirm the potential of our approach to facilitate bee foraging behavior analysis and automated bee monitoring.

Plant identification poses a challenge for end-users, including novice botanists, due to the high similarity between species, particularly those within the same genus and family, as well as the considerable variation among trees of the same species. To tackle to this challenge, botanists frequently integrate multiple pieces of information about plants to determine their species. Inspired by this observation, in this paper we propose a robust plant identification method that integrate the taxonomic information and multiple images observed from the plant. To achieve this, we define different taxonomic losses to capture the relationship of species within the same genus and of genera within the same family. These loss functions are combined with various deep learning network backbones. Additionally, to leverage the multiple observed images of the plant, we utilize late fusion to merge the identification results obtained for each image. Numerous experiments have been conducted on VietForest - a dataset collected for 156 species in PhuTho province in Vietnam to evaluate the effectiveness of the proposed method. The experimental results demonstrate that the proposed method outperforms the existing methods on VietForest dataset, yielding the accuracy of 84.88%, 82.26% and 81.14% for Family, Genus and Species respectively when using one input image. In multiple image-based plant identification, the proposed method achieves the best results of 97.45%, 91.77% and 89.35% for Family, Genus and Species with five images. Thanks to the superior results achieved by the proposed method, the plant identification module has been integrated into an application dedicated to plant management in PhuTho province.

Hand-raising gesture is one of the most popular signs of communication, whose frequency is related to the classroom’s atmosphere, the attractiveness of the subject, and the level of interactions between students and teachers. However, automatic hand-raising gesture detection and recognition remains still a challenging problem mainly due to low hand resolution, hand occlusion, various backgrounds, and viewpoint changes. While majority of the existing methods focus on static hand-raising posture detection, in this paper, we propose a framework for dynamic hand gesture recognition from classroom videos consisting of two main stages: hand posture detection and dynamic hand gesture recognition. In hand posture detection stage, we extend the previous work by adding relative position-aware in non-local network. After detecting the hand-raising posture on static pixels, which in turn accelerates the performance images, at the dynamic hand gesture recognition stage, we incorporate object detection and tracking to associate the hand-raising detection results at consecutive frames to supplement the missing detection due to the occlusion issue and obtain hand-raising gesture recognition at the event level. The experimental results show that the proposed method outperforms three benchmark models for static hand posture detection that are Faster-RCNN, Libra-RCNN, and Libra-RCNN+RDA on our dataset with the gains obtained for mAP being 7.68%, 5.76%, and 0.35% higher, respectively. In terms of event-level recognition results, the proposed method achieves the value of frame-wise accuracy, temporal_IoU, F1-score@0.3, and Levenshtein-score are 90.0%, 84.4%, 83.2%, and 84.3%, respectively. The code and dataset used in the paper will be made publicity available for research community.

In this paper, we present an unified framework for understanding hand action from the first-person video. The proposed framework composes two main components. The first component estimates three-dimensional (3D) hand joints from RGB images. Two network structures derived from the baseline HopeNet network are proposed: convolutional neural networks (CNNs) which are traditional multi-layer CNN and CNN combining with GraphCNN to perform 3D hand pose estimation, without the use of GraphUNet as in baseline HopeNet method. The second component of the framework recognizes hand action from skeleton stream. We first deploy two recent advanced neuronal networks that are PA-ResGCN and Double-feature Double-motion (DDNet). To focus more on the hand pose changes, we improve DDNet with two normalization strategies of the hand joints. Finally, we fuse PA-ResGCN with our improved DDNet to still boost the recognition performance. We evaluate our proposed methods on First-Person Hand Action Benchmark dataset. Experiments show that our model for 3D hand joints estimation achieves the best precision (36.6 mm). Our hand joint normalization strategies improve the original DDNet from 0.71% to 4.05% of accuracy with the ground-truth hand pose while the improvement is significantly larger (from 2.96% to 10.98%) with the estimated hand pose. The late fusion schemes outperform different state-of-the-art methods for the hand action recognition with the highest accuracy of 86.67%. These experimental results show potential and extendable possibilities for developing practical first-person vision applications.

In this work, we study the ability to use hand gestures for human-machine interaction from wrist-worn sensors. Towards this goal, we design a wrist-worn prototype to capture RGB video stream of hand gestures. Then we built a new wrist-worn gesture dataset (named WiGes) with various subjects in interaction with home appliances in different environments. To the best of our knowledge, this is the first benchmark released for studying hand gestures from a wrist-worn camera. We then evaluate various CNN models for vision-based recognition. Furthermore, we deeply analyze the models that produce the best trade-off between accuracy, memory requirement, and computational cost. We point out that among studied architectures, MoviNet produces the highest accuracy. Then, we introduce a new MoviNet-based two-stream architecture that takes both RGB and optical flow into account. Our proposed architecture increases the Top-1 accuracy by 1.36% and 3.67% according to two evaluation protocols. Our dataset, baselines, and proposed model analysis give instructive recommendations for human-machine interaction using hand-held devices.

Abstract Skeleton‐based human action recognition has emerged recently thanks to its compactness and robustness to appearance variations. Although impressive results have been obtained in recent years, the performance of skeleton‐based action recognition methods has to be improved to be deployed in real‐time applications. Recently, a lightweight network structure named Double‐feature Double‐motion Network (DD‐Net) has been proposed for the skeleton‐based human action recognition. With high speed, the DD‐Net achieves state‐of‐the‐art performance on hand and body actions. The DD‐Net could not distinguish actions if they have a weak connection with the global trajectories. However, the DD‐Net is suitable for human action recognition where actions strongly correlate to the global trajectories. In this paper, the authors propose TD‐Net, an improved version of the DD‐Net in which a new branch is added. The new branch takes the normalised coordinates of joints (NCJ) to enrich the spatial information. On five datasets for skeleton‐based human activity recognition that are MSR‐Action3D, CMDFall, JHMDB, FPHAB, and NTU RGB + D, the TD‐Net consistently obtains superior performance compared with the baseline model DD‐Net. The proposed method outperforms different state‐of‐the‐art methods, including both hand‐designed and deep learning‐based methods on four datasets (MSR‐Action3D, CMDFall, JHMDB, and FPHAB). Furthermore, the generalisation of the proposed method is confirmed through cross‐dataset evaluation. To illustrate the potential use of the model for real‐time human action recognition, the authors have deployed an application on an edge device. The experimental result shows that the application can process up to 40 fps for pose estimation using MediaPipe. It takes only 0.04 ms to recognise an action from skeleton sequences.

Fusion techniques with the aim to leverage the discriminative power of different appearance features for person representation have been widely applied in person re-identification. They are performed by concatenating all feature vectors (known as early fusion) or by combining matching scores of different classifiers (known as late fusion). Previous studies have proved that late fusion techniques achieve better results than early fusion ones. However, majority of the studies focus on determining the suitable weighting schemes that can reflect the role of each feature. The determined weights are then integrated in conventional similarity functions, such as Cosine [L. Zheng, S. Wang, L. Tian, F. He, Z. Liu and Q. Tian, Queryadaptive late fusion for image search and person reidentification, in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2015, pp. 1741–1750]. The contribution of this paper is two-fold. First, a robust person re-identification method by combining the metric learning with late fusion techniques is proposed. The metric learning method Cross-view Quadratic Discriminant Analysis (XQDA) is employed to learn a discriminant low dimensional subspace to minimize the intra-person distance while maximize the inter-person distance. Moreover, product rule-based and sum rule-based late fusion techniques are applied on these distances. Second, concerning feature engineering, the ResNet extraction process has been modified in order to extract local features of different stripes in person images. To show the effectiveness of the proposed method, both single-shot and multi-shot scenarios are considered. Three state-of-the-art features that are Gaussians of Gaussians (GOG), Local Maximal Occurrence (LOMO) and deep-learned features extracted through a Residual network (ResNet) are extracted from person images. The experimental results on three benchmark datasets that are iLIDS-VID, PRID-2011 and VIPeR show that the proposed method obtains+11.86%,+3.48% and+2.22% of improvement over the best results obtained with the single feature. The proposed method that achieves the accuracy of 85.73%, 93.82% and 50.85% at rank-1 for iLIDS-VID, PRID-2011 and VIPeR, respectively, outperforms different SOTA methods including deep learning ones. Source code is publicly available to facilitate the development of person re-ID system.

Human action recognition (HAR) under different camera viewpoints is the most critical requirement for practical deployment. In this paper, we propose a novel method that leverages successful deep learning-based features for action representation and multi-view analysis to accomplish robust HAR under viewpoint changes. Specifically, we investigate various deep learning techniques, from 2D CNNs to 3D CNNs to capture spatial and temporal characteristics of actions at each separated camera view. A common feature space is then constructed to keep view-invariant features among extracted streams. This is carried out by learning a set of linear transformations that project private features into the common space in which the classes are well distinguished from each other. To this end, we first adopt Multi-view Discriminant Analysis (MvDA). The original MvDA suffers from odd situations in which the most class-discrepant common space could not be found because its objective is overly concentrated on pushing classes from the global mean but unaware of the distance between specific pairs of adjoining classes. We then introduce a pairwise-covariance maximizing extension that takes pairwise distances between classes into account, namely pc-MvDA. The novel method also differs in the way that could be more favorably applied for large high-dimensional multi-view datasets. Extensive experimental results on four datasets (IXMAS, MuHAVi, MICAGes, NTU RGB+D) show that pc-MvDA achieves consistent performance gain, especially for harder classes. The code is publicly available for research purpose at https://github.com/inspiros/pcmvda.

Recently, skeleton-based gesture and action recognition have emerged thanks to the progress in human pose estimation. Gesture representation using skeletal data is robust since skeletal data are invariant to the individual’s appearance. Among different approaches proposed for skeleton-based action/gesture recognition, Graph Convolutional Network (GCN) and its variations have obtained great attention thanks to its ability to capture the graph essence of the skeletal data. In this paper, we aim to design an efficient scheme using relative joints of skeleton sequences adapted in a GCN framework. Both spatial features (i.e., joint positions) and temporal ones (i.e., the velocity of joints) are combined to form the input of Attention-enhanced Adaptive GCN (AAGCN). The proposed framework deals with limitations of the original AAGCN when it works on challenging datasets with incomplete and noisy skeletal data. Extensive experiments are carried out on three datasets CMDFALL, MICA-Action3D, NTU-RGBD. Experimental results show that the proposed method achieves superior performance compared with existing methods. Moreover, to illustrate the application of the proposed method in real-time traffic control gesture recognition for autonomous vehicles, we have evaluated the proposed method on the TCG dataset. The obtained results show that the proposed method offers real-time computation capability and good recognition results. These results suggest a promising solution to deploy a real-time and robust recognition technique for gesture-based traffic control in autonomous vehicles.

Mathematical expressions have been widely used in scientific documents. In order to analyze the documents, automatic detection of mathematical expressions is a crucial step. The paper presents a unified system for the detection of mathematical expressions including both inline and isolated expressions in scientific document images that usually consist of heterogeneous components (e.g., figures, tables, text and expressions). In the system, a hybrid method of two stages is proposed for the effective detection of mathematical expressions. First, the layout analysis of entire document images is introduced to improve the accuracy of text line and word segmentation. Then, both isolated and inline expressions in document images are detected. Both hand-crafted and deep learning features are extensively investigated and combined to improve the detection accuracy. Furthermore, a generic performance metric is applied to evaluate the system comprehensively. The proposed method has been evaluated on two public benchmark datasets (Marmot and GTDB). The obtained accuracies of isolated and inline expressions in the Marmot dataset are 91.18% and 81.35% while those in the GTDB dataset are 89.51% and 80.20%, respectively. The performance comparison is carried out with the conventional methods to show the outstanding effectiveness of the proposed system. Moreover, extensive experiments have been performed in order to point out the effect of document image resolution and post processing techniques on mathematical expression detection.

Recently, the recent advancement of deep learning with the capacity to perform automatic high-level feature extraction has achieved promising performance for sensor-based human activity recognition (HAR). Among different deep learning methods, Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM) have been widely adopted. However, scalar outputs and pooling in CNN only allow to get the invariance but not the equivariance. The capsule networks (CapsNet) with the vector output and routing by agreement is able to capture the equivariance. In this paper, we propose a method for recognizing human activity from wearable sensors based on a capsule network named SensCapsNet. The architecture of SensCapsNet is designed to be suitable for spatial-temporal data coming from wearable sensors. Experimental results show that the proposed network outperforms CNN and LSTM methods. The performance of the proposed CapsNet architecture is assessed by altering dynamic routing between capsule layers. The proposed SensCapsNet yields improved accuracy values of 77.7% and 70.5% for 1 routing on two testing datasets in comparison with the baseline methods based on CNN and LSTM that yields the F1-score of 67.7% and 69.2% for the first dataset and 65.3% and 67.6% for the second dataset respectively. Moreover, even several human activity datasets are available, privacy invasion and obtrusive concerns have not been carefully taken in to consideration in dataset building. Toward to build a non-obstructive sensing based human activity recognition method, in this paper, a dataset named 19NonSens is designed and collected from twelve subjects wearing e-Shoes and a smart watch to perform 19 activities under multiple contexts. This dataset will be made publicity available. Finally, thanks to the promising results obtained by the proposed method, we develop a life logging application which achieves a real-time computation and the accuracy rate greater than 80% for 5 common upper body activities.

This paper focuses on improving the performance of image-to-video person re-identification through feature fusion. In this study, image-to-video person re-identification is formulated as a classification-based information retrieval in which a pedestrian appearance model is learned in the training phase, and the identity of an interested person is determined based on the probability his/her probe image belongs to the model. Four state-of-the-art features belonging to two categories: hand-designed features and learned features are investigated for person image representation. They are Kernel Descriptor, Gaussian of Gaussian, features extracted from two famous convolutional neural networks (GoogleNet and ResNet). Furthermore, three fusion schemes that are early fusion, product-rule and query-adaptive late fusion are proposed. To evaluate the performance of the chosen features for person appearance representation as well as their combination in three proposed fusion schemes, 114 experiments on two public benchmark datasets (CAVIAR4REID and RAiD) have been conducted. The experiments confirm the robustness and effectiveness of the proposed fusion schemes. The proposed schemes obtain improvement of +7.16%, +5.42%, and +6.30% at rank-1 over those of single feature in case A, case B of CAVIAR4REID, and RAiD, respectively.