Search

This paper presents a real-time obstacle perception method (ROPM) for unmanned aerial vehicles (UAVs) with an RGB-D camera, which aims to address the difficulty of perceiving obstacles in real-time for UAVs in low-light environments. First of all, a novel obstacle detector is designed based on the ensemble detection strategy to rapidly detect dynamic and static obstacles in low-light environments. Moreover, a new tracker is constructed based on Kernel Correlation Filter (KCF), which uses the detection results to obtain obstacle features for regression matching. At the same time, a constant-acceleration Kalman filter is used to estimate the state of the dynamic obstacles in order to achieve the objective of constant and stable dynamic tracking. Furthermore, an obstacle reposition method in the region of obstacle tracking loss is designed, in order to address the problem of occlusion during obstacle perception in unknown low-light environments. Finally, extensive validation conducted in both indoor and outdoor under various lighting conditions proves the effectiveness of our proposed method for highly dynamic and low-light environments. [ABSTRACT FROM AUTHOR]

This paper aims to address the challenge of precise robotic grasping of molecular sieve drying bags during automated packaging by proposing a six-dimensional (6D) pose estimation method based on an red green blue-depth (RGB-D) camera. The method consists of three components: point cloud pre-segmentation, target extraction, and pose estimation. A minimum bounding box-based pre-segmentation method was designed to minimize the impact of packaging wrinkles and skirt curling. Orientation filtering combined with Euclidean clustering and Principal Component Analysis (PCA)-based iterative segmentation was employed to accurately extract the target body. Lastly, a multi-target feature fusion method was applied for pose estimation to compute an accurate grasping pose. To validate the effectiveness of the proposed method, 102 sets of experiments were conducted and compared with classical methods such as Fast Point Feature Histograms (FPFH) and Point Pair Features (PPF). The results showed that the proposed method achieved a recognition rate of 99.02%, processing time of 2 s, pose error rate of 1.31%, and spatial position error of 3.278 mm, significantly outperforming the comparative methods. These findings demonstrated the effectiveness of the method in addressing the issue of accurate 6D pose estimation of molecular sieve drying bags, with potential for future applications to other complex-shaped objects. [ABSTRACT FROM AUTHOR]

In unstructured tea garden environments, accurate recognition and pose estimation of tea bud leaves are critical for autonomous harvesting robots. Due to variations in imaging distance, tea bud leaves exhibit diverse scale and pose characteristics in camera views, which significantly complicates the recognition and pose estimation process. This study proposes a method using an RGB-D camera for precise recognition and pose estimation of tea bud leaves. The approach first constructs an for tea bud leaves, followed by a dynamic weight estimation strategy to achieve adaptive pose estimation. Quantitative experiments demonstrate that the instance segmentation model achieves an mAP@50 of 92.0% for box detection and 91.9% for mask detection, improving by 3.2% and 3.4%, respectively, compared to the YOLOv8s-seg instance segmentation model. The pose estimation results indicate a maximum angular error of 7.76°, a mean angular error of 3.41°, a median angular error of 3.69°, and a median absolute deviation of 1.42°. The corresponding distance errors are 8.60 mm, 2.83 mm, 2.57 mm, and 0.81 mm, further confirming the accuracy and robustness of the proposed method. These results indicate that the proposed method can be applied in unstructured tea garden environments for non-destructive and precise harvesting with autonomous tea bud-leave harvesting robots. [ABSTRACT FROM AUTHOR]

BACKGROUND: In radiography procedures, radiographers' suboptimal positioning and exposure parameter settings may necessitate image retakes, subjecting patients to unnecessary ionizing radiation exposure. Reducing retakes is crucial to minimize patient X-ray exposure and conserve medical resources. OBJECTIVE: We propose a Digital Radiography (DR) Pre-imaging All-round Assistant (PIAA) that leverages Artificial Intelligence (AI) technology to enhance traditional DR. METHODS: PIAA consists of an RGB-Depth (RGB-D) multi-camera array, an embedded computing platform, and multiple software components. It features an Adaptive RGB-D Image Acquisition (ARDIA) module that automatically selects the appropriate RGB camera based on the distance between the cameras and patients. It includes a 2.5D Selective Skeletal Keypoints Estimation (2.5D-SSKE) module that fuses depth information with 2D keypoints to estimate the pose of target body parts. Thirdly, it also uses a Domain expertise (DE) embedded Full-body Exposure Parameter Estimation (DFEPE) module that combines 2.5D-SSKE and DE to accurately estimate parameters for full-body DR views. RESULTS: Optimizes DR workflow, significantly enhancing operational efficiency. The average time required for positioning patients and preparing exposure parameters was reduced from 73 seconds to 8 seconds. CONCLUSIONS: PIAA shows significant promise for extension to full-body examinations. [ABSTRACT FROM AUTHOR]

Efficient and accurate 3D reconstruction of rescue shafts in mining accidents is a critical and challenging task, particularly in low-texture environments. This paper proposes a novel method for real-time 3D model reconstruction, deformation detection, and accessibility analysis of rescue shafts using an RGB-D camera. The approach captures depth and color data from the shaft’s low-texture walls and employs advanced feature extraction and matching algorithms to generate a high-precision 3D point cloud. A hybrid Iterative Closest Point-Perspective n Point (ICP-PNP) algorithm ensures precise camera pose estimation, and motion errors between adjacent frames are minimized to optimize the 3D point cloud. The reconstructed model is refined using Poisson surface reconstruction, achieving millimeter-level pose estimation accuracy and a global trajectory consistency error within 2%. Experimental results demonstrate the superiority of the Speeded Up Robust Features (SURF) algorithm in feature extraction and the effectiveness of the Random Sample Consensus (RANSAC) algorithm in filtering mismatched points. The method also provides deformation profiles and accessibility predictions, with diameter estimates ranging from 510 mm to 540 mm, enabling accurate assessments of shaft usability and deformation trends. This framework enhances the precision and efficiency of rescue operations, offering a robust tool for real-time decision-making in mining emergencies.

In mainstream simultaneous localization and mapping (SLAM) algorithms, feature points are commonly utilized to represent image features. However, the quantity and quality of these feature points are contingent upon the environmental texture, lighting conditions, and motion speed. Although existing algorithms enhance adaptability by extracting point-line features simultaneously, the presence of trivial short lines resulting from environmental noise and object occlusion can adversely affect system robustness. Therefore, in this study, we propose a line feature fusion strategy along with a model incorporating an adaptive length suppression parameter for line features. A new line feature residual model is defined, and the mathematical analytical form of line feature Jacobian matrix is derived in detail. Additionally, the point features are organized into a lattice structure and utilized to construct a global pointcloud map in a dedicated thread, aiming to enhance the semantic comprehension of environmental information. Finally, our algorithm is compared against state-of-the-art algorithms on the publicly available datasets TUM RGB-D and ICL-NUIM. Through quantitative trajectory error analysis and qualitative trajectory effect and mapping quality analysis, the final results indicate that the algorithm proposed in this paper achieves superior positioning accuracy and mapping quality, enabling robust 3D reconstruction of indoor scenes.

As power system equipment gradually ages, the automated disassembly of transformers has become a critical area of research to enhance both efficiency and safety. This paper presents a transformer disassembly system designed for power systems, leveraging multimodal perception and collaborative processing. By integrating 2D images and 3D point cloud data captured by RGB-D cameras, the system enables the precise recognition and efficient disassembly of transformer covers and internal components through multimodal data fusion, deep learning models, and control technologies. The system employs an enhanced YOLOv8 model for positioning and identifying screw-fastened covers while also utilizing the STDC network for segmentation and cutting path planning of welded covers. In addition, the system captures 3D point cloud data of the transformer's interior using multi-view RGB-D cameras and performs multimodal semantic segmentation and object detection via the ODIN model, facilitating the high-precision identification and cutting of complex components such as windings, studs, and silicon steel sheets. Experimental results show that the system achieves a recognition accuracy of 99% for both cover and internal component disassembly, with a disassembly success rate of 98%, demonstrating its high adaptability and safety in complex industrial environments. [ABSTRACT FROM AUTHOR]

Estimating human joint angles is a crucial task in motion analysis, gesture recognition, and motion intention prediction. This paper presents a novel model-based approach for generating reliable and accurate human joint angle estimation using a dual-branch network. The proposed network leverages combined features derived from encoded sEMG signals and RGB-D image data. To ensure the accuracy and reliability of the estimation algorithm, the proposed network employs a convolutional autoencoder to generate a high-level compression of sEMG features aimed at motion prediction. Considering the variability in the distribution of sEMG signals, the proposed network introduces a vision-based joint regression network to maintain the stability of combined features. Taking into account latency, occlusion, and shading issues with vision data acquisition, the feature fusion network utilizes high-frequency sEMG features as weights for specific features extracted from image data. The proposed method achieves effective human body joint angle estimation for motion analysis and motion intention prediction by mitigating the effects of non-stationary sEMG signals. [ABSTRACT FROM AUTHOR]

Multiple view object recognition is challenged by the impact of various view-angles on intra-class relationships. Visually impaired individuals can benefit from accurate navigation services with a navigation system that enables them to avoid obstacles to their destination. An indoor object detection framework called RSIGConv, based on an integrated Region proposal and Spatial Information Guided Convolution network, is proposed in this paper for visually impaired people. To obtain mutual complementarity (MC) features, the RGB and HHA feature maps are fused using the Information Translation Module (ITM). The hyper-parameters are optimized using the Bayesian optimization Algorithm (BOA) to reduce train error and the gap between train error and validation error. The proposed object detection framework is evaluated using the publicly available SUN RGB-D dataset and compared with previous prediction models. The simulation outputs demonstrate that the model overtakes existing approaches, achieving an accuracy of 97.77%. [ABSTRACT FROM AUTHOR]

Purpose: This work presents the implementation of an RGB-D camera as a surrogate signal for liver respiratory-induced motion estimation. This study aims to validate the feasibility of RGB-D cameras as a surrogate in a human subject experiment and to compare the performance of different correspondence models. Methods: The proposed approach uses an RGB-D camera to compute an abdominal surface reconstruction and estimate the liver respiratory-induced motion. Two sets of validation experiments were conducted, first, using a robotic liver phantom and, secondly, performing a clinical study with human subjects. In the clinical study, three correspondence models were created changing the conditions of the learning-based model. Results: The motion model for the robotic liver phantom displayed an error below 3 mm with a coefficient of determination above 90% for the different directions of motion. The clinical study presented errors of 4.5, 2.5, and 2.9 mm for the three different motion models with a coefficient of determination above 80% for all three cases. Conclusion: RGB-D cameras are a promising method to accurately estimate the liver respiratory-induced motion. The internal motion can be estimated in a non-contact, noninvasive and flexible approach. Additionally, three training conditions for the correspondence model are studied to potentially mitigate intra- and inter-fraction motion. [ABSTRACT FROM AUTHOR]

3D reconstruction of seedling can provide comprehensive and quantitative spatial structure information, offering an effective digital tool for breeding research. However, accurate and efficient reconstruction of seedling is still a challenging work due to limited performance of depth sensor for seedling with small-size stem and unavoidable error for multi-view point cloud registration. Therefore, in this paper, we propose an accurate multi-view 3D reconstruction method for seedling using 2D image contour to constrain 3D point cloud. The rotation axis is calibrated and optimised by minimising point-to-contour distance between 2D image contour and projected exterior points from 3D point cloud. Then, to remove outliers and noise, we introduce the seedling mask of 2D image to constrained and delete projected outlier points of 3D model from corresponding view. Furthermore, we propose a residual-guided method to recognise missing region for 3D model and complete 3D model of small-size stem. Finally, we can obtain an accurate 3D model of seedling. The reconstruction accuracy is evaluated by average distance between projected contour of 3D model and 2D image contour of all views (0.3185 mm). Then, the phenotypic parameters were calculated from 3D model and the results are close to manual measurements (Plant height: R2 = 0.98, RMSE = 2.3 mm, rRMSE = 1.52%; Petioles inclination angle: R2 = 0.99, RMSE = 0.73°, rRMSE = 1.41%; Leaf area: R2 = 0.66, RMSE = 1.05 cm2, rRMSE = 7.63%; Leaf inclination angle: R2 = 0.99, RMSE = 1.01°, rRMSE = 1.72%; Stem diameter: R2 = 0.95, RMSE = 0.12 mm, rRMSE = 5.43%). Breeders can improve the selection of more resilient varieties and cultivars to different growing conditions starting from the dynamic analysis of their phenotype. • Introducing a novel 3D reconstruction method for seedling with small-size stem. • Using multi-view 2D image contour to constrain 3D point cloud. • Denoising and completing point cloud for 3D reconstruction. • Achieving accurate 3D reconstruction about 0.3 mm. [ABSTRACT FROM AUTHOR]

This paper presents the theoretical foundation, practical implementation, and empirical evaluation of a glove for interaction with 3-D virtual environments. At the dawn of the “Spatial Computing Era”, where users continuously interact with 3-D Virtual and Augmented Reality environments, the need for a practical and intuitive interaction system that can efficiently engage 3-D elements is becoming pressing. Over the last few decades, there have been attempts to provide such an interaction mechanism using a glove. However, glove systems are currently not in widespread use due to their high cost and, we propose, due to their inability to sustain high levels of performance under certain situations. Performance deterioration has been observed due to the distortion of the local magnetic field caused by ordinary ferromagnetic objects present near the glove’s operating space. There are several areas where reliable hand-tracking gloves could provide a next generation of improved solutions, such as American Sign Language training and automatic translation to text and training and evaluation for activities that require high motor skills in the hands (e.g., playing some musical instruments, training of surgeons, etc.). While the use of a hand-tracking glove toward these goals seems intuitive, some of the currently available glove systems may not meet the accuracy and reliability levels required for those use cases. This paper describes our concept of an interaction glove instrumented with miniature magnetic, angular rate, and gravity (MARG) sensors and aided by a single camera. The camera used is an off-the-shelf red, green, and blue–depth (RGB-D) camera. We describe a proof-of-concept implementation of the system using our custom “GMVDK” orientation estimation algorithm. This paper also describes the glove’s empirical evaluation with human-subject performance tests. The results show that the prototype glove, using the GMVDK algorithm, is able to operate without performance losses, even in magnetically distorted environments.

In unstructured tea garden environments, accurate recognition and pose estimation of tea bud leaves are critical for autonomous harvesting robots. Due to variations in imaging distance, tea bud leaves exhibit diverse scale and pose characteristics in camera views, which significantly complicates the recognition and pose estimation process. This study proposes a method using an RGB-D camera for precise recognition and pose estimation of tea bud leaves. The approach first constructs an for tea bud leaves, followed by a dynamic weight estimation strategy to achieve adaptive pose estimation. Quantitative experiments demonstrate that the instance segmentation model achieves an mAP@50 of 92.0% for box detection and 91.9% for mask detection, improving by 3.2% and 3.4%, respectively, compared to the YOLOv8s-seg instance segmentation model. The pose estimation results indicate a maximum angular error of 7.76°, a mean angular error of 3.41°, a median angular error of 3.69°, and a median absolute deviation of 1.42°. The corresponding distance errors are 8.60 mm, 2.83 mm, 2.57 mm, and 0.81 mm, further confirming the accuracy and robustness of the proposed method. These results indicate that the proposed method can be applied in unstructured tea garden environments for non-destructive and precise harvesting with autonomous tea bud-leave harvesting robots.

Recognition of the boundaries of farmland plow areas has an important guiding role in the operation of intelligent agricultural equipment. To precisely recognize these boundaries, a detection method for unmanned tractor plow areas based on RGB-Depth (RGB-D) cameras was proposed, and the feasibility of the detection method was analyzed. This method applied advanced computer vision technology to the field of agricultural automation. Adopting and improving the YOLOv5-seg object segmentation algorithm, first, the Convolutional Block Attention Module (CBAM) was integrated into Concentrated-Comprehensive Convolution Block (C3) to form C3CBAM, thereby enhancing the ability of the network to extract features from plow areas. The GhostConv module was also utilized to reduce parameter and computational complexity. Second, using the depth image information provided by the RGB-D camera combined with the results recognized by the YOLOv5-seg model, the mask image was processed to extract contour boundaries, align the contours with the depth map, and obtain the boundary distance information of the plowed area. Last, based on farmland information, the calculated average boundary distance was corrected, further improving the accuracy of the distance measurements. The experiment results showed that the YOLOv5-seg object segmentation algorithm achieved a recognition accuracy of 99% for plowed areas and that the ranging accuracy improved with decreasing detection distance. The ranging error at 5.5 m was approximately 0.056 m, and the average detection time per frame is 29 ms, which can meet the real-time operational requirements. The results of this study can provide precise guarantees for the autonomous operation of unmanned plowing units. [ABSTRACT FROM AUTHOR]

Visual odometry endows robots with the ability of autonomous positioning and building environmental maps, and is widely used in various unmanned devices. Visual odometry involves a large amount of image processing and calculation, but most of its deployment platforms only have extremely limited computational resources, limiting its application scope. In response to the I/O bottleneck of existing low-power visual odometry, this paper proposes a high-speed low-power visual odometry, named ELPVO, based on RGB-D cameras for the STM32F7 embedded platform. ELPVO fully considers the hardware resources of the STM32F7 platform, improves the processor utilization efficiency through DMA transmission, and further enhances the processing speed without changing the algorithm accuracy. On the STM32F767 embedded platform equipped with a 216 MHz ARM Cortex®-M7 processor, with the TUM RGB-D dataset as the testing benchmark, ELPVO can achieve a processing speed of 26 frames per second for images with a resolution of 320 x 240, with an overall run speed improved by 84% and run power consumption maintained at 0.7 watts. [ABSTRACT FROM AUTHOR]

Postural instability is one of the most disabling symptoms of Parkinson's Disease, with important impacts on people safety and quality of life since it increases the risk of falls and injuries. Home monitoring of changes in postural stability, as a consequence of therapies and disease progression, is highly desirable for the safety of the patient and better disease management. In this context, we present a system for the automatic evaluation of postural stability that is suitable for self-managing by people with motor impairment directly at home. The system is based on an optical RGB-Depth device, which tracks the body movements both for system's interaction, thanks to a gesture-based human-machine interface, and the automated assessment of postural stability. A set of tasks, based on standard clinical scales, has been designed for the assessment. The user controls the delivery of the tasks through the system interface. A machine learning approach is adopted, and some kinematic parameters that characterize the user's performance during each task execution are estimated and used by supervised classifiers for the automatic assessment. Data collected during experimental clinical trials were used to train the classifiers. This approach supports the compliance of the classifier assessments with respect to the clinical ones. The system prototype and the preliminary results on its accuracy in the assessment of postural stability are presented and discussed. [ABSTRACT FROM AUTHOR]

In this study, a simultaneous localization and mapping (SLAM) system for a two-wheeled mobile robot was developed in an indoor environment using RGB images, depth images, and wheel odometry. The proposed SLAM system applies planar motion constraints performed by a robot in a two-dimensional space to robot poses parameterized in a three-dimensional space. The formulation of these constraints is based on a conventional study. However, in this study, the information matrices that weigh the planar motion constraints are given dynamically based on the wheel odometry model and the number of feature matches. These constraints are implemented into the SLAM graph optimization framework. In addition, to effectively apply these constraints, the system estimates two of the rotation components between the robot and camera coordinates during SLAM initialization using a point cloud to construct a floor recovered from a depth image. The system implements feature-based Visual SLAM software. The experimental results show that the proposed system improves the localization accuracy and robustness in dynamic environments and changes the camera-mounted angle. In addition, we show that planar motion constraints enable the SLAM system to generate a consistent voxel map, even in an environment of several tens of meters. [ABSTRACT FROM AUTHOR]

The dataset contains RGB, depth, segmentation images of the scenes and information about the camera poses that can be used to create a full 3D model of the scene and develop methods that reconstruct objects from a single RGB-D camera view. Data were collected in the custom simulator that loads random graspable objects and random tables from the ShapeNet dataset. The graspable object is placed above the table in a random position. Then, the scene is simulated using the PhysX engine to make sure that the scene is physically plausible. The simulator captures images of the scene from a random pose and then takes the second image from the camera pose that is on the opposite side of the scene. The second subset was created using Kinect Azure and a set of real objects located on the ArUco board that was used to estimate the camera pose.

Accurate estimation of dual walking trajectories remains a challenge in human gait tracking systems due to limitations in sensor precision and data integration methods. To address these issues, this paper presents a novel human gait tracking system that integrates a downward-looking waist-mounted red-green-blue-depth (RGB-D) camera with two inertial measurement units (IMUs) mounted on each foot. Our approach utilizes a fully convolutional network (FCN) for precise foot detection from RGB-D images. The positions of both feet are then computed using the detected foot and the camera’s rotation matrix relative to the floor plane. These position estimates are incorporated into a Kalman filter, with a quadratic optimization-based smoothing method applied to improve accuracy. Experimental results demonstrate a significant improvement in dual trajectory estimation, achieving a root mean square error (RMSE) of 3.3 cm in stride length estimation. This system enhances the accuracy and reliability of gait analysis, effectively addressing the limitations of existing methods.

Modern consumer-grade RGB-D cameras provide intensive depth estimation and high frame rates. They have been widely used in agriculture. However, depth anamorphose occurs when using the RGB-D camera. In order to address this issue, this paper proposes a novel approach to correct the distorted depth images that fit the relationship between the true distances and the depth values from depth images. This study considers the structured light camera ASUS Xtion PRO LIVE as example to develop a system for obtaining a series of depth images at different distances from the target plane. A comparison analysis is conducted between the images before and after correction to evaluate the performance of the distortion correction. The point cloud image of corrected maize leaves is well coincident with the original plant. This approach improves the accuracy of depth measurement and optimizes the subsequent use of the depth camera in crop reconstruction and phenotyping studies.

negatively impacting the quality of life.Despite the benefits of initial specific post-acute treatments at the hospitals,motor functions, and physicalmobility need to be constantly stimulated to avoid regression and subsequent hospitalizations for further rehabilitation treatments. Method: This preliminary study proposes using gamified tasks in a virtual environment to stimulate and maintain upper limb mobility through a single RGB-D camera-based vision system (using Microsoft Azure Kinect DK). This solution is suitable for easy deployment and use in home environments. A cohort of 10 post-stroke subjects attended a 2-week gaming protocol consisting of LateralWeightlifting (LWL) and FrontalWeightlifting (FWL) gamified tasks and gait as the instrumental evaluation task. Results and discussion: Despite its short duration, there were statistically significant results (p < 0.05) between the baseline (T0) and the end of the protocol (TF) for Berg Balance Scale and Time Up-and-Go (9.8 and -12.3%, respectively). LWL and FWL showed significant results for unilateral executions: rate in FWL had an overall improvement of 38.5% (p < 0.001) and 34.9% (p < 0.01) for the paretic and non-paretic arm, respectively; similarly, rate in LWL improved by 19.9% (p < 0.05) for the paretic arm and 29.9% (p < 0.01) for non-paretic arm. Instead, bilateral executions had significant results for rate and speed: considering FWL, there was an improvement in rate with p < 0.01 (31.7% for paretic arm and 37.4% for non-paretic arm), whereas speed improved by 31.2% (p < 0.05) and 41.7% (p < 0.001) for the paretic and non- paretic arm, respectively; likewise, LWL showed improvement in rate with p < 0.001 (29.0% for paretic arm and 27.8% for non-paretic arm) and in speed with 23.6% (p < 0.05) and 23.5% (p < 0.01) for the paretic and non-paretic arms, respectively. No significant results were recorded for gait task, although an overall good improvement was detected for arm swing asymmetry (-22.6%). Hence, this study suggests the potential benefits of continuous stimulation of upper limb function through gamified exercises and performance monitoring over medium-long periods in the home environment, thus facilitating the patient's general mobility in daily activities. [ABSTRACT FROM AUTHOR]

In this paper, an intelligent blind guide system based on 2D LiDAR and RGB-D camera sensing is proposed, and the system is mounted on a smart cane. The intelligent guide system relies on 2D LiDAR, an RGB-D camera, IMU, GPS, Jetson nano B01, STM32, and other hardware. The main advantage of the intelligent guide system proposed by us is that the distance between the smart cane and obstacles can be measured by 2D LiDAR based on the cartographer algorithm, thus achieving simultaneous localization and mapping (SLAM). At the same time, through the improved YOLOv5 algorithm, pedestrians, vehicles, pedestrian crosswalks, traffic lights, warning posts, stone piers, tactile paving, and other objects in front of the visually impaired can be quickly and effectively identified. Laser SLAM and improved YOLOv5 obstacle identification tests were carried out inside a teaching building on the campus of Hainan Normal University and on a pedestrian crossing on Longkun South Road in Haikou City, Hainan Province. The results show that the intelligent guide system developed by us can drive the omnidirectional wheels at the bottom of the smart cane and provide the smart cane with a self-leading blind guide function, like a "guide dog", which can effectively guide the visually impaired to avoid obstacles and reach their predetermined destination, and can quickly and effectively identify the obstacles on the way out. The mapping and positioning accuracy of the system's laser SLAM is 1 m ± 7 cm, and the laser SLAM speed of this system is 25~31 FPS, which can realize the short-distance obstacle avoidance and navigation function both in indoor and outdoor environments. The improved YOLOv5 helps to identify 86 types of objects. The recognition rates for pedestrian crosswalks and for vehicles are 84.6% and 71.8%, respectively; the overall recognition rate for 86 types of objects is 61.2%, and the obstacle recognition rate of the intelligent guide system is 25–26 FPS. [ABSTRACT FROM AUTHOR]

As robots have become more pervasive in our daily life, natural human-robot interaction (HRI) has had a positive impact on the development of robotics. Thus, there has been growing interest in the development of vision-based hand gesture recognition for HRI to bridge human-robot barriers. The aim is for interaction with robots to be as natural as that between individuals. Accordingly, incorporating hand gestures in HRI is a significant research area. Hand gestures can provide natural, intuitive, and creative methods for communicating with robots. This paper provides an analysis of hand gesture recognition using both monocular cameras and RGB-D cameras for this purpose. Specifically, the main process of visual gesture recognition includes data acquisition, hand gesture detection and segmentation, feature extraction and gesture classification, which are discussed in this paper. Experimental evaluations are also reviewed. Furthermore, algorithms of hand gesture recognition for human-robot interaction are examined in this study. In addition, the advances required for improvement in the present hand gesture recognition systems, which can be applied for effective and efficient human-robot interaction, are discussed. [ABSTRACT FROM AUTHOR]

RGB-D cameras provide depth and color information and are widely used in 3D reconstruction and computer vision. In the majority of existing RGB-D cameras, a considerable portion of depth values is often lost due to severe occlusion or limited camera coverage, thereby adversely impacting the precise localization and three-dimensional reconstruction of objects. In this paper, to address the issue of poor-quality in-depth images captured by RGB-D cameras, a depth image hole repair algorithm based on non-local means is proposed first, leveraging the structural similarities between grayscale and depth images. Second, while considering the cumbersome parameter tuning associated with the non-local means hole repair method for determining the size of structural blocks for depth image hole repair, an intelligent block factor is introduced, which automatically determines the optimal search and repair block sizes for various hole sizes, resulting in the development of an adaptive block-based non-local means algorithm for repairing depth image holes. Furthermore, the proposed algorithm's performance are evaluated using both the Middlebury stereo matching dataset and a self-constructed RGB-D dataset, with performance assessment being carried out by comparing the algorithm against other methods using five metrics: RMSE, SSIM, PSNR, DE, and ALME. Finally, experimental results unequivocally demonstrate the innovative resolution of the parameter tuning complexity inherent in-depth image hole repair, effectively filling the holes, suppressing noise within depth images, enhancing image quality, and achieving elevated precision and accuracy, as affirmed by the attained results. [ABSTRACT FROM AUTHOR]

Abstract As robots have become more pervasive in our daily life, natural human-robot interaction (HRI) has had a positive impact on the development of robotics. Thus, there has been growing interest in the development of vision-based hand gesture recognition for HRI to bridge human-robot barriers. The aim is for interaction with robots to be as natural as that between individuals. Accordingly, incorporating hand gestures in HRI is a significant research area. Hand gestures can provide natural, intuitive, and creative methods for communicating with robots. This paper provides an analysis of hand gesture recognition using both monocular cameras and RGB-D cameras for this purpose. Specifically, the main process of visual gesture recognition includes data acquisition, hand gesture detection and segmentation, feature extraction and gesture classification, which are discussed in this paper. Experimental evaluations are also reviewed. Furthermore, algorithms of hand gesture recognition for human-robot interaction are examined in this study. In addition, the advances required for improvement in the present hand gesture recognition systems, which can be applied for effective and efficient human-robot interaction, are discussed.

IntroductionPost-stroke hemiplegia commonly occurs in stroke survivors, negatively impacting the quality of life. Despite the benefits of initial specific post-acute treatments at the hospitals, motor functions, and physical mobility need to be constantly stimulated to avoid regression and subsequent hospitalizations for further rehabilitation treatments.MethodThis preliminary study proposes using gamified tasks in a virtual environment to stimulate and maintain upper limb mobility through a single RGB-D camera-based vision system (using Microsoft Azure Kinect DK). This solution is suitable for easy deployment and use in home environments. A cohort of 10 post-stroke subjects attended a 2-week gaming protocol consisting of Lateral Weightlifting (LWL) and Frontal Weightlifting (FWL) gamified tasks and gait as the instrumental evaluation task.Results and discussionDespite its short duration, there were statistically significant results (p < 0.05) between the baseline (T0) and the end of the protocol (TF) for Berg Balance Scale and Time Up-and-Go (9.8 and −12.3%, respectively). LWL and FWL showed significant results for unilateral executions: rate in FWL had an overall improvement of 38.5% (p < 0.001) and 34.9% (p < 0.01) for the paretic and non-paretic arm, respectively; similarly, rate in LWL improved by 19.9% (p < 0.05) for the paretic arm and 29.9% (p < 0.01) for non-paretic arm. Instead, bilateral executions had significant results for rate and speed: considering FWL, there was an improvement in rate with p < 0.01 (31.7% for paretic arm and 37.4% for non-paretic arm), whereas speed improved by 31.2% (p < 0.05) and 41.7% (p < 0.001) for the paretic and non-paretic arm, respectively; likewise, LWL showed improvement in rate with p < 0.001 (29.0% for paretic arm and 27.8% for non-paretic arm) and in speed with 23.6% (p < 0.05) and 23.5% (p < 0.01) for the paretic and non-paretic arms, respectively. No significant results were recorded for gait task, although an overall good improvement was detected for arm swing asymmetry (−22.6%). Hence, this study suggests the potential benefits of continuous stimulation of upper limb function through gamified exercises and performance monitoring over medium-long periods in the home environment, thus facilitating the patient's general mobility in daily activities.

This paper proposes a method for the recognition of the opened/closed states of automatic sliding glass doors to allow for automatic robot-controlled movement from outdoors to indoors and vice versa by a robot. The proposed method uses an RGB-D camera as a sensor for extraction of the automatic sliding glass doors region and image recognition to determine whether the door is opened or closed. The RGB-D camera measures the distance between the opened or moving door frames, thereby facilitating outdoor to indoor movement and vice versa. Several automatic sliding glass doors under different experimental conditions are experimentally investigated to demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Human environments are designed and managed by humans for humans. Thus, adding robots to interact with humans and perform specific tasks appropriately is an essential topic in robotics research. In recent decades, object recognition, human skeletal, and face recognition frameworks have been implemented to support the tasks of robots. However, recognition of activities and interactions between humans and surrounding objects is an ongoing and more challenging problem. Therefore, this study proposed a graph neural network (GNN) approach to directly recognize human activity at home using vision and speech teaching data. Focus was given to the problem of classifying three activities, namely, eating, working, and reading, where these activities were conducted in the same environment. From the experiments, observations, and analyses, this proved to be quite a challenging problem to solve using only traditional convolutional neural networks (CNN) and video datasets. In the proposed method, an activity classification was learned from a 3D detected object corresponding to the human position. Next, human utterances were used to label the activity from the collected human and object 3D positions. The experiment, involving data collection and learning, was demonstrated by using human-robot communication. It was shown that the proposed method had the shortest training time of 100.346 seconds with 6000 positions from the dataset and was able to recognize the three activities more accurately than the deep layer aggregation (DLA) and X3D networks with video datasets.

In order to solve the problem of three-dimensional (3D) feature measurement in grading of jujubes, a volume and surface area measurement method based on 3D point cloud registration and reconstruction was proposed. First, use the RGB-D camera to collect the multi angle point cloud data of jujubes and perform filtering processing on it. Then, the Random Sample Consensus (RANSAC) algorithm was used to fit the plane and cylinder. After fitting, divide and remove the plane point cloud, obtain the central axis parameters of the cylinder, rotate the jujube around the central axis of the cylinder at a fixed angle, and collect the images of jujube from the same perspective, The Fast Point Feature Histogram (FPFH) and Sample Consciousness Initial Alignment (SAC-IA) algorithms were used to complete the 1/4 point cloud registration, the point to surface Iterative Closest Point (ICP) algorithm is used to complete the entire 1/2 registration, and the reconstructed jujube point cloud model is intercepted by the pass through algorithm. Finally, the jujubes point cloud model was smoothed and was filled of holes. The 3D coordinate method and convex hull method were used to calculate the jujube volume surface area. The correlation coefficients of the volume were 74.4 % and 74.5 %, and the correlation coefficients of the surface area was 83.2 % and 77.7 %. The experimental results show that this method can effectively calculate the 3D characteristic parameters of jujube, and can provide a reference for phenotype measurement and classification of jujube.

The accuracy of RGB-D SLAM systems is sensitive to the image quality, and can be significantly compromised in adverse situations such as when input images are blurry, lacking in texture features, or overexposed. In this paper, based on Continuous Direct Sparse Visual Odometry (CVO), we present a novel Keyframe-based CVO (KF-CVO) with intrinsic keyframe selection mechanism that effectively reduces the tracking error. We then extend KF-CVO to a RGB-D SLAM system, CVO SLAM, equipped with place recognition via ORB features, and joint bundle adjustment & pose graph optimization. Comprehensive evaluations on publicly available benchmarks show that the proposed RGB-D SLAM system achieves a higher success rate than current state-of-the-art-methods. The proposed system is more robust to difficult benchmark sequences than current state-of-the-art methods, where adverse situations such as rapid camera motions, environments lacking in texture, and overexposed images when strong illumination exists.

AKFruitYield is a modular software that allows orchard data from RGB-D Azure Kinect cameras to be processed for fruit size and fruit yield estimation. Specifically, two modules have been developed: i) AK_SW_BENCHMARKER that makes it possible to apply different sizing algorithms and allometric yield prediction models to manually labelled color and depth tree images; and ii) AK_VIDEO_ANALYSER that analyses videos on which to automatically detect apples, estimate their size and predict yield at the plot or per hectare scale using the appropriate algorithms. Both modules have easy-to-use graphical interfaces and provide reports that can subsequently be used by other analysis tools.

With the continuous development of the economy and society, plastic pollution in rivers, lakes, oceans, and other bodies of water is increasingly severe, posing a serious challenge to underwater ecosystems. Effective cleaning up of underwater litter by robots relies on accurately identifying and locating the plastic waste. However, it often causes significant challenges such as noise interference, low contrast, and blurred textures in underwater optical images. A weighted fusion-based algorithm for enhancing the quality of underwater images is proposed, which combines weighted logarithmic transformations, adaptive gamma correction, improved multi-scale Retinex (MSR) algorithm, and the contrast limited adaptive histogram equalization (CLAHE) algorithm. The proposed algorithm improves brightness, contrast, and color recovery and enhances detail features resulting in better overall image quality. A network framework is proposed in this article based on the YOLOv5 model. MobileViT is used as the backbone of the network framework, detection layer is added to improve the detection capability for small targets, self-attention and mixed-attention modules are introduced to enhance the recognition capability of important features. The cross stage partial (CSP) structure is employed in the spatial pyramid pooling (SPP) section to enrich feature information, and the complete intersection over union (CIOU) loss is replaced with the focal efficient intersection over union (EIOU) loss to accelerate convergence while improving regression accuracy. Experimental results proved that the target recognition algorithm achieved a recognition accuracy of 0.913 and ensured a recognition speed of 45.56 fps/s. Subsequently, Using red, green, blue and depth (RGB-D) camera to construct a system for identifying and locating underwater plastic waste. Experiments were conducted underwater for recognition, localization, and error analysis. The experimental results demonstrate the effectiveness of the proposed method for identifying and locating underwater plastic waste, and it has good localization accuracy. [ABSTRACT FROM AUTHOR]

In Japanese agriculture, labor shortages are becoming increasingly severe due to the lack of farmers and aging. Therefore, extensive research has been conducted on the automation of cabbage harvesting. In automatic cabbage harvesting, cabbage detection is performed using deep learning. In the evening hours, if the backlight enters the camera, cabbage detection is not possible. Moreover, cabbages in the back row that are not to be harvested are detected and targeted for harvest. To solve these problems, we have proposed new recognition methods in this paper. We performed cabbage detection using the lower half of the cabbage and performed cabbage selection using an RGB-D camera. Moreover, sliding-mode control was incorporated to enable automatic harvesting in the soft soil. The experimental results demonstrate the effectiveness of these proposed methods. [ABSTRACT FROM AUTHOR]

For this study, a new point cloud alignment method is proposed for extracting corner points and aligning them at the geometric level. It can align point clouds that have low overlap and is more robust to outliers and noise. First, planes are extracted from the raw point cloud, and the corner points are defined as the intersection of three planes. Next, graphs are constructed for subsequent point cloud registration by treating corners as vertices and sharing planes as edges. The graph-matching algorithm is then applied to determine correspondence. Finally, point clouds are registered by aligning the corresponding corner points. The proposed method was investigated by utilizing pertinent metrics on datasets with differing overlap. The results demonstrate that the proposed method can align point clouds that have low overlap, yielding an RMSE of about 0.05 cm for datasets with 90% overlap and about 0.2 cm when there is only about 10% overlap. In this situation, the other methods failed to align point clouds. In terms of time consumption, the proposed method can process a point cloud comprising  10 4  points in 4 s when there is high overlap. When there is low overlap, it can also process a point cloud comprising  10 6  points in 10 s. The contributions of this study are the definition and extraction of corner points at the geometric level, followed by the use of these corner points to register point clouds. This approach can be directly used for low-precision applications and, in addition, for coarse registration in high-precision applications. [ABSTRACT FROM AUTHOR]

The dynamic changes in occupants' presence, movements and other behaviours could affect the actual operation and energy consumption of buildings. The design and operation of buildings can be adjusted to be more energy efficient by thinking over the actual distribution of occupants and application scenarios. Nevertheless, accurate, nonintrusive and applicable occupancy positioning systems, especially for complex and densely populated scenarios, have not yet been established. Herein, we propose a novel indoor positioning system based on a red green blue-depth (RGB-D) camera (CIOPS-RGBD). This system utilizes multiple RGB-D cameras to capture colour and depth images from different views. A data fusion algorithm is developed according to the result from a human pose estimation method. Then, the proposed CIOPS-RGBD system was setup and verified within a multifunction room for steady and dynamic accuracy estimation. The accuracy was verified within 10 cm in most cases. Finally, the system was tested under different application scenarios with more than 25 occupants in an 86 m2 space. The results demonstrate that the system can provide high-quality occupancy positioning and body orientation information for these scenarios in almost real-time, providing a solid basis to improve the actual operation and design of indoor environment creation systems. [ABSTRACT FROM AUTHOR]

Deformation measurement of hull structure is of great significance for enhancing the quality of shipbuilding. In this paper, a three-dimensional (3D) reconstruction method based on RGB-D images is presented for non-structural surface of a stiffened hull plate. The algorithm for object surface region nodes (OSRN) is proposed to extract the features of point clouds and generate higher-order fitting surface model. Firstly, random sample consensus (RANSAC) algorithm and least square method are applied to eliminate the noise of point clouds. Spatial position of targets in RGB image and singular value decomposition (SVD) are utilized to accomplish the registration of point clouds from multiple camera views. Then, the point clouds data are clustered and fused to fit the quadric surface. On this basis, the 3D surface model is reconstructed through higher-order panel element. Two comparative experiments, including the measurements via the RGB-D camera and 3D Laser Scanner, are carried out to verify the applicability and accuracy of the proposed method. [ABSTRACT FROM AUTHOR]

The use of robots for machining operations has become very popular in the last few decades. However, the challenge of the robotic-based machining process, such as surface finishing on curved surfaces, still persists. Prior studies (non-contact- and contact-based) have their own limitations, such as fixture error and surface friction. To cope with these challenges, this study proposes an advanced technique for path correction and normal trajectory generation while tracking a curved workpiece's surface. Initially, a key-point selection approach is used to estimate a reference workpiece's coordinates using a depth measuring tool. This approach overcomes the fixture errors and enables the robot to track the desired path, i.e., where the surface normal trajectory is needed. Subsequently, this study employs an attached RGB-D camera on the end-effector of the robot for determining the depth and angle between the robot and the contact surface, which nullifies surface friction issues. The point cloud information of the contact surface is employed by the pose correction algorithm to guarantee the robot's perpendicularity and constant contact with the surface. The efficiency of the proposed technique is analyzed by carrying out several experimental trials using a 6 DOF robot manipulator. The results reveal a better normal trajectory generation than previous state-of-the-art research, with an average angle and depth error of 1.8 degrees and 0.4 mm. [ABSTRACT FROM AUTHOR]