Search

Pill image recognition by machine vision can reduce the risk of taking the wrong medications, a severe healthcare problem. Automated dispensing machines or home applications both need reliable image processing techniques to compete with the problem of changing viewing conditions, large number of classes, and the similarity in pill appearance. The problem is attacked with a multi‐stream, two‐phase metric embedding neural model. To enhance the metric learning procedure, dynamic margin setting is introduced into the loss function. Moreover, it is shown that besides the visual features of drug samples, even free text of drug leaflets (processed with a natural language model) can be used to set the value of the margin in the triplet loss and thus increase the recognition accuracy of testing. Thus, besides using the conventional metric learning approach, the given discriminating features can be explicitly injected into the metric model using the NLP of the free text of pill leaflets or descriptors of images of selected pills. The performance on two datasets is analysed and a 1.6% (two‐sided) and 2.89% (one‐sided) increase in Top‐1 accuracy on the CURE dataset is reported compared to existing best results. The inference time on CPU and GPU makes the proposed model suitable for different kinds of applications in medical pill verification; moreover, the approach applies to other areas of object recognition where few‐shot problems arise. The proposed high‐level feature injection method (into a low‐level metric learning model) can also be exploited in other cases, where class features can be well described with textual or visual cues. [ABSTRACT FROM AUTHOR]

This study introduces an advanced infrared scene detection algorithm, enhancing the YOLOv8 model for aiding visually impaired individuals in navigation. The focus is on the neck network, integrating attention scale sequences to boost multi‐level perception, particularly for small object detection. This is achieved by adding upsampling and downsampling in the P2 module. Additionally, the CIoU loss function is refined with Inner‐SIoU, elevating bounding box detection precision. A distinctive feature of the approach is its monocular distance and velocity measurement integration, which operates independently of external devices, providing direct navigation support for visually impaired people. Further, the enhanced YOLOv8 is adapted for mobile use, employing pruning and lightweight methods, which substantially enhance its practicality. The experimental results on the FLIR and WOTR datasets demonstrate that, compared to the original YOLOv8n, the improved algorithm has achieved a 2.1% and 3.2% increase in mAP0.5$\text{mAP}_{0.5}$, respectively. Furthermore, the mAP0.5--0.95$\text{mAP}_{0.5\text{--}0.95}$ has seen a 2.2% and 3.8% improvement. Concurrently, the model size has been reduced by 55% and 60%, and the number of parameters has decreased by 60% and 67%. Compared to other assistive travel methods for visually impaired individuals, our work demonstrates superior practicality. [ABSTRACT FROM AUTHOR]

Abstract In response to the inadequate utilization of prior information in current mural inpainting processes, leading to issues such as semantically unreliable inpaintings and the presence of artifacts in the inpainting area, a Dunhuang mural inpainting method based on reference guidance and multi‐scale feature fusion is proposed. First, the simulated broken mural, the mask image, and the reference mural are input into the model to complete the multi‐level embedding of patches and align the multi‐scale fine‐grained features of damaged murals and reference murals. Following the patch embedding module, a hybrid residual module is added based on hybrid attention to fully extract mural features. In addition, by continuing the residual concatenation of outputs of the hierarchical embedding module improves the ability of the model to represent deeper features, and improves the robustness and generalisation of the model. Second, the encoded features are fed into the decoder to generate decoded features. Finally, the convolutional tail is employed to propagate them and complete the mural painting. Experimental validation on the Dunhuang mural dataset demonstrates that, compared to other algorithms, this model exhibits higher evaluation metrics in the inpainting of extensively damaged murals and demonstrates overall robustness. In terms of visual effects, the results of this model in the inpainting process exhibit finer textures, richer semantic information, more coherent edge structures, and a closer resemblance to authentic murals.

Abstract Research on low‐light image enhancement focuses on improving image quality in dim conditions. Recently, deep learning has driven significant advancements, with many studies using neural networks to enhance low‐light images. However, most focus on complex network designs to increase nonlinearity, often neglecting the implicit information from local image transformations. This paper introduces an improved U‐net‐based method for low‐light enhancement, retaining the original encoding network and adding branch links in the decoding network. The method uses attention feature fusion to handle image noise and gradients separately, adjusting brightness through a gradient‐adaptive transform. This approach optimizes performance using loss functions like peak signal‐to‐noise ratio and colour consistency. Unlike previous methods, the approach emphasizes extracting implicit information from image gradients, achieving enhancement that aligns with the original brightness distribution. The result is enhanced images with high detail similarity to the original, achieved through end‐to‐end inference in experiments.

Abstract Most multiple object tracking algorithms depend on the output of the detector. Aiming at the problem that the higher detection quality model is restricted by the computing power, and the robustness of the lightweight detection model is easily affected by motion blur, this paper proposes a lightweight moving object detector based on improved YOLOv8 combined with dynamic confidence compensation algorithm. The algorithm combines various technical means such as network structure optimization, lightweight design, self‐knowledge distillation, loss function improvement and dynamic confidence compensation. ByteTrack is used as a tracker to conduct experiments on PASCAL VOC07+12 data set and UA‐DETRAC test sequence. Compared with the baseline YOLOv8n+ByteTrack, the proposed algorithm improves the HOTA by 1.3% when the single frame tracking delay is reduced by 1.1%. Mostly tracked target is improved by 79.7%, mostly lost target is reduced by 10.9%, and the detection effect is better than the original detector and other popular object detectors. The YOLODCC model achieves a balance between lightweight and multi‐object motion blur.

Abstract Intracranial hematoma, a severe brain injury caused by trauma or cerebrovascular disease, can result in blood accumulation and compression of brain tissue. Untreated cases can cause headaches, impaired consciousness, and even brain tissue damage or death. Therefore, early and accurate diagnosis is crucial. Traditional segmentation methods require physicians with extensive clinical experience and expertise to manually mark out the hematoma region, but for hematoma cases with irregular shapes and uneven grey levels, this process is cumbersome, and the segmentation results are not good. Existing deep learning‐based methods are more likely to perform binary segmentation, considering all hematomas as a class and segmenting them, but this segmentation cannot capture more detailed information and lacks the analysis of different types of hematomas. To address these problems, an ICH segmentation network combining CNN and Transformer Encoder is proposed for accurate segmentation of different types of hematomas. The network incorporated edge information and long‐range context into the segmentation process. Experimental results using the CQ500 dataset demonstrate comparable performance to existing methods, with mIoU (0.8705), TPR (0.9273), mAP (0.9300), and DSC (0.9286) as the best metrics achieved by this paper's method.

Abstract Accurate estimation of wind power is essential for predicting and maintaining the power balance in the power system. This paper proposes a novel approach to enhance the accuracy of wind power estimation through a hybrid model integrating neural networks and error discrimination‐correction techniques. In order to improve the accuracy of estimation, a bidirectional gating recurrent unit is developed, forming an initial wind power estimation curve through training. Additionally, a sequential model‐based algorithmic configuration optimizes bidirectional gating recurrent unit's network hyperparameters. To tackle estimation errors, a multi‐layer perceptron combined with sequential model‐based algorithmic configuration is employed to create a classification model that automatically discerns the quality of estimates. Subsequently, an innovative correction model, based on grey relevancy degree and relevancy errors, is devised to rectify erroneous estimates. The final estimates result from a summation of the initial estimates and the values derived from error corrections. By analysing the real data from a wind farm in northwest China, a simulation test validates the proposed hybrid model. Experimental results demonstrate a substantial improvement in modelling accuracy when compared to the initial model.

Abstract Human action recognition based on graph convolutional networks (GCNs) is one of the hotspots in computer vision. However, previous methods generally rely on handcrafted graph, which limits the effectiveness of the model in characterising the connections between indirectly connected joints. The limitation leads to weakened connections when joints are separated by long distances. To address the above issue, the authors propose a skeleton simplification method which aims to reduce the number of joints and the distance between joints by merging adjacent joints into simplified joints. Group convolutional block is devised to extract the internal features of the simplified joints. Additionally, the authors enhance the method by introducing multi‐scale modelling, which maps inputs into sequences across various levels of simplification. Combining with spatial temporal graph convolution, a multi‐scale skeleton simplification GCN for skeleton‐based action recognition (M3S‐GCN) is proposed for fusing multi‐scale skeleton sequences and modelling the connections between joints. Finally, M3S‐GCN is evaluated on five benchmarks of NTU RGB+D 60 (C‐Sub, C‐View), NTU RGB+D 120 (X‐Sub, X‐Set) and NW‐UCLA datasets. Experimental results show that the authors’ M3S‐GCN achieves state‐of‐the‐art performance with the accuracies of 93.0%, 97.0% and 91.2% on C‐Sub, C‐View and X‐Set benchmarks, which validates the effectiveness of the method.

Abstract Head pose estimation is an especially challenging task due to the complexity nonlinear mapping from 2D feature space to 3D pose space. To address the above issue, this paper presents a novel and efficient head pose estimation framework based on particle swarm optimized contrastive learning and multimodal entangled graph convolution network. Firstly, a new network, the region and difference‐aware feature pyramid network (RD‐FPN), is proposed for 2D keypoints detection to alleviate the background interference and enhance the feature expressiveness. Then, particle swarm optimized contrastive learning is constructed to alternatively match 2D and 3D keypoints, which takes the multimodal keypoints matching accuracy as the optimization objective, while considering the similarity of cross‐modal positive and negative sample pairs from contrastive learning as a local contrastive constraint. Finally, multimodal entangled graph convolution network is designed to enhance the ability of establishing geometric relationships between keypoints and head pose angles based on second‐order bilinear attention, in which point‐edge attention is introduced to improve the representation of geometric features between multimodal keypoints. Compared with other methods, the average error of our method is reduced by 8.23%, indicating the accuracy, generalization, and efficiency of our method on the 300W‐LP, AFLW2000, BIWI datasets.

Abstract The ability to effectively classify human emotion states is critically important for human‐computer or human‐robot interactions. However, emotion classification with physiological signals is still a challenging problem due to the diversity of emotion expression and the characteristic differences in different modal signals. A novel learning‐based network architecture is presented that can exploit four‐modal physiological signals, electrocardiogram, electrodermal activity, electromyography, and blood volume pulse, and make a classification of emotion states. It features two kinds of attention modules, feature‐level, and semantic‐level, which drive the network to focus on the information‐rich features by mimicking the human attention mechanism. The feature‐level attention module encodes the rich information of each physiological signal. While the semantic‐level attention module captures the semantic dependencies among modals. The performance of the designed network is evaluated with the open‐source Wearable Stress and Affect Detection dataset. The developed emotion classification system achieves an accuracy of 83.88%. Results demonstrated that the proposed network could effectively process four‐modal physiological signals and achieve high accuracy of emotion classification.

Research on low‐light image enhancement focuses on improving image quality in dim conditions. Recently, deep learning has driven significant advancements, with many studies using neural networks to enhance low‐light images. However, most focus on complex network designs to increase nonlinearity, often neglecting the implicit information from local image transformations. This paper introduces an improved U‐net‐based method for low‐light enhancement, retaining the original encoding network and adding branch links in the decoding network. The method uses attention feature fusion to handle image noise and gradients separately, adjusting brightness through a gradient‐adaptive transform. This approach optimizes performance using loss functions like peak signal‐to‐noise ratio and colour consistency. Unlike previous methods, the approach emphasizes extracting implicit information from image gradients, achieving enhancement that aligns with the original brightness distribution. The result is enhanced images with high detail similarity to the original, achieved through end‐to‐end inference in experiments. [ABSTRACT FROM AUTHOR]

In response to the inadequate utilization of prior information in current mural inpainting processes, leading to issues such as semantically unreliable inpaintings and the presence of artifacts in the inpainting area, a Dunhuang mural inpainting method based on reference guidance and multi‐scale feature fusion is proposed. First, the simulated broken mural, the mask image, and the reference mural are input into the model to complete the multi‐level embedding of patches and align the multi‐scale fine‐grained features of damaged murals and reference murals. Following the patch embedding module, a hybrid residual module is added based on hybrid attention to fully extract mural features. In addition, by continuing the residual concatenation of outputs of the hierarchical embedding module improves the ability of the model to represent deeper features, and improves the robustness and generalisation of the model. Second, the encoded features are fed into the decoder to generate decoded features. Finally, the convolutional tail is employed to propagate them and complete the mural painting. Experimental validation on the Dunhuang mural dataset demonstrates that, compared to other algorithms, this model exhibits higher evaluation metrics in the inpainting of extensively damaged murals and demonstrates overall robustness. In terms of visual effects, the results of this model in the inpainting process exhibit finer textures, richer semantic information, more coherent edge structures, and a closer resemblance to authentic murals. [ABSTRACT FROM AUTHOR]

Accurate estimation of wind power is essential for predicting and maintaining the power balance in the power system. This paper proposes a novel approach to enhance the accuracy of wind power estimation through a hybrid model integrating neural networks and error discrimination‐correction techniques. In order to improve the accuracy of estimation, a bidirectional gating recurrent unit is developed, forming an initial wind power estimation curve through training. Additionally, a sequential model‐based algorithmic configuration optimizes bidirectional gating recurrent unit's network hyperparameters. To tackle estimation errors, a multi‐layer perceptron combined with sequential model‐based algorithmic configuration is employed to create a classification model that automatically discerns the quality of estimates. Subsequently, an innovative correction model, based on grey relevancy degree and relevancy errors, is devised to rectify erroneous estimates. The final estimates result from a summation of the initial estimates and the values derived from error corrections. By analysing the real data from a wind farm in northwest China, a simulation test validates the proposed hybrid model. Experimental results demonstrate a substantial improvement in modelling accuracy when compared to the initial model. [ABSTRACT FROM AUTHOR]

Multivariate time series forecasting is widely used in various fields in real life. Many time series prediction models have been proposed. The current forecasting model lacks the mining of correlation between sequences based on different periods and correlation of periodical features between different periods when dealing with data. In this paper, we propose a multivariate data prediction model FAFTransformer based on the reorganization of multi‐periodic features, which first extracts the multi‐periodic information of the time series using the method of frequency domain analysis. The temporal dependencies within sequences are then captured using convolution based on different periods, and the correlations between sequences are learned by combining the multivariate attention mechanism to obtain the intra‐sequence and inter‐sequence correlations under the same period. Finally, period fusion is proposed to capture the correlation of period characteristics between different periods. The experimental results show that the model achieves the best results on multiple datasets compared to the latest seven predictive models for time‐series data. [ABSTRACT FROM AUTHOR]

Most multiple object tracking algorithms depend on the output of the detector. Aiming at the problem that the higher detection quality model is restricted by the computing power, and the robustness of the lightweight detection model is easily affected by motion blur, this paper proposes a lightweight moving object detector based on improved YOLOv8 combined with dynamic confidence compensation algorithm. The algorithm combines various technical means such as network structure optimization, lightweight design, self‐knowledge distillation, loss function improvement and dynamic confidence compensation. ByteTrack is used as a tracker to conduct experiments on PASCAL VOC07+12 data set and UA‐DETRAC test sequence. Compared with the baseline YOLOv8n+ByteTrack, the proposed algorithm improves the HOTA by 1.3% when the single frame tracking delay is reduced by 1.1%. Mostly tracked target is improved by 79.7%, mostly lost target is reduced by 10.9%, and the detection effect is better than the original detector and other popular object detectors. The YOLODCC model achieves a balance between lightweight and multi‐object motion blur. [ABSTRACT FROM AUTHOR]

Intracranial hematoma, a severe brain injury caused by trauma or cerebrovascular disease, can result in blood accumulation and compression of brain tissue. Untreated cases can cause headaches, impaired consciousness, and even brain tissue damage or death. Therefore, early and accurate diagnosis is crucial. Traditional segmentation methods require physicians with extensive clinical experience and expertise to manually mark out the hematoma region, but for hematoma cases with irregular shapes and uneven grey levels, this process is cumbersome, and the segmentation results are not good. Existing deep learning‐based methods are more likely to perform binary segmentation, considering all hematomas as a class and segmenting them, but this segmentation cannot capture more detailed information and lacks the analysis of different types of hematomas. To address these problems, an ICH segmentation network combining CNN and Transformer Encoder is proposed for accurate segmentation of different types of hematomas. The network incorporated edge information and long‐range context into the segmentation process. Experimental results using the CQ500 dataset demonstrate comparable performance to existing methods, with mIoU (0.8705), TPR (0.9273), mAP (0.9300), and DSC (0.9286) as the best metrics achieved by this paper's method. [ABSTRACT FROM AUTHOR]

Human action recognition based on graph convolutional networks (GCNs) is one of the hotspots in computer vision. However, previous methods generally rely on handcrafted graph, which limits the effectiveness of the model in characterising the connections between indirectly connected joints. The limitation leads to weakened connections when joints are separated by long distances. To address the above issue, the authors propose a skeleton simplification method which aims to reduce the number of joints and the distance between joints by merging adjacent joints into simplified joints. Group convolutional block is devised to extract the internal features of the simplified joints. Additionally, the authors enhance the method by introducing multi‐scale modelling, which maps inputs into sequences across various levels of simplification. Combining with spatial temporal graph convolution, a multi‐scale skeleton simplification GCN for skeleton‐based action recognition (M3S‐GCN) is proposed for fusing multi‐scale skeleton sequences and modelling the connections between joints. Finally, M3S‐GCN is evaluated on five benchmarks of NTU RGB+D 60 (C‐Sub, C‐View), NTU RGB+D 120 (X‐Sub, X‐Set) and NW‐UCLA datasets. Experimental results show that the authors' M3S‐GCN achieves state‐of‐the‐art performance with the accuracies of 93.0%, 97.0% and 91.2% on C‐Sub, C‐View and X‐Set benchmarks, which validates the effectiveness of the method. [ABSTRACT FROM AUTHOR]

Head pose estimation is an especially challenging task due to the complexity nonlinear mapping from 2D feature space to 3D pose space. To address the above issue, this paper presents a novel and efficient head pose estimation framework based on particle swarm optimized contrastive learning and multimodal entangled graph convolution network. Firstly, a new network, the region and difference‐aware feature pyramid network (RD‐FPN), is proposed for 2D keypoints detection to alleviate the background interference and enhance the feature expressiveness. Then, particle swarm optimized contrastive learning is constructed to alternatively match 2D and 3D keypoints, which takes the multimodal keypoints matching accuracy as the optimization objective, while considering the similarity of cross‐modal positive and negative sample pairs from contrastive learning as a local contrastive constraint. Finally, multimodal entangled graph convolution network is designed to enhance the ability of establishing geometric relationships between keypoints and head pose angles based on second‐order bilinear attention, in which point‐edge attention is introduced to improve the representation of geometric features between multimodal keypoints. Compared with other methods, the average error of our method is reduced by 8.23%, indicating the accuracy, generalization, and efficiency of our method on the 300W‐LP, AFLW2000, BIWI datasets. [ABSTRACT FROM AUTHOR]

The floating‐gate transistor is commonly employed as a non‐volatile memory device, leveraging a floating node at its gate to store electrical charge over extended periods. This stored charge effectively alters the threshold voltage of the transistor. Utilizing standard CMOS technologies, floating‐gate transistors can be designed and fabricated using conventional CMOS processes. This study focuses on characterizing the performance of a PMOS‐based floating‐gate transistor, specifically fabricated using the open‐source Skywater 130 nm process. The modulation of charge on the floating node is explored through both hot‐electron injection and Fowler–Nordheim tunnelling, providing insight into the resolution of these programming mechanisms. Additionally, the study includes a preliminary analysis of the retention time of the programmed charge in these devices. This work contributes to the open‐source electronics community by detailing the design and programming techniques of floating‐gate transistors developed with an open‐source process design kit, and makes the corresponding FG cell designs available for public use. [ABSTRACT FROM AUTHOR]

The ability to effectively classify human emotion states is critically important for human‐computer or human‐robot interactions. However, emotion classification with physiological signals is still a challenging problem due to the diversity of emotion expression and the characteristic differences in different modal signals. A novel learning‐based network architecture is presented that can exploit four‐modal physiological signals, electrocardiogram, electrodermal activity, electromyography, and blood volume pulse, and make a classification of emotion states. It features two kinds of attention modules, feature‐level, and semantic‐level, which drive the network to focus on the information‐rich features by mimicking the human attention mechanism. The feature‐level attention module encodes the rich information of each physiological signal. While the semantic‐level attention module captures the semantic dependencies among modals. The performance of the designed network is evaluated with the open‐source Wearable Stress and Affect Detection dataset. The developed emotion classification system achieves an accuracy of 83.88%. Results demonstrated that the proposed network could effectively process four‐modal physiological signals and achieve high accuracy of emotion classification. [ABSTRACT FROM AUTHOR]

Abstract This research investigates the Circle of Willis, a critical vascular structure vital for cerebral blood supply. A modified novel dual‐pathway multi‐scale hierarchical upsampling network (HUNet) is presented, tailored explicitly for accurate segmentation of Circle of Willis anatomical components from medical imaging data. Evaluating both the multi‐label magnetic resonance angiography region of interest and the multi‐label magnetic resonance angiography whole brain‐case datasets, HUNet consistently outperforms the convolutional U‐net model, demonstrating superior capabilities and achieving higher accuracy across various classes. Additionally, the HUNet model achieves an exceptional dice similarity coefficient of 98.61 and 97.95, along with intersection over union scores of 73.32 and 85.76 in both the multi‐label magnetic resonance angiography region of interest and the multi‐label magnetic resonance angiography whole brain‐case datasets, respectively. These metrics highlight HUNet's exceptional performance in achieving precise and accurate segmentation of anatomical structures within the Circle of Willis, underscoring its robustness in medical image segmentation tasks. Visual representations substantiate HUNet's efficacy in delineating Circle of Willis structures, offering comprehensive insights into its superior performance.

Abstract In response to the challenges of Multi‐Object Tracking (MOT) in sports scenes, such as severe occlusions, similar appearances, drastic pose changes, and complex motion patterns, a deep‐learning framework CTGMOT (CNN‐Transformer‐GNN‐based MOT) specifically for multiple athlete tracking in sports videos that performs joint modelling of detection, appearance and motion features is proposed. Firstly, a detection network that combines Convolutional Neural Networks (CNN) and Transformers is constructed to extract both local and global features from images. The fusion of appearance and motion features is achieved through a design of parallel dual‐branch decoders. Secondly, graph models are built using Graph Neural Networks (GNN) to accurately capture the spatio‐temporal correlations between object and trajectory features from inter‐frame and intra‐frame associations. Experimental results on the public sports tracking dataset SportsMOT show that the proposed framework outperforms other state‐of‐the‐art methods for MOT in complex sport scenes. In addition, the proposed framework shows excellent generality on benchmark datasets MOT17 and MOT20.

Abstract The multifaceted nature of sensor data has long been a hurdle for those seeking to harness its full potential in the field of 3D object detection. Although the utilisation of point clouds as input has yielded exceptional results, the challenge of effectively combining the complementary properties of multi‐sensor data looms large. This work presents a new approach to multi‐model 3D object detection, called adaptive voxel‐image feature fusion (AVIFF). Adaptive voxel‐image feature fusion is an end‐to‐end single‐shot framework that can dynamically and adaptively fuse point cloud and image features, resulting in a more comprehensive and integrated analysis of the camera sensor and the LiDar sensor data. With the aid of the adaptive feature fusion module, spatialised image features can be adroitly fused with voxel‐based point cloud features, while the Dense Fusion module ensures the preservation of the distinctive characteristics of 3D point cloud data through the use of a heterogeneous architecture. Notably, the authors’ framework features a novel generalised intersection over union loss function that enhances the perceptibility of object localsation and rotation in 3D space. Comprehensive experimentation has validated the efficacy of the authors’ proposed modules, firmly establishing AVIFF as a novel framework in the field of 3D object detection.

This research investigates the Circle of Willis, a critical vascular structure vital for cerebral blood supply. A modified novel dual‐pathway multi‐scale hierarchical upsampling network (HUNet) is presented, tailored explicitly for accurate segmentation of Circle of Willis anatomical components from medical imaging data. Evaluating both the multi‐label magnetic resonance angiography region of interest and the multi‐label magnetic resonance angiography whole brain‐case datasets, HUNet consistently outperforms the convolutional U‐net model, demonstrating superior capabilities and achieving higher accuracy across various classes. Additionally, the HUNet model achieves an exceptional dice similarity coefficient of 98.61 and 97.95, along with intersection over union scores of 73.32 and 85.76 in both the multi‐label magnetic resonance angiography region of interest and the multi‐label magnetic resonance angiography whole brain‐case datasets, respectively. These metrics highlight HUNet's exceptional performance in achieving precise and accurate segmentation of anatomical structures within the Circle of Willis, underscoring its robustness in medical image segmentation tasks. Visual representations substantiate HUNet's efficacy in delineating Circle of Willis structures, offering comprehensive insights into its superior performance. [ABSTRACT FROM AUTHOR]

The multifaceted nature of sensor data has long been a hurdle for those seeking to harness its full potential in the field of 3D object detection. Although the utilisation of point clouds as input has yielded exceptional results, the challenge of effectively combining the complementary properties of multi‐sensor data looms large. This work presents a new approach to multi‐model 3D object detection, called adaptive voxel‐image feature fusion (AVIFF). Adaptive voxel‐image feature fusion is an end‐to‐end single‐shot framework that can dynamically and adaptively fuse point cloud and image features, resulting in a more comprehensive and integrated analysis of the camera sensor and the LiDar sensor data. With the aid of the adaptive feature fusion module, spatialised image features can be adroitly fused with voxel‐based point cloud features, while the Dense Fusion module ensures the preservation of the distinctive characteristics of 3D point cloud data through the use of a heterogeneous architecture. Notably, the authors' framework features a novel generalised intersection over union loss function that enhances the perceptibility of object localsation and rotation in 3D space. Comprehensive experimentation has validated the efficacy of the authors' proposed modules, firmly establishing AVIFF as a novel framework in the field of 3D object detection. [ABSTRACT FROM AUTHOR]

In response to the challenges of Multi‐Object Tracking (MOT) in sports scenes, such as severe occlusions, similar appearances, drastic pose changes, and complex motion patterns, a deep‐learning framework CTGMOT (CNN‐Transformer‐GNN‐based MOT) specifically for multiple athlete tracking in sports videos that performs joint modelling of detection, appearance and motion features is proposed. Firstly, a detection network that combines Convolutional Neural Networks (CNN) and Transformers is constructed to extract both local and global features from images. The fusion of appearance and motion features is achieved through a design of parallel dual‐branch decoders. Secondly, graph models are built using Graph Neural Networks (GNN) to accurately capture the spatio‐temporal correlations between object and trajectory features from inter‐frame and intra‐frame associations. Experimental results on the public sports tracking dataset SportsMOT show that the proposed framework outperforms other state‐of‐the‐art methods for MOT in complex sport scenes. In addition, the proposed framework shows excellent generality on benchmark datasets MOT17 and MOT20. [ABSTRACT FROM AUTHOR]

Single image super‐resolution (SISR) is a promising research direction in computer vision and image processing for improving the visual perception of low‐quality images. In recent years, deep learning algorithms have driven tremendous development in SR, and SR methods based on various network architectures have significantly improved the quality of reconstructed images. Although there has been a large amount of reviews focusing on SISR, few studies have focused specifically on network architectures for SISR. This paper aims to provide a systematic overview of the design ideas of SISR using multiple architectures, including Convolutional Neural Networks (CNN), Generative Adversarial Networks (GAN), Transformer, and Diffusion model. In addition, an experimental analysis and comparison of state‐of‐the‐art SR algorithms have been performed on publicly available quantitative and qualitative datasets. Finally, some future directions are discussed that may help other community researchers. [ABSTRACT FROM AUTHOR]

Perception systems in autonomous vehicles need to accurately detect and classify objects within their surrounding environments. Numerous types of sensors are deployed on these vehicles, and the combination of such multimodal data streams can significantly boost performance. The authors introduce a novel sensor fusion framework using deep convolutional neural networks. The framework employs both camera and LiDAR sensors in a multimodal, multiview configuration. The authors leverage both data types by introducing two new innovative fusion mechanisms: element‐wise multiplication and multimodal factorised bilinear pooling. The methods improve the bird's eye view moderate average precision score by +4.97% and +8.35% on the KITTI dataset when compared to traditional fusion operators like element‐wise addition and feature map concatenation. An in‐depth analysis of key design choices impacting performance, such as data augmentation, multi‐task learning, and convolutional architecture design is offered. The study aims to pave the way for the development of more robust multimodal machine vision systems. The authors conclude the paper with qualitative results, discussing both successful and problematic cases, along with potential ways to mitigate the latter. [ABSTRACT FROM AUTHOR]

Abstract An adaptive neuro‐fuzzy inference system is presented which is optimized by a genetic algorithm to classify normal and abnormal brain tumours. The classifier is fast and simple, named genetic algorithm‐adaptive neuro‐fuzzy inference system, and the determined learning rules minimize its error and improve its accuracy. The presented system follows five steps including preprocessing, morphological operation, feature extraction, feature selection, and classification. Morphological operators segment the abnormal regions and calculate the tumour area. The statistical features and the grey‐level co‐occurrence matrix are employed for feature extraction. Magnetic resonance images are considered and 12 statistical features are extracted, then the genetic algorithm‐based selection technique helps to select features and reduce the extracted features and improves the accuracy and decision time. So, the high dimensionality and the computational complexity of the adaptive neuro‐fuzzy inference system are reduced, and the classifier decides more efficiently. The input data are the figshare brain tumour dataset with 670 abnormal and 670 normal magnetic resonance images, and the classifier requires 10.788 s for classification. The efficient performance of the genetic algorithm‐adaptive neuro‐fuzzy inference system is confirmed by the accuracy of 99.85%, sensitivity of 99.7%, specificity of 100%, precision of 100%, and mean square error of 0.0027.

Abstract In response to the suboptimal efficiency observed in the network configuration and administration of 5G photovoltaic base stations (PVBSs), as well as the inherent limitations in accurately forecasting photovoltaic power during inclement weather conditions, this research article introduces a concise and effective method for short‐term power prediction of PVBSs when subjected to non‐sunny weather conditions, leveraging the paradigm of software defined networking. This method ingeniously amalgamates the improved Northern Goshawk optimized back propagation neural network (INGO‐BP) and recurrent generative adversarial networks (RGAN). The investigation commences by delineating a comprehensive PVBS system framework underpinned by SDN principles. Subsequently, it employs the Pearson correlation coefficient technique and principal component analysis for the judicious selection of pertinent features from the PVBS photovoltaic dataset, thereby ameliorating the issue of multicollinearity within the predictive model. To culminate the research endeavour, the INGO algorithm is enlisted to optimize the weightings and bias parameters of the BP neural network, while RGAN is harnessed for the acquisition of distinctive distribution patterns in photovoltaic data across diverse meteorological conditions. This augmentation enhances the model's capacity to discern the intricate long‐term mappings between photovoltaic power and meteorological data. The empirical findings substantiate the elevated accuracy of the INGO‐BP prediction model across sunny, cloudy, and rainy weather conditions, with a discernible enhancement in predictive accuracy for PVBS performance under non‐sunny weather conditions facilitated by data augmentation via RGAN.

In response to the suboptimal efficiency observed in the network configuration and administration of 5G photovoltaic base stations (PVBSs), as well as the inherent limitations in accurately forecasting photovoltaic power during inclement weather conditions, this research article introduces a concise and effective method for short‐term power prediction of PVBSs when subjected to non‐sunny weather conditions, leveraging the paradigm of software defined networking. This method ingeniously amalgamates the improved Northern Goshawk optimized back propagation neural network (INGO‐BP) and recurrent generative adversarial networks (RGAN). The investigation commences by delineating a comprehensive PVBS system framework underpinned by SDN principles. Subsequently, it employs the Pearson correlation coefficient technique and principal component analysis for the judicious selection of pertinent features from the PVBS photovoltaic dataset, thereby ameliorating the issue of multicollinearity within the predictive model. To culminate the research endeavour, the INGO algorithm is enlisted to optimize the weightings and bias parameters of the BP neural network, while RGAN is harnessed for the acquisition of distinctive distribution patterns in photovoltaic data across diverse meteorological conditions. This augmentation enhances the model's capacity to discern the intricate long‐term mappings between photovoltaic power and meteorological data. The empirical findings substantiate the elevated accuracy of the INGO‐BP prediction model across sunny, cloudy, and rainy weather conditions, with a discernible enhancement in predictive accuracy for PVBS performance under non‐sunny weather conditions facilitated by data augmentation via RGAN. [ABSTRACT FROM AUTHOR]

Time‐varying linear systems (TVLS) is a vital mathematical problem in many engineering fields. In this letter, a Milne‐Hamming discretized zeroing neural networks (MHDZNN) is presented to solve the TVLS, where high accurate convergence is got by combing Milne‐Hamming (MH) method. Theoretical analyses of its convergence and precision are discussed to illustrate the validity of the Milne‐Hamming DZNN (MHDZNN). Furthermore, numerical experiments compared with the two existing DZNN models are simulated to verify the superiority of the proposed MHDZNN model. [ABSTRACT FROM AUTHOR]

An adaptive neuro‐fuzzy inference system is presented which is optimized by a genetic algorithm to classify normal and abnormal brain tumours. The classifier is fast and simple, named genetic algorithm‐adaptive neuro‐fuzzy inference system, and the determined learning rules minimize its error and improve its accuracy. The presented system follows five steps including preprocessing, morphological operation, feature extraction, feature selection, and classification. Morphological operators segment the abnormal regions and calculate the tumour area. The statistical features and the grey‐level co‐occurrence matrix are employed for feature extraction. Magnetic resonance images are considered and 12 statistical features are extracted, then the genetic algorithm‐based selection technique helps to select features and reduce the extracted features and improves the accuracy and decision time. So, the high dimensionality and the computational complexity of the adaptive neuro‐fuzzy inference system are reduced, and the classifier decides more efficiently. The input data are the figshare brain tumour dataset with 670 abnormal and 670 normal magnetic resonance images, and the classifier requires 10.788 s for classification. The efficient performance of the genetic algorithm‐adaptive neuro‐fuzzy inference system is confirmed by the accuracy of 99.85%, sensitivity of 99.7%, specificity of 100%, precision of 100%, and mean square error of 0.0027. [ABSTRACT FROM AUTHOR]

Abstract Time‐varying electricity tariffs provide consumers the flexibility to adjust their consumption patterns in response to price variations to reduce the cost of electricity while at the same time contributing to grid operation. As more homes and buildings utilize time‐varying tariffs, utilities and regulators must seek ways to model demand‐side flexibilities to predict future demands and design new incentives. This paper proposes a novel end‐to‐end deep learning framework that simultaneously identifies demand baselines and the price‐response model from the net demand measurements and price signals. A gradient‐descent approach is then proposed that backpropagates the net demand forecast errors to update the weights of the price‐response model and the weights of the baseline demand forecast jointly. The effectiveness of the approach is demonstrated through computation experiments with synthetic demand response traces and a large‐scale real‐world DR dataset. The results show that the approach accurately identifies the DR model, even without prior knowledge about the baseline demand.

Abstract Different from object detection in natural images, aerial‐view object detection faces special challenges with large changes in object orientation and wide multi‐scale distribution. Many methods based on Oriented Bounding Boxes (OBB) can reach accurate results, yet may face the problem of parameter mutation and axis prediction deviation. To address these problems, a two‐stage detection framework is proposed for arbitrary‐oriented traffic participant detection and axis prediction, named Axis Prediction Network. First, a Deformable Convolution Fusion Module (DCF‐Module) is proposed to enhance the ability of FPN to extract multi‐scale semantic features, for dealing with the multi‐scale change of objects. Then, the axis heat‐map prediction head network is proposed to fit the long axis of oriented objects labeled with Gaussian model, which uses pixel‐by‐pixel prediction heatmap to calculate the long axis of the object, avoiding the angle mutation of an OBB. Last, the long and short side prediction head network is proposed to predict the shape of an oriented object, avoiding the mutation of the width and height of an OBB. The experiments are conducted on the new built Crossroad dataset and the public DOTA dataset. Experiment results show that the proposed method achieve good performance in arbitrary‐oriented traffic participant detection and axis prediction in aerial view.

Time‐varying electricity tariffs provide consumers the flexibility to adjust their consumption patterns in response to price variations to reduce the cost of electricity while at the same time contributing to grid operation. As more homes and buildings utilize time‐varying tariffs, utilities and regulators must seek ways to model demand‐side flexibilities to predict future demands and design new incentives. This paper proposes a novel end‐to‐end deep learning framework that simultaneously identifies demand baselines and the price‐response model from the net demand measurements and price signals. A gradient‐descent approach is then proposed that backpropagates the net demand forecast errors to update the weights of the price‐response model and the weights of the baseline demand forecast jointly. The effectiveness of the approach is demonstrated through computation experiments with synthetic demand response traces and a large‐scale real‐world DR dataset. The results show that the approach accurately identifies the DR model, even without prior knowledge about the baseline demand. [ABSTRACT FROM AUTHOR]

This letter presents a novel technique to optimize latency and accuracy for the inference based on binary neural network (BNN). The effects of the spatial locality in feature maps on latency and accuracy are analyzed in the BNN inference with the previous operation‐skipping method. A regularization‐based technique is proposed to adjust the locality with the aim of further reducing latency and improving accuracy of the previous method. In the CIFAR10 classification task, 11.62% latency reduction or 0.77% accuracy increase can be achieved when optimizing for each individually. When optimizing for both simultaneously, 5.58% latency reduction and 0.59% accuracy increase can be achieved. [ABSTRACT FROM AUTHOR]

Light‐field (LF) images offer the potential to improve feature capture in live scenes from multiple perspectives, and also generate additional normal vectors for performing super‐resolution (SR) image processing. With the benefit of machine learning, established AI‐based deep CNN models for LF image SR often individualize the models for various resolutions. However, the rigidity of these approaches for actual LF applications stems from the considerable diversity in angular resolution among LF instruments. Therefore, an advanced neural network proposal is required to utilize a CNN‐based model for super‐resolving LF images with different resolutions obtained from provided features. In this work, a preprocessing to calculate the depth channel from given LF information is first presented, and then a multiple‐decouple and fusion module is introduced to integrate the VGGreNet for the LF image SR, which collects global‐to‐local information according to the CNN kernel size and dynamically constructs each view through a global view module. Besides, the generated features are transformed to a uniform space to perform final fusion, achieving global alignment for precise extraction of angular information. Experimental results show that the proposed method can handle benchmark LF datasets with various angular and different resolutions, reporting the effectiveness and potential performance of the method. [ABSTRACT FROM AUTHOR]

In many deep neural network (DNN) applications, the difficulty of gathering high‐quality data in industry fields hinders the practical use of DNN. Thus, the concept of transfer learning (TL) has emerged, which leverages the pretrained knowledge of the DNN which was built based on large‐scale datasets. For this TL objective, this paper suggests two‐stage architectural fine‐tuning for reducing the costs and time while exploring the most efficient DNN model, inspired by neural architecture search (NAS). The first stage is mutation, which reduces the search costs using a priori architectural information. Moreover, the next stage is early‐stopping, which reduces NAS costs by terminating the search process in the middle of computation. The data‐intensive experimental results verify that the proposed method outperforms benchmarks. [ABSTRACT FROM AUTHOR]

An adaptive neural network dynamic surface control method for strict feedback non‐linear systems with model uncertainty is proposed in this paper. The uncertain parts are approached by RBF neural network; meanwhile, the virtual controller is designed to make system stable according to dynamic surface control. The method of Lyapunov is used to prove the stability and convergence of the system. The simulation results prove the feasibility of this controller and prove that the controller has an advantage to approach the uncertain non‐linear system and make system have well convergence and traceability. [ABSTRACT FROM AUTHOR]

Different from object detection in natural images, aerial‐view object detection faces special challenges with large changes in object orientation and wide multi‐scale distribution. Many methods based on Oriented Bounding Boxes (OBB) can reach accurate results, yet may face the problem of parameter mutation and axis prediction deviation. To address these problems, a two‐stage detection framework is proposed for arbitrary‐oriented traffic participant detection and axis prediction, named Axis Prediction Network. First, a Deformable Convolution Fusion Module (DCF‐Module) is proposed to enhance the ability of FPN to extract multi‐scale semantic features, for dealing with the multi‐scale change of objects. Then, the axis heat‐map prediction head network is proposed to fit the long axis of oriented objects labeled with Gaussian model, which uses pixel‐by‐pixel prediction heatmap to calculate the long axis of the object, avoiding the angle mutation of an OBB. Last, the long and short side prediction head network is proposed to predict the shape of an oriented object, avoiding the mutation of the width and height of an OBB. The experiments are conducted on the new built Crossroad dataset and the public DOTA dataset. Experiment results show that the proposed method achieve good performance in arbitrary‐oriented traffic participant detection and axis prediction in aerial view. [ABSTRACT FROM AUTHOR]

Abstract Human keypoints detection is different from general detection tasks and requires networks that can learn visual information and anatomical constraints. Since CNN is excellent in extracting texture features of images and transformer can learn the correlation among keypoints well, many CTPNets (CNN+transformer type human pose estimation networks) have emerged. However, these networks are unconcerned with the processing of the features extracted from the CNN and naturally expand only from the channel dimension, ignoring the spatial features in the visual information that are essential for complex detection tasks like pose estimation. So the channel spatial integrated transformer for human pose estimation, termed CSIT, is proposed. The visual information are summarized as texture and spatial information, and a parallel network is used to expand the feature maps in the channel and spatial dimensions to learn texture features and spatial features respectively. In addition, anatomically constrained information is learned by keypoint embeddings. At the end of the network, the 1D vector representation method with more advanced performance and more compatible with transformer's characteristics is used to predict keypoints. Experiments show that CSIT outperforms the mainstream CTPNets on the COCO test‐dev dataset, and also show satisfactory results on the MPII dataset.

Abstract Point cloud completion is prevalent due to the insufficient results from current point cloud acquisition equipments, where a large number of point data failed to represent a relatively complete shape. Existing point cloud completion algorithms, mostly encoder‐decoder structures with grids transform (also presented as folding operation), can hardly obtain a persuasive representation of input clouds due to the issue that their bottleneck‐shape result cannot tell a precise relationship between the global and local structures. For this reason, this article proposes a novel point cloud completion model based on a Stack‐Style Folding Network (SSFN). Firstly, to enhance the deep latent feature extraction, SSFN enhances the exploitation of shape feature extractor by integrating both low‐level point feature and high‐level graphical feature. Next, a precise presentation is obtained from a high dimensional semantic space to improve the reconstruction ability. Finally, a refining module is designed to make a more evenly distributed result. Experimental results shows that our SSFN produces the most promising results of multiple representative metrics with a smaller scale parameters than current models.

Abstract Semantic segmentation is a critical topic in computer vision, and it has numerous practical applications, including mobile devices, autonomous driving, and many other fields. However, in these application scenarios, it is often essential for the segmentation models to achieve a balance between efficiency and performance. A lightweight attention‐guided redundancy‐reuse network (LARNet) was proposed to address this challenge in this paper. Specifically, the multi‐scale asymmetric redundancy reuse (MAR) module was designed as the main component of the encoder for dense encoding of contextual semantic features. Furthermore, the efficient attention fusion (EAF) module was established for multi‐scale information fusion via the channel and spatial attention mechanisms in the decoder. A series of experiments were conducted to verify the proposed network. The results of tests on multiple datasets suggest that the network has higher accuracy and faster speed than the existing real‐time semantic segmentation methods.

Abstract Traffic forecasting, as a fundamental and challenging problem of intelligent transportation systems (ITS), has always been the focus of researchers. Nevertheless, accurate traffic forecasting still exists some problems due to the complex spatial‐temporal dependencies and irregularities of traffic flows. Most of the existing methods typically use the spatial adjacency matrix and complicated mechanism to model spatial‐temporal relationships separately, while ignoring the latent spatial‐temporal correlations. In this paper, a novel architecture is proposed named spatial‐temporal correlation graph convolutional networks (STCGCN) for traffic prediction. First, an informative fused graph structure is constructed to better learn the complex spatial‐temporal correlations, which breaks the limitation that the general spatial adjacency matrix cannot reflect temporal correlations. Moreover, spatial‐temporal correlation graph convolution and gated temporal convolution are performed in parallel and they are integrated into a unified layer, which enables capturing both local and global spatial‐temporal dependencies simultaneously. By stacking multiple layers, STCGCN can learn more long‐range spatial‐temporal dependencies. Experimental results on five public traffic datasets demonstrate the effectiveness and robustness of the proposed STCGCN in urban traffic forecasting.

Abstract Most existing Siamese Network trackers rely on a predefined anchor box to predict object position. However, they require complicated hyperparameter settings. The authors directly forecast the object boundary by using the fully convolutional network as the head of the tracking network to solve this issue and simplify the application. The end‐to‐end design avoids the setting of hyperparameters and candidate boxes. The authors also discovered that the validation loss decreased less than the training loss throughout Siamese training. The authors changed the normalisation layer from batch normalisation to group normalisation to solve this issue. It solved the problem that the loss function is difficult to decrease and increased training efficiency. Test experiments on the tracking dataset, including VOT2019 and GOT10k, show that the authors’ network outperforms DaSiamRPN and SiamFC regarding precision under the same network size and runs at 24 FPS on an AMD 4800 CPU. It also runs at 306 FPS on a 3090 GPU.

Abstract Real‐time semantic segmentation is a crucial technology in automatic driving scenarios, which needs to meet both high precision and real‐time. The authors observe that learning complex correlations between object categories is vital in the real‐time semantic segmentation task. Moreover, image spatial detail information plays an important role in small object segmentation and preserving edges and textures. A Semantics Recalibration and Detail Enhancement Network for real‐time semantic segmentation based on BiSeNet V2 is proposed. On the one hand, a lightweight Semantics Recalibration module is designed to effectively extract global semantic contextual information, which combines pyramid segmentation and adaptive recalibration operations to learn the correlations between object categories. On the other hand, a Detail Enhancement module takes the feature maps of the shallow layers in the semantics branch as input and refines the feature maps to highlight the detail information. Finally, quantitative and qualitative analyses on Cityscapes and CamVid datasets demonstrate the effectiveness and generalisation of the proposed method.

Abstract The increase in neural network recognition accuracy is accompanied by a significant increase in the scales of networks and computations. To make deep learning frameworks widely used on mobile platforms, model acceleration has become extremely important in computer vision. In this study, a novel neural network acceleration method based on selective activation is proposed. First, as the algebraic basis for selective activation, mask general matrix multiplication is used to reduce matrix multiplication calculations. Second, to screen and remove activated neurons and reduce the number of calculations, we introduce an Activation Management Unit that includes two different strategies, Selective Activation with Primary Weights (SAPW) and Selective Activation with Primary Inputs (SAPI). SAPW greatly reduces the number of calculations of the fully connected layer and self‐attention and better guarantees detection accuracy. SAPI has the best performance on convolutional architectures, which can significantly reduce the amount of convolutional computation while maintaining the image classification accuracy. We present result of extensive experiments on computational and accuracy tradeoffs and show strong performance for CIFAR‐10 classification and Pascal VOC2012 detection. Compared with the dense method, the proposed selective activation method significantly reduces the number of neural network calculations with equal accuracy.

Abstract Temporal action localisation is a challenging task, and video context is crucial to localisation actions. Most existing cases that incorporate temporal and semantic contexts into video features suffer from single contextual representation and blurred temporal boundaries. In this study, a multi‐temporal resolution pyramid structure model is proposed. Firstly, a temporal‐semantic context aggregation module (TSCF) is designed to assign different attention weights to temporal contexts and combine them with multi‐level semantics into video features. Secondly, for the problem of large differences in the time span between different actions in the video, a local‐global attention module is designed to combine local and global temporal dependencies for each temporal point to obtain a more flexible and robust representation of contextual relations. The redundant representation of the convolution kernel is reduced by modifying the convolution and the arithmetic power is redeployed at a microscopic granularity. To verify the effectiveness of the model, extensive experiments on three challenging datasets are performed. On THUMOS14, the best performance is obtained in IoU@0.3–0.6 with an average mAP of 47.02%. On ActivityNet‐1.3, an average mAP of 34.94% was obtained. On HACS, an average mAP of 28.46% was achieved.

Human keypoints detection is different from general detection tasks and requires networks that can learn visual information and anatomical constraints. Since CNN is excellent in extracting texture features of images and transformer can learn the correlation among keypoints well, many CTPNets (CNN+transformer type human pose estimation networks) have emerged. However, these networks are unconcerned with the processing of the features extracted from the CNN and naturally expand only from the channel dimension, ignoring the spatial features in the visual information that are essential for complex detection tasks like pose estimation. So the channel spatial integrated transformer for human pose estimation, termed CSIT, is proposed. The visual information are summarized as texture and spatial information, and a parallel network is used to expand the feature maps in the channel and spatial dimensions to learn texture features and spatial features respectively. In addition, anatomically constrained information is learned by keypoint embeddings. At the end of the network, the 1D vector representation method with more advanced performance and more compatible with transformer's characteristics is used to predict keypoints. Experiments show that CSIT outperforms the mainstream CTPNets on the COCO test‐dev dataset, and also show satisfactory results on the MPII dataset. [ABSTRACT FROM AUTHOR]