Search

Camera-based measurement of vital signs utilizes imaging equipment to measure physiological changes by analyzing human body images. The contactless physiological measurement provides benefits across various fields, from healthcare and clinical trial to telemedicine and exercise scenarios. On the basis of advances in medicine, optics, and computer vision, these technologies have made significant progress. Research in this area has grown exponentially in recent years. In this review, we provide a comprehensive and meticulous overview of camera-based measurements of physiological vital signs, including four aspects: data acquisition, methodology, vital signs and applications. It covers the measurement of heart rate, heart rate variability, respiratory rate, blood pressure, and oxygen saturation, using various technologies including motion detection, signal processing, and deep learning. We also present newly released datasets and applications in new areas. At last, future promising prospects and potential research directions are also discussed. [ABSTRACT FROM AUTHOR]

Person Re-identification (ReID) is an important yet challenging task in computer vision. Due to diverse background clutters, variations of viewpoints and body poses, it is far from being solved. How to extract discriminative and robust features invariant to background clutters is one of the core problems. In this paper, we first introduce a set of binary segmentation masks to construct synthetic RGB-Mask pairs as inputs, and then design a mask-guided contrastive attention model (MGCAM) to learn features separately from the body and background regions. Moreover, we propose a novel region-level triplet loss to guide the features learning, i.e., pulling the features from the full image and body region close, whereas pushing the features from backgrounds away. To learn the similarities from multiple features of the proposed MGCAM, we further introduce the instance-level two-stream metric co-learning (TSMCL) to help learn pair-wise relations between features from not only different regions but also different instances. TSMCL could help learn more compact features across full and body streams, enhancing the performance of MGCAM. We evaluate the proposed method on four public datasets, including MARS, Market-1501, CUHK03, and DukeMTMC-reID. Extensive experiments show that the proposed method is effective and achieves satisfying results. [ABSTRACT FROM AUTHOR]

Objective: The utilization of hyperspectral imaging (HSI) in real-time tumor segmentation during a surgery have recently received much attention, but it remains a very challenging task. Methods: In this work, we propose semantic segmentation methods, and compare them with other relevant deep learning algorithms for tongue tumor segmentation. To the best of our knowledge, this is the first work using deep learning semantic segmentation for tumor detection in HSI data using channel selection, and accounting for more spatial tissue context, and global comparison between the prediction map, and the annotation per sample. Results, and Conclusion: On a clinical data set with tongue squamous cell carcinoma, our best method obtains very strong results of average dice coefficient, and area under the ROC-curve of 0.891 ± 0.053, and 0.924 ± 0.036, respectively on the original spatial image size. The results show that a very good performance can be achieved even with a limited amount of data. We demonstrate that important information regarding tumor decision is encoded in various channels, but some channel selection, and filtering is beneficial over the full spectra. Moreover, we use both visual (VIS), and near-infrared (NIR) spectrum, rather than commonly used only VIS spectrum; although VIS spectrum is generally of higher significance, we demonstrate NIR spectrum is crucial for tumor capturing in some cases. Significance: The HSI technology augmented with accurate deep learning algorithms has a huge potential to be a promising alternative to digital pathology or a doctors’ supportive tool in real-time surgeries. [ABSTRACT FROM AUTHOR]

Ultrasound-guided procedures have been applied in many clinical therapies, such as cardiac catheterization and regional anesthesia. Medical instrument detection in 3D Ultrasound (US) is highly desired, but the existing approaches are far from real-time performance. Our objective is to investigate an efficient instrument detection method in 3D US for practical clinical use. We propose a novel Multi-dimensional Mixed Network for efficient instrument detection in 3D US, which extracts the discriminating features at 3D full-image level by a 3D encoder, and then applies a specially designed dimension reduction block to reduce the spatial complexity of the feature maps by projecting from 3D space into 2D space. A 2D decoder is adopted to detect the instrument along the specified axes. By projecting the predicted 2D outputs, the instrument is detected or visualized in the 3D volume. Furthermore, to enable the network to better learn the discriminative information, we propose a multi-level loss function to capture both pixel- and image-level differences. We carried out extensive experiments on two datasets for two tasks: (1) catheter detection for cardiac RF-ablation and (2) needle detection for regional anesthesia. Our experiments show that our proposed method achieves a detection error of 2-3 voxels with an efficiency of about 0.12 sec per 3D US volume. The proposed method is 3-8 times faster than the state-of-the-art methods, leading to real-time performance. The results show that our proposed method has significant clinical value for real-time 3D US-guided intervention. [ABSTRACT FROM AUTHOR]

Infants are particularly vulnerable to the effects of pain and discomfort, which can lead to abnormal brain development, yielding long-term adverse neurodevelopmental outcomes. In this study, we propose a video-based method for automated detection of their discomfort. The infant face is first detected and normalized. A two-phase classification workflow is then employed, where Phase 1 is subject-independent, and Phase 2 is subject-dependent. Phase 1 derives geometric and appearance features, while Phase 2 incorporates facial landmark-based template matching. An SVM classifier is finally applied to video frames to recognize facial expressions of comfort or discomfort. The method is evaluated using videos from 22 infants. Experimental results show an AUC of 0.87 for the subject-independent phase and 0.97 for the subject-dependent phase, which is promising for clinical use. [ABSTRACT FROM AUTHOR]

Abstract Hyperspectral images (HSIs) are often corrupted by noise during the acquisition process, thus degrading the HSI's discriminative capability significantly. Therefore, HSI denoising becomes an essential preprocess step before application. This paper proposes a new HSI denoising approach connecting Partial Sum of Singular Values (PSSV) and superpixels segmentation named as SS-PSSV, which can remove the noise effectively. Based on the fact that there is a high correlation between different bands of the same signal, it is easy to know the property of low rank between distinct bands. To this end, PSSV is utilized, and in order to better tap the low-rank attribute of pixels, we introduce the superpixels segmentation method, which allows pixels in HSI with high similarity to be grouped in the same sub-block as much as possible. Extensive experiments display that the proposed algorithm outperforms the state-of-the-art. [ABSTRACT FROM AUTHOR]

Accurate and efficient catheter segmentation in 3D ultrasound (US) is essential for ultrasound-guided cardiac interventions. State-of-the-art segmentation algorithms, based on convolutional neural networks (CNNs), suffer from high computational cost and large 3D data size for GPU implementation, which are far from satisfactory for real-time applications. In this paper, we propose a novel approach for efficient catheter segmentation in 3D US. Instead of using Cartesian US, our approach performs catheter segmentation in Frustum US (i.e., the US data before scan conversion). Compared to Cartesian US, Frustum US has a much smaller volume size, therefore the catheter can be segmented more efficiently in Frustum US. However, annotating the irregular and deformed Frustum images is challenging, and it is laborious to obtain the voxel-level annotation. To address this, we propose a weakly supervised learning framework, which requires only bounding-box annotations. The labels of the voxels are generated by incorporating class activation maps with line filtering, which are iteratively updated during the training cycles. Our experimental results show that, compared to Cartesian US, the catheter can be segmented much more efficiently in Frustum US (i.e., 0.25 s per volume) with better accuracy. Extensive experiments also validate the effectiveness of the proposed weakly supervised learning method. • Catheter segmentation in 3D Frustum ultrasound images with low computation cost. • Novel scheme for semantic segmentation training based on bounding box annotation. • Extensive validations show our method's effectiveness and has comparable performance to fully supervised learning. [ABSTRACT FROM AUTHOR]

Abstract: Fast robust background subtraction under sudden lighting changes is a challenging problem in many applications. In this paper, we propose a real-time approach, which combines the Eigenbackground and Statistical Illumination method to address this issue. The first algorithm is used to reconstruct the background frame, while the latter improves the foreground segmentation. In addition, we introduce an online spatial likelihood model by detecting reliable background pixels. Extensive quantitative experiments illustrate our approach consistently achieves significantly higher precision at high recall rates, compared to several state-of-the-art illumination invariant background subtraction methods. [Copyright &y& Elsevier]

Abstract: In this paper we propose to adopt a learning-based encoding method for age estimation under unconstrained imaging conditions. A similar approach [Cao et al., 2010] is applied to face recognition in real-life face images. However, the feature vectors are encoded in hard manner i.e. each feature vector is assigned to one code. The face is divided into patches where a code histogram is built for each patch. However, the codebook is learned using sample features from the entire face. Therefore, we propose an approach to extract robust and discriminative facial features and use soft encoding. Instead of learning a codebook from the entire face, we extract and learn multiple codebooks for individual face patches. The encoding is done by a weighting scheme in which each pixel is softly assigned to multiple candidate codes. Finally, orientation histogram of local gradients in neighborhood has been introduced as feature vector for code learning. On a large scale face dataset which contains 2744 real-life faces, the age group classification using our method achieves an absolute(relative) improvement of 3.6%(6.5%) over the best reported results [Shan, 2010]. [Copyright &y& Elsevier]

Abstract: Gender recognition is one of fundamental face analysis tasks. Most of the existing studies have focused on face images acquired under controlled conditions. However, real-world applications require gender classification on real-life faces, which is much more challenging due to significant appearance variations in unconstrained scenarios. In this paper, we investigate gender recognition on real-life faces using the recently built database, the Labeled Faces in the Wild (LFW). Local Binary Patterns (LBP) is employed to describe faces, and Adaboost is used to select the discriminative LBP features. We obtain the performance of 94.81% by applying Support Vector Machine (SVM) with the boosted LBP features. The public database used in this study makes future benchmark and evaluation possible. [Copyright &y& Elsevier]

Smile detection in face images captured in unconstrained real-world scenarios is an interesting problem with many potential applications. This paper presents an efficient approach to smile detection, in which the intensity differences between pixels in the grayscale face images are used as features. We adopt AdaBoost to choose and combine weak classifiers based on intensity differences to form a strong classifier. Experiments show that our approach has similar accuracy to the state-of-the-art method but is significantly faster. Our approach provides 85% accuracy by examining 20 pairs of pixels and 88% accuracy with 100 pairs of pixels. We match the accuracy of the Gabor-feature-based support vector machine using as few as 350 pairs of pixels. [ABSTRACT FROM PUBLISHER]

Local binary pattern (LBP) is a nonparametric descriptor, which efficiently summarizes the local structures of images. In recent years, it has aroused increasing interest in many areas of image processing and computer vision and has shown its effectiveness in a number of applications, in particular for facial image analysis, including tasks as diverse as face detection, face recognition, facial expression analysis, and demographic classification. This paper presents a comprehensive survey of LBP methodology, including several more recent variations. As a typical application of the LBP approach, LBP-based facial image analysis is extensively reviewed, while its successful extensions, which deal with various tasks of facial image analysis, are also highlighted. [ABSTRACT FROM PUBLISHER]

Spectral regression discriminant analysis (SRDA) is an efficient subspace learning method proposed recently. One important unsolved issue of SRDA is how to automatically determine an appropriate regularization parameter. In this letter, we present a method to estimate the optimal regularization parameter for SRDA. We test our method in different applications including head pose estimation, face recognition, and text categorization. Our extensive experiments evidently illustrate the effectiveness and efficiency of our approach. [ABSTRACT FROM AUTHOR]

Abstract: Automatic facial expression analysis is an interesting and challenging problem, and impacts important applications in many areas such as human–computer interaction and data-driven animation. Deriving an effective facial representation from original face images is a vital step for successful facial expression recognition. In this paper, we empirically evaluate facial representation based on statistical local features, Local Binary Patterns, for person-independent facial expression recognition. Different machine learning methods are systematically examined on several databases. Extensive experiments illustrate that LBP features are effective and efficient for facial expression recognition. We further formulate Boosted-LBP to extract the most discriminant LBP features, and the best recognition performance is obtained by using Support Vector Machine classifiers with Boosted-LBP features. Moreover, we investigate LBP features for low-resolution facial expression recognition, which is a critical problem but seldom addressed in the existing work. We observe in our experiments that LBP features perform stably and robustly over a useful range of low resolutions of face images, and yield promising performance in compressed low-resolution video sequences captured in real-world environments. [Copyright &y& Elsevier]

Abstract: Computer vision-based gender classification is an interesting and challenging problem, and has potential applications in visual surveillance and human–computer interaction systems. In this paper, we investigate gender classification from human gaits in image sequences, a relatively understudied problem. Moreover, we propose to fuse gait and face for improved gender discrimination. We exploit canonical correlation analysis (CCA), a powerful tool that is well suited for relating two sets of measurements, to fuse the two modalities at the feature level. Experiments demonstrate that our multimodal gender recognition system achieves the superior recognition performance of 97.2% in large data sets. [Copyright &y& Elsevier]

Abstract: Particle filtering and mean shift (MS) are two successful approaches to visual tracking. Both have their respective strengths and weaknesses. In this paper, we propose to integrate advantages of the two approaches for improved tracking. By incorporating the MS optimization into particle filtering to move particles to local peaks in the likelihood, the proposed mean shift embedded particle filter (MSEPF) improves the sampling efficiency considerably. Our work is conducted in the context of developing a hand control interface for a robotic wheelchair. We realize real-time hand tracking in dynamic environments of the wheelchair using MSEPF. Extensive experimental results demonstrate that MSEPF outperforms the MS tracker and the conventional particle filter in hand tracking. Our approach produces reliable tracking while effectively handling rapid motion and distraction with roughly 85% fewer particles. We also present a simple method for dynamic gesture recognition. The hand control interface based on the proposed algorithms works well in dynamic environments of the wheelchair. [Copyright &y& Elsevier]

Image Copy Detection (ICD) is developed to identify and track duplicated or manipulated images. The majority of existing methods rely on Convolutional Neural Networks (CNNs) and are trained using unsupervised learning techniques, which leads to subpar performance. We discover that by carefully designing the training process, Vision Transformer (ViT) backbones yield superior results. Specifically, directly training a ViT for ICD often leads to overfitting on the training images, which in turn results in poor generalization to unseen (test) images. Consequently, we initially train a CNN (such as ResNet-50), and during the ViT training, the distances between the features of CNN and ViT are regularized. We also incorporate an active learning method to further enhance performance. Notably, due to the visual discrepancy between auto-generated transformations and those used in the query set, we incorporate a small number (approximately 0.5% of unlabeled training images) of manually produced and labeled positive pairs. Training models on these pairs results in a significant performance boost though with little cost. Experimental findings demonstrate the effectiveness of our approach, and our method achieves state-of-the-art performance. Our code is available at: https://github.com/WangWenhao0716/ViT4ICD. [ABSTRACT FROM AUTHOR]

• A lightweight CNN is designed for efficient gait recognition. • A joint knowledge distillation algorithm is proposed for boosting the performance of the simplified model. • Extensive experiments on two public datasets demonstrate the effectiveness of the proposed method. Gait recognition has made significant progress recently. However, most of existing methods utilize complicated neural networks, which lead to high computation cost. In this paper, a lightweight model named Distilled Light GaitSet (DLGS) is proposed for efficient gait recognition. More specifically, a lightweight CNN is designed for efficient computation, and a Joint Knowledge Distillation algorithm is proposed to boost the accuracy of the simplified model. Extensive experiments on the CASIA-B dataset and the OU-MVLP dataset show that the proposed DLGS can reduce the number of parameters and computation cost significantly while achieving the state-of-the-art performance. [ABSTRACT FROM AUTHOR]

• Robust and efficient medical instrument segmentation in 3D cardiac ultrasound images • Automatic instrument detection to guide the cardiac intervention using 3D ultrasound imaging • Accurate and faster coarse-to-fine segmentation based on convolutional neural networks • Extensive validation shows our method has better performance than the state-of-the-art methods by a large margin. Instrument segmentation plays a vital role in 3D ultrasound (US) guided cardiac intervention. Efficient and accurate segmentation during the operation is highly desired since it can facilitate the operation, reduce the operational complexity, and therefore improve the outcome. Nevertheless, current image-based instrument segmentation methods are not efficient nor accurate enough for clinical usage. Lately, fully convolutional neural networks (FCNs), including 2D and 3D FCNs, have been used in different volumetric segmentation tasks. However, 2D FCN cannot exploit the 3D contextual information in the volumetric data, while 3D FCN requires high computation cost and a large amount of training data. Moreover, with limited computation resources, 3D FCN is commonly applied with a patch-based strategy, which is therefore not efficient for clinical applications. To address these, we propose a POI-FuseNet, which consists of a patch-of-interest (POI) selector and a FuseNet. The POI selector can efficiently select the interested regions containing the instrument, while FuseNet can make use of 2D and 3D FCN features to hierarchically exploit contextual information. Furthermore, we propose a hybrid loss function, which consists of a contextual loss and a class-balanced focal loss, to improve the segmentation performance of the network. With the collected challenging ex-vivo dataset on RF-ablation catheter, our method achieved a Dice score of 70.5%, superior to the state-of-the-art methods. In addition, based on the pre-trained model from ex-vivo dataset, our method can be adapted to the in-vivo dataset on guidewire and achieves a Dice score of 66.5% for a different cardiac operation. More crucially, with POI-based strategy, segmentation efficiency is reduced to around 1.3 seconds per volume, which shows the proposed method is promising for clinical use. [ABSTRACT FROM AUTHOR]

Current methods for skeleton-based human action recognition usually work with complete skeletons. However, in real scenarios, it is inevitable to capture incomplete or noisy skeletons, which could significantly deteriorate the performance of current methods when some informative joints are occluded or disturbed. To improve the robustness of action recognition models, a multi-stream graph convolutional network (GCN) is proposed to explore sufficient discriminative features spreading over all skeleton joints, so that the distributed redundant representation reduces the sensitivity of the action models to non-standard skeletons. Concretely, the backbone GCN is extended by a series of ordered streams which is responsible for learning discriminative features from the joints less activated by preceding streams. Here, the activation degrees of skeleton joints of each GCN stream are measured by the class activation maps (CAM), and only the information from the unactivated joints will be passed to the next stream, by which rich features over all active joints are obtained. Thus, the proposed method is termed richly activated GCN (RA-GCN). Compared to the state-of-the-art (SOTA) methods, the RA-GCN achieves comparable performance on the standard NTU RGB+D 60 and 120 datasets. More crucially, on the synthetic occlusion and jittering datasets, the performance deterioration due to the occluded and disturbed joints can be significantly alleviated by utilizing the proposed RA-GCN. [ABSTRACT FROM AUTHOR]

Human parsing, which is a task of labeling pixels in human images into different fine-grained semantic parts, has achieved significant progress during the past decade. However, there are still several challenges in human parsing, due to occlusions, varying poses and similar appearance between the left/right parts. To tackle these problems, a Human Kinematic Skeleton Graph Layer (HKSGL) is proposed to augment regular neural networks with human kinematic skeleton information. The HKSGL has two major components: kinematic skeleton graph and interconnected modular neural layer. The kinematic skeleton graph is a user pre-defined skeleton graph, which models the interconnections between different semantic parts. Then the skeleton graph is passed to the interconnected modular neural layer which is composed of a set of modular blocks corresponding to different semantic parts. The HKSGL is a lightweight, low costs layer which can be easily attached to any existing neural networks. To demonstrate the power of the HKSGL, a new dataset on human parsing in occlusions is also collected, termed the RAP-Occ. Extensive experiments have been performed on four datasets on human parsing, including the LIP, the CIHP, the ATR and the RAP-Occ. And two popular baselines, i.e., the Deeplab V3+ and the CE2P, are agumented by the proposed HKSGL. Competitive performance of the augmented models has been achieved in comparison with state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Objective: Surgical navigation is a well-established tool in endoscopic skull base surgery. However, navigational and endoscopic views are usually displayed on separate monitors, forcing the surgeon to focus on one or the other. Aiming to provide real-time integration of endoscopic and diagnostic imaging information, we present a new navigation technique based on augmented reality with fusion of intraoperative cone beam computed tomography (CBCT) on the endoscopic view. The aim of this study was to evaluate the accuracy of the method. Material and methods: An augmented reality surgical navigation system (ARSN) with 3D CBCT capability was used. The navigation system incorporates an optical tracking system (OTS) with four video cameras embedded in the flat detector of the motorized C-arm. Intra-operative CBCT images were fused with the view of the surgical field obtained by the endoscope's camera. Accuracy of CBCT image co-registration was tested using a custom-made grid with incorporated 3D spheres. Results: Co-registration of the CBCT image on the endoscopic view was performed. Accuracy of the overlay, measured as mean target registration error (TRE), was 0.55 mm with a standard deviation of 0.24 mm and with a median value of 0.51mm and interquartile range of 0.39˗˗0.68 mm. Conclusion: We present a novel augmented reality surgical navigation system, with fusion of intraoperative CBCT on the endoscopic view. The system shows sub-millimeter accuracy. [ABSTRACT FROM AUTHOR]

Zero-shot learning aims to recognize image categories which are "unseen" in the training phase of image classification models. The key to this task is to transfer the learned knowledge from "seen" classes to "unseen" classes. In order to make the knowledge transfer process more effective, we propose to exploit both the visual and semantic attention mechanisms simultaneously in zero-shot learning tasks. Specifically, a dual-focus transfer network (DFTN) model is proposed to implement attention mechanisms from both the visual and semantic ends in a mapping based zero-shot learning framework with a visual focus transfer (VFT) module and a semantic focus transfer (SFT) module. The VFT module is composed by multi-head self-attention networks, which endows salient parts of images with greater weights at different resolutions of the feature maps. The SFT module generates semantic weights to re-weight semantic attribute features with the guidance of visual representations, where the semantic attributes corresponding to more visual discrimination capability will obtain greater weights. Extensive experiments of zero-shot learning and generalized zero-shot learning on five representative benchmarks demonstrate the superiority of the proposed DFTN model, compared to other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

By exploiting the complementary information of RGB modality and thermal modality, RGB-thermal (RGB-T) semantic segmentation is robust to adverse lighting conditions. When fusing features from RGB images and thermal images, the existing methods design different feature fusion strategies, but most of these methods overlook the modality differences caused by different imaging mechanisms. This may result in insufficient usage of complementary information. To address this issue, we propose a novel Mask-guided Modality Difference Reduction Network (MMDRNet), where the mask is utilized in the image reconstruction to ensure that the modality discrepancy within foreground regions is minimized. Doing so enables the generation of more discriminative representations for foreground pixels, thus facilitating the segmentation task. On top of this, we present a Dynamic Task Balance (DTB) method to balance the modality difference reduction task and semantic segmentation task dynamically. The experimental results on the MFNet dataset and the PST900 dataset demonstrate the superiority of the proposed mask-guided modality difference reduction strategy and the effectiveness of the DTB method. [ABSTRACT FROM AUTHOR]

• A dual-branch self-attention network is proposed for pedestrian attribute recognition. • A second-order self-attention module is introduced to exploit global information of a pedestrian with reduced feature dimensions. • Extensive experiments on three public datasets demonstrate the effectiveness of the proposed method. Pedestrian attribute recognition (PAR) is still a challenging task in real surveillance scenes, where the difficulties, such as occlusion, complex background, and varying views, degrade the recognition accuracy. To fully exploit attribute correlation and regional context, we propose a dual-branch self-attention network for PAR: (1) For the attribute branch, the second-order self-attention module (SO-SAM) is first introduced to derive the second-order feature maps; they are then fused with the first-order information to learn unique features for each attribute using the constrained loss function. (2) For the context branch, multiple adaptive visual tokens and a group of multi-head context self-attention modules (C-SAM) are exploited to describe the image and explore the relationships between different regions. The experimental results on three main public benchmarks, RAP, PA100K, and PETA datasets, demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Imaging the acoustical coefficient of nonlinearity, β, is of interest in several healthcare interventional applications. It is an important feature that can be used for discriminating tissues. In this paper, we propose a nonlinearity characterization method with the goal of locally estimating the coefficient of nonlinearity. The proposed method is based on a 1-D solution of the nonlinear lossy Westerfelt equation, thereby deriving a local relation between β and the pressure wave field. Based on several assumptions, a β imaging method is then presented that is based on the ratio between the harmonic and fundamental fields, thereby reducing the effect of spatial amplitude variations of the speckle pattern. By testing the method on simulated ultrasound pressure fields and an in vitro B-mode ultrasound acquisition, we show that the designed algorithm is able to estimate the coefficient of nonlinearity, and that the tissue types of interest are well discriminable. The proposed imaging method provides a new approach to β estimation, not requiring a special measurement setup or transducer, that seems particularly promising for in vivo imaging. [ABSTRACT FROM AUTHOR]

Automated segmentation of brain glioma plays an active role in diagnosis decision, progression monitoring and surgery planning. Based on deep neural networks, previous studies have shown promising technologies for brain glioma segmentation. However, these approaches lack powerful strategies to incorporate contextual information of tumor cells and their surrounding, which has been proven as a fundamental cue to deal with local ambiguity. In this work, we propose a novel approach named Context-Aware Network (CANet) for brain glioma segmentation. CANet captures high dimensional and discriminative features with contexts from both the convolutional space and feature interaction graphs. We further propose context guided attentive conditional random fields which can selectively aggregate features. We evaluate our method using publicly accessible brain glioma segmentation datasets BRATS2017, BRATS2018 and BRATS2019. The experimental results show that the proposed algorithm has better or competitive performance against several State-of-The-Art approaches under different segmentation metrics on the training and validation sets. [ABSTRACT FROM AUTHOR]

An introduction is presented in which the editors discuss various reports within the issue on topics including computer vision, remote sensing and foreground detection.

Background: Minimally invasive spine surgery is dependent on accurate navigation. Computer-assisted navigation is increasingly used in minimally invasive surgery (MIS), but current solutions require the use of reference markers in the surgical field for both patient and instruments tracking.Purpose: To improve reliability and facilitate clinical workflow, this study proposes a new marker-free tracking framework based on skin feature recognition.Methods: Maximally Stable Extremal Regions (MSER) and Speeded Up Robust Feature (SURF) algorithms are applied for skin feature detection. The proposed tracking framework is based on a multi-camera setup for obtaining multi-view acquisitions of the surgical area. Features can then be accurately detected using MSER and SURF and afterward localized by triangulation. The triangulation error is used for assessing the localization quality in 3D.Results: The framework was tested on a cadaver dataset and in eight clinical cases. The detected features for the entire patient datasets were found to have an overall triangulation error of 0.207 mm for MSER and 0.204 mm for SURF. The localization accuracy was compared to a system with conventional markers, serving as a ground truth. An average accuracy of 0.627 and 0.622 mm was achieved for MSER and SURF, respectively.Conclusions: This study demonstrates that skin feature localization for patient tracking in a surgical setting is feasible. The technology shows promising results in terms of detected features and localization accuracy. In the future, the framework may be further improved by exploiting extended feature processing using modern optical imaging techniques for clinical applications where patient tracking is crucial. [ABSTRACT FROM AUTHOR]

RGB-induced salient object detection has recently witnessed substantial progress, which is attributed to the superior feature learning capability of deep convolutional neural networks (CNNs). However, such detections suffer from challenging scenarios characterized by cluttered backgrounds, low-light conditions and variations in illumination. Instead of improving RGB based saliency detection, this paper takes advantage of the complementary benefits of RGB and thermal infrared images. Specifically, we propose a novel end-to-end network for multi-modal salient object detection, which turns the challenge of RGB-T saliency detection to a CNN feature fusion problem. To this end, a backbone network (e.g., VGG-16) is first adopted to extract the coarse features from each RGB or thermal infrared image individually, and then several adjacent-depth feature combination (ADFC) modules are designed to extract multi-level refined features for each single-modal input image, considering that features captured at different depths differ in semantic information and visual details. Subsequently, a multi-branch group fusion (MGF) module is employed to capture the cross-modal features by fusing those features from ADFC modules for a RGB-T image pair at each level. Finally, a joint attention guided bi-directional message passing (JABMP) module undertakes the task of saliency prediction via integrating the multi-level fused features from MGF modules. Experimental results on several public RGB-T salient object detection datasets demonstrate the superiorities of our proposed algorithm over the state-of-the-art approaches, especially under challenging conditions, such as poor illumination, complex background and low contrast. [ABSTRACT FROM AUTHOR]

Zero-shot learning aims to recognize objects which do not appear in the training dataset. Previous prevalent mapping-based zero-shot learning methods suffer from the projection domain shift problem due to the lack of image classes in the training stage. In order to alleviate the projection domain shift problem, a deep unbiased embedding transfer (DUET) model is proposed in this paper. The DUET model is composed of a deep embedding transfer (DET) module and an unseen visual feature generation (UVG) module. In the DET module, a novel combined embedding transfer net which integrates the complementary merits of the linear and nonlinear embedding mapping functions is proposed to connect the visual space and semantic space. What’s more, the end-to-end joint training process is implemented to train the visual feature extractor and the combined embedding transfer net simultaneously. In the UVG module, a visual feature generator trained with a conditional generative adversarial framework is used to synthesize the visual features of the unseen classes to ease the disturbance of the projection domain shift problem. Furthermore, a quantitative index, namely the score of resistance on domain shift (ScoreRDS), is proposed to evaluate different models regarding their resistance capability on the projection domain shift problem. The experiments on five zero-shot learning benchmarks verify the effectiveness of the proposed DUET model. As demonstrated by the qualitative and quantitative analysis, the unseen class visual feature generation, the combined embedding transfer net and the end-to-end joint training process all contribute to alleviating projection domain shift in zero-shot learning. [ABSTRACT FROM AUTHOR]

Integration of multi-level contextual information, such as feature maps and side outputs, is crucial for Convolutional Neural Networks (CNNs)-based salient object detection. However, most existing methods either simply concatenate multi-level feature maps or calculate element-wise addition of multi-level side outputs, thus failing to take full advantages of them. In this paper, we propose a new strategy for guiding multi-level contextual information integration, where feature maps and side outputs across layers are fully engaged. Specifically, shallower-level feature maps are guided by the deeper-level side outputs to learn more accurate properties of the salient object. In turn, the deeper-level side outputs can be propagated to high-resolution versions with spatial details complemented by means of shallower-level feature maps. Moreover, a group convolution module is proposed with the aim to achieve high-discriminative feature maps, in which the backbone feature maps are divided into a number of groups and then the convolution is applied to the channels of backbone feature maps within each group. Eventually, the group convolution module is incorporated in the guidance module to further promote the guidance role. Experiments on three public benchmark datasets verify the effectiveness and superiority of the proposed method over the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Highlights • A local tree-structured low-rank representation (TS-LRR) model for salient object detection is introduced. • The foreground consistency within potential foreground regions is considered. • A primitive background dictionary is constructed for the proposed model. • Experimental results on public datasets demonstrate the effectiveness of the proposed model over the state-of-the-art methods. Abstract We propose a local tree-structured low-rank representation (TS-LRR) model to detect salient objects under the complicated background with diverse local regions, which is problematic for most low-rank matrix recovery (LRMR) based salient object detection methods. We first impose a local tree-structured low-rank constraint on the representation coefficients matrix to capture the complicated background. Specifically, a primitive background dictionary is constructed for TS-LRR to promote its background representation ability, and thus enlarge the gap between the salient objects and the background. We then impose a group-sparsity constraint on the sparse error matrix with the intention to ensure the saliency consistency among patches with similar features. At last, a foreground consistency is introduced to identically highlight the distinctive regions within the salient object. Experimental results on three public benchmark datasets demonstrate the effectiveness and superiority of the proposed model over the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The primary treatment for malignant brain tumors is surgical resection. While gross total resection improves the prognosis, a supratotal resection may result in neurological deficits. On the other hand, accurate intraoperative identification of the tumor boundaries may be very difficult, resulting in subtotal resections. Histological examination of biopsies can be used repeatedly to help achieve gross total resection but this is not practically feasible due to the turn-around time of the tissue analysis. Therefore, intraoperative techniques to recognize tissue types are investigated to expedite the clinical workflow for tumor resection and improve outcome by aiding in the identification and removal of the malignant lesion. Hyperspectral imaging (HSI) is an optical imaging technique with the power of extracting additional information from the imaged tissue. Because HSI images cannot be visually assessed by human observers, we instead exploit artificial intelligence techniques and leverage a Convolutional Neural Network (CNN) to investigate the potential of HSI in twelve in vivo specimens. The proposed framework consists of a 3D–2D hybrid CNN-based approach to create a joint extraction of spectral and spatial information from hyperspectral images. A comparison study was conducted exploiting a 2D CNN, a 1D DNN and two conventional classification methods (SVM, and the SVM classifier combined with the 3D–2D hybrid CNN) to validate the proposed network. An overall accuracy of 80% was found when tumor, healthy tissue and blood vessels were classified, clearly outperforming the state-of-the-art approaches. These results can serve as a basis for brain tumor classification using HSI, and may open future avenues for image-guided neurosurgical applications. [ABSTRACT FROM AUTHOR]

Surgical navigation systems are increasingly used for complex spine procedures to avoid neurovascular injuries and minimize the risk for reoperations. Accurate patient tracking is one of the prerequisites for optimal motion compensation and navigation. Most current optical tracking systems use dynamic reference frames (DRFs) attached to the spine, for patient movement tracking. However, the spine itself is subject to intrinsic movements which can impact the accuracy of the navigation system. In this study, we aimed to detect the actual patient spine features in different image views captured by optical cameras, in an augmented reality surgical navigation (ARSN) system. Using optical images from open spinal surgery cases, acquired by two gray-scale cameras, spinal landmarks were identified and matched in different camera views. A computer vision framework was created for preprocessing of the spine images, detecting and matching local invariant image regions. We compared four feature detection algorithms, Speeded Up Robust Feature (SURF), Maximal Stable Extremal Region (MSER), Features from Accelerated Segment Test (FAST), and Oriented FAST and Rotated BRIEF (ORB) to elucidate the best approach. The framework was validated in 23 patients and the 3D triangulation error of the matched features was < 0.5 mm. Thus, the findings indicate that spine feature detection can be used for accurate tracking in navigated surgery. [ABSTRACT FROM AUTHOR]