Search

Computer-Aided Diagnosis (CADx) systems for characterization of Narrow-Band Imaging (NBI) videos of suspected lesions in Barrett’s Esophagus (BE) can assist endoscopists during endoscopic surveillance. The real clinical value and application of such CADx systems lies in real-time analysis of endoscopic videos inside the endoscopy suite, placing demands on robustness in decision making and insightful classification matching with the clinical opinions. In this paper, we propose a lightweight int8-based quantized neural network architecture supplemented with an efficient stability function on the output for real-time classification of NBI videos. The proposed int8-architecture has low-memory footprint (4.8 MB), enabling operation on a range of edge devices and even existing endoscopy equipment. Moreover, the stability function ensures robust inclusion of temporal information from the video to provide a continuously stable video classification. The algorithm is trained, validated and tested with a total of 3,799 images and 284 videos of in total 598 patients, collected from 7 international centers. Several stability functions are experimented with, some of them being clinically inspired by weighing low-confidence predictions. For the detection of early BE neoplasia, the proposed algorithm achieves a performance of 92.8% accuracy, 95.7% sensitivity, and 91.4% specificity, while only 5.6% of the videos are without a final video classification. This work shows a robust, lightweight and effective deep learning-based CADx system for accurate automated real-time endoscopic video analysis, suited for embedding in endoscopy clinical practice.

Finding potential driving paths on unstructured roads is a challenging problem for autonomous driving and robotics applications. Although the rise of autonomous driving has resulted in massive public datasets, most of these datasets focus on urban environments and feature almost exclusively paved roads. To circumvent the problem of limited public datasets of unpaved roads, we combine seven public vehicle-mounted-camera datasets with a very small private dataset and train a neural network to achieve accurate road segmentation on almost any type of road. This trained network vastly outperforms networks trained on individual datasets when validated on our unpaved road datasets, with only a minor performance reduction on the highly challenging public WildDash dataset, which is mostly urban. Finally, we develop an algorithm to robustly transform these road segmentations to road centerlines, used to automatically control a vehicle-mounted PTZ camera.

The field of Out-of-Distribution (OOD) detection aims to separate OOD data from in-distribution (ID) data in order to make safe predictions. With the increasing application of Convolutional Neural Networks (CNNs) in sensitive environments such as autonomous driving and security, this field is bound to become indispensable in the future. Although the OOD detection field has made some progress in recent years, a fundamental understanding of the underlying phenomena enabling the separation of datasets remains lacking. We find that the OOD detection relies heavily on the covariate shift of the data and not so much on the semantic shift, i.e. a CNN does not carry explicit semantic information and relies solely on differences in features. Although these features can be affected by the underlying semantics, this relation does not seem strong enough to rely on. Conversely, we found that since the CNN training setup determines what features are learned, that it is an important factor for the OOD performance. We found that variations in the model training can lead to an increase or decrease in the OOD detection performance. Through this insight, we obtain an increase in OOD detection performance on the common OOD detection benchmarks by changing the training procedure and using the simple Maximum Softmax Probability (MSP) model introduced by (Hendrycks and Gimpel, 2016). We hope to inspire others to look more closely into the fundamental principles underlying the separation of two datasets. The code for reproducing our results can be found at https://github.com/SimonMariani/OOD- detection.

Barrett’s Esophagus (BE) is a well-known precursor for Esophageal Adenocarcinoma (EAC). Endoscopic detection and diagnosisof early BE neoplasia is performed in two steps: primary detection of asuspected lesion in overview and a targeted and detailed inspection of thespecific area using Narrow-Band Imaging (NBI). Despite the improvedvisualization of tissue by NBI and clinical classification systems, endoscopists have difficulties with correct characterization of the imagery.Computer-aided Diagnosis (CADx) may assist endoscopists in the classification of abnormalities in NBI imagery. We propose an endoscopydriven pre-trained deep learning-based CADx, for the characterization of NBI imagery of BE. We evaluate the performance of the algorithm on images as well as on videos, for which we use several post-hoc and real-time video analysis methods. The proposed real-time methods outperform the post-hoc methods on average by 1.2% and 2.3% for accuracy and specificity, respectively. The obtained results show promising methods towards real-time endoscopic video analysis and identifies steps forfurther development.

Introduction In an earlier study by King’s College London (KCL), a framework for the automatic segmentation of vestibular schwannomas (VS) from MRI scans was proposed. They demonstrated that artificial intelligence is capable of annotating and calculating VS tumor volumes, which can benefit tumor progression monitoring. In this study, their algorithm is validated using data from another institution. The obtained results are subsequently compared to a multi-institutional model, in which the data from both centers is combined. Methods In addition to the original KCL dataset of 375 contrast-enhanced T1-weighted (ceT1) MRI scans, a total of 1,115 ceT1 scans of individual VS patients from the ETZ hospital in Tilburg, the Netherlands, were used. All tumors were manually annotated by a neurosurgeon at time of treatment. Employing the original 2.5D convolutional neural network, two different models were subsequently trained. First, one model is trained solely on 176 scans from the KCL dataset, using an additional 20 and 46 scans for hyper-parameter tuning and internal validation, respectively. This single-institution model is externally validated on the entire ETZ dataset. The second model employed 242 scans from the KCL dataset and all of the ETZ data from before 2015 (733 scans), with the remainder of the ETZ data equally distributed between a tuning and validation set. The remaining 133 scans from the KCL dataset were also used for validation. Results The single-institutional model achieved internal and external validation mean Dice scores of 92.0±5.1% and 64.5±32.4%, respectively. The external validation set included 175 scans where the model failed to recognize any VS, resulting in a Dice score of zero for these scans, which skews the results. The second model yielded validation mean dice scores of 92.5±5.1% for the ETZ scans and 89.1±9.6% for the KCL scans. During validation, most of the low-scoring segmentations are of tumors that are either very small or cystic. Conclusions The significant difference in performance between internal and external validation of the first model, thereby surmising a poor generalization, can be explained by variations in spatial resolution and image acquisition parameters between the two datasets. The second model shows performances approaching inter-observer variability of human annotators (cf. 93.8±3.1%). The results show that creating a robust and well-generalizing model from single-institution data is challenging. However, the multi-institutional results empower the earlier demonstrated capabilities of the framework for the automatic segmentation of VS.

Temporal localization of driving actions plays a crucial role in advanced driver-assistance systems and naturalistic driving studies. However, this is a challenging task due to strict requirements for robustness, reliability and accurate localization. In this work, we focus on improving the overall performance by efficiently utilizing video action recognition networks and adapting these to the problem of action localization. To this end, we first develop a density-guided label smoothing technique based on label probability distributions to facilitate better learning from boundary videosegments that typically include multiple labels. Second, we design a post-processing step to efficiently fuse information from video-segments and multiple camera views into scene-level predictions, which facilitates elimination of false positives. Our methodology yields a competitive performance on the A2 test set of the naturalistic driving action recognition track of the 2022 NVIDIA AI City Challenge with an F1 score of 0.271.

Deep learning-based pose estimation algorithms can successfully estimate the pose of objects in an image, especially in the field of color images. 6D Object pose estimation based on deep learning models for X-ray images often use custom architectures that employ extensive CAD models and simulated data for training purposes. Recent RGB-based methods opt to solve pose estimation problems using small datasets, making them more attractive for the X-ray domain where medical data is scarcely available. We refine an existing RGB-based model (SingleShotPose) to estimate the 6D pose of a marked cube from grayscale X-ray images by creating a generic solution trained on only real X-ray data and adjusted for X-ray acquisition geometry. The model regresses 2D control points and calculates the pose through 2D/3D correspondences using Perspective-n-Point(PnP), allowing a single trained model to be used across all supporting cone-beam-based X-ray geometries. Since modern X-ray systems continuously adjust acquisition parameters during a procedure, it is essential for such a pose estimation network to consider these parameters in order to be deployed successfully and find a real use case. With a 5-cm/5-degree accuracy of 93% and an average 3D rotation error of 2.2 degrees, the results of the proposed approach are comparable with state-of-the-art alternatives, while requiring significantly less real training examples and being applicable in real-time applications., Published at SPIE Medical Imaging 2022

Accurate and efficient catheter segmentation in 3D ultrasound (US) is essential for cardiac intervention. Currently, the state-of-the-art segmentation algorithms are based on convolutional neural networks (CNNs), which achieved remarkable performances in a standard Cartesian volumetric data. Nevertheless, these approaches suffer the challenges of low efficiency and GPU unfriendly image size. Therefore, such difficulties and expensive hardware requirements become a bottleneck to build accurate and efficient segmentation models for real clinical application. In this paper, we propose a novel Frustum ultrasound based catheter segmentation method. Specifically, Frustum ultrasound is a polar coordinate based image, which includes same information of standard Cartesian image but has much smaller size, which overcomes the bottleneck of efficiency than conventional Cartesian images. Nevertheless, the irregular and deformed Frustum images lead to more efforts for accurate voxel-level annotation. To address this limitation, a weakly supervised learning framework is proposed, which only needs 3D bounding box annotations overlaying the region-of-interest to training the CNNs. Although the bounding box annotation includes noise and inaccurate annotation to mislead to model, it is addressed by the proposed pseudo label generated scheme. The labels of training voxels are generated by incorporating class activation maps with line filtering, which is iteratively updated during the training. Our experimental results show the proposed method achieved the state-of-the-art performance with an efficiency of 0.25 second per volume. More crucially, the Frustum image segmentation provides a much faster and cheaper solution for segmentation in 3D US image, which meet the demands of clinical applications.

Since the introduction of the GDPR and CCPA privacy legislation, both public and private facial image datasets are increasingly scrutinized. Several datasets have been taken offline completely and some have been anonymized. However, it is unclear how anonymization impacts face detection performance. To our knowledge, this paper presents the first empirical study on the effect of image anonymization on supervised training of face detectors. We compare conventional face anonymiz-ers with three state-of-the-art Generative Adversarial Network-based (GAN) methods, by training an off-the-shelf face detector on anonymized data. Our experiments investigate the suitability of anonymization methods for maintaining face detector performance, the effect of detectors overtraining on anonymization artefacts, dataset size for training an anonymizer, and the effect of training time of anonymization GANs. A final experiment investigates the correlation between common GAN evaluation metrics and the performance of a trained face detector. Although all tested anonymization methods lower the performance of trained face detectors, faces anonymized using GANs cause far smaller performance degradation than conventional methods. As the most important finding, the best-performing GAN, DeepPrivacy, removes identifiable faces for a face detector trained on anonymized data, resulting in a modest decrease from 91.0 to 88.3 mAP. In the last few years, there have been rapid improvements in realism of GAN-generated faces. We expect that further progression in GAN research will allow the use of Deep Fake technology for privacy-preserving Safe Fakes, without any performance degradation for training face detectors.

Pancreatic cancer is one of the global leading causes of cancer-related deaths. Despite the success of Deep Learning in computer-aideddiagnosis and detection (CAD) methods, little attention has been paidto the detection of Pancreatic Cancer. We propose a method for de-tecting pancreatic tumor that utilizes clinically-relevant features in thesurrounding anatomical structures, thereby better aiming to exploit theradiologist’s knowledge compared to other, conventional deep learningapproaches. To this end, we collect a new dataset consisting of 99 caseswith pancreatic ductal adenocarcinoma (PDAC) and 97 control caseswithout any pancreatic tumor. Due to the growth pattern of pancre-atic cancer, the tumor may not be always visible as a hypodense lesion,therefore experts refer to the visibility of secondary external features thatmay indicate the presence of the tumor. We propose a method based ona U-Net-like Deep CNN that exploits the following external secondaryfeatures: the pancreatic duct, common bile duct and the pancreas, alongwith a processed CT scan. Using these features, the model segmentsthe pancreatic tumor if it is present. This segmentation for classificationand localization approach achieves a performance of 99% sensitivity (onecase missed) and 99% specificity, which realizes a 5% increase in sensi-tivity over the previous state-of-the-art method. The model additionallyprovides location information with reasonable accuracy and a shorterinference time compared to previous PDAC detection methods. Theseresults offer a significant performance improvement and highlight theimportance of incorporating the knowledge of the clinical expert whendeveloping novel CAD methods.

When merging existing similar datasets, it would be attractive to benefit from a higher detection rate of objects and the additional partial ground-truth samples for improving object classification. To this end, a novel CNN detector with a hierarchical binary classification system is proposed. The detector is based on the Single-Shot multibox Detector (SSD) and inspired by the hierarchical classification used in the YOLO9000 detector. Localization and classification are separated during training, by introducing a novel loss term that handles hierarchical classification in the loss function (SSD-ML). We experiment with the proposed SSD-ML detector on the generic PASCAL VOC dataset and show that additional super-categories can be learned with minimal impact on the overall accuracy. Furthermore, we find that not all objects are required to have classification label information as classification performance only drops from 73.3 % to 70.6 % while 60 % of the label information is removed. The flexibility of the detector with respect to the different levels of details in label definitions is investigated for a traffic surveillance application, involving public and proprietary datasets with non-overlapping class definitions. Including classification label information from our dataset raises the performance significantly from 70.7 % to 82.2 %. The experiments show that the desired hierarchical labels can be learned from the public datasets, while only using box information from our dataset. In general, this shows that it is possible to combine existing datasets with similar object classes and partial annotations and benefit in terms of growth of detection rate and improved class categorization performance.

Melanoma is a serious form of skin cancer with high mortality rate at later stages. Fortunately, when detected early, the prognosis of melanoma is promising and malignant melanoma incidence rates are relatively low. As a result, datasets are heavily imbalanced which complicates training current state-of-the-art supervised classification AI models. We propose to use generative models to learn the benign data distribution and detect Out-of-Distribution (OOD) malignant images through density estimation. Normalizing Flows (NFs) are ideal candidates for OOD detection due to their ability to compute exact likelihoods. Nevertheless, their inductive biases towards apparent graphical features rather than semantic context hamper accurate OOD detection. In this work, we aim at using these biases with domain-level knowledge of melanoma, to improve likelihood-based OOD detection of malignant images. Our encouraging results demonstrate potential for OOD detection of melanoma using NFs. We achieve a 9% increase in Area Under Curve of the Receiver Operating Characteristics by using wavelet-based NFs. This model requires significantly less parameters for inference making it more applicable on edge devices. The proposed methodology can aid medical experts with diagnosis of skin-cancer patients and continuously increase survival rates. Furthermore, this research paves the way for other areas in oncology with similar data imbalance issues (Code available at: https://github.com/A-Vzer/WaveletFlowPytorch).

Computer-Aided Diagnosis (CADx) systems can play a crucial role as a second opinion for endoscopists to improve the overall optical diagnostic performance of colonoscopies. While such supportive systems hold great potential, optimal clinical implementation is currently impeded, since deep neural network-based systems often tend to overestimate the confidence about their decisions. In other words, these systems are poorly calibrated, and, hence, may assign high prediction scores to samples associated with incorrect model predictions. For the optimal clinical workflow integration and physician-AI collaboration, a reliable CADx system should provide accurate and well-calibrated classification confidence. An important application of these models is characterization of Colorectal polyps (CRPs), that are potential precursor lesions of Colorectal cancer (CRC). An improved optical diagnosis of CRPs during the colonoscopy procedure is essential for an appropriate treatment strategy. In this paper, we incorporate Bayesian variational inference and investigate the performance of a hybrid Bayesian neural network-based CADx system for the characterization of CRPs. Results of conducted experiments demonstrate that this Bayesian variational inference-based approach is capable of quantifying model uncertainty along with calibration confidence. This framework is able to obtain classification accuracy comparable to the deterministic version of the network, while achieving a 24.65% and 9.14% lower Expected Calibration Error (ECE) compared to the uncalibrated and calibrated deterministic network using a post-processing calibration technique, respectively.

It is important to detect negative behavior of animals for breeding in order to improve their health and welfare. In this work, AI is employed to assist individual animal detection and tracking, which enables the future analysis of behavior for individual animals. The study involves animal groups of pigs and laying hens. First, two state-of-the-art deep learning-based Multi-Object Tracking (MOT) methods are investigated, namely Joint Detection and Embedding (JDE) and FairMOT. Both models detect and track individual animals automatically and continuously. Second, a weighted association algorithm is proposed, which is feasible for both MOT methods to optimize the object re-identification (re-ID), thereby improving the tracking performance. The proposed methods are evaluated on manually annotated datasets. The best tracking performance on pigs is obtained by FairMOT with the weighted association, resulting in an IDF1 of 90.3%, MOTA of 90.8%, MOTP of 83.7%, number of identity switches of 14, and an execution rate of 20.48 fps. For the laying hens, FairMOT with the weighted association also achieves the best tracking performance, with an IDF1 of 88.8%, MOTA of 86.8%, MOTP of 72.8%, number of identity switches of 2, and an execution rate of 21.01 fps. These results show a promising high accuracy and robustness for the individual animal tracking.

Person re-identification (re-ID) is a significant problem of computer vision with increasing scientific attention. To date, numerous studies have been conducted to improve the accuracy and robustness of person re-ID to meet the practical demands. However, most of the previous efforts concentrated on solving the closed-set variant of the problem, where a query is assumed to always have a correct match within the set of known people (the gallery set). However, this assumption is usually not valid for the industrial re-ID use cases. In this study, we focus on the open-set person re-ID problem, where, in addition to the similarity ranking, the solution is expected to detect the presence or absence of a given query identity within the gallery set. To determine good practices and to assess the practicality of the person re-ID in industrial applications, first, we convert popular closed-set person re-ID datasets into the open-set scenario. Second, we compare performance of eight state-of-the-art closed-set person re-ID methods under the open-set conditions. Third, we experimentally determine the efficiency of using different loss function combinations for the open-set problem. Finally, we investigate the impact of a statistics-driven gallery refinement approach on the open-set person re-ID performance in the low false-acceptance rate (FAR) region, while simultaneously reducing the computational demands of retrieval. Results show an average detection and identification rate increase of 8.38% and 3.39% on the DukeMTMC-reID and Market1501 datasets, respectively, for a FAR of 1%.

In the past few years, several datasets that include privacy-sensitive information have been taken offline due to privacy concerns and new laws such as the GDPR regulation in Europe. Medical images contain patient information that is regulated even more strictly. Deep learning has become the standard for Computer Aided Diagnosis (CAD) based on medical images and requires large amounts of data to achieve good performance. The current standard in medical imaging is to print patient text directly over the measurement image, sometimes even partly occluding tissue pixels. This privacy-sensitive patient data is not required for general medical imaging research. Thus, anonymizing these medical images enables researchers to use this data for research purposes. This paper describes a method to automatically detect and remove text from medical images at high processing speeds. We base our method on the EAST text detector [1] and make the following four contributions: (1) open-source implementation of the anonymization tool; (2) method to generate large amounts of synthetic training text; (3) multiple optimizations to improve the processing speed of anonymization

MR-guided high-intensity focused ultrasound (MR-HIFU) is a non-invasive therapeutic technology which has demonstrated clinical potential for tissue ablation. The application of this therapeutic approach facilitated to be a promising option to achieve faster pain palliation in patients with bone metastasis. However, its clinical adoption is still hampered by a lack of workflow integration. Currently, to ensure sufficient positioning, MR images have to be repeatedly acquired in between patient re-positioning tasks, leading to a time-consuming preparation phase of at least 30 minutes, involving extra costs and time to the available treatment time. Augmented reality (AR) is a promising technology that enables the fusion of medical images, such as MRI, with the view of an external camera. AR represents a valid tool for a faster localization and visualization of the lesion during positioning. The aim of this work is the implementation of a novel AR setup for accelerating the patient positioning during MRHIFU treatments by enabling adequate target positioning outside the MRI scanner. A marker-based approach was investigated for fusing the MR data with video data for providing an augmented view. Initial experiments on four volunteers show that MR images were overlaid on the camera views with an average re-projection error of 3.13 mm, which matches the clinical requirements for this specific application. It can be concluded that the implemented system is suitable for MR-HIFU procedures and supports its clinical adoption by improving the patient positioning, thereby offering potential for faster treatment time.

Surgery is a crucial treatment for malignant brain tumors where gross total resection improves the prognosis. Tissue samples taken during surgery are either subject to a preliminary intraoperative histological analysis, or sent for a full pathological evaluation which can take days or weeks. Whereas a lengthy complete pathological analysis includes an array of techniques to be executed, a preliminary tissue analysis on frozen tissue is performed as quickly as possible (30-45 minutes on average) to provide fast feedback to the surgeon during the surgery. The surgeon uses the information to confirm that the resected tissue is indeed tumor and may, at least in theory, initiate repeated biopsies to help achieve gross total resection. However, due to the total turn-around time of the tissue inspection for repeated analyses, this approach may not be feasible during a single surgery. In this context, intraoperative image-guided techniques can improve the clinical workflow for tumor resection and improve outcome by aiding in the identification and removal of the malignant lesion. Hyperspectral imaging (HSI) is an optical imaging technique with the potential to extract combined spectral-spatial information. By exploiting HSI for human brain-tissue classification in 13 in-vivo hyperspectral images from 9 patients, a brain-tissue classifier is developed. The framework consists of a hybrid 3D-2D CNN-based approach and a band-selection step to enhance the capability of extracting both spectral and spatial information from the hyperspectral images. An overall accuracy of 77% was found when tumor, normal and hyper-vascularized tissue are classified, which clearly outperforms the state-of-the-art approaches (SVM, 2D-CNN). These results may open an attractive future perspective for intraoperative brain-tumor classification using HSI.

A Dual-Energy CT (DECT) with a spectral detector greatly extends the capabilities of CT by incorporating energy-dependent information of the X-ray attenuation. In order to fully exploit DECT capabilities, it is required to perform a process known as spectral decomposition. However, this process is sensitive to noise, suffers from reduced photon count per layer in DECT scans and generates anti-correlated noise in the estimated material specific images. In order to overcome these problems, the Anti-Correlated Rudin, Osher and Fatemi (AC-ROF) model is applied for noise reduction, exploiting the relationship between the material-specific images. However, this model deteriorates the structural information with intense noise. In this paper we propose to extend this method by integrating it into an iterative reconstruction procedure to improve the noise reduction performance. The resulting algorithm is called Iterative Reconstruction AC-ROF, or IR-AC-ROF. We have tested AC-ROF and IR-AC-ROF algorithms with realistic brain simulation phantoms and show encouraging results indicating that the resulting material-specific images of IR-AC-ROF can generate better mono-energetic images with improved brain structure visibility. This demonstrates the benefit of including the noise reduction constraints within the reconstruction procedure, rather than using them in a post-processing step.

Peripheral Artery Diseases (PAD) are caused by the occlusions of arteries in the peripheral locations of the circulatory system. The severity of PAD is usually assessed using the Ankle Brachial Index (ABI) and the Ultrasound Doppler. Non-contact Photoplethysmography (PPG) imaging is a recent emerging technology capable of monitoring skin perfusion. Using an off-The-shelf camera and a light source, is possible to remotely detect the dynamic changes in blood volume in the skin and derive a map correlated to the blood perfusion. The aim of this study is the evaluation of a PPG imaging system (iPPG) for the assessment of Peripheral Arterial Diseases. Reduced blood flow is simulated on 21 volunteers by increasing the pressure in a pressure cuff. For each volunteer, measurements with iPPG, ultrasound, Laser Speckle Contrast Analysis (LASCA) and ABI were acquired. Our experiments show that iPPG can detect reduced perfusion levels, and correlates well with the other measurement systems.