Search

There is considerable evidence that evaluating the subjective risk level of driving decisions can improve the safety of Autonomous Driving Systems (ADS) in both typical and complex driving scenarios. In this paper, we propose a novel data-driven approach that uses scene-graphs as intermediate representations for modeling the subjective risk of driving maneuvers. Our approach includes a Multi-Relation Graph Convolution Network, a Long-Short Term Memory Network, and attention layers. To train our model, we formulate subjective risk assessment as a supervised scene classification problem. We evaluate our model on both synthetic lane-changing datasets and real-driving datasets with various driving maneuvers. We show that our approach achieves a higher classification accuracy than the state-of-the-art approach on both large (96.4% vs. 91.2%) and small (91.8% vs. 71.2%) lane-changing synthesized datasets, illustrating that our approach can learn effectively even from small datasets. We also show that our model trained on a lane-changing synthesized dataset achieves an average accuracy of 87.8% when tested on a real-driving lane-changing dataset. In comparison, the state-of-the-art model trained on the same synthesized dataset only achieved 70.3% accuracy when tested on the real-driving dataset, showing that our approach can transfer knowledge more effectively. Moreover, we demonstrate that the addition of spatial and temporal attention layers improves our model's performance and explainability. Finally, our results illustrate that our model can assess the risk of various driving maneuvers more accurately than the state-of-the-art model (86.5% vs. 58.4%, respectively).

This article analyzes the effects of approximate multiplication when performing inferences on deep convolutional neural networks (CNNs). The approximate multiplication can reduce the cost of the underlying circuits so that CNN inferences can be performed more efficiently in hardware accelerators. The study identifies the critical factors in the convolution, fully-connected, and batch normalization layers that allow more accurate CNN predictions despite the errors from approximate multiplication. The same factors also provide an arithmetic explanation of why bfloat16 multiplication performs well on CNNs. The experiments are performed with recognized network architectures to show that the approximate multipliers can produce predictions that are nearly as accurate as the FP32 references, without additional training. For example, the ResNet and Inception-v4 models with Mitch-ww6 multiplication produces Top-5 errors that are within 0.2 percent compared to the FP32 references. A brief cost comparison of Mitch-ww6 against bfloat16 is presented where a MAC operation saves up to 80 percent of energy compared to the bfloat16 arithmetic. The most far-reaching contribution of this article is the analytical justification that multiplications can be approximated while additions need to be exact in CNN MAC operations.

This paper presents a dual camera system for high spatiotemporal resolution (HSTR) video acquisition, where one camera shoots a video with high spatial resolution and low frame rate (HSR-LFR) and another one captures a low spatial resolution and high frame rate (LSR-HFR) video. Our main goal is to combine videos from LSR-HFR and HSR-LFR cameras to create an HSTR video. We propose an end-to-end learning framework, AWnet, mainly consisting of a FlowNet and a FusionNet that learn an adaptive weighting function in pixel domain to combine inputs in a frame recurrent fashion. To improve the reconstruction quality for cameras used in reality, we also introduce noise regularization under the same framework. Our method has demonstrated noticeable performance gains in terms of both objective PSNR measurement in simulation with different publicly available video and light-field datasets and subjective evaluation with real data captured by dual iPhone 7 and Grasshopper3 cameras. Ablation studies are further conducted to investigate and explore various aspects, such as reference structure, camera parallax, exposure time, etc) of our system to fully understand its capability for potential applications.

Lensless cameras provide a framework to build thin imaging systems byreplacing the lens in a conventional camera with an amplitude or phase masknear the sensor. Existing methods for lensless imaging can recover the depthand intensity of the scene, but they require solving computationally-expensiveinverse problems. Furthermore, existing methods struggle to recover densescenes with large depth variations. In this paper, we propose a lenslessimaging system that captures a small number of measurements using differentpatterns on a programmable mask. In this context, we make three contributions.First, we present a fast recovery algorithm to recover textures on a fixednumber of depth planes in the scene. Second, we consider the mask designproblem, for programmable lensless cameras, and provide a design template foroptimizing the mask patterns with the goal of improving depth estimation.Third, we use a refinement network as a post-processing step to identify andremove artifacts in the reconstruction. These modifications are evaluatedextensively with experimental results on a lensless camera prototype toshowcase the performance benefits of the optimized masks and recoveryalgorithms over the state of the art.

Many neurological diseases are characterized by gradual deterioration of brain structure andfunction. Large longitudinal MRI datasets have revealed such deterioration, in part, by applying machine and deep learning to predict diagnosis. A popular approach is to apply Convolutional Neural Networks (CNN) to extract informative features from each visit of the longitudinal MRI and then use those features to classify each visit via Recurrent Neural Networks (RNNs). Such modeling neglects the progressive nature of the disease, which may result in clinically implausible classifications across visits. To avoid this issue, we propose to combine features across visits by coupling feature extraction with a novel longitudinal pooling layer and enforce consistency of the classification across visits in line with disease progression. We evaluate the proposed method on the longitudinal structural MRIs from three neuroimaging datasets: Alzheimer's Disease Neuroimaging Initiative (ADNI, N=404), a dataset composed of 274 normal controls and 329 patients with Alcohol Use Disorder (AUD), and 255 youths from the National Consortium on Alcohol and NeuroDevelopment in Adolescence (NCANDA). In allthree experiments our method is superior to other widely used approaches for longitudinal classification thus making a unique contribution towards more accurate tracking of the impact of conditions on the brain. The code is available at https://github.com/ouyangjiahong/longitudinal-pooling.

The plug-and-play priors (PnP) and regularization by denoising (RED) methodshave become widely used for solving inverse problems by leveraging pre-traineddeep denoisers as image priors. While the empirical imaging performance and thetheoretical convergence properties of these algorithms have been widelyinvestigated, their recovery properties have not previously been theoreticallyanalyzed. We address this gap by showing how to establish theoretical recoveryguarantees for PnP/RED by assuming that the solution of these methods lies nearthe fixed-points of a deep neural network. We also present numerical resultscomparing the recovery performance of PnP/RED in compressive sensing againstthat of recent compressive sensing algorithms based on generative models. Ournumerical results suggest that PnP with a pre-trained artifact removal networkprovides significantly better results compared to the existing state-of-the-artmethods.

Disentangling the sources of visual motion in a dynamic scene during self-movement or ego motion is important for autonomous navigation and tracking. In the dynamic image segments of a video frame containing independently moving objects, optic flow relative to the next frame is the sum of the motion fields generated due to camera and object motion. The traditional ego-motion estimation methods assume the scene to be static, and the recent deep learning-based methods do not separate pixel velocities into object- and ego-motion components. We propose a learning-based approach to predict both ego-motion parameters and object-motion field (OMF) from image sequences using a convolutional autoencoder while being robust to variations due to the unconstrained scene depth. This is achieved by: 1) training with continuous ego-motion constraints that allow solving for ego-motion parameters independently of depth and 2) learning a sparsely activated overcomplete ego-motion field (EMF) basis set, which eliminates the irrelevant components in both static and dynamic segments for the task of ego-motion estimation. In order to learn the EMF basis set, we propose a new differentiable sparsity penalty function that approximates the number of nonzero activations in the bottleneck layer of the autoencoder and enforces sparsity more effectively than L1- and L2-norm-based penalties. Unlike the existing direct ego-motion estimation methods, the predicted global EMF can be used to extract OMF directly by comparing it against the optic flow. Compared with the state-of-the-art baselines, the proposed model performs favorably on pixelwise object- and ego-motion estimation tasks when evaluated on real and synthetic data sets of dynamic scenes.

PurposeTo organize a multi-institute knee MRI segmentation challenge for characterizing the semantic and clinical efficacy of automatic segmentation methods relevant for monitoring osteoarthritis progression.Materials and methodsA dataset partition consisting of three-dimensional knee MRI from 88 retrospective patients at two time points (baseline and 1-year follow-up) with ground truth articular (femoral, tibial, and patellar) cartilage and meniscus segmentations was standardized. Challenge submissions and a majority-vote ensemble were evaluated against ground truth segmentations using Dice score, average symmetric surface distance, volumetric overlap error, and coefficient of variation on a holdout test set. Similarities in automated segmentations were measured using pairwise Dice coefficient correlations. Articular cartilage thickness was computed longitudinally and with scans. Correlation between thickness error and segmentation metrics was measured using the Pearson correlation coefficient. Two empirical upper bounds for ensemble performance were computed using combinations of model outputs that consolidated true positives and true negatives.ResultsSix teams (T 1-T 6) submitted entries for the challenge. No differences were observed across any segmentation metrics for any tissues (P = .99) among the four top-performing networks (T 2, T 3, T 4, T 6). Dice coefficient correlations between network pairs were high (> 0.85). Per-scan thickness errors were negligible among networks T 1-T 4 (P = .99), and longitudinal changes showed minimal bias (< 0.03 mm). Low correlations (ρ < 0.41) were observed between segmentation metrics and thickness error. The majority-vote ensemble was comparable to top-performing networks (P = .99). Empirical upper-bound performances were similar for both combinations (P = .99).ConclusionDiverse networks learned to segment the knee similarly, where high segmentation accuracy did not correlate with cartilage thickness accuracy and voting ensembles did not exceed individual network performance.See also the commentary by Elhalawani and Mak in this issue.Keywords: Cartilage, Knee, MR-Imaging, Segmentation © RSNA, 2020Supplemental material is available for this article.

Distortion quantification of point clouds plays a stealth, yet vital role ina wide range of human and machine perception tasks. For human perception tasks,a distortion quantification can substitute subjective experiments to guide 3Dvisualization; while for machine perception tasks, a distortion quantificationcan work as a loss function to guide the training of deep neural networks forunsupervised learning tasks. To handle a variety of demands in manyapplications, a distortion quantification needs to be distortion discriminable,differentiable, and have a low computational complexity. Currently, however,there is a lack of a general distortion quantification that can satisfy allthree conditions. To fill this gap, this work proposes multiscale potentialenergy discrepancy (MPED), a distortion quantification to measure point cloudgeometry and color difference. By evaluating at various neighborhood sizes, theproposed MPED achieves global-local tradeoffs, capturing distortion in amultiscale fashion. Extensive experimental studies validate MPED's superiorityfor both human and machine perception tasks.

In this paper, we propose a new distortion quantification method for pointclouds, the multiscale potential energy discrepancy (MPED). Currently, there isa lack of effective distortion quantification for a variety of point cloudperception tasks. Specifically, for dense point clouds, a distortionquantification method is used to predict human subjective scores and optimizethe selection of human perception tasks parameters, such as compression andenhancement. For sparse point clouds, a distortion quantification methods iswork as a loss function to guide the training of deep neural networks forunsupervised learning tasks (e.g., point cloud reconstruction, completion andupsampling). Therefore, an effective distortion quantification should bedifferentiable, distortion discriminable and have a low computationalcomplexity. However, current distortion quantification cannot satisfy all threeconditions. To fill this gap, we propose a new point cloud feature descriptionmethod, the point potential energy (PPE), inspired by the classical physics. Weregard the point clouds are systems that have potential energy and thedistortion can change the total potential energy. By evaluating at variousneighborhood sizes, the proposed MPED achieves global-local tradeoffs,capturing distortion in a multiscale fashion. We further theoretically showthat classical Chamfer distance is a special case of our MPED. Extensiveexperiments show the proposed MPED superior to current methods on both humanand machine perception tasks. Our code is avaliable athttps://github.com/Qi-Yangsjtu/MPED.

The Posit Number System was introduced in 2017 as a replacement forfloating-point numbers. Since then, the community has explored its applicationin Neural Network related tasks and produced some unit designs which are stillfar from being competitive with their floating-point counterparts. This paperproposes a Posit Logarithm-Approximate Multiplication (PLAM) scheme tosignificantly reduce the complexity of posit multipliers, the most power-hungryunits within Deep Neural Network architectures. When comparing withstate-of-the-art posit multipliers, experiments show that the proposedtechnique reduces the area, power, and delay of hardware multipliers up to72.86%, 81.79%, and 17.01%, respectively, without accuracy degradation.

Retinal degenerative diseases cause profound visual impairment in more than10 million people worldwide, and retinal prostheses are being developed torestore vision to these individuals. Analogous to cochlear implants, thesedevices electrically stimulate surviving retinal cells to evoke visual percepts(phosphenes). However, the quality of current prosthetic vision is stillrudimentary. Rather than aiming to restore "natural" vision, there is potentialmerit in borrowing state-of-the-art computer vision algorithms as imageprocessing techniques to maximize the usefulness of prosthetic vision. Here wecombine deep learning--based scene simplification strategies with apsychophysically validated computational model of the retina to generaterealistic predictions of simulated prosthetic vision, and measure their abilityto support scene understanding of sighted subjects (virtual patients) in avariety of outdoor scenarios. We show that object segmentation may bettersupport scene understanding than models based on visual saliency and monoculardepth estimation. In addition, we highlight the importance of basingtheoretical predictions on biologically realistic models of phosphene shape.Overall, this work has the potential to drastically improve the utility ofprosthetic vision for people blinded from retinal degenerative diseases.

Fiber-reinforced ceramic-matrix composites are advanced materials resistantto high temperatures, with application to aerospace engineering. Their analysisdepends on the detection of embedded fibers, with semi-supervised techniquesusually employed to separate fibers within the fiber beds. Here we present anopen computational pipeline to detect fibers in ex-situ X-ray computedtomography fiber beds. To separate the fibers in these samples, we tested fourdifferent architectures of fully convolutional neural networks. When comparingour neural network approach to a semi-supervised one, we obtained Dice andMatthews coefficients greater than $92.28 \pm 9.65\%$, reaching up to $98.42\pm 0.03 \%$, showing that the network results are close to thehuman-supervised ones in these fiber beds, in some cases separating fibers thathuman-curated algorithms could not find. The software we generated in thisproject is open source, released under a permissive license, and can be freelyadapted and re-used in other domains. All data and instructions on how todownload and use it are also available.

Most person re-identification methods, being supervised techniques, suffer from the burden of massive annotation requirement. Unsupervised methods overcome this need for labeled data, but perform poorly compared to the supervised alternatives. In order to cope with this issue, we introduce the problem of learning person re-identification models from videos with weak supervision. The weak nature of the supervision arises from the requirement of video-level labels, i.e. person identities who appear in the video, in contrast to the more precise frame-level annotations. Towards this goal, we propose a multiple instance attention learning framework for person re-identification using such video-level labels. Specifically, we first cast the video person re-identification task into a multiple instance learning setting, in which person images in a video are collected into a bag. The relations between videos with similar labels can be utilized to identify persons, on top of that, we introduce a co-person attention mechanism which mines the similarity correlations between videos with person identities in common. The attention weights are obtained based on all person images instead of person tracklets in a video, making our learned model less affected by noisy annotations. Extensive experiments demonstrate the superiority of the proposed method over the related methods on two weakly labeled person re-identification datasets.

A plethora of research advances have emerged in the fields of optics and photonics that benefit from harnessing the power of machine learning. Specifically, there has been a revival of interest in optical computing hardware due to its potential advantages for machine learning tasks in terms of parallelization, power efficiency and computation speed. Diffractive deep neural networks (D2NNs) form such an optical computing framework that benefits from deep learning-based design of successive diffractive layers to all-optically process information as the input light diffracts through these passive layers. D2NNs have demonstrated success in various tasks, including object classification, the spectral encoding of information, optical pulse shaping and imaging. Here, we substantially improve the inference performance of diffractive optical networks using feature engineering and ensemble learning. After independently training 1252 D2NNs that were diversely engineered with a variety of passive input filters, we applied a pruning algorithm to select an optimized ensemble of D2NNs that collectively improved the image classification accuracy. Through this pruning, we numerically demonstrated that ensembles of N = 14 and N = 30 D2NNs achieve blind testing accuracies of 61.14 ± 0.23% and 62.13 ± 0.05%, respectively, on the classification of CIFAR-10 test images, providing an inference improvement of >16% compared to the average performance of the individual D2NNs within each ensemble. These results constitute the highest inference accuracies achieved to date by any diffractive optical neural network design on the same dataset and might provide a significant leap to extend the application space of diffractive optical image classification and machine vision systems.

Our computational developments and analyses on experimental images are designed to evaluate the effectiveness of chemical spraying via unmanned aerial vehicle (UAV). Our evaluations are in accord with the two perspectives of color-complexity: color variety within a color system and color distributional geometry on an image. First, by working within RGB and HSV color systems, we develop a new color-identification algorithm relying on highly associative relations among three color-coordinates to lead us to exhaustively identify all targeted color-pixels. A color-dot is then identified as one isolated network of connected color-pixel. All identified color-dots vary in shapes and sizes within each image. Such a pixel-based computing algorithm is shown robustly and efficiently accommodating heterogeneity due to shaded regions and lighting conditions. Secondly, all color-dots with varying sizes are categorized into three categories. Since the number of small color-dot is rather large, we spatially divide the entire image into a 2D lattice of rectangular. As such, each rectangle becomes a collective of color-dots of various sizes and is classified with respect to its color-dots intensity. We progressively construct a series of minimum spanning trees (MST) as multiscale 2D distributional spatial geometries in a decreasing-intensity fashion. We extract the distributions of distances among connected rectangle-nodes in the observed MST and simulated MSTs generated under the spatial uniformness assumption. We devise a new algorithm for testing 2D spatial uniformness based on a Hierarchical clustering tree upon all involving MSTs. This new tree-based p-value evaluation has the capacity to become exact.

Traditionally, the regime of mental healthcare has followed an episodicpsychotherapy model wherein patients seek care from a provider through aprescribed treatment plan developed over multiple provider visits. Recentadvances in wearable and mobile technology have generated increased interest indigital mental healthcare that enables individuals to address episodic mentalhealth symptoms. However, these efforts are typically reactive andsymptom-focused and do not provide comprehensive, wrap-around, customizedtreatments that capture an individual's holistic mental health model as itunfolds over time. Recognizing that each individual is unique, we present thenotion of Personalized Mental Health Navigation (MHN): a therapist-in-the-loop,cybernetic goal-based system that deploys a continuous cyclic loop ofmeasurement, estimation, guidance, to steer the individual's mental healthstate towards a healthy zone. We outline the major components of MHN that ispremised on the development of an individual's personal mental health state,holistically represented by a high-dimensional cover of multiple knowledgelayers such as emotion, biological patterns, sociology, behavior, andcognition. We demonstrate the feasibility of the personalized MHN approach viaa 12-month pilot case study for holistic stress management in college studentsand highlight an instance of a therapist-in-the-loop intervention using MHN formonitoring, estimating, and proactively addressing moderately severe depressionover a sustained period of time. We believe MHN paves the way to transformmental healthcare from the current passive, episodic, reactive process (whereindividuals seek help to address symptoms that have already manifested) to acontinuous and navigational paradigm that leverages a personalized model of theindividual, promising to deliver timely interventions to individuals in aholistic manner.

Histological staining is a vital step in diagnosing various diseases and has been used for more than a century to provide contrast in tissue sections, rendering the tissue constituents visible for microscopic analysis by medical experts. However, this process is time consuming, labour intensive, expensive and destructive to the specimen. Recently, the ability to virtually stain unlabelled tissue sections, entirely avoiding the histochemical staining step, has been demonstrated using tissue-stain-specific deep neural networks. Here, we present a new deep-learning-based framework that generates virtually stained images using label-free tissue images, in which different stains are merged following a micro-structure map defined by the user. This approach uses a single deep neural network that receives two different sources of information as its input: (1) autofluorescence images of the label-free tissue sample and (2) a "digital staining matrix", which represents the desired microscopic map of the different stains to be virtually generated in the same tissue section. This digital staining matrix is also used to virtually blend existing stains, digitally synthesizing new histological stains. We trained and blindly tested this virtual-staining network using unlabelled kidney tissue sections to generate micro-structured combinations of haematoxylin and eosin (H&E), Jones' silver stain, and Masson's trichrome stain. Using a single network, this approach multiplexes the virtual staining of label-free tissue images with multiple types of stains and paves the way for synthesizing new digital histological stains that can be created in the same tissue cross section, which is currently not feasible with standard histochemical staining methods.

Polarized light microscopy provides high contrast to birefringent specimen and is widely used as a diagnostic tool in pathology. However, polarization microscopy systems typically operate by analyzing images collected from two or more light paths in different states of polarization, which lead to relatively complex optical designs, high system costs, or experienced technicians being required. Here, we present a deep learning-based holographic polarization microscope that is capable of obtaining quantitative birefringence retardance and orientation information of specimen from a phase-recovered hologram, while only requiring the addition of one polarizer/analyzer pair to an inline lensfree holographic imaging system. Using a deep neural network, the reconstructed holographic images from a single state of polarization can be transformed into images equivalent to those captured using a single-shot computational polarized light microscope (SCPLM). Our analysis shows that a trained deep neural network can extract the birefringence information using both the sample specific morphological features as well as the holographic amplitude and phase distribution. To demonstrate the efficacy of this method, we tested it by imaging various birefringent samples including, for example, monosodium urate and triamcinolone acetonide crystals. Our method achieves similar results to SCPLM both qualitatively and quantitatively, and due to its simpler optical design and significantly larger field-of-view this method has the potential to expand the access to polarization microscopy and its use for medical diagnosis in resource limited settings.

In many intelligent systems, a network of agents collaboratively perceivesthe environment for better and more efficient situation awareness. As theseagents often have limited resources, it could be greatly beneficial to identifythe content overlapping among camera views from different agents and leverageit for reducing the processing, transmission and storage ofredundant/unimportant video frames. This paper presents a consensus-baseddistributed multi-agent video fast-forwarding framework, named DMVF, thatfast-forwards multi-view video streams collaboratively and adaptively. In ourframework, each camera view is addressed by a reinforcement learning basedfast-forwarding agent, which periodically chooses from multiple strategies toselectively process video frames and transmits the selected frames atadjustable paces. During every adaptation period, each agent communicates witha number of neighboring agents, evaluates the importance of the selected framesfrom itself and those from its neighbors, refines such evaluation together withother agents via a system-wide consensus algorithm, and uses such evaluation todecide their strategy for the next period. Compared with approaches in theliterature on a real-world surveillance video dataset VideoWeb, our methodsignificantly improves the coverage of important frames and also reduces thenumber of frames processed in the system.

Fourier phase retrieval is a classical problem that deals with the recoveryof an image from the amplitude measurements of its Fourier coefficients.Conventional methods solve this problem via iterative (alternating)minimization by leveraging some prior knowledge about the structure of theunknown image. The inherent ambiguities about shift and flip in the Fouriermeasurements make this problem especially difficult; and most of the existingmethods use several random restarts with different permutations. In this paper,we assume that a known (learned) reference is added to the signal beforecapturing the Fourier amplitude measurements. Our method is inspired by theprinciple of adding a reference signal in holography. To recover the signal, weimplement an iterative phase retrieval method as an unrolled network. Then weuse back propagation to learn the reference that provides us the bestreconstruction for a fixed number of phase retrieval iterations. We performed anumber of simulations on a variety of datasets under different conditions andfound that our proposed method for phase retrieval via unrolled network andlearned reference provides near-perfect recovery at fixed (small) computationalcost. We compared our method with standard Fourier phase retrieval methods andobserved significant performance enhancement using the learned reference.

In most real world scenarios, a policy trained by reinforcement learning inone environment needs to be deployed in another, potentially quite differentenvironment. However, generalization across different environments is known tobe hard. A natural solution would be to keep training after deployment in thenew environment, but this cannot be done if the new environment offers noreward signal. Our work explores the use of self-supervision to allow thepolicy to continue training after deployment without using any rewards. Whileprevious methods explicitly anticipate changes in the new environment, weassume no prior knowledge of those changes yet still obtain significantimprovements. Empirical evaluations are performed on diverse simulationenvironments from DeepMind Control suite and ViZDoom, as well as real roboticmanipulation tasks in continuously changing environments, taking observationsfrom an uncalibrated camera. Our method improves generalization in 31 out of 36environments across various tasks and outperforms domain randomization on amajority of environments.

Recent advances in Convolutional Neural Network (CNN) model interpretabilityhave led to impressive progress in visualizing and understanding modelpredictions. In particular, gradient-based visual attention methods have drivenmuch recent effort in using visual attention maps as a means for visualexplanations. A key problem, however, is these methods are designed forclassification and categorization tasks, and their extension to explaininggenerative models, e.g. variational autoencoders (VAE) is not trivial. In thiswork, we take a step towards bridging this crucial gap, proposing the firsttechnique to visually explain VAEs by means of gradient-based attention. Wepresent methods to generate visual attention from the learned latent space, andalso demonstrate such attention explanations serve more than just explainingVAE predictions. We show how these attention maps can be used to localizeanomalies in images, demonstrating state-of-the-art performance on the MVTec-ADdataset. We also show how they can be infused into model training, helpingbootstrap the VAE into learning improved latent space disentanglement,demonstrated on the Dsprites dataset.

Even as deep neural networks have become very effective for tasks in vision and perception, it remains difficult to explain and debug their behavior. In this paper, we present a programmatic and semantic approach to explaining, understanding, and debugging the correct and incorrect behaviors of a neural network-based perception system. Our approach is semantic in that it employs a high-level representation of the distribution of environment scenarios that the detector is intended to work on. It is programmatic in that scenario representation is a program in a domain-specific probabilistic programming language which can be used to generate synthetic data to test a given perception module. Our framework assesses the performance of a perception module to identify correct and incorrect detections, extracts rules from those results that semantically characterizes the correct and incorrect scenarios, and then specializes the probabilistic program with those rules in order to more precisely characterize the scenarios in which the perception module operates correctly or not. We demonstrate our results using the SCENIC probabilistic programming language and a neural network-based object detector. Our experiments show that it is possible to automatically generate compact rules that significantly increase the correct detection rate (or conversely the incorrect detection rate) of the network and can thus help with understanding and debugging its behavior.

Most of the existing works in video synthesis focus on generating videos using adversarial learning. Despite their success, these methods often require input reference frame or fail to generate diverse videos from the given data distribution, with little to no uniformity in the quality of videos that can be generated. Different from these methods, we focus on the problem of generating videos from latent noise vectors, without any reference input frames. To this end, we develop a novel approach that jointly optimizes the input latent space, the weights of a recurrent neural network and a generator through non-adversarial learning. Optimizing for the input latent space along with the network weights allows us to generate videos in a controlled environment, i.e., we can faithfully generate all videos the model has seen during the learning process as well as new unseen videos. Extensive experiments on three challenging and diverse datasets well demonstrate that our proposed approach generates superior quality videos compared to the existing state-of-the-art methods.

Purpose: Proximal femur image analyses based on quantitative computedtomography (QCT) provide a method to quantify the bone density and evaluateosteoporosis and risk of fracture. We aim to develop a deep-learning-basedmethod for automatic proximal femur segmentation. Methods and Materials: Wedeveloped a 3D image segmentation method based on V-Net, an end-to-end fullyconvolutional neural network (CNN), to extract the proximal femur QCT imagesautomatically. The proposed V-net methodology adopts a compound loss function,which includes a Dice loss and a L2 regularizer. We performed experiments toevaluate the effectiveness of the proposed segmentation method. In theexperiments, a QCT dataset which included 397 QCT subjects was used. For theQCT image of each subject, the ground truth for the proximal femur wasdelineated by a well-trained scientist. During the experiments for the entirecohort then for male and female subjects separately, 90% of the subjects wereused in 10-fold cross-validation for training and internal validation, and toselect the optimal parameters of the proposed models; the rest of the subjectswere used to evaluate the performance of models. Results: Visual comparisondemonstrated high agreement between the model prediction and ground truthcontours of the proximal femur portion of the QCT images. In the entire cohort,the proposed model achieved a Dice score of 0.9815, a sensitivity of 0.9852 anda specificity of 0.9992. In addition, an R2 score of 0.9956 (p

In uncertain domains, the goals are often unknown and need to be predicted by the organism or system. In this paper, contrastive Excitation Backprop (c-EB) was used in two goal-driven perception tasks - one with pairs of noisy MNIST digits and the other with a robot in an action-based attention scenario. The first task included attending to even, odd, low, and high digits, whereas the second task included action goals, such as "eat", "work-on-computer", "read", and "say-hi" that led to attention to objects associated with those actions. The system needed to increase attention to target items and decrease attention to distractor items and background noise. Because the valid goal was unknown, an online learning model based on the cholinergic and noradrenergic neuromodulatory systems was used to predict a noisy goal (expected uncertainty) and re-adapt when the goal changed (unexpected uncertainty). This neurobiologically plausible model demonstrates how neuromodulatory systems can predict goals in uncertain domains and how attentional mechanisms can enhance the perception for that goal.

Knee Osteoarthritis (OA) is a common musculoskeletal disorder in the United States. When diagnosed at early stages, lifestyle interventions such as exercise and weight loss can slow OA progression, but at later stages, only an invasive option is available: total knee replacement (TKR). Though a generally successful procedure, only 2/3 of patients who undergo the procedure report their knees feeling "normal" post-operation, and complications can arise that require revision. This necessitates a model to identify a population at higher risk of TKR, particularly at less advanced stages of OA, such that appropriate treatments can be implemented that slow OA progression and delay TKR. Here, we present a deep learning pipeline that leverages MRI images and clinical and demographic information to predict TKR with AUC 0.834 ± 0.036 (p

We introduce the isoperimetric loss as a regularization criterion for learning the map from a visual representation to a semantic embedding, to be used to transfer knowledge to unknown classes in a zero-shot learning setting. We use a pre-trained deep neural network model as a visual representation of image data, a Word2Vec embedding of class labels, and linear maps between the visual and semantic embedding spaces. However, the spaces themselves are not linear, and we postulate the sample embedding to be populated by noisy samples near otherwise smooth manifolds. We exploit the graph structure defined by the sample points to regularize the estimates of the manifolds by inferring the graph connectivity using a generalization of the isoperimetric inequalities from Riemannian geometry to graphs. Surprisingly, this regularization alone, paired with the simplest baseline model, outperforms the state-of-the-art among fully automated methods in zero-shot learning benchmarks such as AwA and CUB. This improvement is achieved solely by learning the structure of the underlying spaces by imposing regularity.

In recent years, numerous effective multi-object tracking (MOT) methods aredeveloped because of the wide range of applications. Existing performanceevaluations of MOT methods usually separate the object tracking step from theobject detection step by using the same fixed object detection results forcomparisons. In this work, we perform a comprehensive quantitative study on theeffects of object detection accuracy to the overall MOT performance, using thenew large-scale University at Albany DETection and tRACking (UA-DETRAC)benchmark dataset. The UA-DETRAC benchmark dataset consists of 100 challengingvideo sequences captured from real-world traffic scenes (over 140,000 frameswith rich annotations, including occlusion, weather, vehicle category,truncation, and vehicle bounding boxes) for object detection, object trackingand MOT system. We evaluate complete MOT systems constructed from combinationsof state-of-the-art object detection and object tracking methods. Our analysisshows the complex effects of object detection accuracy on MOT systemperformance. Based on these observations, we propose new evaluation tools andmetrics for MOT systems that consider both object detection and object trackingfor comprehensive analysis.

PurposeTo investigate the feasibility of automatic identification and classification of hip fractures using deep learning, which may improve outcomes by reducing diagnostic errors and decreasing time to operation.Materials and methodsHip and pelvic radiographs from 1118 studies were reviewed, and 3026 hips were labeled via bounding boxes and classified as normal, displaced femoral neck fracture, nondisplaced femoral neck fracture, intertrochanteric fracture, previous open reduction and internal fixation, or previous arthroplasty. A deep learning-based object detection model was trained to automate the placement of the bounding boxes. A Densely Connected Convolutional Neural Network (or DenseNet) was trained on a subset of the bounding box images, and its performance was evaluated on a held-out test set and by comparison on a 100-image subset with two groups of human observers: fellowship-trained radiologists and orthopedists; senior residents in emergency medicine, radiology, and orthopedics.ResultsThe binary accuracy for detecting a fracture of this model was 93.7% (95% confidence interval [CI]: 90.8%, 96.5%), with a sensitivity of 93.2% (95% CI: 88.9%, 97.1%) and a specificity of 94.2% (95% CI: 89.7%, 98.4%). Multiclass classification accuracy was 90.8% (95% CI: 87.5%, 94.2%). When compared with the accuracy of human observers, the accuracy of the model achieved an expert-level classification, at the very least, under all conditions. Additionally, when the model was used as an aid, human performance improved, with aided resident performance approximating unaided fellowship-trained expert performance in the multiclass classification.ConclusionA deep learning model identified and classified hip fractures with expert-level performance, at the very least, and when used as an aid, improved human performance, with aided resident performance approximating that of unaided fellowship-trained attending physicians.Supplemental material is available for this article.© RSNA, 2020.

As recent neural networks are being improved to be more accurate, their model's size is exponentially growing. Thus, a huge number of parameters requires to be loaded and stored from/in memory hierarchy and computed in processors to perform training or inference phase of neural network processing. Increasing the number of parameters causes a big challenge for real-time deployment since the memory bandwidth improvement's trend cannot keep up with models' complexity growing trend. Although some operations in neural networks processing are computational intensive such as convolutional layer computing, computing dense layers face with memory bandwidth bottleneck. To address the issue, the paper has proposed Partition Pruning for dense layers to reduce the required parameters while taking into consideration parallelization. We evaluated the performance and energy consumption of parallel inference of partitioned models, which showed a 7.72x speedup of performance and a 2.73x reduction in the energy used for computing pruned fully connected layers in TinyVGG16 model in comparison to running the unpruned model on a single accelerator. Besides, our method showed a limited reduction in accuracy while partitioning fully connected layers.

High-resolution (HR) magnetic resonance imaging (MRI) provides detailedanatomical information that is critical for diagnosis in the clinicalapplication. However, HR MRI typically comes at the cost of long scan time,small spatial coverage, and low signal-to-noise ratio (SNR). Recent studiesshowed that with a deep convolutional neural network (CNN), HR generic imagescould be recovered from low-resolution (LR) inputs via single imagesuper-resolution (SISR) approaches. Additionally, previous works have shownthat a deep 3D CNN can generate high-quality SR MRIs by using learned imagepriors. However, 3D CNN with deep structures, have a large number of parametersand are computationally expensive. In this paper, we propose a novel 3D CNNarchitecture, namely a multi-level densely connected super-resolution network(mDCSRN), which is light-weight, fast and accurate. We also show that with thegenerative adversarial network (GAN)-guided training, the mDCSRN-GAN providesappealing sharp SR images with rich texture details that are highly comparablewith the referenced HR images. Our results from experiments on a large publicdataset with 1,113 subjects showed that this new architecture outperformedother popular deep learning methods in recovering 4x resolution-downgradedimages in both quality and speed.

As recent neural networks are being improved to be more accurate, their model's size is exponentially growing. Thus, a huge number of parameters requires to be loaded and stored from/in memory hierarchy and computed in processors to perform training or inference phase of neural network processing. Increasing the number of parameters causes a big challenge for real-time deployment since the memory bandwidth improvement's trend cannot keep up with models' complexity growing trend. Although some operations in neural networks processing are computational intensive such as convolutional layer computing, computing dense layers face with memory bandwidth bottleneck. To address the issue, the paper has proposed Partition Pruning for dense layers to reduce the required parameters while taking into consideration parallelization. We evaluated the performance and energy consumption of parallel inference of partitioned models, which showed a 7.72x speedup of performance and a 2.73x reduction in the energy used for computing pruned fully connected layers in TinyVGG16 model in comparison to running the unpruned model on a single accelerator. Besides, our method showed a limited reduction in accuracy while partitioning fully connected layers.

An image dataset of 10 different size molecules, where each molecule has 2,000 structural variants, is generated from the 2D cross-sectional projection of Molecular Dynamics trajectories. The purpose of this dataset is to provide a benchmark dataset for the increasing need of machine learning, deep learning and image processing on the study of scattering, imaging and microscopy.

Mask-based lensless cameras replace the lens of a conventional camera with a custom mask. These cameras can potentially be very thin and even flexible. Recently, it has been demonstrated that such mask-based cameras can recover light intensity and depth information of a scene. Existing depth recovery algorithms either assume that the scene consists of a small number of depth planes or solve a sparse recovery problem over a large 3D volume. Both these approaches fail to recover the scenes with large depth variations. In this paper, we propose a new approach for depth estimation based on an alternating gradient descent algorithm that jointly estimates a continuous depth map and light distribution of the unknown scene from its lensless measurements. We present simulation results on image and depth reconstruction for a variety of 3D test scenes. A comparison between the proposed algorithm and other method shows that our algorithm is more robust for natural scenes with a large range of depths. We built a prototype lensless camera and present experimental results for reconstruction of intensity and depth maps of different real objects.

Generative models have received considerable attention in signal processing and compressive sensing for their ability to generate high-dimensional natural image using low-dimensional codes. In the context of compressive sensing, if the unknown image belongs to the range of a pretrained generative network, then we can recover the image by estimating the underlying compact latent code from the available measurements. In practice, however, a given pretrained generator can only reliably generate images that are similar to the training data. To overcome this challenge, a number of methods have been proposed recently to use untrained generator structure as prior while solving the signal recovery problem. In this paper, we propose a similar method for jointly updating the weights of the generator and latent codes while recovering a video sequence from compressive measurements. We use a single generator to generate the entire video. To exploit the temporal redundancy in a video sequence, we use a low-rank constraint on the latent codes that imposes a low-dimensional manifold model on the generated video sequence. We evaluate the performance of our proposed methods on different video compressive sensing problems under different settings and compared them against some state-of-the-art methods. Our results demonstrate that our proposed methods provide better or comparable accuracy and low computational and memory complexity compared to the existing methods.

Computer-Aided-Diagnosis (CADx) systems assist radiologists with identifyingand classifying potentially malignant pulmonary nodules on chest CT scans usingmorphology and texture-based (radiomic) features. However, radiomic featuresare sensitive to differences in acquisitions due to variations in dose levelsand slice thickness. This study investigates the feasibility of generating anormalized scan from heterogeneous CT scans as input. We obtained projectiondata from 40 low-dose chest CT scans, simulating acquisitions at 10%, 25% and50% dose and reconstructing the scans at 1.0mm and 2.0mm slice thickness. A 3Dgenerative adversarial network (GAN) was used to simultaneously normalizereduced dose, thick slice (2.0mm) images to normal dose (100%), thinner slice(1.0mm) images. We evaluated the normalized image quality using peaksignal-to-noise ratio (PSNR), structural similarity index (SSIM) and LearnedPerceptual Image Patch Similarity (LPIPS). Our GAN improved perceptualsimilarity by 35%, compared to a baseline CNN method. Our analysis also showsthat the GAN-based approach led to a significantly smaller error (p-value <0.05) in nine studied radiomic features. These results indicated that GANscould be used to normalize heterogeneous CT images and reduce the variabilityin radiomic feature values.

Even as deep neural networks have become very effective for tasks in vision and perception, it remains difficult to explain and debug their behavior. In this paper, we present a programmatic and semantic approach to explaining, understanding, and debugging the correct and incorrect behaviors of a neural network-based perception system. Our approach is semantic in that it employs a high-level representation of the distribution of environment scenarios that the detector is intended to work on. It is programmatic in that scenario representation is a program in a domain-specific probabilistic programming language which can be used to generate synthetic data to test a given perception module. Our framework assesses the performance of a perception module to identify correct and incorrect detections, extracts rules from those results that semantically characterizes the correct and incorrect scenarios, and then specializes the probabilistic program with those rules in order to more precisely characterize the scenarios in which the perception module operates correctly or not. We demonstrate our results using the SCENIC probabilistic programming language and a neural network-based object detector. Our experiments show that it is possible to automatically generate compact rules that significantly increase the correct detection rate (or conversely the incorrect detection rate) of the network and can thus help with understanding and debugging its behavior.

In high-stakes applications of machine learning models, interpretabilitymethods provide guarantees that models are right for the right reasons. Inmedical imaging, saliency maps have become the standard tool for determiningwhether a neural model has learned relevant robust features, rather thanartefactual noise. However, saliency maps are limited to local modelexplanation because they interpret predictions on an image-by-image basis. Wepropose aggregating saliency globally, using semantic segmentation masks, toprovide quantitative measures of model bias across a dataset. To evaluateglobal saliency methods, we propose two metrics for quantifying the validity ofsaliency explanations. We apply the global saliency method to skin lesiondiagnosis to determine the effect of artefacts, such as ink, on model bias.

We propose regularization strategies for learning discriminative models thatare robust to in-class variations of the input data. We use the Wasserstein-2geometry to capture semantically meaningful neighborhoods in the space ofimages, and define a corresponding input-dependent additive noise dataaugmentation model. Expanding and integrating the augmented loss yields aneffective Tikhonov-type Wasserstein diffusion smoothness regularizer. Thisapproach allows us to apply high levels of regularization and train functionsthat have low variability within classes but remain flexible across classes. Weprovide efficient methods for computing the regularizer at a negligible cost incomparison to training with adversarial data augmentation. Initial experimentsdemonstrate improvements in generalization performance under adversarialperturbations and also large in-class variations of the input data.

We consider the problem of accurately identifying cell boundaries and labeling individual cells in confocal microscopy images, specifically, 3D image stacks of cells with tagged cell membranes. Precise identification of cell boundaries, their shapes, and quantifying inter-cellular space leads to a better understanding of cell morphogenesis. Towards this, we outline a cell segmentation method that uses a deep neural network architecture to extract a confidence map of cell boundaries, followed by a 3D watershed algorithm and a final refinement using a conditional random field. In addition to improving the accuracy of segmentation compared to other state-of-the-art methods, the proposed approach also generalizes well to different datasets without the need to retrain the network for each dataset. Detailed experimental results are provided, and the source code is available on GitHub 1.

While deep learning methods have demonstrated performance comparable to human readers in tasks such as computer-aided diagnosis, these models are difficult to interpret, do not incorporate prior domain knowledge, and are often considered as a "black-box." The lack of model interpretability hinders them from being fully understood by end users such as radiologists. In this paper, we present a novel interpretable deep hierarchical semantic convolutional neural network (HSCNN) to predict whether a given pulmonary nodule observed on a computed tomography (CT) scan is malignant. Our network provides two levels of output: 1) low-level semantic features; and 2) a high-level prediction of nodule malignancy. The low-level outputs reflect diagnostic features often reported by radiologists and serve to explain how the model interprets the images in an expert-interpretable manner. The information from these low-level outputs, along with the representations learned by the convolutional layers, are then combined and used to infer the high-level output. This unified architecture is trained by optimizing a global loss function including both low- and high-level tasks, thereby learning all the parameters within a joint framework. Our experimental results using the Lung Image Database Consortium (LIDC) show that the proposed method not only produces interpretable lung cancer predictions but also achieves significantly better results compared to using a 3D CNN alone.

With advanced image journaling tools, one can easily alter the semantic meaning of an image by exploiting certain manipulation techniques such as copy clone, object splicing, and removal, which mislead the viewers. In contrast, the identification of these manipulations becomes a very challenging task as manipulated regions are not visually apparent. This paper proposes a high-confidence manipulation localization architecture that utilizes resampling features, long short-term memory (LSTM) cells, and an encoder-decoder network to segment out manipulated regions from non-manipulated ones. Resampling features are used to capture artifacts, such as JPEG quality loss, upsampling, downsampling, rotation, and shearing. The proposed network exploits larger receptive fields (spatial maps) and frequency-domain correlation to analyze the discriminative characteristics between the manipulated and non-manipulated regions by incorporating the encoder and LSTM network. Finally, the decoder network learns the mapping from low-resolution feature maps to pixel-wise predictions for image tamper localization. With the predicted mask provided by the final layer (softmax) of the proposed architecture, end-to-end training is performed to learn the network parameters through back-propagation using the ground-truth masks. Furthermore, a large image splicing dataset is introduced to guide the training process. The proposed method is capable of localizing image manipulations at the pixel level with high precision, which is demonstrated through rigorous experimentation on three diverse datasets.

We propose a novel, simple and effective method to integrate lesion prior anda 3D U-Net for improving brain tumor segmentation. First, we utilize theground-truth brain tumor lesions from a group of patients to generate theheatmaps of different types of lesions. These heatmaps are used to create thevolume-of-interest (VOI) map which contains prior information about brain tumorlesions. The VOI map is then integrated with the multimodal MR images and inputto a 3D U-Net for segmentation. The proposed method is evaluated on a publicbenchmark dataset, and the experimental results show that the proposed featurefusion method achieves an improvement over the baseline methods. In addition,our proposed method also achieves a competitive performance compared tostate-of-the-art methods.

For visual tracking methods based on kernel support vector machines (SVMs), data sampling is usually adopted to reduce the computational cost in training. In addition, budgeting of support vectors is required for computational efficiency. Instead of sampling and budgeting, recently the circulant matrix formed by dense sampling of translated image patches has been utilized in kernel correlation filters for fast tracking. In this paper, we derive an equivalent formulation of a SVM model with the circulant matrix expression and present an efficient alternating optimization method for visual tracking. We incorporate the discrete Fourier transform with the proposed alternating optimization process, and pose the tracking problem as an iterative learning of support correlation filters (SCFs). In the fully-supervision setting, our SCF can find the globally optimal solution with real-time performance. For a given circulant data matrix with n2 samples of n ×n pixels, the computational complexity of the proposed algorithm is O(n2 logn) whereas that of the standard SVM-based approaches is at least O(n4). In addition, we extend the SCF-based tracking algorithm with multi-channel features, kernel functions, and scale-adaptive approaches to further improve the tracking performance. Experimental results on a large benchmark dataset show that the proposed SCF-based algorithms perform favorably against the state-of-the-art tracking methods in terms of accuracy and speed.

We have developed a convolutional neural network that can make a pixel-level prediction of objects in image data recorded by a liquid argon time projection chamber (LArTPC) for the first time. We describe the network design, training techniques, and software tools developed to train this network. The goal of this work is to develop a complete deep neural network based data reconstruction chain for the MicroBooNE detector. We show the first demonstration of a network's validity on real LArTPC data using MicroBooNE collection plane images. The demonstration is performed for stopping muon and a νμ charged-current neutral pion data samples.

The effective identification of clouds and monitoring of their evolution are important toward more accurate quantitative precipitation estimation and forecast. In this study, a new gradient-based cloud-image segmentation algorithm is developed using image processing techniques. This method integrates morphological image gradient magnitudes to separate cloud systems and patches boundaries. A varying scale kernel is implemented to reduce the sensitivity of image segmentation to noise and to capture objects with various finenesses of the edges in remote sensing images. The proposed method is flexible and extendable from single to multispectral imagery. Case studies were carried out to validate the algorithm by applying the proposed segmentation algorithm to synthetic radiances for channels of the Geostationary Operational Environmental Satellite (GOES-16) simulated by a high-resolution weather prediction model. The proposed method compares favorably with the existing cloud-patch-based segmentation technique implemented in the Precipitation Estimation from Remotely Sensed Information Using Artificial Neural Networks-Cloud Classification System (PERSIANN-CCS) rainfall retrieval algorithm. Evaluation of event-based images indicates that the proposed algorithm has potentials comparing to the conventional segmentation technique used in PERSIANN-CCS to improve rain detection and estimation skills with an accuracy rate of up to 98% in identifying cloud regions.

We introduce the isoperimetric loss as a regularization criterion for learning the map from a visual representation to a semantic embedding, to be used to transfer knowledge to unknown classes in a zero-shot learning setting. We use a pre-trained deep neural network model as a visual representation of image data, a Word2Vec embedding of class labels, and linear maps between the visual and semantic embedding spaces. However, the spaces themselves are not linear, and we postulate the sample embedding to be populated by noisy samples near otherwise smooth manifolds. We exploit the graph structure defined by the sample points to regularize the estimates of the manifolds by inferring the graph connectivity using a generalization of the isoperimetric inequalities from Riemannian geometry to graphs. Surprisingly, this regularization alone, paired with the simplest baseline model, outperforms the state-of-the-art among fully automated methods in zero-shot learning benchmarks such as AwA and CUB. This improvement is achieved solely by learning the structure of the underlying spaces by imposing regularity.