Search

Summary: This work introduces the EXSCLAIM! toolkit for the automatic extraction, separation, and caption-based natural language annotation of images from scientific literature. EXSCLAIM! is used to show how rule-based natural language processing and image recognition can be leveraged to construct an electron microscopy dataset containing thousands of keyword-annotated nanostructure images. Moreover, it is demonstrated how a combination of statistical topic modeling and semantic word similarity comparisons can be used to increase the number and variety of keyword annotations on top of the standard annotations from EXSCLAIM! With large-scale imaging datasets constructed from scientific literature, users are well positioned to train neural networks for classification and recognition tasks specific to microscopy—tasks often otherwise inhibited by a lack of sufficient annotated training data. The bigger picture: Due to recent improvements in image resolution and acquisition speed, materials microscopy is experiencing an explosion of published imaging data. The standard publication format, while sufficient for data ingestion scenarios where a selection of images can be critically examined and curated manually, is not conducive to large-scale data aggregation or analysis, hindering data sharing and reuse. Most images in publications are part of a larger figure, with their explicit context buried in the main body or caption text; so even if aggregated, collections of images with weak or no digitized contextual labels have limited value. The tool developed in this work establishes a scalable pipeline for meaningful image-/language-based information curation from scientific literature.

Tactile sensing is essential for a variety of daily tasks. Inspired by the event-driven nature and sparse spiking communication of the biological systems, recent advances in event-driven tactile sensors and Spiking Neural Networks (SNNs) spur the research in related fields. However, SNN-enabled event-driven tactile learning is still in its infancy due to the limited representation abilities of existing spiking neurons and high spatio-temporal complexity in the event-driven tactile data. In this paper, to improve the representation capability of existing spiking neurons, we propose a novel neuron model called “location spiking neuron,” which enables us to extract features of event-based data in a novel way. Specifically, based on the classical Time Spike Response Model (TSRM), we develop the Location Spike Response Model (LSRM). In addition, based on the most commonly-used Time Leaky Integrate-and-Fire (TLIF) model, we develop the Location Leaky Integrate-and-Fire (LLIF) model. Moreover, to demonstrate the representation effectiveness of our proposed neurons and capture the complex spatio-temporal dependencies in the event-driven tactile data, we exploit the location spiking neurons to propose two hybrid models for event-driven tactile learning. Specifically, the first hybrid model combines a fully-connected SNN with TSRM neurons and a fully-connected SNN with LSRM neurons. And the second hybrid model fuses the spatial spiking graph neural network with TLIF neurons and the temporal spiking graph neural network with LLIF neurons. Extensive experiments demonstrate the significant improvements of our models over the state-of-the-art methods on event-driven tactile learning, including event-driven tactile object recognition and event-driven slip detection. Moreover, compared to the counterpart artificial neural networks (ANNs), our SNN models are 10× to 100× energy-efficient, which shows the superior energy efficiency of our models and may bring new opportunities to the spike-based learning community and neuromorphic engineering. Finally, we thoroughly examine the advantages and limitations of various spiking neurons and discuss the broad applicability and potential impact of this work on other spike-based learning applications.

One of the holy grails of neuroscience is to record the activity of every neuron in the brain while an animal moves freely and performs complex behavioral tasks. While important steps forward have been taken recently in large-scale neural recording in rodent models, single neuron resolution across the entire mammalian brain remains elusive. In contrast the larval zebrafish offers great promise in this regard. Zebrafish are a vertebrate model with substantial homology to the mammalian brain, but their transparency allows whole-brain recordings of genetically-encoded fluorescent indicators at single-neuron resolution using optical microscopy techniques. Furthermore zebrafish begin to show a complex repertoire of natural behavior from an early age, including hunting small, fast-moving prey using visual cues. Until recently work to address the neural bases of these behaviors mostly relied on assays where the fish was immobilized under the microscope objective, and stimuli such as prey were presented virtually. However significant progress has recently been made in developing brain imaging techniques for zebrafish which are not immobilized. Here we discuss recent advances, focusing particularly on techniques based on light-field microscopy. We also draw attention to several important outstanding issues which remain to be addressed to increase the ecological validity of the results obtained.

Here, the authors present Synthetic Wavelength Holography, an approach for Non-Line-of-Sight imaging. By exploiting spectral correlations in scattered light, the authors transform real world surfaces such as walls or scatterers into High-Resolution, Wide-Field-of-View imaging portals that provide holograms of objects obscured from view.

Abstract Optical coherence tomography (OCT) is an optical technique which allows for volumetric visualization of the internal structures of translucent materials. Additional information can be gained by measuring the rate of signal attenuation in depth. Techniques have been developed to estimate the rate of attenuation on a voxel by voxel basis. This depth resolved attenuation analysis gives insight into tissue structure and organization in a spatially resolved way. However, the presence of speckle in the OCT measurement causes the attenuation coefficient image to contain unrealistic fluctuations and makes the reliability of these images at the voxel level poor. While the distribution of speckle in OCT images has appeared in literature, the resulting voxelwise corruption of the attenuation analysis has not. In this work, the estimated depth resolved attenuation coefficient from OCT data with speckle is shown to be approximately exponentially distributed. After this, a prior distribution for the depth resolved attenuation coefficient is derived for a simple system using statistical mechanics. Finally, given a set of depth resolved estimates which were made from OCT data in the presence of speckle, a posterior probability distribution for the true voxelwise attenuation coefficient is derived and a Bayesian voxelwise estimator for the coefficient is given. These results are demonstrated in simulation and validated experimentally.

Abstract This work presents the development of a flexible microscopic chemical imaging platform for cultural heritage that utilizes wavelength-tunable oblique illumination from a point source to obtain per-pixel reflectance spectra in the VIS–NIR range. The microscope light source can be adjusted on two axes allowing for a hemisphere of possible illumination directions. The synthesis of multiple illumination angles allows for the calculation of surface normal vectors, similar to phase gradients, and axial optical sectioning. The extraction of spectral reflectance images with high spatial resolutions from these data is demonstrated through the analysis of a replica cross-section, created from known painting reference materials, as well as a sample extracted from a painting by Pablo Picasso entitled La Miséreuse accroupie (1902). These case studies show the rich microscale molecular information that may be obtained using this microscope and how the instrument overcomes challenges for spectral analysis commonly encountered on works of art with complex matrices composed of both inorganic minerals and organic lakes.