Search

The rapid advancement of machine learning and artificial intelligence (AI) has transformednumerous fields, including microscopy imaging, where it forms the foundation of com- putational microscopy. This approach overcomes the physical limitations of traditional optical microscopy, simplifying imaging systems and enabling breakthroughs across various imaging modalities, such as bright-field, fluorescence, and holography microscopy. Compu- tational microscopy has significantly advanced tasks such as super-resolution, refocusing, and volumetric imaging. However, concerns remain regarding the generalizability and reliability of AI models, particularly when applied to real-world scenarios in fields like biological research and medical diagnostics. In the first part of my dissertation, novel methodologies have been developed to further enhance the capability of AI in computational microscopy, including a convo- lutional recurrent neural network designed to efficiently utilize sequential data and a Fourier-domain-based network, termed FIN, which demonstrates superior generalization to previously unseen sample types. Next, a physics-informed, self-supervised learning framework for inverse problems in computational microscopy has been introduced, whose effectiveness is validated in holographic imaging. A notable innovation is GedankenNet, a network trained exclusively on artificially generated data that generalizes effectively to real-world samples without requiring transfer learning or fine-tuning. This innovation stands for a significant step toward generalizable AI in computational microscopy. To address the concern regarding reliability of AI in computational microscopy, an uncertainty quantification technique based on forward-backward cycles, termed AQuA, has been proposed, enabling the detection of failure modes in virtual staining models with superhuman accuracy. AQuA’s application further extends to quality assessment for both virtual staining and histochemical stained images. These advancements contribute toward the development of generalizable and reliable AI systems in computational microscopy, offering significant potential for future research and practical applications in biological and medical imaging.

Fluorescence microscopy has become an indispensable technology to push fundamental neuroscience by recovering labeled neural structures with high resolution. To enable these studies, the field has adopted the use of low-cost widefield 1-photon epi-fluorescence microscopes to image fixed samples and miniaturized head-mounted miniscopes to monitor neural activity in freely behaving animals. However, fluorescence imaging platforms face a number of challenges such as a limited depth of field (DoF), lack of optical sectioning, and susceptibility to scattering and aberrations which compromises the image quality and signal fidelity. As a result, neural studies are often constrained to a shallow volume near the surface of the sample and are limited by high noise and background. To overcome these challenges, this thesis introduces two novel frameworks that combine pupil engineering with computational imaging to push the performance of miniaturized and tabletop fluorescence neural imaging platforms. These strategies will directly optimize and integrate custom phase elements on the often-vacant pupil plane to enable the encoding of extended fluorescence signals by designing a point spread function (PSF) that exhibits an extended depth of field (EDoF) in scattering media. Next, these strategies will use tailored post processing algorithms to recover that extended information from the resulting images. As a result, this strategy allows for the recovery of sources in an extended neural volume without compromising the optical resolution or imaging speed on the underlying platform. First, this thesis introduces EDoF-Miniscope, a miniaturized neural imaging platform which utilizes a novel physics-informed genetic algorithm to optimize a lightweight binary diffractive optical element (DOE) on the pupil plane. By integrating the binary DOE into a prototype platform, EDoF-Miniscope is able to achieve a 2.8x extension in the DoF between twin imaging foci in neural samples. To enable the recovery of the extended sources, this thesis utilizes a straightforward post-processing filter, which can recover neuronal signals with an SBR down to 1.08. Overall, this framework introduces a generalizable, compact and lightweight solution for augmenting miniscopes with a computational EDoF. Next, I improve upon the proposed framework by designing a flexible 1-photon widefield tabletop platform, entitled EDoF-Tabletop, that exhibits comparable field-of-view (FoV, FoV = 0.6x0.6mm), numerical aperture (NA, NA = 0.5) and aberrations to a miniscope. This platform utilizes a spatial light modulator (SLM) on the pupil plane to rapidly deploy optimized pupil phase profiles without the need of manufacturing, aligning and integrating miniaturized optics. EDoF-Tabletop incorporates a deep optics pipeline, which utilizes novel physical modeling, initialization and training strategies to simultaneously and reliably learn a user-defined EDoFs and a reconstruction using synthetic-only data. As a result, EDoF-Tabletop is able to encode and recover signals from EDoFs up to 140-microns deep in neural samples and 400-microns deep in non-scattering samples. By combining pupil engineering with computational imaging, EDoF-Miniscope and EDoF-Tabletop showcase the potential to enhance neural imaging platforms by extracting information from extended volumes in the brain. By focusing on flexible optimization algorithms and rapid prototyping capabilities, the advancements introduced in this thesis promise broader utility across fluorescence microscopy, where capturing detailed information from complex biological samples is essential for advancing scientific understanding.

The continuous advancement in microscopy has been unveiling the hidden world of tissues, cells, and molecules. In the quest for deeper spatiotemporal insights into biological processes, light field microscopy (LFM) has emerged as a powerful and intriguing tool. Unlike traditional imaging systems that capture focused images, LFM records multiplexed signals with single snapshot that encodes information within a three-dimensional (3D) volume. By leveraging computational reconstruction algorithms, this approach enables the observation of transient volumetric dynamics with remarkable efficiency and speed. This thesis presents a series of efforts to apply LFM in functional imaging, enabling researchers to monitor real-time changes in live organisms, including ion fluxes, electrical signaling, and cells interactions. The exceptional temporal resolution makes LFM a unique tool to visualize and analyze rapid processes that are difficult to capture with conventional 3D microscopy. We demonstrated calcium imaging of motor neurons in freely moving C. elegans and tracked flowing blood cells in-vivo within a beating zebrafish heart. The excessive and unpredictable motion observed in these processes requires capturing hundreds of 3D volumes per second, a demanding but necessary task that provides insights into the underlying mechanisms of neural and cardiac functions. We further moved forward to voltage imaging, a frontier in neuroscience, which directly measures the neural action potential as well as sub-thresholding activities. Our LFM provides kilohertz volumetric imaging on leech ganglion and mouse hippocampus. It measures 7.3 gigavoxels per second in a 3D field of view of 550×550×300 μm^3, which makes it capable of recording the accurate timing and waveform of neural spikes across entire volume. These demonstrations are achieved through several innovative redesigns of LFM, detailed in Chapter 3 to 5. The first approach, VCD-LFM, addresses the inherent trade-off between spatial resolution and depth information in light field imaging by introducing a learning-based reconstruction algorithm. By incorporating data priors and constraints, this method aims to mitigate the issues of low spatial resolution and artifacts in conventional LFM without compromising imaging speed. The second approach, Squeezed Light Field Microscopy (SLIM), leverages data redundancy in light fields and revises the optical hardware to achieve kilohertz volume rate. Designed to meet the high-speed demands of voltage imaging, SLIM offers a powerful and robust imaging tool for sparse volumetric processes. Lastly, the third approach, Light Field Tomography (LIFT), adapts LFM for one-dimensional (1D) measurements through optical Radon transformation. This method enables the use of low-dimensional detectors, such as line sensor, to capture high-dimensional light fields, resulting in enhanced sensitivity, reduced cost and even greater temporal resolution.

Most imaging systems were developed to capture images for static objects that do not move during the image acquisition time. As a result, motion is considered as a primary source of imaging artifacts, which limits the observation of fast moving samples. The most common way to suppress motion artifacts is to shorten the acquisition time, thereby minimizing motion during the observation. However, this comes with a cost of less signal and increased noise in the measurements. While most imaging systems assume static scenes and well-calibrated system motion during image acquisition, this thesis pioneers an alternative approach that algorithmically estimates unknown motion. With accurate motion estimation, we can computationally correct motion artifacts during the image reconstruction process, opening opportunities to design imaging systems specifically for dynamic scenes. The core idea of this thesis is to simultaneously reconstruct both the object and its motion using optimization techniques. We find this approach to be effective and versatile, demonstrating it across various imaging modalities, object scales, and applications. This joint object and motion optimization can be done in post-processing without altering image acquisition, enabling the reconstruction of dynamic scenes from existing datasets affected by motion artifacts. In addition to recovering dynamic information from static imaging systems, we also explore an opposite problem: recovering static scenes using an event camera that only detects changes in the scene. By studying the triggering mechanism of noise events, we develop a statistical noise model for the event camera that explains its illuminance-dependent noise characteristics. With this understanding, we propose to form an image of a static scene using only noise events, providing rich contextual information about static scenes to the dynamic sensor, without requiring any change on its hardware.This thesis demonstrates these concepts using various novel imaging systems, including an event camera, a lensless camera, a quantitative phase microscope, a 3D refractive-index microscope, and a super-resolution fluorescence microscope. By accurately modeling both unknown motion and noise, we aim to demonstrate how computational methods can bridge the gap between static and dynamic imaging.

With the continuous advancement of imaging technologies, imaging devices are no longer limited to the exclusive measurement of optical intensity (at the expense of all other degrees of freedom of the incident light) in a standard single-aperture configuration. Increasingly demanding applications are currently driving the exploration of more complex imaging capabilities, such as phase contrast imaging, wave front sensing, optical spatial filtering, and compound-eye vision. Many of these applications also require highly integrated, lightweight, and compact designs without sacrificing performance. Thanks to recent developments in micro- and nanophotonics, planar devices such as metasurfaces have emerged as a powerful new paradigm to construct optical elements with extreme miniaturization and high design flexibility. Sophisticated simulation tools and high-resolution fabrication techniques have also become available to enable the implementation of these compact subwavelength structures in academic and industrial labs. In this dissertation, I will present my work aimed at achieving directional light sensing by directly integrating composite plasmonic metasurfaces on the illumination windows of standard planar photodetectors. The devices developed in this work feature sharp detection peaks in their angular response with three different types of behaviors: symmetric around the device surface normal, asymmetric with nearly linear angular variations around normal incidence, and geometrically tunable single peaks up to over 60 degrees. The performance of the proposed metasurfaces has been optimized by full-wave numerical simulations, and experimental devices have been fabricated and tested with a custom-designed measurement setup. The measured angular characteristics were then used to computationally demonstrate incoherent edge enhancement for computer vision and quantitative phase-contrast imaging for biomedical microscopy. Importantly, the device fabrication process has also been upgraded to wafer scale, further promoting the possibility of batch-production of our devices.

In recent years, significant progress has been made in passive non-line-of-sight imaging, which looks around corners at hidden objects using just scattered light off a rough surface. Since passive non-line-of-sight imaging can obtain information about the surrounding environment that was previously deemed irrecoverable, it has many far-reaching applications, such as improving autonomous vehicles, aiding search-and-rescue operations, and performing military surveillance. While the bulk of progress has focused on improving the resolution and capabilities of existing non-line-of-sight imaging algorithms, most methods assume a high signal-to-noise ratio and low amounts of background signals. These conditions are impossible to find outside of highly-controlled laboratories and do not correspond to real-world applications. This thesis considers "low-signal'' passive non-line-of-sight imaging, where the desired imaging signal is either swamped by stochastic noise or structured background signals that interfere with conventional non-line-of-sight reconstructions. These scenarios mimic what is found in real-world environments. To image in these difficult low-signal scenarios, this thesis delves into two main works: unmixing spectral content of the scattered light and denoising thermal scattered imagery. As a result of these methods, non-line-of-sight imaging can be performed in many scenarios previously deemed too difficult, and future low-signal imaging can build off the principles laid in this thesis.

Image Deconvolution is a well-studied problem that seeks to restore the original sharp image from a blurry image formed in the imaging system. The Point Spread function(PSF) of a particular system can be used to infer the original sharp image given the blurred image. However, such a problem is usually simplified by making the shift-invariant assumption over the Field of View (FOV). Realistic systems are shift-variant; the optical system’s point spread function depends on the position of the object point from the principal axis. For example, asymmetrical lenses can cause space variant aberration. In this paper, we first simulate our shift-variant aberrations by generating Point Spread Functions using the Seidel Aberration polynomial and use a shift-variant forward blur model to generate our shift-variant blurred image pairs. We then introduce, ShiVaNet. It is a two-stage architecture that builds upon the Learnable Wiener Deconvolution block as described in Yanny, Monakhova, Shuai, and Waller (Yanny et al.) by introducing Simplified Channel Attention and Transpose Attention to improve the performance of the module. We also devise a novel UNet refinement block by fusing a ConvNext-V2 block with Channel Attention and coupling with Transposed Attention Zamir, Arora, Khan, Hayat, Khan, and Yang (Zamir et al.). Our model performs better than state-of-the-art restoration models by a factor of 0.2 dB Peak Signal to Noise Ratio.

Optical computing and information processing, known for its energy efficiency, scalability, low latency, and parallelism, stands at the forefront of revolutionizing computational systems and machine vision. Among different optical computing designs, diffractive optical neural networks represent a free-space-based framework that can be used to perform computation, statistical inference, and inverse design of optical elements. A diffractive neural network is composed of multiple transmissive and/or reflective diffractive layers (or surfaces), which leverage light-matter interactions to jointly perform modulation of the input light field to generate the desired output field. These passive diffractive layers, each containing numerous spatially engineered diffractive features, are optimized in a computer using deep-learning tools. Post-training, these layers form a passive optical processing unit powered solely by illumination light, capable of processing based on 2D/3D input information at the speed of light and analyzing various light properties e.g., amplitude, phase and polarization. Building upon these foundations, this thesis presents multiple strategies and designs to enhance the performance and capacity of diffractive networks for all-optical information processing. First, a differential detection scheme is introduced to improve the classification performance by addressing the non-negativity constraint in opto-electronic detectors. Additionally, the parallel training of multiple networks, each targeting different object classes or trained diversely, further boosts the overall inference performance. Extending the framework to harnessing broadband light, the encoding of spatial information into the power spectrum of diffracted light is presented, facilitating classification and reconstruction with a single-pixel spectroscopic detector. Furthermore, diffractive networks have been theoretically and numerically proven as a universal linear transformer, capable of all-optically implementing arbitrary complex-valued linear transformations between its input and output field-of-views. Introducing the wavelength multiplexing scheme into a broadband diffractive processor allows simultaneous execution of a large group of linear transformations across different wavelength channels, thereby enhancing the computational throughput of the framework. Another significant advancement is made with the integration of polarization-sensitive elements into isotropic diffractive networks, which enables universal polarization transformations of spatially varying polarization fields, surpassing the limitations of traditional polarization control methods. In summary, this thesis significantly advances diffractive neural networks as a versatile all-optical information processing platform that is ideal for efficient, high-speed and intelligent machine vision systems, also laying the groundwork for future innovations.

The rapid development of machine learning has transformed conventional optical imaging processes, setting new benchmarks in computational imaging tasks. In this dissertation, we delve into the transformative impact of recent advancements in machine learning on optical imaging processes, focusing on how these technologies revolutionize computational imaging tasks. Specifically, this dissertation centers on two major topics: deep learning-enabled computational microscopy and the all-optical diffractive networks. Optical microscopy has long served as the benchmark technique for diagnosing various diseases over centuries. However, its reliance on high-end optical components and accessories, necessary to adapt to various imaging samples and conditions, often limits its applicability and throughput. Recent advancements in computational imaging techniques utilizing deep learning methods have transformed conventional microscopic imaging modalities, delivering both enhanced speed and superior image quality without introducing extra complexity of the optical systems. In the first topic of this dissertation, we demonstrate that deep learning-enabled image translation approach can significantly benefit a wide range of applications for microscopic imaging. We start with introducing a customized system for single-shot quantitative polarization imaging, capable of reconstructing comprehensive birefringent maps from a single image capture, which offers enhanced sensitivity and specificity in diagnosing crystal-induced diseases. Utilizing these quantitative birefringent maps as a baseline, we employ deep learning tools to convert phase-recovered holograms into quantitative birefringence maps, thereby improving the throughput of crystal detection with simplified system complexity. Extending this concept of deep learning-enabled image translation, we also explore its applications in histopathology. Our technique, termed as “virtual histological staining”, transforms unstained biological samples into visually rich, stained-like images without the need for chemical agents. This innovation minimizes costs, labor, and diagnostic delays as well as opens up new possibilities in histopathology workflow. The evolution of deep learning tools not only facilities the optical image analysis and processing, but also provides guidance in design and enhancement of optical systems. The second topic of this dissertation is the development and application of diffractive deep neural networks (D2NN). Developed with deep learning, D2NNs execute given computational tasks by manipulating light diffraction through a series of engineered surfaces, which is completed at the speed of light propagation with negligible power consumption. Based on this framework, a lot of novel computational tasks can be executed in an all-optical way, which is beyond the capabilities of the traditional optics design approaches. We introduce several all-optical computational imaging applications based on D2NN, including class-specific imaging, class-specific image encryption, and unidirectional image magnification and demagnification, demonstrating the versatility of this promising framework.

Label-free imaging modalities offer numerous advantages, such as the ability to avoid the time-consuming and potentially disruptive process of physical staining. However, one challenge that arises in label-free imaging is the limited ability to extract specific structural or molecular information from the acquired images. To overcome this limitation, a novel approach known as digital staining or digital labeling has emerged. Digital staining leverages the power of deep learning algorithms to virtually introduce labels or stains into label-free images, thereby enabling the extraction of detailed information that would typically require physical staining. The integration of digital staining with label-free imaging holds great promise in expanding the capabilities of imaging techniques, facilitating improved analysis, and advancing our understanding of biological systems at both the cellular and tissue level. In this thesis, I explore supervised and semi-supervised methodologies of digital staining and the applications in augmenting label-free imaging modalities, particularly in the context of cell imaging and brain imaging. In the first part of the thesis, I demonstrate the novel integration of multi-contrast dark-field reflectance microscopy and supervised deep learning to enable subcellular immunofluorescence labeling and cell cytometry from label-free imaging. By leveraging the rich structural information and sensitivity of reflectance microscopy, this method accurately predicts subcellular features without the need for physical staining. As a result of the use of a novel multi-contrast modality, the digital labeling approach demonstrates significant improvements over the state-of-the-art techniques, achieving up to 3× prediction accuracy. In addition to fluorescence prediction, the method successfully reproduces single-cell level structural phenotypes related to cell cycles. The multiplexed readouts obtained through digital labeling enable accurate multi-parametric single-cell profiling across a large cell population. In the second part, I investigated a novel digital staining optical coherence tomography (DS-OCT) modality combining advantages of serial sectioning OCT and semi-supervised deep learning and demonstrated several advantages for the application of 3D histological brain imaging. The DS model is trained using a semi-supervised learning framework that incorporates unpaired translation, a biophysical model, and cross-modality image registration, which manifests broad applicability to other weakly-paired bioimaging modalities. The DS model enables the translation of S-OCT images to Gallyas silver staining, providing consistent staining quality across different samples. I further show that DS enhances contrast across cortical layer boundaries and enables reliable cortical layer differentiation. Additionally, DS-OCT preserves 3D-geometry on centimeter-scale brain tissue blocks. My pilot study demonstrates promising results on other anatomical regions acquired from different S-OCT systems, highlighting its potential for generalization in various imaging contexts. Overall, I investigate the problems of augmenting label-free imaging modalities with deep learning generated digital stains. I explored both supervised and semi-supervised methods for building novel DS frameworks. My work showcased two important applications in the field of immunofluorescence cell imaging and 3D histological brain imaging. On the one hand, the integration of DS techniques with multi-contrast microscopy has the potential to enhance the throughput of single-cell imaging cytometry, and phenotyping. On the other hand, integrating DS techniques with S-OCT holds great potential for high-throughput human brain imaging, enabling comprehensive studies on the structure and function of the brain. Through the exploration, I aim to shed light on the impact of digital staining in the field of computational imaging and its implications for various scientific disciplines.

Over the past decade, the possibility of forming images of objects hidden from line-of-sight (LOS) view has emerged as an intriguing and potentially important expansion of computational imaging and computer vision technology. This capability could help soldiers anticipate danger in a tunnel system, autonomous vehicles avoid collision, and first responders safely traverse a building. In many scenarios where non-line-of-sight (NLOS) vision is desired, the LOS view is obstructed by a wall with a vertical edge. In this thesis we show that through modeling and computation, the impediment to LOS itself can be exploited for enhanced resolution of the hidden scene. NLOS methods may be active, where controlled illumination of the hidden scene is used, or passive, relying only on already present light sources. In both active and passive NLOS imaging, measured light returns to the sensor after multiple diffuse bounces. Each bounce scatters light in all directions, eliminating directional information. When the scene is hidden behind a wall with a vertical edge, that edge occludes light as a function of its incident azimuthal angle around the edge. Measurements acquired on the floor adjacent to the occluding edge thus contain rich azimuthal information about the hidden scene. In this thesis, we explore several edge-resolved NLOS imaging systems that exploit the occlusion provided by a vertical edge. In addition to demonstrating novel edge-resolved NLOS imaging systems with real experimental data, this thesis includes modeling, performance bound analyses, and inversion algorithms for the proposed systems. We first explore the use of a single vertical edge to form a 1D (in azimuthal angle) reconstruction of the hidden scene. Prior work demonstrated that temporal variation in a video of the floor may be used to image moving components of the hidden scene. In contrast, our algorithm reconstructs both moving and stationary hidden scenery from a single photograph, without assuming uniform floor albedo. We derive a forward model that describes the measured photograph as a nonlinear combination of the unknown floor albedo and the light from behind the wall. The inverse problem, which is the joint estimation of floor albedo and a 1D reconstruction of the hidden scene, is solved via optimization, where we introduce regularizers that help separate light variations in the measured photograph due to floor pattern and hidden scene, respectively. Next, we combine the resolving power of a vertical edge with information from the relationship between intensity and radial distance to form 2D reconstructions from a single passive photograph. We derive a new forward model, accounting for radial falloff, and propose two inversion algorithms to form 2D reconstructions from a single photograph of the penumbra. The performances of both algorithms are demonstrated on experimental data corresponding to several different hidden scene configurations. A Cramer-Rao bound analysis further demonstrates the feasibility and limitations of this 2D corner camera. Our doorway camera exploits the occlusion provided by the two vertical edges of a doorway for more robust 2D reconstruction of the hidden scene. This work provides and demonstrates a novel inversion algorithm to jointly estimate two views of change in the hidden scene, using the temporal difference between photographs acquired on the visible side of the doorway. A Cramer-Rao bound analysis is used to demonstrate the 2D resolving power of the doorway camera over other passive acquisition strategies and to motivate the novel biangular reconstruction grid. Lastly, we present the active corner camera. Most existing active NLOS methods illuminate the hidden scene using a pulsed laser directed at a relay surface and collect time-resolved measurements of returning light. The prevailing approaches are inherently limited by the need for laser scanning, a process that is generally too slow to image hidden objects in motion. Methods that avoid laser scanning track the moving parts of the hidden scene as one or two point targets. In this work, based on more complete optical response modeling yet still without multiple illumination positions, we demonstrate accurate reconstructions of objects in motion and a `map’ of the stationary scenery behind them. This new ability to count, localize, and characterize the sizes of hidden objects in motion, combined with mapping of the stationary hidden scene could greatly improve indoor situational awareness in a variety of applications.

Multi-modal camera-radar fusion has improved performance in 3D depth reconstruction, 3D object detection, and 3D object localization, as well as robustness against adverse sensing conditions and sensor failure. However, existing multi-modal fusion methods are not designed for joint optimization of performance and bias in medical human-centered sensing. This is a problem for clinical remote plethysmography, the contactless monitoring of blood flow information via subtle skin color variations (camera-based imaging photoplethysmography) or surface vibrations on the chest wall (radar plethysmography). Prior work has established a Pareto trade-off in performance and bias between these two modalities, where millimeter wave (mmWave) radar signals exhibit less performance bias across skin tones, but camera-based signals provide superior blood flow estimation performance. We propose a near Pareto optimal multi-modal fusion framework that leverages a debiasing discriminator architecture to jointly optimize over both performance and bias. We show state-of-the-art performance and equity results relative to competitive benchmarks. We also open-source our custom-built millimeter wave radar platform and the largest multi-modal remote heart-rate estimation dataset of paired camera and radar measurements with a focus on skin tone representation.

Remote plethysmography, also known as contactless heart rate monitoring, has gained significant prominence, with contemporary research mitigating errors to a few beats per minute. However, these techniques have not explored finer facets such as skin tone bias and out-of-distribution generalization. This, therefore, becomes a barrier to deploying these systems. This work aims to alleviate these concerns and take the first step towards fairer and generalizable remote plethysmography systems. Here, we show the existence of bias against darker skin tones and further propose a solution, through hardware and software alterations, to mitigate this bias. We employ a novel multimodal fusion framework consisting of a camera and radar to mitigate the bias in camera-based approaches. Furthermore, we also propose a novel camera-based approach to enable generalization to out-of-distribution data. We aim to mitigate false ``hallucinations'', enabling inference in real-world settings. We propose a solution through the perspective of functional decompositions motivated by implicit networks. In contrast to contemporary learning-based techniques, our model generalizes to unseen and challenging data samples.

Contactless vital sensing is gaining prominence with applications in disease control, health monitoring, and medicine; airports are beginning to use infrared thermometers to screen for fevers, automobile companies are researching how cars can wirelessly detect drowsy drivers, and the medical field is exploring the benefits of how cameras can be used to remotely monitor neonates or detect diseases such as atrial fibrillation or sleep apnea. However, prior research to a large extent has not explored when remote vital sensing methods fail and if they may be disadvantageous to certain physiologies more than others such as age, weight, or gender. New methods in the field should strive to determine the impact of these variables as well as rectify inaccuracies in sensing that may occur if possible. This work explores how skin tone can adversely impact heart-rate detection with cameras and temperature evaluation with thermal cameras. Multimodal fusion and algorithmic techniques are proposed to improve skin tone equity while improving performance of contactless vital sensing methods.

Magnetic resonance imaging (MRI) is an effective imaging modality offering tremendous benefits to both science and medicine. It provides exceptional contrast for visualizing soft tissue, can capture images from any orientation, and does not involve any ionizing radiation. Its remarkable versatility enables a wide range of applications, including assessing blood flow, imaging brain activity with functional MRI (fMRI), and quantifying susceptibility mapping, ushering in a new era of clinical diagnosis and brain research. However, due to its physics limitations, acquiring MRI data is inherently time-consuming, which significantly extends scan times and limits throughput in hospitals. As a result, there is great interest in reconstructing diagnostic-quality images from limited measurements (k-space data) to shorten scan times.For instance, parallel imaging (PI) capitalizes on spatially sensitive receive coil arrays to simultaneously acquire multiple MRI measurements. Compressed Sensing (CS) techniques have been employed to iteratively reconstruct under-sampled data into high-quality images by utilizing sparse priors. More recently, end-to-end deep learning (DL) based reconstruction techniques have been introduced, leveraging deep neural networks to learn the reconstruction pipeline directly from extensive training datasets, rather than relying on hand-crafted prior knowledge.Although DL-based methods have demonstrated significant success surpassing PI and CS capabilities, several challenges persist that limit the fidelity and efficiency, for example: 1) Loss functions used in DL-based reconstruction are mostly hand-crafted, either pixel-wise or based on local statistics (e.g., SSIM loss), inadequately capture perceptual information, leading to compromised image quality and blurring; 2) Memory constraints during network training restrict the applicability of DL reconstruction for high-dimensional MRI (e.g., 2D+time, 3D, 3D+time); 3) The confidence or reliability of reconstructed structures remains insufficiently investigated, posing a challenge for DL-based approaches in clinical applications. 4) Unlike natural images, MRI data is inherently complex-valued and faces challenges due to the limited availability of fully-sampled ground truth. This constraint inevitably restricts the applications of deep learning-based MRI techniques to tasks without access to adequate ground truth.In this dissertation, we introduce a series of projects aimed at overcoming existing obstacles and achieving enhanced fidelity and efficiency in Magnetic Resonance (MR) image reconstruction. Chapter 3 begins by reconstructing high-fidelity contrast-weighted images from highly under-sampled Magnetic Resonance Fingerprinting (MRF) scan. It introduces a supervised learning method that directly synthesizes contrast-weighted images (T1-weighted, T2-weighted, and FLAIR) from an MRF scan. This technique generates multi-contrast images with significantly reduced scan times, as detailed in the.Chapters 4-6 feature physics-informed DL-based reconstruction from undersampled k-space data. First, A novel patch-based Unsupervised Feature Loss (UFLoss) is proposed as a novel perceptual loss function and incorporated into the training of DL-based reconstruction frameworks in order to preserve perceptual similarity and high-order statistics. Next, I employ our previously proposed memory-efficient learning framework to minimize the memory required for backpropagation, facilitating the training of DL-based unrolled reconstructions for large-scale 3D MRI and 2D+time cardiac cine MRI.Then, I present our uncertainty estimation framework to identify when and where a reconstruction model is producing potentially misleading results. Our framework produces confidence intervals at each pixel of a reconstruction image with a rigorous finite-sample statistical guarantee.Our in-vivo knee and brain results probe the quality of our uncertainty estimation model, which allows us to identify specific regions where the model performs poorly. A distinctive aspect of MRI lies in its inherently complex-valued data. The final section of this dissertation concentrates on the representation learning of complex-valued data. In contrast to deep learning applied to natural images, MRI faces the challenge of a scarcity of fully-sampled ground truth and well-annotated data.To tackle this challenge, I introduce Complex-valued Scattering Representation (CSR) as a universal complex-valued representation, which so far demonstrates superior performance in both real-valued (e.g., RGB image) and complex-valued (e.g., MRI) image classification tasks compared to its counterparts, particularly when training samples are limited. Although this dissertation has not applied CSR to DL-based reconstruction, it represents a promising direction for future research.Collectively, these approaches embody the central theme and progress toward MR image reconstruction with high fidelity, high efficiency, and high reliability.

Computational imaging as a field has exploded in the past three decades, as new methods and techniques have been developed to overcome traditional digital imaging limitations. The focus of most of this progress has been in either the pre-sensor optics and illumination, or in the post-sensor software and data processing. Little attention has been given to the sensor itself. This work aims to fill that gap. I present two novel CMOS image sensors for use in computational cameras which are designed to detect time as a descriptive dimension of light. The first sensor exploits the wave nature of light as well as the ability to build nano-scale diffraction gratings directly into the integrated circuit. The prototype demonstrates a technique to directly capture phase shifts of a light field, which can be used for holography or optical coherence tomography applications. The second is based upon the particle nature of light, and addresses the practical challenges of data bandwidth and silicon footprint for developing megapixel arrays of single-photon sensors. This work builds towards the goal of developing full imaging systems that can, in a compact format, see through scattering media such as fog or human tissue.

Inverse problems in computational imaging seek to recover an unknown image of interest from observed measurements acquired using a known forward model. These inverse problems are often ill-conditioned, requiring some form of regularization. The corresponding objective function for inverse problems in computational imaging can often be solved using iterative optimization approaches that alternate between two sub-problems that enforce data consistency and promote the regularization approach, respectively. Such inverse problems arise in a multitude of imaging modalities, in particular in magnetic resonance imaging (MRI), which is the main application area for this thesis. Lengthy scan times remain a challenge in MRI, thus accelerating MRI scans has remained an open research problem over decades. Conventional accelerated MRI techniques acceleration rate is limited by noise amplification and residual artifacts. Recently, deep learning has emerged as an alternative approach for accelerated MRI. Among deep learning techniques, physics-guided deep learning (PG-DL) has drawn great interest, as it incorporates the known physical forward model into the network architecture. PG-DL unrolls a conventional iterative algorithm for solving a regularized least squares problem for a fixed number of iterations, and replaces the proximal operation corresponding to the regularizer implicitly with neural networks. These unrolled networks are trained end-to-end, with the goal of minimizing the difference between the network output and the corresponding reference data. Most of the existing deep learning approaches in MRI reconstruction are based on supervised learning, which requires ground-truth/ fully-sampled data for training. However, acquisition of fully-sampled data is infeasible in many applications due to physiological constraints, such as organ motion, or physical constraints, such as signal decay. In several other scenarios, such as high-resolution anatomical brain imaging, it is impractical to acquire fully-sampled datasets as the scan time becomes extremely lengthy. Therefore, enabling the training of PG-DL reconstruction without fully-sampled data is essential for the integration of deep learning reconstruction into clinical MRI practice. The present thesis introduces novel frameworks to enable the training of deep learning reconstruction methods for inverse imaging problems in the absence of ground-truth/fully-sampled data. First, we introduce self-supervised learning via data undersampling (SSDU) approach to enable database training without fully-sampled data. Succinctly, SSDU partitions available measurements into two disjoint sets. One of these sets is used in the data consistency units of the unrolled network, while the other is used to define the loss in the measurement domain. Subsequently, we extend SSDU for processing 3D datasets and provide solutions for GPU memory constraints and data scarcity issues faced in 3D processing. To cope with potential performance degradation at very high acceleration rates, we develop a multi-mask self-supervised learning approach, which retrospectively splits available measurements into multiple 2-tuples of disjoint sets to perform training and define a loss function. Furthermore, we introduce a zero-shot self-supervised learning approach to enable training from a single scan/sample without any external training databases. ZS-SSL partitions the available measurements from a single scan into three disjoint sets. Two of these sets are used to enforce data consistency and define loss during training for self-supervision, while the last set serves to self-validate, establishing an early stopping criterion. Finally, we introduce a self-supervised learning algorithm for referenceless image denoising. Self-supervised deep learning algorithms split the pixels for each image into two disjoint sets to perform training and defining loss. In existent self-supervised denoising approaches which are purely data-driven, the set of pixels used as input to the network is not re-utilized in the end-to-end training since the network is only comprised of a neural network. Reusing the pixels within the network would promote consistency with acquired measurements, thus leading to a more robust and improved denoising performance. To tackle this challenge, we build upon existent self-supervised learning algorithms and recast the denoising problem into a regularized image inpainting framework which allows use of algorithm unrolling for denoising.

We live in a world where imaging systems are ubiquitous. From the cell phones in our pockets to our cars and doorbells and on to telescopes and medical scanners, imaging has changed how we share, document, and understand our world. There is an increasing demand to make these systems more efficient by producing higher-quality images with fewer and fewer resources. When we need an image that is of higher quality than can be directly constructed from its samples, we have an inverse problem and must rely on nonlinear reconstruction schemes. These reconstruction algorithms fill the gap between the measured samples and the desired high-quality images by leveraging a model that attempts to distinguish image content from artifacts. Many modern image models are optimized — or trained — to produce high-quality results on a large dataset of images. These methods have shown a large improvement over classical fixed models, but require a lot of computation and high-quality data for the dataset. This thesis applies and develops adaptive image models which are trained during reconstruction for a particular image and can optionally be pre-trained on a dataset. These adaptive methods are formulated as part of a regularization term of a minimization problem and enabled by efficient model updates. In particular, we further develop a union of subspaces model and a dynamic subspace model which are both enabled by an efficient generalized Procrustes update. Beyond subspace models, this work also investigates the application of neural networks trained at reconstruction time on image patches. The proposed methods are applied to problems in light-field imaging and MRI.

A key aspect of many computational imaging systems, from compressive cameras to low light photography, are the algorithms used to uncover the signal from encoded or noisy measurements. Some computational cameras encode higher-dimensional information (e.g.\ different wavelengths of light, 3D, time) onto a 2-dimensional sensor, then use algorithms to decode and recover this high-dimensional information. Others capture measurements that are extremely noisy, or degraded, and require algorithms to extract the signal and make the images usable by people, or by higher-level downstream algorithms. In each case, the algorithms used to decode and extract information from raw measurements are critical and necessary to make computational cameras function. Over the years the predominant methods, classic methods, to recover information from computational cameras have been based on minimizing an optimization problem consisting of a data term and hand-picked prior term. More recently, deep learning has been applied to these problems, but often has no way to incorporate known optical characteristics, requires large training datasets, and results in black-box models that cannot easily be interpreted. In this dissertation, we present physics-informed machine learning for computational imaging, which is a middle ground approach that combines elements of classic methods with deep learning. We show how to incorporate knowledge of the imaging system physics into neural networks to improve image quality and performance beyond what is feasible with either classic or deep methods for several computational cameras. We show several different ways to incorporate imaging physics into neural networks, including algorithm unrolling, differentiable optical models, unsupervised methods, and through generative adversarial networks. For each of these methods, we focus on a different computational camera with unique challenges and modeling considerations. First, we introduce an unrolled, physics-informed network that improves the quality and reconstruction time of lensless cameras, improving these cameras and showing photorealistic image quality on a variety of scenes. Building up on this, we demonstrate a new reconstruction network that can improve the reconstruction time for compressive, single-shot 3D microscopy with spatially-varying blur by 1,600X, enabling interactive previewing of the scene. In cases where training data is hard to acquire, we show that an untrained physics-informed network can improve image quality for compressive single-shot video and hyperspectral imaging without the need for training data. Finally, we design a physics-informed noise generator that can realistically synthesize noise at extremely high-gain, low-light settings. Using this learned noise model, we show how we can push a camera past its typical limit and take photorealistic videos at starlight levels of illumination for the first time. Each case highlights how using physics-informed machine learning can improve computational cameras and push them to their limits.

Traditional cameras and microscopes are often optimized to produce sharp 2D images of the object. These 2D images miss important information about the world (e.g. depth and spectrum). Access to this information can make a significant impact on fields such as neuroscience, medicine, and robotics. For example, volumetric neural imaging in freely moving animals requires compact head-mountable 3D microscopes and tumor classification in tissue benefits from access to spectral information. Modifications that enable capturing these extra dimensions often result in bulky, expensive, and complex imaging setups. In this dissertation, I focus on designing compact single-shot computational imaging systems that can capture high dimensional information (depth and spectrum) about the world. This is achieved by using a multiplexing optic as the image capture hardware and formulating image recovery as a convex optimization problem. First, I discuss designing a single-shot compact miniature 3D fluorescence microscope, termed Miniscope3D. By placing an optimized multifocal phase mask at the objective’s exit pupil, 3D fluorescence intensity is encoded into a single 2D measurement and the 3D volume can be recovered by solving a sparsity constrained inverse problem. This enables a 2.76 micron lateral and 15 micron axial resolution across 900x700x390 micron cubed volume at 40 volumes per second in a device smaller than a U.S. quarter. Second, I discuss designing a single-shot hyperspectral camera, termed Spectral DiffuserCam, by combining a diffuser with a tiled spectral filter array. This enables recovering a hyperspectral volume with higher spatial resolution than the spectral filter alone. The system is compact, flexible, and can be designed with contiguous or non-contiguous spectral filters tailored to a given application. Finally, the iterative reconstruction methods generally used for compressed sensing take thousands of iterations to converge and rely on hand-tuned priors. I discuss a deep learning architecture, termed MultiWienerNet, that uses multiple differentiable Wiener filters paired with a convolutional neural network to take into account the system’s spatially-varying point spread functions. The result is a 625-1600X increase in speed compared to iterative methods with spatially-varying models and better reconstruction quality than deep learning methods that assume shift invariance.

We present a selection of topics relating to modeling, designing, and measuring EUV (Extreme Ultraviolet) photomasks, with implications for high-volume nanofabrication of integrated circuits. These EUV photomasks must be accurately designed, but rigorously modeling large domains is extremely computationally intensive; we introduce an approximateFresnel Double Scattering model which is 10,000x faster. This approximation can predict the trend of phase vs pitch, which is critical to designing EUV phase shift masks (PSMs). We also explore novel mask architectures to improve efficiency and contrast, such as an etched multilayer PSM (up to 6x throughput but restrictive applicability), aperiodic multilayers (up to +22% throughput and more general applicability), and multilayers with minimal propagation distance at certain angles (lower throughput but higher contrast with minimized 3D effects). Finally we explore computational metrology with EUV reflectometry, scatterometry, and imaging for probing the phase and amplitude response of an EUV mask, with experimental demonstrations at the Advanced Light Source synchrotron. We perform reflectometry experiments on 3 masks with different architectures to infer approximately 25 physical film parameters each. Another reflectometry application to contamination monitoring achieved single-picometer precision for thickness (3σ < 6pm) and sub-degree precision for phase (3σ < 0.2deg). We compare two implementations of phase scatterometry, either applying nonlinear optimization with approximate scattering, or linearizing the rigorous scattering relationship between intensity and phase; linearization is shown to generally be more accurate, but both methods have similar precision. We apply novel software and hardware for phase imaging, using PhaseLift convex phase retrieval, combined with a set of custom Zernike Phase Contrast (ZPC) zone plates. We perform hyperspectral ZPC phase imaging on 3 masks, where we see promising agreement with reflectometry in the trend of phase vs wavelength.

Non-line-of-sight (NLOS) imaging is the inference of the properties of objects or scenes outside of the direct line-of-sight of the observer. Such inferences can range from a 2D photograph-like image of a hidden area, to determining the position, motion or number of hidden objects, to 3D reconstructions of a hidden volume. NLOS imaging has many enticing potential applications, such as leveraging the existing hardware in many automobiles to identify hidden pedestrians, vehicles or other hazards and hence plan safer trajectories. Other potential application areas include improving navigation for robots or drones by anticipating occluded hazards, peering past obstructions in medical settings, or in surveying unreachable areas in search-and-rescue operations. Most modern NLOS imaging methods fall into one of two categories: active imaging methods that have some control of the illumination of the hidden area, and passive methods that simply measure light that already exists. This thesis introduces two NLOS imaging methods, one of each category, along with modeling and data processing techniques that are more broadly applicable. The methods are linked by their use of objects (‘occluders’) that reside somewhere between the observer and the hidden scene and block some possible light paths. Computational periscopy, a passive method, can recover the unknown position of an occluding object in the hidden area and then recover an image of the hidden scene behind it. It does so using only a single photograph of a blank relay wall taken by an ordinary digital camera. We develop also a framework using an optimized preconditioning matrix to improve the speed at which these reconstructions can be made and greatly improve the robustness to ambient light. Lastly, we develop tools necessary to demonstrate recovery of scenes at multiple unknown depths – paving the way towards three-dimensional reconstructions. Edge-resolved transient imaging, an active method, enables the formation of 2.5D representations – a plan view plus heights – of large-scale scenes. A pulsed laser illuminates spots along a small semi-circle on the floor, centered on the edge of a vertical wall such as in a doorway. The wall edge occludes some light paths, only allowing the laser light reflecting off of the floor to illuminate certain portions of the hidden area beyond the wall, depending on where along the semi-circle it is illuminating. The time at which photons return following a laser pulse is recorded. The occluding wall edge provides angular resolution, and time-resolved sensing provides radial resolution. This novel acquisition strategy, along with a scene response model and reconstruction algorithm, allow for 180° field of view reconstructions of large-scale scenes unlike other active imaging methods. Lastly, we introduce a sparsity penalty named mutually exclusive group sparsity (MEGS), that can be used as a constraint or regularization in optimization problems to promote solutions in which certain components are mutually exclusive. We explore how this penalty relates to other similar penalties, develop fast algorithms to solve MEGS-regularized problems, and demonstrate how enforcing mutual exclusivity structure can provide great utility in NLOS imaging problems.

Biological nanoparticles have enormous utility as well as potential adverse impacts in biotechnology, human health, and medicine. The physical and chemical properties of these nanoparticles have strong implications on their distribution, circulation, and clearance in vivo. Accurate morphological visualization and chemical characterization of nanoparticles by label-free (direct) optical microscopy would provide valuable insights into their natural and intrinsic properties. However, three major challenges related to label-free nanoparticle imaging must be overcome: (i) weak contrast due to exceptionally small size and low-refractive-index difference with the surrounding medium, (ii) inadequate spatial resolution to discern nanoscale features, and (iii) lack of chemical specificity. Advances in common-path interferometric microscopy have successfully overcome the weak contrast limitation and enabled direct detection of low-index biological nanoparticles down to single proteins. However, interferometric light microscopy does not overcome the diffraction limit, and studying the nanoparticle morphology at sub-wavelength spatial resolution remains a significant challenge. Moreover, chemical signature and composition are inaccessible in these interferometric optical measurements. This dissertation explores innovations in common-path interferometric microscopy to provide enhanced spatial resolution and chemical specificity in high-throughput imaging of individual nanoparticles. The dissertation research effort focuses on a particular modality of interferometric imaging, termed “single-particle interferometric reflectance (SPIR) microscopy”, that uses an oxide-coated silicon substrate for enhanced coherent detection of the weakly scattered light. We seek to advance three specific aspects of SPIR microscopy: sensitivity, spatial resolution, and chemical specificity. The first one is to enhance particle visibility via novel optical and computational methods that push optical detection sensitivity. The second one is to improve the lateral resolution beyond the system's classical limit by a new computational imaging method with an engineered illumination function that accesses high-resolution spatial information at the nanoscale. The last one is to extract a distinctive chemical signature by probing the mid-infrared absorption-induced photothermal effect. To realize these goals, we introduce new theoretical models and experimental concepts. This dissertation makes the following four major contributions in the wide-field common-path interferometric microscopy field: (1) formulating vectorial-optics based linear forward model that describes interferometric light scattering near planar interfaces in the quasi-static limit, (2) developing computationally efficient image reconstruction methods from defocus images to detect a single 25 nm dielectric nanoparticle, (3) developing asymmetric illumination based computational microscopy methods to achieve direct morphological visualization of nanoparticles at 150 nm, and (4) developing bond-selective interferometric microscopy to enable multispectral chemical imaging of sub-wavelength nanoparticles in the vibrational fingerprint region. Collectively, through these research projects, we demonstrate significant advancement in the wide-field common-path interferometric microscopy field to achieve high-resolution and accurate visualization and chemical characterization of a broad size range of individual biological nanoparticles with high sensitivity.

Stimulated Raman scattering (SRS) imaging is a chemical imaging scheme that can visualize cellular content based on intrinsic chemical bond vibrations. To resolve chemicals with overlapping Raman bands, spectroscopic SRS platforms have been developed. To date, endeavors on high-speed instrumentation have achieved spectral acquisition at the microsecond level, enabling in vivo imaging of cells and tissues. Nevertheless, due to the extremely small Raman cross-sections, the current performance of SRS is bounded by a design space that trades off speed, signal fidelity, and spectral bandwidth. The lack of tailored data mining algorithms further limits the chemical information one can extract from the spectroscopic images. My thesis work focuses on developing computational SRS imaging approaches to break the physical tradeoffs and novel data analytical tools to decipher essential chemical information from stimulated Raman spectroscopic images. Utilizing data redundancy of spectroscopic images, we developed two compressive sensing schemes to improve the imaging speed by one order of magnitude without information loss. To break the sensitivity limit, we proposed an ultrafast spectroscopic SRS system and further integrated it with a deep neural network to synergistically achieve microsecond level imaging in the fingerprint region. To improve the chemical specificity and content levels, we implemented a sparsity-regularized spectral unmixing algorithm, realizing multiplexed imaging of up to six major metabolites in a cell. Finally, enabled by advances in low-exposure imaging and spectral unmixing, longitudinal imaging of biofuel synthesis in live cells with sophisticated chemical information is demonstrated.

Heart rate (HR) is an essential clinical measure for the assessment of cardiorespiratory instability. The growing telemedicine market opens up the urgent requirement for scalable yet affordable remote HR estimation. Smartphones that use in-built camera modules to measure HR from facial videos offer a more economical solution in comparison to mass deployment of wearable sensors. However, existing computer vision methods that estimate HR from facial videos exhibit biased performance against dark skin tones. This is a major concern, since communities of color are disproportionately affected by both COVID-19 and cardiovascular disease. We identify the origin of this bias and present a novel physics-driven algorithm that boosts performance on darker skin tones in our reported data. We assess the performance of our method through the creation of the first telemedicine-focused remote vital signs dataset, the VITAL dataset. 472 videos (~944 minutes) of 59 subjects with diverse skin tones are recorded under realistic scene conditions with corresponding vital sign data. Our method reduces errors due to lighting changes, shadows, and specular highlights and imparts unbiased performance gains across skin tones, setting the stage for making non-contact HR sensing technologies a viable reality for patients across skin tones, using just smartphone cameras.

Biomedical research relies upon quantitative imaging methods to measure functional and structural data about microscopic organisms. Recently-developed quantitative phase imaging (QPI) methods use jointly designed optical and computational systems to recover structural quantitative phase information for biological samples. However, these methods have not seen wide adoption in biological research because the optical systems can be difficult to use and the computational algorithms often require expert operation for consistently high-quality results. QPI systems are usually developed under a computational imaging framework, where the optical measurement system is jointly designed with the computational reconstruction algorithm. Designing QPI systems for robust and practical real-world use is often difficult, however, because each imaging and computational configuration has unique and difficult-to-quantify practical implications for the end-user.In this dissertation, I present three frameworks for increasing the robustness and practicality of computational imaging systems, and I demonstrate the usefulness of these three frameworks by applying them to 2D and 3D quantitative phase imaging systems. First, algorithmic self-calibration directly recovers imaging system parameters from data measurements, doing away with the need for extensive pre-calibration steps and ensuring greater calibration accuracy for non-ideal, real-world systems. I present a robust and efficient self-calibration algorithm for angled coherent illumination, which has enabled new QPI system designs for 2D Fourier ptychographic microscopy (FPM) and 3D intensity optical diffraction tomography (ODT) that would have otherwise been infeasible. Second, increased measurement diversity better encodes useful information across measurements, which can reduce imaging system complexity, data requirements, and computation time. I present a novel pupil-coded intensity ODT system designed to increase measurement diversity of 3D refractive index (RI) information by including joint illumination- and detection-side coding for improved volumetric RI reconstructions. Finally, physics-based machine learning uses a data-driven approach to directly optimize imaging system parameters, which can improve imaging reconstructions and build intuition for better designs of complicated computational imaging systems. I show results from a physics-based machine learning algorithm to optimize pupil coding masks for 3D RI reconstructions of thick cell clusters in the pupil-coded intensity ODT system. In addition, I provide practical methods for the design, calibration, and operation of Fourier ptychography, intensity-only ODT, and pupil-coded intensity ODT microscopes to aid in the future development of robust QPI systems. I additionally present a validation of joint system pupil recovery using FPM and a comparison of the accuracy and computational complexity of coherent light propagation models that are commonly used in 3D quantitative phase imaging. I also compare field-based 3D RI reconstructions to intensity-based RI reconstructions, concluding that the proposed pupil-coded intensity ODT system captures similarly diverse phase information to field-based ODT microscopes.Throughout this work, I demonstrate that by using the frameworks of algorithmic self-calibration, increased system measurement diversity, and physics-based machine learning for computational imaging system design, we can develop more robust quantitative phase imaging systems that are practical for real-world use.

In computational imaging systems (e.g. tomographic systems, computational optics, magnetic resonance imaging) the acquisition of data and reconstruction of images are co-designed to retrieve information which is not traditionally accessible. The performance of such systems is characterized by how information is encoded to (forward process) and decoded from (inverse problem) the measurements. Recently, critical aspects of these systems, such as their signal prior, have been optimized using deep neural networks formed from unrolling the iterations of a physics-based image reconstruction.In this dissertation, I will detail my work, physics-based learned design, to optimize the performance of the entire computational imaging system by jointly learning aspects of its experimental design and computational reconstruction. As an application, I introduce how the LED-array microscope performs super-resolved quantitative phase imaging and demonstrate how physics-based learning can optimize a reduced set of measurements without sacrificing performance to enable the imaging of live fast moving biology.In this dissertation’s latter half, I will discuss how to overcome some of the computational challenges encountered in applying physics-based learning concepts to large-scale computational imaging systems. I will describe my work, memory-efficient learning, that makes physics-based learning for large-scale systems feasible on commercially-available graphics processing units. I demonstrate this method on two large-scale real-world systems: 3D multi-channel compressed sensing MRI and super-resolution optical microscopy.

Efficiently capturing high-dimensional optical signals, such as temporal dynamics, depth, perspective, or spectral content is a difficult imaging challenge. Because image sensors are inherently two-dimensional, direct sampling of the many dimensions that completely describe a scene presents a significant engineering challenge. Computational imaging is a design approach in which imaging hardware and digital signal processing algorithms are designed jointly to achieve performance not possible with partitioned design schemes. Within this paradigm, the sensing hardware is viewed as an encoder, coding the information of interest into measurements that can be captured with conventional sensors. Algorithms are then used to decode the information. In this dissertation, I explore the connection between optical imaging system design and compressed sensing, demonstrating that extra dimensions of optical signals (time, depth, and perspective) can be encoded into a single 2D measurement, then extracted using sparse recovery methods. The key to these capabilities is exploiting the inherent multiplexing properties of diffusers, pseudorandom free-form phase optics that scramble incident light. Contrary to their intended use, I show that certain classes of diffuser encode high-dimensional information about the incident light field into high-contrast, pseudorandom intensity patterns (caustics). Sparse recovery methods can then decode these patterns, recovering 3D images from snapshot 2D measurements. This transforms a diffuser into a computational imaging element for high-dimensional capture at video rates. Efficient physical models are introduced that reduce the computational burden for image recovery as compared to explicit matrix approaches (the computational cost remains high, however). Lastly, analysis and theory is developed that enables optimization of customized diffusers for miniaturized 3D fluorescence microscopy.

Computational imaging and sensing aim to redesign optical systems from the ground up, jointly considering both hardware/sensors and software/reconstruction algorithms to enable new modalities with superior capabilities, speed, cost, and/or footprint. Often systems can be optimized with targeted applications in mind, such as low-light imaging or remote sensing in a specific spectral regime. For medical diagnostics in particular, computational sensing could enable more portable, cost-effective systems and in turn improve access to care. In the last decade, the increased availability of data and cost-effective computational resources coupled with the commodification of neural networks has accelerated and expanded the potential for these computational sensing systems.First, I will present my work on a cost-effective system for quantifying antimicrobial resistance, which could be of particular use in resource-limited settings, where poverty, population density, and lack of healthcare infrastructure lead to the emergence of some of the most resistant strains of bacteria. The device uses optical fibers to spatially subsample all 96 wells of a standard microplate without any scanning components, and a neural network identifies bacterial growth from the optical intensity information captured by the fibers. Our accelerated antimicrobial susceptibility testing system can interface with the current laboratory workflow and, when blindly tested on patient bacteria at UCLA Health, was able to identify bacterial growth after an average of 5.72 h, as opposed to the gold standard method requiring 18–24 h. The system is completely automated, avoiding the need for a trained medical technologist to manually inspect each well of a standard 96-well microplate for growth. Second, I will discuss a deep learning-enabled spectrometer framework using localized surface plasmon resonance. By fabricating an array of periodic nanostructures with varying geometries, we created a “spectral encoder chip” whose spatial transmission intensity depends upon the incident spectrum of light. A neural network uses the transmitted intensities captured by a CMOS image sensor to faithfully reconstruct the underlying spectrum. Unlike conventional diffraction-based spectrometers, this framework is scalable to large areas through imprint lithography, conducive to compact, lightweight designs, and, crucially, does not suffer from the resolution–signal strength tradeoff inherent to grating-based designs.

Machine learning has been transforming many fields including optics by creating a new avenue for designing optical sensors and devices. This new paradigm takes a data driven approach, without focusing on underlying physics of the design. This new alternative and yet powerful method brings new advancements to traditional design tools and opens up numerous opportunities.This dissertation introduces machine learning-enabled optical sensors and devices in which computational imaging and deep learning based design of devices tackle various challenges. First, a cost-effective and portable platform is presented to non-invasively detect and monitor a bacteria that resides in human ocular microbiome, Staphylococcus aureus. Contact lenses are designed to capture S. aureus using surface chemistry protocol, and sandwich immunoassay with polystyrene microbeads is performed to tag captured bacteria. Lens-free on-chip microscope is used to obtain a single hologram of the contact lens surface and 3D surface of it is computationally reconstructed. Support vector machine based machine learning algorithm is employed to detect and count the amount of bacteria on contact lens surface. This platform, which only weighs 77 g, is controlled by laptop and provides ~16 bacteria/�L detection limit. This wearable sensor platform can be used to analyze and monitor other viruses and bacteria in tear with the appropriate modification to its surface chemistry protocol. Second, a novel physical mechanism is introduced, diffractive optical networks, to perform all-optical machine learning using passive diffractive layers that work together to implement various functions. This framework merges wave-optics with deep learning to all optically perform different tasks. A classification of handwritten digits and fashion products were demonstrated with 3D-printed diffractive optical networks. Moreover, a diffractive optical network is designed to function as an imaging lens in terahertz spectrum. This scalable platform can execute various functions at the speed of light with low power and help us to design exotic optical components. Third, terahertz pulse shaping architecture using diffractive optical surfaces is introduced. This platform engineers arbitrary broadband input pulse into desired waveform. Synthesis of various pulses has been demonstrated by designing and fabricating diffractive layers. This works constitutes the first demonstration of direct pulse shaping in terahertz spectrum with precise control of amplitude and phase of input broadband light over a wide frequency range. This approach can also find applications in other fields like optical communications, spectroscopy and ultra-fast imaging.

CamIoT is an eort to add Articial intelligence into the emerging technologies of SmartHome services. The Features embedded in the device also enable CamIoT to be used as an assistive technology for visually impaired people. CamIoT is the rst wrist-worn platform that uses an outward-facing camera to recognize and interact with distant IoT objects. It provides a novel technique using the index nger's orientation to locate an IoT object in the camera view while supporting disambiguating selection in the presence of multiple objects. Our study ends with the full evaluation of the contributed methods and CamIoT's performance, highlighting the best case appliance recognition accuracy of 96%.

Despite its desirability, capturing and displaying higher dimensional content is still a novelty since image sensors and display panels are inherently 2D. A popular option is to use scanning mechanisms to sequentially capture 3D data or display content at a variety of depths. This approach is akin to directly measuring (or displaying) the content of interest, which has low computational cost but sacrifices temporal resolution and requires complex physical hardware with moving parts. The exacting specifications on the hardware make it challenging to miniaturize these optical systems for demanding applications such as neural imaging in animals or head-mounted augmented reality displays.In this dissertation, I propose moving the burden of 3D capture from hardware into computation by replacing the physical scanning mechanisms with a simple static diffuser (a transparent optical element with pseudorandom thickness) and formulating image recovery as an optimization problem. First, I highlight the versatility of the diffuser by showing that it can replace a lens to create an easy-to-assemble, compact camera that is robust to missing pixels; although the raw data is not intelligible by a human, it contains information that we extract with optimization using an efficient physically-based model of the optics. Next, I show that the randomness of the diffuser makes the system well-suited for compressed sensing; we leverage this to recover 3D volumes from a single acquisition of raw data. Additionally, I extend our lensless 3D imaging system to fluorescence microscopy and introduce a new diffuser design with improved noise performance. Finally, I show how incorporating the diffuser in a 3D holographic display expands the field-of-view, and I demonstrate state-of-the-art performance by using perceptually inspired loss functions when optimizing the display panel pattern. These results show how randomness in the optical system in conjunction with optimization-based algorithms can both improve the physical form factor and expand the capabilities of cameras, microscopes, and displays.

A novel computational imaging approach to sensor protection based on point spread function (PSF) engineering is designed to suppress harmful laser irradiance without significant loss of image fidelity of a background scene. PSF engineering is accomplished by modifying a traditional imaging system with a lossless linear phase mask at the pupil which diffracts laser light over a large area of the imaging sensor. The approach provides the additional advantage of an instantaneous response time across a broad region of the electromagnetic spectrum. As the mask does not discriminate between the laser and desired scene, a post-processing image reconstruction step is required, which may be accomplished in real time, that both removes the laser spot and improves the image fidelity. This thesis includes significant experimental and numerical advancements in the determination and demonstration of optimized phase masks. Analytic studies of PSF engineering systems and their fundamental limits were conducted. An experimental test-bed was designed using a spatial light modulator to create digitally-controlled phase masks to image a target in the presence of a laser source. Experimental results using already known phase masks: axicon, vortex and cubic are reported. New methods for designing phase masks are also reported including (1) a numeric differential evolution algorithm, (2) a “PSF reverse engineering” method, and (3) a hardware based simulated annealing experiment. Broadband performance of optimized phase masks were also evaluated in simulation. Optimized phase masks were shown to provide three orders of magnitude laser suppression while simultaneously providing high fidelity imaging a background scene.

There is an unprecedented need in neuroscience and medical research for the precise imaging of individual neurons and their interconnectivity in an effort to achieve a more complete understanding of neurological illness and cognitive growth. While several imaging architectures successfully detect active neural tissue, fluorescent imaging through head-mounted microscopes is becoming a standard method of imaging neural circuitry in freely behaving animals. At Boston University, the Gardner Group developed a miniaturized, open-source, single-photon ‘finch-scope’ to spur rapid prototyping in head-mounted miniscope technology. While experimentally convenient, the finch-scope and other miniscope platforms are limited by their native depth of field and may only detect a thin layer of active neurons in a neurological volume. In this Master’s Thesis Project, I will investigate utilizing optical phase masks integrated in the Fourier plane of the finch-scope to invoke a less-diffractive Bessel point spread function. Next, I will experimentally justify the extended depth of field nature of these phase masks by imaging the axial profile of a 10μm fluorescent pinhole object with a modified finch-scope.

Exponential advancements in computational resources and algorithms have given birth to the new paradigm in imaging that rely on computation to digitally reconstruct and enhance images. These computational imaging modalities have enabled higher resolution, larger throughput and/or automatic detection capabilities for optical microscopy. An example is lens-less digital holographic microscope, which enables snapshot imaging of volumetric samples over wide field-of-view without using imaging lenses. Recent developments in the field of deep learning have further opened up exciting avenues for computational imaging, which offer unprecedented performance thanks to their capability to robustly learn content-specific complex image priors.This dissertation introduces a novel and universal modeling framework of deep learning -based image reconstruction technique to tackle various challenges in optical microscopic imaging, including digital holography reconstructions and 3D fluorescence microscopy. Firstly, auto-focusing and phase recovery in holography reconstruction are conventionally challenging and time-consuming to digitally perform. A convolutional neural network (CNN) based approach was developed that solves both problems rapidly in parallel, enabling extended depth-of-field holographic reconstruction with significantly improved time complexity from O(mn) to O(1). Secondly, to fuse advantages of snapshot volumetric capability in digital holography and speckle- and artifact-free image contrast in bright-field microscopy, a CNN was used to transform across microscopy modalities from holographic image reconstructions to their equivalent high contrast bright-field microscopic images. Thirdly, 3D fluorescence microscopy generally requires axial scanning. A CNN was trained to learn defocuses of fluorescence and digitally refocusing a single 2D fluorescence image onto user-defined 3D surfaces within the sample volume, which extends depth-of-field of fluorescence microscopy by 20-fold without any axial scanning, additional hardware, or a trade-off of imaging resolution or speed. This enables high-speed volumetric imaging and digital aberration correction for live samples. Based on deep learning powered computational microscopy, a hand-held device was also developed to measure the particulate matters and bio-aerosols in the air using the lens-less digital holographic microscopic imaging geometry. This device, named c-Air, demonstrates accurate, high-throughput and automatic detection, sizing and classification of the particles in the air, which opens new opportunities in deep learning based environmental sensing and personalized and/or distributed air quality monitoring.

There is an ever present demand to increase the capability and performance of optical imagers while decreasing their physical size and cost. In order to achieve this, these devices must be engineered as complete systems and integrated together, rather than treated as disparate independent elements. This dissertation presents several examples of this concept. Specifically, it describes the integration process, including the design, simulation, fabrication, assembly, and characterization of several micro-optic imaging systems.These systems include a wink-controlled polarization-switched telescopic contact lens, which switches between magnified and unmagnified vision by detecting a user's winks with an external pair of liquid crystal shutter glasses. A folded monocentric imager with deformable mirror focus splits a monocentric lens in half by utilizing symmetry and refocuses by changing the curvature of a flexible fold mirror in contact with a gel core. A sequential-capture panoramic light field camera uses a high resolution image relay to transfer the light field passing through a monocentric lens and microlens array to an image sensor. The entire relay is rotated around the monocentric objective lens to capture a series of exposures which are combined to produce the final panoramic light field image. A panoramic single-aperture multi-sensor light field camera uses an array of consolidating light field image sensors behind a monocentric lens to create a contiguous panoramic light field image in a single capture.In addition to these systems, several future directions under ongoing research are described. First, a three-orthogonal discrete field light field camera which senses depth information in three separate directions using a single monocentric lens for visual odometry is presented. Finally, preliminary lens designs for a full field and field-expanded single-aperture sensor array cameras are documented. These lenses produce images that can be directly captured, and one of which extends the field of view beyond the hemisphere. In addition to their optical performance, considerations around cost reduction, manufacturability, and mass production for the consumer market are discussed.

Quantitative optical microscopy continues to be a powerful tool for biomedical research and the sciences at-large. Building upon centuries of optical theory, computational microscopy leverages large-scale numerical computation to dramatically extend and improve the capabilities of existing optical microscopes through high-speed capture, super-resolution, or opening new application spaces such as quantitative phase imaging. This dissertation describes the theory and reduction to practice of several novel computational microscopy techniques which make use of jointly-designed coded illumination hardware and physics-based reconstruction algorithms. We first introduce label-free quantitative phase imaging using differential phase contrast, and demonstrate a novel single-shot variant using color-multiplexing. Second, we explore coded illumination for high-throughput imaging, and demonstrate a temporal-coding technique which enables significantly higher SNR for high-speed slide scanning and neuropathology applications. For fluorescent imaging, this method can provide up to a 10x improvement in reconstruction SNR compared to conventional high-speed imaging techniques. Next, we explore the design and fabrication of LED illumination devices, including a quasi-dome LED illuminator which enables high-angle illumination for a variety of applications. To address practical calibration concerns for computational microscopy systems, we demonstrate two examples of algorithmic self-calibration. These include aberration recovery using differential phase contrast as well as source calibration for the quasi-dome LED array, using both image-based calibration and an online method based on Fourier ptychography. These works demonstrate the benefits and practical challenges related to computational optical microscopy with coded illumination.

X-ray imaging traces its roots back to the discovery of x-rays by Ro ̈ntgen in 1901, in the form of simple projection imaging. Several decades later, fabrication of x-ray optics enabled image formation at higher resolutions, leading to microscopy. Nevertheless, microscopy at resolutions on the order of the x-ray wavelength has only become possible in recent years, after a host of enabling technologies, including high-brightness coherent sources, advanced detectors, high-performance computers and algorithms, and precision instrumentation were brought together by multidisciplinary teams of scientists to form one coherent imaging sys- tem. Today, x-ray ptychography microscopes operating at modern synchrotrons can image thick samples at unprecedented resolutions, and yield chemical information about the sam- ple, enabling novel material science studies. The field is rapidly developing, and the advent of new Diffraction Limited Storage Ring (DLSR) sources around the world promises to provide new opportunities with increased coherent brightness.This dissertation presents new advances in instrumentation and analysis at the Advanced Light Source at Lawrence Berkeley National Lab. A new x-ray ptychography microscope is described, compact in size and ultrastable in both vibrations and drift, enabling better than 3nm spatial resolution, and integrating a TEM-compatible in-situ sample holder to enable multimodal, operando, and cryogenic experiments. A new approach to the analysis of spectro-microscopic data is presented, combining low-resolution STXM and high-resolution ptychography images to produce chemical maps at high resolution without a priori reference spectra. This is achieved by collecting STXM data at fine x-ray energy increments, from which spectra are extracted by PCA and clustering. The ptychography data, collected at only a few energies, is then fitted to these extracted spectra. Finally, new ideas are explored to speed up scanning in ptychography, leading to a reduction in scan overhead, dose, and data collection times. This is done in two ways: firstly, a simple, yet effective, scanning algorithm is introduced to adjust feedback parameters for the piezo stage scanning the zone plate, reducing the scan overhead time. Secondly, lower density scan patterns and their effects on quality and resolution of reconstructions are investigated; based on these, ideas are proposed to intelligently reduce the total number of diffraction patterns collected, while only minimally affecting results. Combined with the promise of increased coherent flux with upcoming DLSRs, leading to much shorter exposure times, these speedups will enable much faster ptychography scans, greatly facilitating tomography or in-situ experiments, currently taking many hours or days.

Light microscopy is an important driving force for new biological discoveries because of its capability of visualizing micro-scale cell structures and interactions. Fundamentally, optical resolution is bounded by the diffraction limit, preventing observation of biological events with an even smaller scale. However, computational imaging approaches are efficient tools to surpass this limit. In recent years, optimization formulation has become more popular because of its flexibility and efficacy toward information retrieval. In this thesis, we leverage the power of optimization algorithms and new experimental schemes to tackle super-resolution microscopy, both improving existing methods and developing new techniques. We first apply rigorous optimization algorithm analysis to a super-resolution phase microscopy technique, Fourier ptychography. A more accurate noise model and the self-calibration algorithm ensure a better reconstruction quality for this technique. Next, we incorporate optimization into the study of a super-resolution fluorescence microscopy technique, structured illumination microscopy. Super-resolution reconstruction is achieved even with a series of random unknown illumination patterns, which is not possible without proper optimization formulation. Next, we leverage the experience of the previous two projects to propose a super-resolution microscopy method for phase and fluorescence contrast with multi-fold resolution improvement in both 2D and 3D using an unknown speckle illumination composed of high-angle plane waves from Scotch tape as a patterning element. The result is a practical method for achieving multimodal super-resolution with >2x resolution gain, surpassing the limit of the traditional linear structured illumination microscopy. All these outcomes are within the realm of computational super-resolution microscopy, where the optimization algorithm is jointly designed with optics for efficient information retrieval to achieve super-resolution microscopy.

Online model-based estimation is applied to two major applications in optics: Automated optical component alignment and wavefront reconstruction with simultaneous system parameter estimation. Both applications utilize mechanical perturbation in the optical system to generate phase diversity in real-time stochastic systems. The first part of this study proposes a novel automated alignment method which improves efficiency and increases the flexibility of an optical system. Current optical systems with automated alignment capabilities are typically designed to include a dedicated wavefront sensor. Here, we demonstrate a self-aligning method for a reconfigurable system using only focal plane images. We define reconfigurable and reflective optical systems and simulate the images given misalignment parameters using ZEMAX software. We perform a principal component analysis (PCA) on the simulated dataset to obtain Karhunen-Loeve (KL) modes, which form the basis set whose weights are the system measurements. A model function which maps the state to the measurement is learned using nonlinear least squares fitting and serves as the measurement function for the extended Kalman filter (EKF) and unscented Kalman filter (UKF) used to estimate the state and control the system. The observability and stability of the system are discussed. We present both simulated and experimental results of the full system in operation. The second part of this study presents a novel algorithm for phase retrieval and optical system parameter estimation. Many wavefront reconstruction techniques estimate the amplitude and phase from multiple intensity measurements. One can generate phase diversity among these intensity measurements by varying certain parameters in the optical system. These parameters are subject to noise and disturbances, which might strongly degrade the accuracy of the reconstruction. The parallel algorithm iterative amplitude and phase retrieval (APR) have been proven to accurately reconstruct arbitrary wavefronts from multiple intensity measurements when system parameters are known exactly, given the ability to induce phase diversity between images. Such sets of intensity images with phase diversity can be generated by moving a lens in the optical system, but any position error on the lens will degenerate the reconstruction result. We demonstrate the use of an expectation-maximization (EM) algorithm with Kalman smoothing for recovering both the complex field and the lens position from a stack of intensity images. Our method successfully reduces the mean-squared-error of the estimated wavefront in comparison to an approach without position error estimation. We present and discuss the results of using a Kalman smoother and nonlinear least-square optimization for the estimation of the moving lens position. We modify and extend the system variable estimation method to serial phase retrieval algorithm. We present the use of iterated extended Kalman filter (IEKF) to estimate the system variables in a multiple-image phase retrieval framework. An iterated extended Kalman filter is shown to effectively reduce the normalized mean-square-error of the reconstructed wavefront by estimating the defocus and transverse shifts of a moving camera in simulation. Experiments are conducted using two different test objects, and the results clearly demonstrate the enhancement of detail and contrast of the wavefront when using the filter. A quadratic phase introduced by a convex lens is used with a binary mask as one of the test objects. The focal length estimated from the unwrapped phase agrees with the (1% tolerance) value provided by the manufacturer.

DRAMATIC advances in the field of computational and medical imaging over the past decades have enabled many critical applications such as night vision, medical diagnosis, quality control, and remote sensing. The increasing demand for image quality and its fidelity requires an increase in pixel count and a sophisticated post-processing mechanism to efficiently store, transmit, and analyze this massive data. There is an inherent trade-off between the generation of big data by such imaging systems and efficiency in extraction of useful information within real-time constraints, limiting the efficacy of such sensors in real-time decision-making systems. The traditional imaging system gets burdened by the acquisition, transmission, and storage of surplus data, often bearing redundant information for the given application of interest. Transmission of the irrelevant information requires a high bandwidth and results in consuming extra power to store or transmit. Similarly, post-processing imposes extra latency and suffers under the power constraint, which is troublesome for many low-power, real-time applications, and portable devices. There is a need to address this problem by intelligently acquiring a limited but most important set of data features, and then efficiently processing this abstract information. This, in turn, needs an additional ability of performing computations at the pixel level, within the readout integrated circuit at the front-end of the imager. The most challenging job in this computational part is devising the mechanism to properly select the right set of data features for adequately representing the information which might be corrupted with noise and therefore potentially increase the reconstruction error. In this thesis, we work towards development of an efficient bias selection algorithm to configure the computational imaging hardware. This algorithm not only fulfills the fundamental hardware requirements of memory efficiency, low power consumption and minimal latency, but also results in lesser reconstruction error than the previously used naive approach. This algorithm supports compression during the data acquisition time at the pixel level thus enabling compression even in a noisy environment. This work was carried out in joint collaboration and supervision of two pioneers, Prof. Majeed Hayat and Prof. Paymen Zarkesh-Ha, under whom an integrated hardware and algorithm imaging concept called compressed domain imaging device was developed. The hardware used in this study was developed under the supervision of Prof. Payman Zarkesh-Ha by Dr. Javad Ghasemi, a graduate student from his lab. Details of this ``Readout Integrated Circuit" (ROIC) based imaging hardware are presented in Dr. Ghasemi's Ph.D. dissertation. The developed hardware enables compressed-domain image acquisition by means of projecting the input image onto a series of coded apertures, or spatial intensity masks, that are stored in the ROIC. The real-time, pixel-level projection is achieved by the means of designing a set of coded apertures and programming them in the ROIC as a 2D array of analog biases. The stored bias values at each pixel govern the operating point of the photodetector, thereby affecting the spatial intensity modulation of the image. The pixel values are then summed up within the ROIC. The output of this unique camera is therefore the features (or code-words) arising from the projection of the image onto each of the prescribed coded apertures. Hence, the camera does not generate and store the actual image at any point.The image can be reconstructed using a reconstruction algorithm that employs the features computed by the camera. However, due to the limited dynamic range and sensitivity of the hardware towards the noise, the overall performance could degrade significantly during computation of the Discrete Cosine Transform (DCT) and Compressed Sensing (CS) coefficients for data compression; hence the significance and applicability of the unique hardware could diminish considerably at times. This specific type of hardware limitation has opened the scope for this thesis work, where we have performed theoretical and experimental systematic study of the hardware by modeling its response and characterizing the noise statistically. Thus, in this work we develop a robust Minimum Mean Square Error (MMSE) based bias selection algorithm to overcome the effect of noise and formally demonstrate how use of superior signal processing techniques on the statistical model of the hardware response enhances the performance of computational imaging hardware systems.

This dissertation addresses the components necessary for simulation of an image-based recovery of the position of a target using orbital image sensors. Each component is considered in detail, focusing on the effect that design choices and system parameters have on the accuracy of the position estimate. Changes in sensor resolution, varying amounts of blur, differences in image noise level, selection of algorithms used for each component, and lag introduced by excessive processing time all contribute to the accuracy of the result regarding recovery of target coordinates using orbital sensor data. Using physical targets and sensors in this scenario would be cost-prohibitive in the exploratory setting posed, therefore a simulated target path is generated using Bezier curves which approximate representative paths followed by the targets of interest. Orbital trajectories for the sensors are designed on an elliptical model representative of the motion of physical orbital sensors. Images from each sensor are simulated based on the position and orientation of the sensor, the position of the target, and the imaging parameters selected for the experiment (resolution, noise level, blur level, etc.). Post-processing of the simulated imagery seeks to reduce noise and blur and increase resolution. The only information available for calculating the target position by a fully implemented system are the sensor position and orientation vectors and the images from each sensor. From these data we develop a reliable method of recovering the target position and analyze the impact on near-realtime processing. We also discuss the influence of adjustments to system components on overall capabilities and address the potential system size, weight, and power requirements from realistic implementation approaches.

Optical microscopy is an essential tool for biological research, as it allows for non-invasive imaging of small animals. However, optical microscopy has its limits. Due to the low light level, fluorescence microscopy prohibits high speed imaging, making it difficult to study fast dynamic biological processes. In addition, optical blur due to the diffraction of light results in limited spatial resolution, particularly when using objective lenses with low numerical apertures. In this thesis, we propose computational imaging methods to overcome these limitations using a combination of novel image acquisition procedures and reconstruction algorithms.The first part of this thesis deals with improving temporal resolution in fluorescence microscopy to image rapid, repeating processes. We take advantage of multiple acquisitions, each taken with different time delays or temporally modulated illumination patterns, to recover high frequency information that is lost with traditional imaging. We demonstrate our method to image the beating heart in live embryonic zebrafish with reduced motion blur and high resolution in time.The second part of this thesis deals with reducing spatial blur in optical projection tomography, a form of optical microscopy that uses multiple 2D projections to reconstruct a 3D image of an object. We propose a method to reduce the optical distortion (as characterized by the system's optical point spread function) that can be implemented with a scanning acquisition approach combined with a modified filtered backprojection algorithm for reconstruction. We demonstrate our method to image blood vessels in larval zebrafish with high spatial resolution and reduced out-of-focus blur.The final part of this thesis deals with the dimensional limitation of 2D sensors for measuring 3D motion in microscopy. We propose a method to combine two-dimensional motion estimates from multiple views to recover out-of-plane velocity and reconstruct a divergence-free, three-dimensional velocity field. We demonstrate our method to measure, for the first time, dynamic blood flow in 3D inside the beating heart of a live zebrafish using optical microscopy.This thesis provides new tools that integrate custom image acquisition procedures and image reconstruction algorithms to overcome the resolution limitations -- temporal, spatial, and out-of-plane velocity resolution -- in optical microscopy. The methods presented in this thesis, in particular the single camera, active illumination method for temporal superresolution in fluorescence microscopy, will be directly applicable to a broad range of biological studies and will open up new perspectives for imaging small organisms in 3D (and time) with high spatio-temporal resolution.

Sparse aperture image synthesis requires proper phasing between sub-apertures. Phasing can be difficult due to hardware misalignments, atmospheric turbulence, and many other causes of optical path differences (OPD). Common synthesis techniques include incoherent and coherent methods. Incoherent methods utilize passive illumination and adaptive optics while coherent methods rely on active illumination and phase reconstruction approaches such as phase retrieval or spatial heterodyne. In this thesis, we present a partially coherent technique with the capability to use either active or passive illumination to digitally correct for piston phase errors. This technique requires an anamorphic pupil relay system and a piston correction algorithm. The anamorphic pupil relay causes two closely spaced sub-apertures in the entrance pupil to appear to be shifted further apart in the exit pupil. Analytic and numerical wave optics models demonstrate the effectiveness of this relay system, matching with experimental results. An analytic model shows that the higher frequency terms are equivalent to scaled cross-correlations of the two sub-apertures, which are shifted due to the anamorphic separation. The constant shifts due to the separation are found experimentally using a registration algorithm with a calibration target. The cross-correlations are dependent on the piston phase errors between sub-apertures. We show that a piston correction algorithm can be used to shift the cross-correlations to their original positions dictated by the entrance pupil, multiply a cross-correlation with the complex conjugate of the auto-correlation, use the summation of this product to calculate the piston, and correct the phase error in each cross-correlation before recombining them with the auto-correlation. Examples show diffraction limited results for both simulated and experimental images that are supported by analytical, numerical, and experimental analysis of the system’s modulation transfer function (MTF). In addition, analysis of the piston for multiple wavelengths reveals two effects of bandwidth on the system. First, the impact of a constant spatial separation resulting in different spatial frequency shifts for different wavelengths, referred to as field dependent contrast (FDC), is addressed. Second, the inverse relationship between the bandwidth of the system and OPD tolerances is shown. In future research, diffraction gratings could eliminate the FDC so that partially coherent illumination with a small enough system bandwidth could relax OPD requirements for the sparse aperture array.

Imaging systems often make use of macroscopic lenses to manipulate light. Modern microfabrication techniques, however, have opened up a pathway to the development of novel arrayed imaging systems. In such systems, centimeter-scale areas can contain thousands to millions of micro-scale optical elements, presenting exciting opportunities for new imaging applications. We show two such applications in this thesis: pressure sensing in microfluidics and high throughput fluorescence microscopy for high content screening. Conversely, we show that arrayed elements are not always needed for three dimensional light field imaging.