Search

Fluorescence lifetime imaging microscopy (FLIM) is a powerful imaging technique that enables the visualization of biological samples at the molecular level by measuring the fluorescence decay rate of fluorescent probes. This provides critical information about molecular interactions, environmental changes, and localization within biological systems. However, creating high-resolution lifetime maps using conventional FLIM systems can be challenging, as it often requires extensive scanning that can significantly lengthen acquisition times. This issue is further compounded in three-dimensional (3D) imaging because it demands additional scanning along the depth axis. To tackle this challenge, we developed a computational imaging technique called light-field tomographic FLIM (LIFT-FLIM). Our approach allows for the acquisition of volumetric fluorescence lifetime images in a highly data-efficient manner, significantly reducing the number of scanning steps required compared to conventional point-scanning or line-scanning FLIM imagers. Moreover, LIFT-FLIM enables the measurement of high-dimensional data using low-dimensional detectors, which are typically low cost and feature a higher temporal bandwidth. We demonstrated LIFT-FLIM using a linear single-photon avalanche diode array on various biological systems, showcasing unparalleled single-photon detection sensitivity. Additionally, we expanded the functionality of our method to spectral FLIM and demonstrated its application in high-content multiplexed imaging of lung organoids. LIFT-FLIM has the potential to open up broad avenues in both basic and translational biomedical research. [ABSTRACT FROM AUTHOR]

Light field cameras can obtain 3D spatial information about objects by acquiring a single image, demonstrating that light field imaging technology does not require complex light path arrangements. Owing to this capability, light field cameras have recently attracted attention from various fields. Besides, particle streak velocimetry (PSV) is a non-contact image-based flow field measurement technique widely used in optical and image processing technology. It has obvious advantages in measuring high-speed flow fields. Hence, this paper combines light field imaging technology with PSV technology and introduces a new 3D velocity measurement system that exploits the advantages of both technologies to measure flow fields with narrow windows, fast flow rates, and difficult-to-arrange measurement equipment. The system developed constructs a threedimensional flow field measurement method based on light field PSV, which is unique as no relevant research on the PSV 3D velocity measurement utilizes light field imaging. Specifically, the motion streak of a 5 μm point light source moving at different speeds was captured using a light field camera, and its 3D velocity field was verified and calculated using a computing system. The results demonstrated that the 3D flow field measurement method based on light field PSV can successfully measure 3D flow fields. [ABSTRACT FROM AUTHOR]

Deep learning is progressively emerging as a vital tool for image reconstruction in light field microscopy. The present review provides a comprehensive examination of the latest advancements in light field image reconstruction techniques based on deep learning algorithms. First, the review briefly introduced the concept of light field and deep learning techniques. Following that, the application of deep learning in light field image reconstruction was discussed. Subsequently, we classified deep learning-based light field microscopy reconstruction algorithms into three types based on the contribution of deep learning, including fully deep learning-based method, deep learning enhanced raw light field image with numerical inversion volumetric reconstruction, and numerical inversion volumetric reconstruction with deep learning enhanced resolution, and comprehensively analyzed the features of each approach. Finally, we discussed several challenges, including deep neural approaches for increasing the accuracy of light field microscopy to predict temporal information, methods for obtaining light field training data, strategies for data enhancement using existing data, and the interpretability of deep neural networks.

Light field tomography, an optical combustion diagnostic technology, has recently attracted extensive attention due to its easy implementation and non-intrusion. However, the conventional iterative methods are high data throughput, low efficiency and time-consuming, and the existing machine learning models use the radiation spectrum information of the flame to realize the parameter field measurement at the current time. It is still an offline measurement and cannot realize the online prediction of the instantaneous structure of the actual turbulent combustion field. In this work, a novel online prediction model of flame temperature instantaneous structure based on deep convolutional neural network and long short-term memory (CNN-LSTM) is proposed. The method uses the characteristics of local perception, shared weight, and pooling of CNN to extract the three-dimensional (3D) features of flame temperature and outgoing radiation images. Moreover, the LSTM is used to comprehensively utilize the ten historical time series information of high dynamic combustion flame to accurately predict 3D temperature at three future moments. A chaotic time-series dataset based on the flame radiation forward model is built to train and validate the performance of the proposed CNN-LSTM model. It is proven that the CNN-LSTM prediction model can successfully learn the evolution pattern of combustion flame and make accurate predictions. [ABSTRACT FROM AUTHOR]

Hyperspectral imaging is an emerging technology that has gained significant attention in the medical field due to its ability to provide precise and accurate imaging of biological tissues. The current methods of hyperspectral imaging, such as filter-wheel, snapshot, line-scanning, and push-broom cameras have limitations such as low spatial and spectral resolution, slow acquisition time. New developments on the field of light field cameras show the potential to overcome these limitations. In this paper, we use a novel hyperspectral lightfield camera and try to combine the capability of hyperspectral and 3D analysis. For this purpose we calibrate our system and test it during two ENT-surgeries to show its potential for improving surgical outcomes. The micro-lenses of the camera map 66 spectral sub-images onto the sensor allowing to reconstruct the spectral behavior of the captured scene in the spectral range of 350-1000nm. In addition, we use the sensor data to apply a 3D camera calibration pipeline to allow 3D surface reconstruction. We captured 26 calibration images and achieved calibration results in accordance to stated company data. The best calibration showed a re-projection error of 0.55 px. Further, we tested the camera during a parotidectomy and a neck-dissection. The extracted reflectance spectra of the selected venal and arterial regions correspond perfectly to the spectrum of oxygenated and deoxygenated hemoglobin. For the first time, up to our knowledge, a hyperspectral lightfield camera has been used during a surgery. We were able to continuously capture images and analyze the reconstructed spectra of specific tissue types. Further, we are able to use the sensor data of the micro-lens projections to calibrate the multilens camera system for later intraoperative measurement tasks.

In order to accurately recover the complete shape of the measured object with partial occlusion, a system for light field 3-D imaging based on multi-line structured light marking was proposed and a structured light field 3-D imaging method based on the Hough transform was proposed for the image processing problem in this system. The extracted sub-pixel fringe center coordinates were used to Hough transform, then the voting distribution in the Hough parameter space was analyzed, and adaptive range voting and adaptive window strategies were designed. Consequently, the parameters of multiple straight lines in the epipolar plane image can be accurately determined by the proposed method, which does not need to encode the fringe level. The experimental results show that the mean deviation and standard deviation of the fitted plane are 0.0096 mm and 0.0074 mm respectively. The complete shape of measured object is accurately obtained, which utilities the occlusion of different sub-aperture images is inconsistent in light field imaging. The result is helpful for solving the occlusion problem in the 3-D measurement process, and this work provides a reference for the measurement method to obtain complete and efficient 3-D data. [ABSTRACT FROM AUTHOR]

Depth estimation for light field (LF) images is the cornerstone of many applications of light field cameras, such as 3D reconstruction, defects inspection, face liveness detection, and so forth. In recent years, convolutional neural network (CNN) has dominated the primary workhorse for depth estimation. However, the interpretability of the network and the accuracy of the depth estimation results still need to be improved. This paper uses the conditional random field (CRF) theory to explain and model the LF depth estimation. Further, from the perspective of sequence analysis, we extract the sequence features of epipolar plane image (EPI) patches with recurrent neural network (RNN) and serve as the unary term of the energy function in the CRF. Then, a unified neural network (called as LFRNN) is designed to solve the CRF and get the disparity map. Our LFRNN builds upon two-stage architecture, involving a local depth estimation and a depth refinement. In the first part, we design an RNN to analyze the vector sequences in EPI patches and obtain local disparity values. There are two thinking behind the design of this part. The first is the general principle that the slope of the straight line in the EPI is inversely proportional to the depth; the second is our unique observation that those straight lines are distributed in vector sequences. In the second part, continuous CRF is used to optimize the output of the first part. We train LFRNN on a synthetic LF dataset and test it on both synthetic and real-world LF datasets. Quantitative and qualitative results validate the superior performance of our LFRNN over the state-of-the-art methods.

Deep learning (DL) provides an effective approach for light field (LF) reconstruction that aims to synthesize novel views from sparsely-sampled views. However, it is challenging to address domain asymmetry when adopting spatial-angular interaction LF reconstruction methods. To overcome this problem, a view-selective angular feature extraction block (VS-LFAFE) is proposed to obtain full-resolution angular features that enumerate whole viewpoints in a macropixel. By applying the VS-LFAFE, a novel LF reconstruction method is proposed, consisting of two subblocks: a spatial-angular feature extraction and fusion block, and an angular upsampling block. Experimental results demonstrate the effectiveness of the VS-LFAFE, and validate that the proposed method can achieve superior performance compared with the state-of-the-art methods.

Robust lane detection is imperative for the realization of intelligent transportation. Recently, vision-based systems that employ deep convolution neural networks (CNNs) for lane detection have made considerable progress. However, for better generalization under various road conditions learning-based methods require excessive training data, which becomes non-trivial in challenging conditions such as illumination variation, shadows, false lane lines, and worn lane markings, etc. In this paper, we propose a light field (LF) based lane detection method that utilizes the additional angular information for improved prediction and increased robustness. Two different LF representations are investigated to study the possibility of maximum performance improvement and minimal additional computation cost and data labeling efforts. Experimental results successfully demonstrate that the proposed approach improves the prediction of the lane line point coordinates and is significantly robust against the aforementioned adverse conditions.

Copyright of Chinese Journal of Liquid Crystal & Displays is the property of Chinese Journal of Liquid Crystal & Displays and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Fast neutron identification and spectroscopy is of great interest to nuclear physics experiments. Using the neutron elastic scattering, the fast neutron momentum can be measured. Wang and Morris introduced the theoretical concept that the initial fast neutron momentum can be derived from up to three consecutive elastic collisions between the neutron and the target, including the information of two consecutive recoil ion tracks and the vertex position of the third collision or two consecutive elastic collisions with the timing information. Here, we also include the additional possibility of measuring the deposited energies from the recoil ions. In this paper, we simulate the neutron elastic scattering using the Monte Carlo N-Particle Transport Code (MCNP) and study the corresponding neutron detection and tracking efficiency. The corresponding efficiency and the scattering distances are simulated with different target materials, especially natural silicon (92.23%28Si, 4.67%29Si, and 3.1%30Si) and helium-4 (4He). The timing of collision and the recoil ion energy are also investigated, which are important characters for the detector design. We also calculate the ion traveling range for different energies using the software, “The Stopping and Range of Ions in Matter (SRIM)”, showing that the ion track can be most conveniently observed in 4He unless sub-micron spatial resolution can be obtained in silicon.

The luminance dynamic range in natural scenes is exceedingly wide. However, capturing the wide dynamic range information of natural scenes through a single exposure is challenging for commercial light field cameras. One solution to expand the luminance dynamic range of captured light field images is to employ multi-exposure imaging techniques. Nevertheless, this approach may introduce various distortions during the image fusion procedure, especially for dynamic scenes. To evaluate the visual quality of multi-exposure fused light field images (MEFLFIs) and compare the performance of different multi-exposure light field fusion methods, we initially conduct a subjective quality assessment on MEFLFIs for dynamic scenes and establish the first MEFLFI benchmark dataset, which comprises 480 fused light field images generated by eleven state-of-the-art multi-exposure fusion algorithms and two tone-mapping algorithms, along with their corresponding subjective scores. Subsequently, a novel specially designed objective quality assessment metric for MEFLFIs is developed, which incorporates both local–global joint features and angular quality-aware features. The key contribution of this work lies in establishing a pioneering benchmark dataset for MEFLFIs along with a novel tailored objective quality metric. Experimental results conducted over our established MEFLFI database demonstrate that the proposed metric achieves superior performance, with Pearson linear correlation coefficient (PLCC) and Spearman rank correlation coefficient (SROCC) values of 0.8895 and 0.8658 respectively, representing a significant improvement of approximately 5.7 % and 4.9 % compared to the second-best specifically-designed LFI quality metric available. Furthermore, ablation experiments have confirmed that the integration of local–global joint features and angular features significantly enhances the performance of our objective metric, resulting in a better alignment with human visual perception. [ABSTRACT FROM AUTHOR]

Compressed sensing using color-coded masks has been recently considered for capturing light fields using a small number of measurements. Such an acquisition scheme is very practical, since any consumer-level camera can be turned into a light field acquisition camera by simply adding a coded mask in front of the sensor. We present an efficient and mathematically grounded deep learning model to reconstruct a light field from a set of measurements obtained using a color-coded mask and a color filter array (CFA). Following the promising trend of unrolling optimization algorithms with learned priors, we formulate our task of light field reconstruction as an inverse problem and derive a principled deep network architecture from this formulation. We also introduce a closed-form extraction of information from the acquisition, while similar methods found in the recent literature systematically use an approximation. Compared to similar deep learning methods, we show that our approach allows for a better reconstruction quality. We further show that our approach is robust to noise using realistic simulations of the sensing acquisition process.

Deep learning is progressively emerging as a vital tool for image reconstruction in light field microscopy. The present review provides a comprehensive examination of the latest advancements in light field image reconstruction techniques based on deep learning algorithms. First, the review briefly introduced the concept of light field and deep learning techniques. Following that, the application of deep learning in light field image reconstruction was discussed. Subsequently, we classified deep learning-based light field microscopy reconstruction algorithms into three types based on the contribution of deep learning, including fully deep learning-based method, deep learning enhanced raw light field image with numerical inversion volumetric reconstruction, and numerical inversion volumetric reconstruction with deep learning enhanced resolution, and comprehensively analyzed the features of each approach. Finally, we discussed several challenges, including deep neural approaches for increasing the accuracy of light field microscopy to predict temporal information, methods for obtaining light field training data, strategies for data enhancement using existing data, and the interpretability of deep neural networks., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2024 Lin, Tian, Zhang, Zhu and Wang.)

Fluorescence lifetime imaging microscopy (FLIM) is a powerful imaging technique that enables the visualization of biological samples at the molecular level by measuring the fluorescence decay rate of fluorescent probes. This provides critical information about molecular interactions, environmental changes, and localization within biological systems. However, creating high-resolution lifetime maps using conventional FLIM systems can be challenging, as it often requires extensive scanning that can significantly lengthen acquisition times. This issue is further compounded in three-dimensional (3D) imaging because it demands additional scanning along the depth axis. To tackle this challenge, we developed a computational imaging technique called light-field tomographic FLIM (LIFT-FLIM). Our approach allows for the acquisition of volumetric fluorescence lifetime images in a highly data-efficient manner, significantly reducing the number of scanning steps required compared to conventional point-scanning or line-scanning FLIM imagers. Moreover, LIFT-FLIM enables the measurement of high-dimensional data using low-dimensional detectors, which are typically low cost and feature a higher temporal bandwidth. We demonstrated LIFT-FLIM using a linear single-photon avalanche diode array on various biological systems, showcasing unparalleled single-photon detection sensitivity. Additionally, we expanded the functionality of our method to spectral FLIM and demonstrated its application in high-content multiplexed imaging of lung organoids. LIFT-FLIM has the potential to open up broad avenues in both basic and translational biomedical research., Competing Interests: Competing interests statement:Edoardo Charbon holds the position of Chief Scientific Officer at Fastree3D, a company that specializes in the manufacturing of LiDARs for the automotive market. Claudio Bruschini and Edoardo Charbon are also cofounders of Pi Imaging Technology. Additionally, Liang Gao has a financial interest in Lift Photonics, which commercializes the LIFT technology for FLIM applications. However, it is important to note that none of these companies were involved in the research presented in this paper. The authors disclose UCLA provisional patent filing. Research support was provided by the National Institutes of Health (R01HL165318 and RF1NS128488).

Abstract Light fields are vector functions that map the geometry of light rays to the corresponding plenoptic attributes. They describe the holographic information of scenes by representing the amount of light flowing in every direction through every point in space. The physical concept of light fields was first proposed in 1936, and light fields are becoming increasingly important in the field of computer graphics, especially with the fast growth of computing capacity as well as network bandwidth. In this article, light field imaging is reviewed from the following aspects with an emphasis on the achievements of the past five years: (1) depth estimation, (2) content editing, (3) image quality, (4) scene reconstruction and view synthesis, and (5) industrial products because the technologies of lights fields also intersect with industrial applications. State-of-the-art research has focused on light field acquisition, manipulation, and display. In addition, the research has extended from the laboratory to industry. According to these achievements and challenges, in the near future, the applications of light fields could offer more portability, accessibility, compatibility, and ability to visualize the world.

To study the three-dimensional motion characteristics of bubbles in a bubble slurry bed, a simultaneous measurement method for bubble size and velocity based on a light field imaging particle tracking velocimetry (PTV) technology was proposed. A light-field PTV bubble measurement system was constructed, the light-field camera was used to collect two consecutive light field images. After refocusing calculation, image processing and sharpness evaluation, the size and three-dimensional coordinates of the bubbles were calculated combined with the calibration results. The PTV technology was used to calculate the three-dimensional velocity of the bubble, and the simultaneous measurement of bubble size and three-dimensional velocity was completed. The evaluation of the light field PTV system and the measurement of bubble size and three-dimensional velocity were carried out. The results show that the relative error of the diameter measurement is less than 1.60%, the repeatability error of the diameter measurement is less than 2. 48%, and the relative error of the depth position measurement is less than 7.14% . When the air flow increases from 0.1 L/min to 0.5 L/min, the average equivalent diameter of the bubbles increases from 3. 85 mm to 4. 36 mm, and the average bubble velocity increases from 0. 34 m/s to 0. 38 m/s. The light field imaging PTV technology can be used for simultaneous measurement of the bubble size and the three-dimensional velocity. [ABSTRACT FROM AUTHOR]

An unfocused plenoptic light field (LF) camera places an array of microlenses in front of an image sensor in order to separately capture different directional rays arriving at an image pixel. Using a conventional Bayer pattern, data captured at each pixel is a single color component (R, G or B). The sensed data then undergoes demosaicking (interpolation of RGB components per pixel) and conversion to an array of sub-aperture images (SAIs). In this paper, we propose a new LF image coding scheme based on graph lifting transform (GLT), where the acquired sensor data are coded in the original captured form without pre-processing. Specifically, we directly map raw sensed color data to the SAIs, resulting in sparsely distributed color pixels on 2D grids, and perform demosaicking at the receiver after decoding. To exploit spatial correlation among the sparse pixels, we propose a novel intra-prediction scheme, where the prediction kernel is determined according to the local gradient estimated from already coded neighboring pixel blocks. We then connect the pixels by forming a graph, modeling the prediction residuals statistically as a Gaussian Markov Random Field (GMRF). The optimal edge weights are computed via a graph learning method using a set of training SAIs. The residual data is encoded via low-complexity GLT. Experiments show that at high PSNRs—important for archiving and instant storage scenarios—our method outperformed significantly a conventional light field image coding scheme with demosaicking followed by High Efficiency Video Coding (HEVC). [ABSTRACT FROM AUTHOR]

Copyright of Frontiers of Information Technology & Electronic Engineering is the property of Springer Nature and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Meta-lens has successfully been developed for a variety of optical functions. We demonstrate a light-field edge detection imaging system with a gallium nitride achromatic meta-lens array. It enables edge detection from one dimension to three dimensions. The designed meta-lens array consists of 60 by 60 achromatic meta-lenses, which operate in the visible range from 400 to 660 nm. All of the light field information of objects in the scene can be captured and computed. The focused edge images from one dimension to three dimensions are extracted with depth estimation by image rendering. Three dimensions edge detection is two dimensions edge imaging with depth information. The focused edge images can be obtained by the sub-image reconstruction of the light field image. Our multidimensional edge detection system by achromatic meta-lens array brings novel advantages, such as broadband detection, data volume reduction, and device miniaturization capacity. Results of our experiments show new insight into applications of biological diagnose and robotic vision.

• A novel approach is proposed to retrieve the 3-D soot absorption coefficient and temperature of flames. • Achieves higher accuracy in 3-D temperature reconstruction and reduces reconstruction time. • Numerical simulations are carried out to evaluate the proposed approach. • The performance of the proposed method is evaluated through ethylene/air diffusion flames. The accuracy and efficiency of 3-D flame temperature field reconstruction using the light field imaging technique heavily depend on soot radiation characteristics. In this study, we employ the line-of-sight attenuation method to reconstruct the soot absorption coefficient distribution in a pure absorbing flame. Utilizing these soot absorption coefficients, the radiative transfer equation is effectively transformed within the framework of the light field imaging technique into a linear inverse problem and also outlines the flame boundary. This proposed strategy reduces the unwanted detection rays significantly, thus eliminating the extensive computational processing. Consequently, the proposed approach substantially enhances the accuracy and efficiency of flame temperature reconstruction. Numerical simulations were carried out on a bimodal asymmetric flame to validate the noise tolerance capabilities, investigate the effects of varying voxel numbers on flame division and carry out a comparative study. Experimental studies were also conducted to reconstruct flame temperature and soot absorption coefficient distributions under different combustion operating conditions. Thermocouple measurements were performed and compared with the reconstructed temperatures. Outcomes achieved from both numerical simulations and experimental studies demonstrate the feasibility, accuracy and robustness of the proposed method. [ABSTRACT FROM AUTHOR]

Fast neutron identification and spectroscopy is of great interest to nuclear physics experiments. Using the neutron elastic scattering, the fast neutron momentum can be measured. Wang and Morris introduced the theoretical concept that the initial fast neutron momentum can be derived from up to three consecutive elastic collisions between the neutron and the target, including the information of two consecutive recoil ion tracks and the vertex position of the third collision or two consecutive elastic collisions with the timing information. Here, we also include the additional possibility of measuring the deposited energies from the recoil ions. In this paper, we simulate the neutron elastic scattering using the Monte Carlo N-Particle Transport Code (MCNP) and study the corresponding neutron detection and tracking efficiency. The corresponding efficiency and the scattering distances are simulated with different target materials, especially natural silicon (92.23% 28 Si, 4.67% 29 Si, and 3.1% 30 Si) and helium-4 ( 4 He). The timing of collision and the recoil ion energy are also investigated, which are important characters for the detector design. We also calculate the ion traveling range for different energies using the software, "The Stopping and Range of Ions in Matter (SRIM)", showing that the ion track can be most conveniently observed in 4 He unless sub-micron spatial resolution can be obtained in silicon. [ABSTRACT FROM AUTHOR]

Compared to conventional photography which allows capturing spatial intensity only, light field imaging can capture angular and spatial information by collecting information from all directions. This massive information could be used in many applications such as depth estimation and post-capture refocusing. In addition, the emergence of consumer light field cameras has resulted in the widespread use of light field imaging. However, its limited resolution constitutes a major drawback to the use of enormous capabilities provided. In this article, we tried to alleviate the effect of this drawback by using a machine learning algorithm. Our proposed network is built upon existing reconstruction techniques that divide the process into disparity estimation and final image reconstruction. This is achieved by using two consecutive neural networks while the whole network is trained simultaneously. We propose to use a predefined convolutional network at the first stage to decrease preprocessing time, and in addition, we use dual disparity vectors to alleviate the interpolation error whereas warping input images. Our system was trained to reconstruct multi-angular resolution images fast and accurately using real light field images for training. Experimental results demonstrate that the proposed system can reconstruct high-quality images faster than the state-of-the-art techniques.

Deep learning undoubtedly has had a huge impact on the computer vision community in recent years. In light field imaging, machine learning-based applications have significantly outperformed their conventional counterparts. Furthermore, multi- and hyperspectral light fields have shown promising results in light field-related applications such as disparity or shape estimation. Yet, a multispectral light field dataset, enabling data-driven approaches, is missing. Therefore, we propose a new synthetic multispectral light field dataset with depth and disparity ground truth. The dataset consists of a training, validation and test dataset, containing light fields of randomly generated scenes, as well as a challenge dataset rendered from hand-crafted scenes enabling detailed performance assessment. Additionally, we present a Python framework for light field deep learning. The goal of this framework is to ensure reproducibility of light field deep learning research and to provide a unified platform to accelerate the development of new architectures. The dataset is made available under dx.doi.org/10.21227/y90t-xk47. The framework is maintained at gitlab.com/iiit-public/lfcnn.

Light field imaging technology is a new imaging method for recording light propagation. It records light mainly through two devices: a camera array and an all-optical camera. The collected light field data can be processed in different ways such as post hoc on-demand imaging and depth estimation. This technology has high application value in the fields of depth extraction, 3D modeling, and virtual reality, and is one of the hot spots of imaging research. However, existing light field acquisition methods are based on angular sampling, which requires special hardware. The angular and spatial resolutions of the light field collected by these methods are mutually constrained. Accordingly, we propose a new acquisition method for the depth sampling of the light field and compare the clarity of light field subaperture images acquired by this method with that of the images acquired by the traditional angular sampling method. The results of the study show that the spatial resolution of the light field images acquired with depth sampling reaches the sensor's pixel dimensions without special hardware, and their sharpness is very close to that of light field images acquired by the angular sampling method. This study provides a more concise approach to the acquisition of light field images in computational imaging techniques. [ABSTRACT FROM AUTHOR]

Light fields are vector functions that map the geometry of light rays to the corresponding plenoptic attributes. They describe the holographic information of scenes by representing the amount of light flowing in every direction through every point in space. The physical concept of light fields was first proposed in 1936, and light fields are becoming increasingly important in the field of computer graphics, especially with the fast growth of computing capacity as well as network bandwidth. In this article, light field imaging is reviewed from the following aspects with an emphasis on the achievements of the past five years: (1) depth estimation, (2) content editing, (3) image quality, (4) scene reconstruction and view synthesis, and (5) industrial products because the technologies of lights fields also intersect with industrial applications. State-of-the-art research has focused on light field acquisition, manipulation, and display. In addition, the research has extended from the laboratory to industry. According to these achievements and challenges, in the near future, the applications of light fields could offer more portability, accessibility, compatibility, and ability to visualize the world. [ABSTRACT FROM AUTHOR]

The ability of camera arrays to efficiently capture higher space-bandwidth product than single cameras has led to various multiscale and hybrid systems. These systems play vital roles in computational photography, including light field imaging, 360 VR camera, gigapixel videography, etc. One of the critical tasks in multiscale hybrid imaging is matching and fusing cross-resolution images from different cameras under perspective parallax. In this paper, we investigate the reference-based super-resolution (RefSR) problem associated with dual-camera or multi-camera systems. RefSR consists of super-resolving a low-resolution (LR) image given an external high-resolution (HR) reference image, where they suffer both a significant resolution gap ($8\times$ 8 × ) and large parallax ($\sim 10\%$ ∼ 10 % pixel displacement). We present CrossNet++, an end-to-end network containing novel two-stage cross-scale warping modules, image encoder and fusion decoder. The stage I learns to narrow down the parallax distinctively with the strong guidance of landmarks and intensity distribution consensus. Then the stage II operates more fine-grained alignment and aggregation in feature domain to synthesize the final super-resolved image. To further address the large parallax, new hybrid loss functions comprising warping loss, landmark loss and super-resolution loss are proposed to regularize training and enable better convergence. CrossNet++ significantly outperforms the state-of-art on light field datasets as well as real dual-camera data. We further demonstrate the generalization of our framework by transferring it to video super-resolution and video denoising. [ABSTRACT FROM AUTHOR]

A reconstruction algorithm for flame 3D temperature distribution based on deep leaning and light field imaging was proposed. The deep feature of flame light field image was extracted by the convolutional neural network, and then the mapping between light field imaging and 3D temperature field is established to achieve the rapid reconstruction of flame 3D temperature distribution. Through the data set of flame light field image and 3D temperature distribution obtained by line-of-sight, the convolutional neural network was trained, and the test set was used to verify and evaluate the training results. The reconstruction results by the convolutional neural network algorithm and the traditional non-negative last squares (NNLS) algorithm were compared. The results show that the flame 3D temperature distribution reconstruction algorithm based on deep leaning and light field imaging can accurately reconstruct the flame temperature distribution with the average relative error of 0.14% and has high computational efficiency(when the flame grid is 10 x8 x15, the reconstruction time by NNLS algorithm is 4 759 s, and that by the deep leaning algorithm is 830 ^s). Also, the proposed algorithm is robust to image noise. [ABSTRACT FROM AUTHOR]

Light field (LF) imaging sensors based on micro-lens array are susceptible to numerous noise pollution when collecting raw 4D LF data due to their own special structural design, which affects visual perception and subsequent applications such as depth estimation. Unfortunately, existing 2D image denoising methods are difficult to directly apply to 4D LF images. To this end, this paper proposes a new LF denoising method based on spatial-angular collaborative encoding network, considering the inherent 4D structure of LF image. Specifically, the convolutions in the spatial and angular branches are first constructed to extract specific 2D spatial and 2D angular features from noisy LF data. Then, a tailored spatial-angular collaborative encoder is designed to co-process spatial-angular features and improve the expression ability of features. After that, the spatial-angular feature fusion module is constructed to fuse the extracted features. Finally, the denoised LF image is reconstructed by a residual prediction module integrating attention mechanism. In particular, the proposed method simultaneously reconstructs all sub-aperture images of LF in one forward inference, so as to preserve the angular consistency of the denoised LF image. Extensive experimental results show that the proposed method outperforms the state-of-the-art methods in both subjective visual perception and objective quality evaluation. Furthermore, the proposed method preserves the parallax structure well, which is beneficial for subsequent LF applications. [ABSTRACT FROM AUTHOR]

• L p -norm relaxation approach combined with non-negative constraint is proposed. • Impact of sampling characteristics on ill-posedness of inverse problem is analyzed. • Robustness and validity of L p -norm are investigated under measurement noise. • Experiment system is built to implement premixed flame 3D temperature measurement. Optical imaging technology based on flame spontaneous radiation is a promising tool for combustion diagnosis. However, the existing radiation thermometry technology faces the problems of low angular resolution and low efficiency in solving radiative transfer. Therefore, a novel optical imaging model is proposed, which combines light field imaging theory with equation-solving integral equation method based on radiation distribution factor to effectively decouple the radiative properties related to transfer process and the temperature-related source terms. Aiming at the problem that traditional method may cause over-smooth effect, L p -norm regularization based on non-negative constraint is established. Results show that L p -norm can improve reconstruction quality and robustness, but p value needs to be reasonably selected according to noise level and flame inhomogeneity. Experimental study of ethylene/air-premixed flames shows that the methodology can realize 3D measurement (900 ∼ 1550 K). Compared with laser absorption spectroscopy and thermocouple, error is better than 3 % at height above burner 15 mm. [ABSTRACT FROM AUTHOR]

• A novel learning-based method is proposed for ghost-free high dynamic range (HDR) light field imaging. • A multi-scale architecture integrating deformable alignment module and angular embedding module is designed. • A new large-scale benchmark dataset is established to serve the HDR light field imaging task for dynamic scenes. • The proposed method achieves superior spatial quality and preserves accurate angular consistency. Light field (LF) imaging is an effective way to enable immersive applications. However, limited by the potential well capacity of the image sensor, the acquired LF images suffer from low dynamic range and are thus prone to under-exposure or over-exposure. High dynamic range (HDR) LF imaging is an efficacious avenue to improve the LF imaging's dynamic range. Unfortunately, for dynamic scenes, existing methods are inclined to produce ghosting artifacts and lose details in the saturated regions, while potentially damaging the parallax structure of generated HDR LF images. To address the above challenges, in this paper, we propose a new ghost-free HDR LF imaging method based on a deformable aggregation and angular embedding network. Specifically, considering the four-dimensional geometric structure of the LF image, a deformable alignment module is first designed to handle dynamic regions in the spatial domain, and then the aligned spatial features are fully fused through an aggregation operation. Subsequently, an angular embedding module is constructed to explore angular information to enhance the aggregated spatial features. Based on this, the above two modules are cascaded in a multi-scale manner to achieve multi-level feature extraction and enhance the feature representation ability. Finally, a decoder is leveraged to recover the ghost-free HDR LF image from the enhanced multi-scale features. For performance evaluation, this paper establishes a large-scale benchmark dataset with multi-exposure inputs and ground truth images. Extensive experimental results show that the proposed method generates visually pleasing HDR LF images while preserving accurate angular consistency. Moreover, the proposed method surpasses the state-of-the-art methods in both quantitative and qualitative comparisons. The code and dataset will be available at https://github.com/YeyaoChen/HDRLFI. [ABSTRACT FROM AUTHOR]

We present a novel multi-flash light field photography (MFLF) technique that couples a light field camera with a ring of four flash lights to combine the benefits of both imaging systems. The multi-flash photography technique enables reliable detection of occlusion edges at individual subaperture images (light field views) in the light field image. We then separately treat occlusion vs. non-occlusion edges in light field stereo matching. For non-occlusion edge pixels, we conduct correspondence matching and depth-from-focus. For pixels on or near the occlusion boundary, we estimate the shape of the occlusion boundary using a novel Support Vector Machine (SVM) approach. We collect an MFLF dataset of synthetic and real scenes that exhibit strong occlusions. The extensive experiments show that our method significantly outperforms the state-of-the-art light field stereo matching methods in accuracy and robustness.

In this paper, we present a deep nonparametric Bayesian method to synthesize a light field from a single image. Conventionally, light-field capture requires special optical architecture, and the gain in angular resolution often comes at the expense of a reduction in spatial resolution. Techniques for computationally generating the light field from a single image can be expanded further to a variety of applications, ranging from microscopy and materials analysis to vision-based robotic control and autonomous vehicles. We treat the light field as multiple sub-aperture views, and to compute the novel viewpoints, our model contains three major components. First, a convolutional neural network is used for predicting the depth probability map from the image. Second, a multi-scale feature dictionary is constructed within a multi-layer dictionary learning network. Third, the novel views are synthesized taking into account both the probabilistic depth map and the multi-scale feature dictionary. The experiments show that our method outperforms several state-of-the-art novel view synthesis methods in delivering good image resolution.

In recent years, optical field imaging technology has received extensive attention in the academic circle for its novel imaging characteristics of shooting first and focusing later, variable depth of field, variable viewpoint, and so on. However, the existing optical field acquisition equipment can only acquire a limited number of discrete angle signals, so image aliasing caused by under sampling of optical field angle signals reduces the quality of optical field images. Based on the camera array system as a platform, this paper studies the optical field imaging and depth estimation method based on the Big Data in Internet of Things obtained from camera array around the angle sampling characteristics of the optical field data set, and has achieved some innovative research results in the following aspects. On the basis of analyzing the characteristics of different depth clues in the optical field data set, a depth estimation method combining parallax method and focusing method is proposed. First, this paper analyzes the disparity clues and focus clues contained in the multi-view data set and the light field refocusing image set of the camera array, respectively, and points out the differences and relationships between the two depth clues extraction methods in the light field sampling frequency domain space, that is, the disparity method focuses on the energy concentration characteristics near the frequency domain spatial angle axis, while the focus method focuses on the high frequency proportion of energy distribution on the angle axis. Then, the weighted linear fusion method based on image gradient is used to fuse the two calculation results, which improves the accuracy and robustness of depth estimation. Finally, the results of depth estimation experiments on different sets of scenes show that compared with the method based on a single depth cue, the method in this paper has higher accuracy in depth calculation in discontinuous areas of scene depth and similar texture areas.

Recognizing a fake pedestrian on a planar surface (2-D fake pedestrian) is an essential and challenging task in the field of machine vision because of its wide range of applications. In this paper, a 2-D fake pedestrian recognition method is proposed based on light-field (LF) imaging and support vector machine (SVM). This method can recognize a 2-D fake pedestrian with only one sensor in a single exposure. To evaluate the method, we construct a new pedestrian dataset comprising more than 1000 samples using LF imaging. The experimental results show that the highest accuracy of this method is greater than 96%. Moreover, due to the efficient SVM classifier and obvious shape difference between a 2-D fake pedestrian and the real one in LF depth images, an accessible accuracy (85%) will be obtained even if the number of training samples is not large (120 training samples). This preliminary work demonstrates that the proposed approach is quite valid for the recognition of a 2-D fake pedestrian and has potential application in the machine vision field.

Owning to the recorded light ray distributions, light field contains much richer information and provides possibilities of some enlightening applications, and it has becoming more and more popular. To facilitate the relevant applications, many light field processing techniques have been proposed recently. These operations also bring the loss of visual quality, and thus there is need of a light field quality metric to quantify the visual quality loss. To reduce the processing complexity and resource consumption, light fields are generally sparsely sampled, compressed, and finally reconstructed and displayed to the users. We consider the distortions introduced in this typical light field processing chain, and propose a full-reference light field quality metric. Specifically, we measure the light field quality from three aspects: global spatial quality based on view structure matching, local spatial quality based on near-edge mean square error, and angular quality based on multi-view quality analysis. These three aspects have captured the most common distortions introduced in light field processing, including global distortions like blur and blocking, local geometric distortions like ghosting and stretching, and angular distortions like flickering and sampling. Experimental results show that the proposed method can estimate light field quality accurately, and it outperforms the state-of-the-art quality metrics which may be effective for light field. [ABSTRACT FROM AUTHOR]

Terahertz focal plane array imaging methods, direct camera imaging and conventional light field imaging methods are incapable of resolving and separating layers of multilayer objects. In this paper, for the purpose of fast, high-resolution and layer-resolving imaging of multilayer structures with different reflection characteristics, a novel angular intensity filtering (AIF) method based on terahertz light-field imaging is purposed. The method utilizes the extra dimensional information from the 4D light field and the reflection characteristics of the imaging object, and the method is capable to resolve and reconstruct layers individually. The feasibility of the method is validated by experiment on both “idealized” and “practical” multilayer samples, and the advantages in performance of the method are proven by quantitative comparison with conventional methods.

The integral imaging system has received considerable research attention because it can be applied to real-time three-dimensional image displays with a continuous view angle without supplementary devices. Most previous approaches place a physical micro-lens array in front of the image, where each lens looks different depending on the viewing angle. A computational integral imaging system with a virtual micro-lens arrays has been proposed in order to provide flexibility for users to change micro-lens arrays and focal length while reducing distortions due to physical mismatches with the lens arrays. However, computational integral imaging methods only represent part of the whole image because the size of virtual lens arrays is much smaller than the given large-scale images when dealing with large-scale images. As a result, the previous approaches produce sub-aperture images with a small field of view and need additional devices for depth information to apply to integral imaging pickup systems. In this paper, we present a single image-based computational RGB-D integral imaging pickup system for a large field of view in real time. The proposed system comprises three steps: deep learning-based automatic depth map estimation from an RGB input image without the help of an additional device, a hierarchical integral imaging system for a large field of view in real time, and post-processing for optimized visualization of the failed pickup area using an inpainting method. Quantitative and qualitative experimental results verify the proposed approach’s robustness.