Search

Histopathology has remained the gold standard for surgical margin assessment for decades. However, routine pathological examination based on formalin-fixed and paraffin-embedded (FFPE) tissues is laborious and time-consuming, failing to guide surgeons intraoperatively. Here, we propose a rapid, label-free, and non-destructive histological imaging method, termed microscopy with ultraviolet single-plane illumination (MUSI). With intrinsic fluorescence from deep ultraviolet excitation, MUSI enables both ex vivo and in vivo imaging of fresh and unprocessed tissues at the subcellular level with an imaging speed of 0.5 mm2/s, generating high-quality optically sectioned surface images from irregular surgical tissues with a long depth-of-field. We demonstrate that MUSI could differentiate between different subtypes of human lung adenocarcinomas (e.g., lepidic, acinar, papillary, and micropapillary), revealing diagnostically important features that are comparable to the gold standard FFPE histology. As an assistive imaging platform, MUSI can provide immediate feedback to surgeons and pathologists for intraoperative decision-making, holding great promise to revolutionize the current clinical practice in surgical pathology. © 2024 Author(s).

Slide-free imaging techniques have shown great promise in improving the histological workflow. For example, computational high-throughput autofluorescence microscopy by pattern illumination (CHAMP) has achieved high resolution with a long depth of field, which, however, requires a costly ultraviolet laser. Here, simply using a low-cost light-emitting diode (LED), we propose a deep learning-assisted framework of enhanced widefield microscopy, termed EW-LED, to generate results similar to CHAMP (the learning target). Comparing EW-LED and CHAMP, EW-LED reduces the cost by 85×, shortening the image acquisition time and computation time by 36× and 17×, respectively. This framework can be applied to other imaging modalities, enhancing widefield images for better virtual histology.

Hematologists evaluate alterations in blood cell enumeration and morphology to confirm peripheral blood smear findings through manual microscopic examination. However, routine peripheral blood smear analysis is both time-consuming and labor-intensive. Here, we propose using smartphone-based autofluorescence microscopy (Smart-AM) for imaging label-free blood smears at subcellular resolution with automatic hematological analysis. Smart-AM enables rapid and label-free visualization of morphological features of normal and abnormal blood cells (including leukocytes, erythrocytes, and thrombocytes). Moreover, assisted with deep-learning algorithms, this technique can automatically detect and classify different leukocytes with high accuracy, and transform the autofluorescence images into virtual Giemsa-stained images which show clear cellular features. The proposed technique is portable, cost-effective, and user-friendly, making it significant for broad point-of-care applications.

Documents are 2-dimensional carriers of written communication, and as such their interpretation requires a multi-modal approach where textual and visual information are efficiently combined. Document Visual Question Answering (Document VQA), due to this multi-modal nature, has garnered significant interest from both the document understanding and natural language processing communities. The state-of-the-art single-page Document VQA methods show impressive performance, yet in multi-page scenarios, these methods struggle. They have to concatenate all pages into one large page for processing, demanding substantial GPU resources, even for evaluation. In this work, we propose a novel method and efficient training strategy for multi-page Document VQA tasks. In particular, we employ a visual-only document representation, leveraging the encoder from a document understanding model, Pix2Struct. Our approach utilizes a self-attention scoring mechanism to generate relevance scores for each document page, enabling the retrieval of pertinent pages. This adaptation allows us to extend single-page Document VQA models to multi-page scenarios without constraints on the number of pages during evaluation, all with minimal demand for GPU resources. Our extensive experiments demonstrate not only achieving state-of-the-art performance without the need for Optical Character Recognition (OCR), but also sustained performance in scenarios extending to documents of nearly 800 pages compared to a maximum of 20 pages in the MP-DocVQA dataset. Our code is publicly available at \url{https://github.com/leitro/SelfAttnScoring-MPDocVQA}., Comment: Accepted to ICDAR2024

Document understanding models have recently demonstrated remarkable performance by leveraging extensive collections of user documents. However, since documents often contain large amounts of personal data, their usage can pose a threat to user privacy and weaken the bonds of trust between humans and AI services. In response to these concerns, legislation advocating ``the right to be forgotten" has recently been proposed, allowing users to request the removal of private information from computer systems and neural network models. A novel approach, known as machine unlearning, has emerged to make AI models forget about a particular class of data. In our research, we explore machine unlearning for document classification problems, representing, to the best of our knowledge, the first investigation into this area. Specifically, we consider a realistic scenario where a remote server houses a well-trained model and possesses only a small portion of training data. This setup is designed for efficient forgetting manipulation. This work represents a pioneering step towards the development of machine unlearning methods aimed at addressing privacy concerns in document analysis applications. Our code is publicly available at \url{https://github.com/leitro/MachineUnlearning-DocClassification}., Comment: Accepted to ICDAR2024

Three-dimensional (3D) histopathology involves the microscopic examination of a specimen, which plays a vital role in studying tissue's 3D structures and the signs of diseases. However, acquiring high-quality histological images of a whole organ is extremely time-consuming (e.g., several weeks) and laborious, as the organ has to be sectioned into hundreds or thousands of slices for imaging. Besides, the acquired images are required to undergo a complicated image registration process for 3D reconstruction. Here, by incorporating a recently developed vibratome-assisted block-face imaging technique with deep learning, we developed a pipeline termed HistoTRUST that can rapidly and automatically generate subcellular whole organ's virtual hematoxylin and eosin (H & E) stained histological images, which can be reconstructed into 3D by simple image stacking (i.e., without registration). The performance and robustness of HistoTRUST have been successfully validated by imaging all six organs (e.g., brain, heart, liver, lung, kidney, and spleen). The imaging process for a whole organ takes hours to days, depending on the volume of imaged samples. The generated 3D dataset has the same color tune as the traditional H & E stained histological images. Therefore, the virtual H & E stained images can be directly analyzed by pathologists. HistoTRUST has a high potential to serve as a new standard in providing 3D histology for research or clinical applications.

High-resolution and label-free histological imaging modalities provide cell nuclear contrast that is analogous to standard hematoxylin and eosin (H&E) histological staining. Recently, a rapid and slide-free imaging technique, termed computational high-throughput autofluorescence microscopy by pattern illumination (CHAMP), has been developed to image thick and unprocessed tissues with subcellular resolution. Here, we propose a fast and low-cost light-emitting diode (LED) based CHAMP imaging system assisted by deep learning. The low-resolution widefield LED images can be translated into high-resolution LED-CHAMP images that highly resemble Laser-CHAMP images by enhanced super-resolution generative adversarial networks (ESRGAN). Moreover, LED-CHAMP images can be further translated into virtual H&E-stained images comparable to standard H&E histology by virtual staining models. The versatility of LEDCHAMP is experimentally demonstrated using mouse brain thin slices and thick sections, which takes only five minutes for imaging tissue surface area with 10 × 10 mm2. The promising LED-CHAMP workflow enables fast, low-cost, and comparable image quality for intraoperative assessment.

Lung cancer is one of the leading causes of cancer mortality worldwide, with an estimated 2.2 million new cancer cases and 1.8 million deaths in 2020. Adenocarcinoma is the most common type of non-small cell lung cancer (NSCLC), which is usually developed with a mixture of histologic subtypes. Surgery to remove the affected tissue or tumor is the most curative treatment option for the early-stage NSCLC currently. The clinical diagnosis of NSCLC based on pathological analysis of formalin-fixed and paraffin-embedded (FFPE) tissues is laborious and time-consuming, failing to guide surgeons intraoperatively. Although frozen section can serve as a rapid alternative to FFPE histology, it still requires a turnaround time of 20–30 minutes during surgery. Besides, the diagnostic accuracy of the frozen section could be affected due to the tissue freezing artifacts and inadequate sampling of resection margins. Here, we propose a rapid histological imaging method, termed microscopy with ultraviolet single-plane illumination (MUSI), which enables label-free and non-destructive imaging of freshly excised and unprocessed tissues. The MUSI system allows the surgical specimens with large irregular surfaces to be scanned in a label-free manner at a speed of 0.65 mm2/s with a subcellular resolution, showing great potential as an assistive imaging platform that can provide immediate feedback to surgeons and pathologists for intraoperative decision-making. We demonstrate that MUSI can differentiate between different subtypes of human lung adenocarcinomas, revealing diagnostically important features that are comparable to the gold standard FFPE histology, holding great promise to revolutionize the current practice of surgical pathology.

Speech Emotion Recognition (SER) is to recognize human emotions in a natural verbal interaction scenario with machines, which is considered as a challenging problem due to the ambiguous human emotions. Despite the recent progress in SER, state-of-the-art models struggle to achieve a satisfactory performance. We propose a self-attention based method with combined use of label-adaptive mixup and center loss. By adapting label probabilities in mixup and fitting center loss to the mixup training scheme, our proposed method achieves a superior performance to the state-of-the-art methods., Comment: Accepted to ICASSP 2023

Document Visual Question Answering (DocVQA) is a fast growing branch of document understanding. Despite the fact that documents contain sensitive or copyrighted information, none of the current DocVQA methods offers strong privacy guarantees. In this work, we explore privacy in the domain of DocVQA for the first time. We highlight privacy issues in state of the art multi-modal LLM models used for DocVQA, and explore possible solutions. Specifically, we focus on the invoice processing use case as a realistic, widely used scenario for document understanding, and propose a large scale DocVQA dataset comprising invoice documents and associated questions and answers. We employ a federated learning scheme, that reflects the real-life distribution of documents in different businesses, and we explore the use case where the ID of the invoice issuer is the sensitive information to be protected. We demonstrate that non-private models tend to memorise, behaviour that can lead to exposing private information. We then evaluate baseline training schemes employing federated learning and differential privacy in this multi-modal scenario, where the sensitive information might be exposed through any of the two input modalities: vision (document image) or language (OCR tokens). Finally, we design an attack exploiting the memorisation effect of the model, and demonstrate its effectiveness in probing different DocVQA models.

Ultraviolet photoacoustic microscopy (UV-PAM) has been investigated to provide label-free and registration-free volumetric histological images for whole organs, offering new insights into complex biological organs. However, because of the high UV absorption of lipids and pigments in tissue, UV-PAM suffers from low image contrast and shallow image depth, hindering its capability for revealing various microstructures in organs. To improve the UV-PAM imaging contrast and imaging depth, here we propose to implement a state-of-the-art optical clearing technique, CUBIC (clear, unobstructed brain/body imaging cocktails and computational analysis), to wash out the lipids and pigments from tissues. Our results show that the UV-PAM imaging contrast and quality can be significantly improved after tissue clearing. With the cleared tissue, multilayers of cell nuclei can also be extracted from time-resolved PA signals. Tissue clearing-enhanced UV-PAM can provide fine details for organ imaging. © 2021 The Authors

Rapid and high-resolution histological imaging with minimal tissue preparation has long been a challenging and yet captivating medical pursuit. Here, the authors propose a promising and transformative histological imaging method, termed computational high-throughput autofluorescence microscopy by pattern illumination (CHAMP). With the assistance of computational microscopy, CHAMP enables high-throughput and label-free imaging of thick and unprocessed tissues with large surface irregularity at an acquisition speed of 10 mm2/10 s with 1.1-µm lateral resolution. Moreover, the CHAMP image can be transformed into a virtually stained histological image (Deep-CHAMP) through unsupervised learning within 15 s, where significant cellular features are quantitatively extracted with high accuracy. The versatility of CHAMP is experimentally demonstrated using mouse brain/kidney and human lung tissues prepared with various clinical protocols, which enables a rapid and accurate intraoperative/postoperative pathological examination without tissue processing or staining, demonstrating its great potential as an assistive imaging platform for surgeons and pathologists to provide optimal adjuvant treatment.

Surgical margin analysis (SMA), an essential procedure to confirm the complete excision of cancerous tissue in tumor resection surgery, requires intraoperative diagnostic tools to avoid repeated surgeries due to a positive surgical margin. Recently, by taking the advantage of the high intrinsic optical absorption of DNA/ RNA at 266 nm wavelength, ultraviolet photoacoustic microscopy (UV-PAM) has been developed to provide high-resolution histological images without labeling, showing great promise as an intraoperative tool for SMA. To enable the development of UV-PAM for SMA, here, a high-speed and open-top UV-PAM system is presented, which can be operated similarly to conventional optical microscopies. The UV-PAM system provides a high lateral resolution of 1.2 µm, and a high imaging speed of 55 kHz A-line rate with one-axis galvanometer mirror scanning. Moreover, to ensure UV-PAM images can be easily interpreted by pathologists without additional training, the original grayscale UV-PAM images are virtually stained by a deep-learning algorithm to mimic the standard hematoxylin-and eosin-stained images, enabling training-free histological analysis. Mouse brain slice imaging is performed to demonstrate the high performance of the open-top UV-PAM system, illustrating its great potential for SMA applications. © 2022 JoVE Journal of Visualized Experiments.

We propose a rapid and label-free three-dimensional imaging technique to analyze paraffin-embedded whole organs without tissue staining or clearing. Various anatomical structures are revealed at subcellular resolution, facilitating comprehensive and volumetric cellular histopathological analysis.

Histopathology based on formalin-fixed and paraffin-embedded tissues has long been the gold standard for surgical margin assessment (SMA). However, routine pathological practice is lengthy and laborious, failing to guide surgeons intraoperatively. In this report, we propose a practical and low-cost histological imaging method with wide-field optical-sectioning microscopy (i.e., High-and-Low-frequency (HiLo) microscopy). HiLo can achieve rapid and non-destructive imaging of freshly-excised tissues at an extremely high acquisition speed of 5 cm2/min with a spatial resolution of 1.3 µm (lateral) and 5.8 µm (axial), showing great potential as an SMA tool that can provide immediate feedback to surgeons and pathologists for intraoperative decision-making. We demonstrate that HiLo enables rapid extraction of diagnostic features for different subtypes of human lung adenocarcinoma and hepatocellular carcinoma, producing surface images of rough specimens with large field-of-views and cellular features that are comparable to the clinical standard. Our results show promising clinical translations of HiLo microscopy to improve the current standard of care.

Rapid and high-resolution histological imaging with minimal tissue preparation has long been a challenging and yet captivating medical pursuit. Here, the authors propose a promising and transformative histological imaging method, termed computational high-throughput autofluorescence microscopy by pattern illumination (CHAMP). With the assistance of computational microscopy, CHAMP enables high-throughput and label-free imaging of thick and unprocessed tissues with large surface irregularity at an acquisition speed of 10 mm2/10 s with 1.1-µm lateral resolution. Moreover, the CHAMP image can be transformed into a virtually stained histological image (Deep-CHAMP) through unsupervised learning within 15 s, where significant cellular features are quantitatively extracted with high accuracy. The versatility of CHAMP is experimentally demonstrated using mouse brain/kidney and human lung tissues prepared with various clinical protocols, which enables a rapid and accurate intraoperative/postoperative pathological examination without tissue processing or staining, demonstrating its great potential as an assistive imaging platform for surgeons and pathologists to provide optimal adjuvant treatment.

Ultraviolet photoacoustic microscopy (UV-PAM) has been investigated to provide label-free and registration-free volumetric histological images for whole organs, offering new insights into complex biological organs. However, because of the high UV absorption of lipids and pigments in tissue, UV-PAM suffers from low image contrast and shallow image depth, hindering its capability for revealing various microstructures in organs. To improve the UV-PAM imaging contrast and imaging depth, here we propose to implement a state-of-the-art optical clearing technique, CUBIC (clear, unobstructed brain/body imaging cocktails and computational analysis), to wash out the lipids and pigments from tissues. Our results show that the UV-PAM imaging contrast and quality can be significantly improved after tissue clearing. With the cleared tissue, multilayers of cell nuclei can also be extracted from time-resolved PA signals. Tissue clearing-enhanced UV-PAM can provide fine details for organ imaging. © 2021 The Authors

Histological image with hematoxylin and eosin (H&E) staining is essential for histopathological diagnosis and has been the gold standard of tumor diagnosis. However, the acquiring of H&E stained images takes around one week in the hospital due to the involved lengthy and laborious steps such as tissue preprocessing, sectioning, and staining. In this paper, I presented two novel deep learning assisted imaging methods that can shorten the acquisition time of H&E stained images from one week to several minutes, greatly increasing the efficacy of histopathological diagnosis. The first imaging method combines deep learning with ultraviolet photoacoustic microscopy (UV-PAM), which can rapidly get the H&E digital staining histological image of unprocessed fresh mouse brain within 15 minutes. The second method integrates deep learning with computational autofluorescence microscopy (CHAMP), which further reduces the time to 1 minute. In addition to the 2D histological imaging, by combing deep learning with a vibratome-assisted 3D ultraviolet imaging system, I further developed a pipeline for the rapid and fully automatic acquiring of H&E stained histopathological images of a whole organ. With the designed method, we rapidly get the 3D H&E histological images of versatile organs, such as brain, liver, kidney, heart, lung, heart, spleen with little human involvement. The proposed method might promote the routine use of 3D histology in research or clinic, as it greatly shortens the acquisition time of 3D H&E histological images from one month to ~3 days. Lastly, I summarize the advances that our three deep learning assisted histological imaging methods have brought to conventional 2D and 3D histology. Some improvements and new perspectives are also discussed.

Handwritten Text Recognition has achieved an impressive performance in public benchmarks. However, due to the high inter- and intra-class variability between handwriting styles, such recognizers need to be trained using huge volumes of manually labeled training data. To alleviate this labor-consuming problem, synthetic data produced with TrueType fonts has been often used in the training loop to gain volume and augment the handwriting style variability. However, there is a significant style bias between synthetic and real data which hinders the improvement of recognition performance. To deal with such limitations, we propose a generative method for handwritten text-line images, which is conditioned on both visual appearance and textual content. Our method is able to produce long text-line samples with diverse handwriting styles. Once properly trained, our method can also be adapted to new target data by only accessing unlabeled text-line images to mimic handwritten styles and produce images with any textual content. Extensive experiments have been done on making use of the generated samples to boost Handwritten Text Recognition performance. Both qualitative and quantitative results demonstrate that the proposed approach outperforms the current state of the art., Comment: Accepted to TPAMI

© 2020 Mikhail Y. Shalaginov et al., published by De Gruyter. Optical metasurfaces, planar subwavelength nanoantenna arrays with the singular ability to sculpt wavefront in almost arbitrary manners, are poised to become a powerful tool enabling compact and high-performance optics with novel functionalities. A particularly intriguing research direction within this field is active metasurfaces, whose optical response can be dynamically tuned postfabrication, thus allowing a plurality of applications unattainable with traditional bulk optics. Designing reconfigurable optics based on active metasurfaces is, however, presented with a unique challenge, since the optical quality of the devices must be optimized at multiple optical states. In this article, we provide a critical review on the active meta-optics design principles and algorithms that are applied across structural hierarchies ranging from single meta-atoms to full meta-optical devices. The discussed approaches are illustrated by specific examples of reconfigurable metasurfaces based on optical phase-change materials.

Rapid multicolor three-dimensional (3D) imaging for centimeter-scale specimens with subcellular resolution remains a challenging but captivating scientific pursuit. Here, we present a fast, cost-effective, and robust multicolor whole-organ 3D imaging method assisted with ultraviolet (UV) surface excitation and vibratomy-assisted sectioning, termed translational rapid ultraviolet-excited sectioning tomography (TRUST). With an inexpensive UV light-emitting diode (UV-LED) and a color camera, TRUST achieves widefield exogenous molecular-specific fluorescence and endogenous content-rich autofluorescence imaging simultaneously while preserving low system complexity and system cost. Formalin-fixed specimens are stained layer by layer along with serial mechanical sectioning to achieve automated 3D imaging with high staining uniformity and time efficiency. 3D models of all vital organs in wild-type C57BL/6 mice with the 3D structure of their internal components (e.g., vessel network, glomeruli, and nerve tracts) can be reconstructed after imaging with TRUST to demonstrate its fast, robust, and high-content multicolor 3D imaging capability. Moreover, its potential for developmental biology has also been validated by imaging entire mouse embryos (similar to 2 days for the embryo at the embryonic day of 15). TRUST offers a fast and cost-effective approach for high-resolution whole-organ multicolor 3D imaging while relieving researchers from the heavy sample preparation workload.

We propose a rapid and label-free three-dimensional imaging technique to analyze paraffin-embedded whole organs without tissue staining or clearing. Various anatomical structures are revealed at subcellular resolution, facilitating comprehensive and volumetric cellular histopathological analysis.

Rapid and high-resolution histological imaging with minimal tissue preparation has long been a challenging and yet captivating medical pursuit. Here, the authors propose a promising and transformative histological imaging method, termed computational high-throughput autofluorescence microscopy by pattern illumination (CHAMP). With the assistance of computational microscopy, CHAMP enables high-throughput and label-free imaging of thick and unprocessed tissues with large surface irregularity at an acquisition speed of 10 mm2/10 s with 1.1-µm lateral resolution. Moreover, the CHAMP image can be transformed into a virtually stained histological image (Deep-CHAMP) through unsupervised learning within 15 s, where significant cellular features are quantitatively extracted with high accuracy. The versatility of CHAMP is experimentally demonstrated using mouse brain/kidney and human lung tissues prepared with various clinical protocols, which enables a rapid and accurate intraoperative/postoperative pathological examination without tissue processing or staining, demonstrating its great potential as an assistive imaging platform for surgeons and pathologists to provide optimal adjuvant treatment.

Surgical margin analysis (SMA), an essential procedure to confirm the complete excision of cancerous tissue in tumor resection surgery, requires intraoperative diagnostic tools to avoid repeated surgeries due to a positive surgical margin. Recently, by taking the advantage of the high intrinsic optical absorption of DNA/ RNA at 266 nm wavelength, ultraviolet photoacoustic microscopy (UV-PAM) has been developed to provide high-resolution histological images without labeling, showing great promise as an intraoperative tool for SMA. To enable the development of UV-PAM for SMA, here, a high-speed and open-top UV-PAM system is presented, which can be operated similarly to conventional optical microscopies. The UV-PAM system provides a high lateral resolution of 1.2 µm, and a high imaging speed of 55 kHz A-line rate with one-axis galvanometer mirror scanning. Moreover, to ensure UV-PAM images can be easily interpreted by pathologists without additional training, the original grayscale UV-PAM images are virtually stained by a deep-learning algorithm to mimic the standard hematoxylin-and eosin-stained images, enabling training-free histological analysis. Mouse brain slice imaging is performed to demonstrate the high performance of the open-top UV-PAM system, illustrating its great potential for SMA applications.

Histopathology based on formalin-fixed and paraffin-embedded tissues has long been the gold standard for surgical margin assessment (SMA). However, routine pathological practice is lengthy and laborious, failing to guide surgeons intraoperatively. In this report, we propose a practical and low-cost histological imaging method with wide-field optical-sectioning microscopy (i.e., High-and-Low-frequency (HiLo) microscopy). HiLo can achieve rapid and non-destructive imaging of freshly-excised tissues at an extremely high acquisition speed of 5 cm2/min with a spatial resolution of 1.3 µm (lateral) and 5.8 µm (axial), showing great potential as an SMA tool that can provide immediate feedback to surgeons and pathologists for intraoperative decision-making. We demonstrate that HiLo enables rapid extraction of diagnostic features for different subtypes of human lung adenocarcinoma and hepatocellular carcinoma, producing surface images of rough specimens with large field-of-views and cellular features that are comparable to the clinical standard. Our results show promising clinical translations of HiLo microscopy to improve the current standard of care.

Ultraviolet photoacoustic microscopy (UV-PAM) has been investigated to provide label-free and registration-free volumetric histological images for whole organs, offering new insights into complex biological organs. However, because of the high UV absorption of lipids and pigments in tissue, UV-PAM suffers from low image contrast and shallow image depth, hindering its capability for revealing various microstructures in organs. To improve the UV-PAM imaging contrast and imaging depth, here we propose to implement a state-of-the-art optical clearing technique, CUBIC (clear, unobstructed brain/body imaging cocktails and computational analysis), to wash out the lipids and pigments from tissues. Our results show that the UV-PAM imaging contrast and quality can be significantly improved after tissue clearing. With the cleared tissue, multilayers of cell nuclei can also be extracted from time-resolved PA signals. Tissue clearing-enhanced UV-PAM can provide fine details for organ imaging. © 2021 The Authors

Laser diodes (LDs) have been considered as cost-effective and compact excitation sources to overcome the requirement of costly and bulky pulsed laser sources that are commonly used in photoacoustic microscopy (PAM). However, the spatial resolution and/or imaging speed of previously reported LD-based PAM systems have not been optimized simultaneously. In this paper, we developed a high-speed and high-resolution LD-based PAM system using a continuous wave LD, operating at a pulsed mode, with a repetition rate of 30 kHz, as an excitation source. A hybrid scanning mechanism that synchronizes a one-dimensional galvanometer mirror and a two-dimensional motorized stage is applied to achieve a fast imaging capability without signal averaging due to the high signal-to-noise ratio. By optimizing the optical system, a high lateral resolution of 4.8 μm has been achieved. In vivo microvasculature imaging of a mouse ear has been demonstrated to show the high performance of our LD-based PAM system. © 2021, The Author(s).

Histopathology based on formalin-fixed and paraffin-embedded tissues has long been the gold standard for surgical margin assessment (SMA). However, routine pathological practice is lengthy and laborious, failing to guide surgeons intraoperatively. In this report, we propose a practical and low-cost histological imaging method with wide-field optical-sectioning microscopy (i.e., High-and-Low-frequency (HiLo) microscopy). HiLo can achieve rapid and non-destructive imaging of freshly-excised tissues at an extremely high acquisition speed of 5 cm2/min with a spatial resolution of 1.3 µm (lateral) and 5.8 µm (axial), showing great potential as an SMA tool that can provide immediate feedback to surgeons and pathologists for intraoperative decision-making. We demonstrate that HiLo enables rapid extraction of diagnostic features for different subtypes of human lung adenocarcinoma and hepatocellular carcinoma, producing surface images of rough specimens with large field-of-views and cellular features that are comparable to the clinical standard. Our results show promising clinical translations of HiLo microscopy to improve the current standard of care.

Histopathology based on formalin-fixed and paraffin-embedded tissues has long been the gold standard for surgical margin assessment (SMA). However, routine pathological practice is lengthy and laborious, failing to guide surgeons intraoperatively. In this report, we propose a practical and low-cost histological imaging method with wide-field optical-sectioning microscopy (i.e., High-and-Low-frequency (HiLo) microscopy). HiLo can achieve rapid and non-destructive imaging of freshly-excised tissues at an extremely high acquisition speed of 5 cm2/min with a spatial resolution of 1.3 µm (lateral) and 5.8 µm (axial), showing great potential as an SMA tool that can provide immediate feedback to surgeons and pathologists for intraoperative decision-making. We demonstrate that HiLo enables rapid extraction of diagnostic features for different subtypes of human lung adenocarcinoma and hepatocellular carcinoma, producing surface images of rough specimens with large field-of-views and cellular features that are comparable to the clinical standard. Our results show promising clinical translations of HiLo microscopy to improve the current standard of care.

Ultraviolet-based photoacoustic microscopy (UV-PAM) has recently been demonstrated as a promising tool to overcome the time-consuming sample preparation procedure in traditional pathological analysis. In order to achieve high-speed UVPAM for clinical usage, we implemented UV-PAM with a single-axis galvo mirror scanner. With our UV laser operating at a repetition rate of 55 kHz, our system produced images ∼5.5 times faster than the previously reported point-by-point raster scanning based UV-PAM, with a lateral resolution of ∼1.0 μm. Histology-like images of a mouse brain slice were acquired by our system, showing its potential as an intraoperative imaging tool for surgical margin assessment. © 2020 SPIE.

By combining an imaging modality (Ultraviolet Photoacoustic Microscopy) with deep learning (unpaired image-to-image translation network), we demonstrate a digital histological imaging method that can acquire the virtual histological images of both thin and thick label-free tissue specimens. Its capability to directly get the virtual histological images of fresh tissue will cut off the laborious tissue preprocessing procedures for traditional histological staining, thus greatly expediting the histological examination for pathologists. In our method, the cycle-consistent adversarial network (Cycle-GAN) is trained to transform the ultraviolet photoacoustic microscopy (UV-PAM) images of label-free tissue to bright-field microscopy images of hematoxylin- and eosin-stained tissue sections. The networks can be trained using unpaired images of UV-PAM label-free tissue images and bright-field microscopy histologically stained tissue images from different tissue sections, which means our method can bypass all the difficulty of acquiring paired images datasets for supervised network training. Compared with other deep learning-based staining methods, we have advanced the deep learning’s application by successfully applying unpaired image-to-image translation methods such as Cycle-GAN in our staining method and successfully achieved the virtual histological staining of thick un-sectioned tissue. We showed the effectiveness of the UV-PAM based virtual histological imaging method on both thin mouse brain sections and thick fresh mouse brain.

We proposed a high-throughput label-free and slide-free imaging technique by ultraviolet-excited autofluorescence, assisted by speckle illumination and a deep learning algorithm, to generate subcellular-resolution histology-like images of unprocessed tissues, simplifying the workflow of standard-of-care histopathology. © OSA 2020 © 2020 The Author(s)

By implementing microscopy with ultraviolet surface excitation using speckle illumination, we have improved the spatial resolution by 1.5 times and preserved the long depth of focus, achieving sharp imaging of thick and rough specimens. © 2020 The Author(s)

By combining an imaging modality (Ultraviolet Photoacoustic Microscopy) with deep learning (unpaired image-to-image translation network), we demonstrate a digital histological imaging method that can acquire the virtual histological images of both thin and thick label-free tissue specimens. Its capability to directly get the virtual histological images of fresh tissue will cut off the laborious tissue preprocessing procedures for traditional histological staining, thus greatly expediting the histological examination for pathologists. In our method, the cycle-consistent adversarial network (Cycle-GAN) is trained to transform the ultraviolet photoacoustic microscopy (UV-PAM) images of label-free tissue to bright-field microscopy images of hematoxylin- and eosin-stained tissue sections. The networks can be trained using unpaired images of UV-PAM label-free tissue images and bright-field microscopy histologically stained tissue images from different tissue sections, which means our method can bypass all the difficulty of acquiring paired images datasets for supervised network training. Compared with other deep learning-based staining methods, we have advanced the deep learning’s application by successfully applying unpaired image-to-image translation methods such as Cycle-GAN in our staining method and successfully achieved the virtual histological staining of thick un-sectioned tissue. We showed the effectiveness of the UV-PAM based virtual histological imaging method on both thin mouse brain sections and thick fresh mouse brain.

We proposed a high-throughput label-free and slide-free imaging technique by ultraviolet-excited autofluorescence, assisted by speckle illumination and a deep learning algorithm, to generate subcellular-resolution histology-like images of unprocessed tissues, simplifying the workflow of standard-of-care histopathology. © OSA 2020 © 2020 The Author(s)

By implementing microscopy with ultraviolet surface excitation using speckle illumination, we have improved the spatial resolution by 1.5 times and preserved the long depth of focus, achieving sharp imaging of thick and rough specimens. © 2020 The Author(s)

By combining an imaging modality (Ultraviolet Photoacoustic Microscopy) with deep learning (unpaired image-to-image translation network), we demonstrate a digital histological imaging method that can acquire the virtual histological images of both thin and thick label-free tissue specimens. Its capability to directly get the virtual histological images of fresh tissue will cut off the laborious tissue preprocessing procedures for traditional histological staining, thus greatly expediting the histological examination for pathologists. In our method, the cycle-consistent adversarial network (Cycle-GAN) is trained to transform the ultraviolet photoacoustic microscopy (UV-PAM) images of label-free tissue to bright-field microscopy images of hematoxylin- and eosin-stained tissue sections. The networks can be trained using unpaired images of UV-PAM label-free tissue images and bright-field microscopy histologically stained tissue images from different tissue sections, which means our method can bypass all the difficulty of acquiring paired images datasets for supervised network training. Compared with other deep learning-based staining methods, we have advanced the deep learning’s application by successfully applying unpaired image-to-image translation methods such as Cycle-GAN in our staining method and successfully achieved the virtual histological staining of thick un-sectioned tissue. We showed the effectiveness of the UV-PAM based virtual histological imaging method on both thin mouse brain sections and thick fresh mouse brain.

We proposed a high-throughput label-free and slide-free imaging technique by ultraviolet-excited autofluorescence, assisted by speckle illumination and a deep learning algorithm, to generate subcellular-resolution histology-like images of unprocessed tissues, simplifying the workflow of standard-of-care histopathology. © OSA 2020 © 2020 The Author(s)

By implementing microscopy with ultraviolet surface excitation using speckle illumination, we have improved the spatial resolution by 1.5 times and preserved the long depth of focus, achieving sharp imaging of thick and rough specimens. © 2020 The Author(s)

The advent of recurrent neural networks for handwriting recognition marked an important milestone reaching impressive recognition accuracies despite the great variability that we observe across different writing styles. Sequential architectures are a perfect fit to model text lines, not only because of the inherent temporal aspect of text, but also to learn probability distributions over sequences of characters and words. However, using such recurrent paradigms comes at a cost at training stage, since their sequential pipelines prevent parallelization. In this work, we introduce a non-recurrent approach to recognize handwritten text by the use of transformer models. We propose a novel method that bypasses any recurrence. By using multi-head self-attention layers both at the visual and textual stages, we are able to tackle character recognition as well as to learn language-related dependencies of the character sequences to be decoded. Our model is unconstrained to any predefined vocabulary, being able to recognize out-of-vocabulary words, i.e. words that do not appear in the training vocabulary. We significantly advance over prior art and demonstrate that satisfactory recognition accuracies are yielded even in few-shot learning scenarios.

Although current image generation methods have reached impressive quality levels, they are still unable to produce plausible yet diverse images of handwritten words. On the contrary, when writing by hand, a great variability is observed across different writers, and even when analyzing words scribbled by the same individual, involuntary variations are conspicuous. In this work, we take a step closer to producing realistic and varied artificially rendered handwritten words. We propose a novel method that is able to produce credible handwritten word images by conditioning the generative process with both calligraphic style features and textual content. Our generator is guided by three complementary learning objectives: to produce realistic images, to imitate a certain handwriting style and to convey a specific textual content. Our model is unconstrained to any predefined vocabulary, being able to render whatever input word. Given a sample writer, it is also able to mimic its calligraphic features in a few-shot setup. We significantly advance over prior art and demonstrate with qualitative, quantitative and human-based evaluations the realistic aspect of our synthetically produced images., Comment: Accepted to ECCV2020

This dissertation investigates how to improve airline fuel efficiency by changing fuel loading behavior of airline dispatchers and reducing unnecessary fuel loading. First, we estimate the potential benefit of fuel loading reduction for six major U.S.-based airlines. We find that the annual monetary savings per airline of avoiding carrying unused fuel in 2012 ranges from $42 million to $605 million, with a total across all six airlines of $1.16 billion. This suggests that there may be significant benefit from reducing unnecessary fuel loading in the U.S. airline industry. Second, to capture that benefit, we first study the behavior of dispatchers – who make fuel loading decisions – based on comprehensive historical fuel burn data provided by a major U.S.-based airline. Risk-neutral and risk-averse newsvendor models are applied to better understand how dispatchers trade off safety concerns due to possibly insufficient fuel loading and extra fuel burn costs due to excess discretionary fuel loading. We find that dispatchers place extremely high priority on safety relative to excess fuel cost in making discretionary fuel loading decisions. Furthermore, combining our results with a dispatcher survey, we also find that dispatchers who are detail oriented and conservationists are likely to load less discretionary fuel. Our results imply that airlines may want to select for these characteristics during dispatcher interview and seek to cultivate such behavior in dispatcher recurrent training. Finally, besides behavioral modeling of dispatchers, we propose two novel discretionary fuel estimation approaches that can assist dispatchers with better discretionary fuel loading decisions. Based on the analysis on our study airline, our approaches are found to substantially reduce unnecessary discretionary fuel loading while maintaining the same safety level compared to the current fuel loading practice. The idea is that by providing dispatchers with more accurate information and be

This dissertation investigates how to improve airline fuel efficiency by changing fuel loading behavior of airline dispatchers and reducing unnecessary fuel loading. First, we estimate the potential benefit of fuel loading reduction for six major U.S.-based airlines. We find that the annual monetary savings per airline of avoiding carrying unused fuel in 2012 ranges from $42 million to $605 million, with a total across all six airlines of $1.16 billion. This suggests that there may be significant benefit from reducing unnecessary fuel loading in the U.S. airline industry. Second, to capture that benefit, we first study the behavior of dispatchers – who make fuel loading decisions – based on comprehensive historical fuel burn data provided by a major U.S.-based airline. Risk-neutral and risk-averse newsvendor models are applied to better understand how dispatchers trade off safety concerns due to possibly insufficient fuel loading and extra fuel burn costs due to excess discretionary fuel loading. We find that dispatchers place extremely high priority on safety relative to excess fuel cost in making discretionary fuel loading decisions. Furthermore, combining our results with a dispatcher survey, we also find that dispatchers who are detail oriented and conservationists are likely to load less discretionary fuel. Our results imply that airlines may want to select for these characteristics during dispatcher interview and seek to cultivate such behavior in dispatcher recurrent training. Finally, besides behavioral modeling of dispatchers, we propose two novel discretionary fuel estimation approaches that can assist dispatchers with better discretionary fuel loading decisions. Based on the analysis on our study airline, our approaches are found to substantially reduce unnecessary discretionary fuel loading while maintaining the same safety level compared to the current fuel loading practice. The idea is that by providing dispatchers with more accurate information and be

Sequence-to-sequence models have recently become very popular for tackling handwritten word recognition problems. However, how to effectively integrate an external language model into such recognizer is still a challenging problem. The main challenge faced when training a language model is to deal with the language model corpus which is usually different to the one used for training the handwritten word recognition system. Thus, the bias between both word corpora leads to incorrectness on the transcriptions, providing similar or even worse performances on the recognition task. In this work, we introduce Candidate Fusion, a novel way to integrate an external language model to a sequence-to-sequence architecture. Moreover, it provides suggestions from an external language knowledge, as a new input to the sequence-to-sequence recognizer. Hence, Candidate Fusion provides two improvements. On the one hand, the sequence-to-sequence recognizer has the flexibility not only to combine the information from itself and the language model, but also to choose the importance of the information provided by the language model. On the other hand, the external language model has the ability to adapt itself to the training corpus and even learn the most commonly errors produced from the recognizer. Finally, by conducting comprehensive experiments, the Candidate Fusion proves to outperform the state-of-the-art language models for handwritten word recognition tasks.

Handwritten Text Recognition (HTR) is still a challenging problem because it must deal with two important difficulties: the variability among writing styles, and the scarcity of labelled data. To alleviate such problems, synthetic data generation and data augmentation are typically used to train HTR systems. However, training with such data produces encouraging but still inaccurate transcriptions in real words. In this paper, we propose an unsupervised writer adaptation approach that is able to automatically adjust a generic handwritten word recognizer, fully trained with synthetic fonts, towards a new incoming writer. We have experimentally validated our proposal using five different datasets, covering several challenges (i) the document source: modern and historic samples, which may involve paper degradation problems; (ii) different handwriting styles: single and multiple writer collections; and (iii) language, which involves different character combinations. Across these challenging collections, we show that our system is able to maintain its performance, thus, it provides a practical and generic approach to deal with new document collections without requiring any expensive and tedious manual annotation step., Comment: Accepted to WACV 2020