Search

We present a new algorithm, Cross-Source-Context Place Recognition (CSCPR), for RGB-D indoor place recognition that integrates global retrieval and reranking into a single end-to-end model. Unlike prior approaches that primarily focus on the RGB domain, CSCPR is designed to handle the RGB-D data. We extend the Context-of-Clusters (CoCs) for handling noisy colorized point clouds and introduce two novel modules for reranking: the Self-Context Cluster (SCC) and Cross Source Context Cluster (CSCC), which enhance feature representation and match query-database pairs based on local features, respectively. We also present two new datasets, ScanNetIPR and ARKitIPR. Our experiments demonstrate that CSCPR significantly outperforms state-of-the-art models on these datasets by at least 36.5% in Recall@1 at ScanNet-PR dataset and 44% in new datasets. Code and datasets will be released.

We present a novel end-to-end algorithm (PoCo) for the indoor RGB-D place recognition task, aimed at identifying the most likely match for a given query frame within a reference database. The task presents inherent challenges attributed to the constrained field of view and limited range of perception sensors. We propose a new network architecture, which generalizes the recent Context of Clusters (CoCs) to extract global descriptors directly from the noisy point clouds through end-to-end learning. Moreover, we develop the architecture by integrating both color and geometric modalities into the point features to enhance the global descriptor representation. We conducted evaluations on public datasets ScanNet-PR and ARKit with 807 and 5047 scenarios, respectively. PoCo achieves SOTA performance: on ScanNet-PR, we achieve R@1 of 64.63%, a 5.7% improvement from the best-published result CGis (61.12%); on Arkit, we achieve R@1 of 45.12%, a 13.3% improvement from the best-published result CGis (39.82%). In addition, PoCo shows higher efficiency than CGis in inference time (1.75X-faster), and we demonstrate the effectiveness of PoCo in recognizing places within a real-world laboratory environment.

Reconstructing transparent objects using affordable RGB-D cameras is a persistent challenge in robotic perception due to inconsistent appearances across views in the RGB domain and inaccurate depth readings in each single-view. We introduce a two-stage pipeline for reconstructing transparent objects tailored for mobile platforms. In the first stage, off-the-shelf monocular object segmentation and depth completion networks are leveraged to predict the depth of transparent objects, furnishing single-view shape prior. Subsequently, we propose Epipolar-guided Optical Flow (EOF) to fuse several single-view shape priors from the first stage to a cross-view consistent 3D reconstruction given camera poses estimated from opaque part of the scene. Our key innovation lies in EOF which employs boundary-sensitive sampling and epipolar-line constraints into optical flow to accurately establish 2D correspondences across multiple views on transparent objects. Quantitative evaluations demonstrate that our pipeline significantly outperforms baseline methods in 3D reconstruction quality, paving the way for more adept robotic perception and interaction with transparent objects., Comment: IEEE-RAS Humanoids 2023 paper, 8 pages, 6 figures

Large multimodal language models (LMMs) have achieved significant success in general domains. However, due to the significant differences between medical images and text and general web content, the performance of LMMs in medical scenarios is limited. In ophthalmology, clinical diagnosis relies on multiple modalities of medical images, but unfortunately, multimodal ophthalmic large language models have not been explored to date. In this paper, we study and construct an ophthalmic large multimodal model. Firstly, we use fundus images as an entry point to build a disease assessment and diagnosis pipeline to achieve common ophthalmic disease diagnosis and lesion segmentation. Then, we establish a new ophthalmic multimodal instruction-following and dialogue fine-tuning dataset based on disease-related knowledge data and publicly available real-world medical dialogue. We introduce visual ability into the large language model to complete the ophthalmic large language and vision assistant (OphGLM). Our experimental results demonstrate that the OphGLM model performs exceptionally well, and it has the potential to revolutionize clinical applications in ophthalmology. The dataset, code, and models will be made publicly available at https://github.com/ML-AILab/OphGLM., Comment: OphGLM:The first ophthalmology large language-and-vision assistant based on instructions and dialogue

Ophthalmologists have used fundus images to screen and diagnose eye diseases. However, different equipments and ophthalmologists pose large variations to the quality of fundus images. Low-quality (LQ) degraded fundus images easily lead to uncertainty in clinical screening and generally increase the risk of misdiagnosis. Thus, real fundus image restoration is worth studying. Unfortunately, real clinical benchmark has not been explored for this task so far. In this paper, we investigate the real clinical fundus image restoration problem. Firstly, We establish a clinical dataset, Real Fundus (RF), including 120 low- and high-quality (HQ) image pairs. Then we propose a novel Transformer-based Generative Adversarial Network (RFormer) to restore the real degradation of clinical fundus images. The key component in our network is the Window-based Self-Attention Block (WSAB) which captures non-local self-similarity and long-range dependencies. To produce more visually pleasant results, a Transformer-based discriminator is introduced. Extensive experiments on our clinical benchmark show that the proposed RFormer significantly outperforms the state-of-the-art (SOTA) methods. In addition, experiments of downstream tasks such as vessel segmentation and optic disc/cup detection demonstrate that our proposed RFormer benefits clinical fundus image analysis and applications. The dataset, code, and models are publicly available at https://github.com/dengzhuo-AI/Real-Fundus, Comment: IEEE J-BHI 2022; The First Benchmark and First Transformer-based Method for Real Clinical Fundus Image Restoration

Body orientation estimation provides crucial visual cues in many applications, including robotics and autonomous driving. It is particularly desirable when 3-D pose estimation is difficult to infer due to poor image resolution, occlusion or indistinguishable body parts. We present COCO-MEBOW (Monocular Estimation of Body Orientation in the Wild), a new large-scale dataset for orientation estimation from a single in-the-wild image. The body-orientation labels for around 130K human bodies within 55K images from the COCO dataset have been collected using an efficient and high-precision annotation pipeline. We also validated the benefits of the dataset. First, we show that our dataset can substantially improve the performance and the robustness of a human body orientation estimation model, the development of which was previously limited by the scale and diversity of the available training data. Additionally, we present a novel triple-source solution for 3-D human pose estimation, where 3-D pose labels, 2-D pose labels, and our body-orientation labels are all used in joint training. Our model significantly outperforms state-of-the-art dual-source solutions for monocular 3-D human pose estimation, where training only uses 3-D pose labels and 2-D pose labels. This substantiates an important advantage of MEBOW for 3-D human pose estimation, which is particularly appealing because the per-instance labeling cost for body orientations is far less than that for 3-D poses. The work demonstrates high potential of MEBOW in addressing real-world challenges involving understanding human behaviors. Further information of this work is available at https://chenyanwu.github.io/MEBOW/., Comment: Accepted by CVPR2020

High speed mid-wave infrared (MWIR) photodetectors have important applications in the emerging areas such high-precision frequency comb spectroscopy and light detection and ranging (LIDAR). In this work, we report a high-speed room-temperature mid-wave infrared interband cascade photodetector (ICIP) based on a type-II InAs/GaSb superlattice. The devices show an optical cut-off wavelength around 5um and a 3-dB bandwidth up to 7.04 GHz. The relatively low dark current density around 9.39 x 10-2 A/cm2 under -0.1 V is also demonstrated at 300 K. These results validate the advantages of ICIPs to achieve both high-frequency operation and low noise at room temperature. Limitations on the high-speed performance of the detector are also discussed based on the S-parameter analysis and other RF performance measurement.

High speed mid-wave infrared (MWIR) photodetectors have applications in the areas such as free space optical communication and frequency comb spectroscopy. However, most of the research on the MWIR photodetectors is focused on how to increase the quantum efficiency and reduce the dark current, in order to improve the detectivity (D*), and the 3dB bandwidth performance of the corresponding MWIR photodetectors is still not fully studied. In this work, we report and characterize a MWIR interband cascade photodetector based on InAs/GaSb type-II superlattice with a 50% cutoff wavelength at ~5.3 um at 300 K. The 3 dB cutoff frequency is 2.4 GHz at 300 K, for a 40 {\mu}m circular diameter device under -5 V applied bias. Limitations on the detector high speed performance are also discussed.High speed mid-wave infrared (MWIR) photodetectors have applications in the areas such as free space optical communication and frequency comb spectroscopy. However, most of the research on the MWIR photodetectors is focused on how to increase the quantum efficiency and reduce the dark current, in order to improve the detectivity (D*), and the 3dB bandwidth performance of the corresponding MWIR photodetectors is still not fully studied. In this work, we report and characterize a MWIR interband cascade photodetector based on InAs/GaSb type-II superlattice with a 50% cutoff wavelength at ~5.3 um at 300 K. The 3 dB cutoff frequency is 2.4 GHz at 300 K, for a 40 um circular diameter device under -5 V applied bias. Limitations on the detector high speed performance are also discussed.

In this paper, mid-wave infrared photodetection based on an InAs/GaSb type-II superlattice p-i-n photodetector grown directly on Si substrate is demonstrated and characterized. Excitation power dependence on integrated intensity from the photoluminescence measurements reveals a power coefficient of P~I0.74, indicating that defects related process is playing an important role in the predominant recombination channel for photogenerated carriers. At 70 K, the device exhibits a dark current density of 2.3 A/cm2 under -0.1 V bias. Arrhenius analysis of dark current shows activation energies much less than half of the active layer bandgap, which suggests that the device is mainly limited by surface leakage and defect-assisted tunneling, consistent with the photoluminescence analysis. The detector shows 50% cutoff wavelength at ~5.5 um at 70 K under bias of -0.1 V. The corresponding peak responsivity and specific detectivity are 1.2 A/W and 1.3*10e9 cm*Hz1/2/W, respectively. Based on these optoelectronics characterization results, reduction of defects by optimizing the III/V-Si interface, and suppression of surface leakage channels are argued to be the main factors for performance improvement in this Si-based T2SL detector towards low cost, large-format MWIR detection system on Si photonics platform.

InP based dilute Bismide InGaAsBi material is emerging as a promising candidate for extending short wavelength infrared detection. One critical factor to limit the performance of these InGaAsBi photodiodes is dark current caused by defects within the material. In this work, low frequency noise spectroscopy (LFNS) and temperature varied photoluminescence was used to characterize the defect levels in the devices. Three deep levels located at Ec -0.33 eV, Ev +0.14 eV, and Ec -0.51 eV were identified from the LFNS spectra, which are consistent with emission peak energy found by photoluminescence spectra of InGaAsBi.

Current optical communication system operating at 1.55 {\mu}m wavelength band may not be able to continually satisfy the growing demand on the data capacity within the next few years. Opening a new spectral window at around 2 {\mu}m wavelength with recently developed hollow-core photonic band gap fiber and thulium-doped fiber amplifier is a promising solution to increase the transmission capacity due to the low loss and wide bandwidth properties of these components at this wavelength. However, as a key component, the already demonstrated high speed photodetectors at 2 {\mu}m wavelength are still not comparable with those at 1.55 {\mu}m wavelength band, which chokes the feasibility of the new spectral window. In this work, we, for the first time, demonstrated a high speed uni-traveling carrier photodiode for 2 {\mu}m applications with InGaAs/GaAsSb type-II multiple quantum wells as the absorption region, which is lattice matched to InP. The device shows a 3dB bandwidth of 25 GHz at -3 V bias voltage and is, to the best of our knowledge, the fastest photodiodes among all group III-V and group IV photodetectors working in 2 {\mu}m wavelength range.