Search

Recent work introduced the concept of human teleoperation (HT), where the remote robot typically considered in conventional bilateral teleoperation is replaced by a novice person wearing a mixed reality head mounted display and tracking the motion of a virtual tool controlled by an expert. HT has advantages in cost, complexity, and patient acceptance for telemedicine in low-resource communities or remote locations. However, the stability, transparency, and performance of bilateral HT are unexplored. In this paper, we therefore develop a mathematical model and simulation of the HT system using test data. We then analyze various control architectures with this model and implement them with the HT system to find the achievable performance, investigate stability, and determine the most promising teleoperation scheme in the presence of time delays. We show that instability in HT, while not destructive or dangerous, makes the system impossible to use. However, stable and transparent teleoperation are possible with small time delays (<200 ms) through 3-channel teleoperation, or with large time delays through model-mediated teleoperation with local pose and force feedback for the novice.

Ultrasound is a hand-held, low-cost, non-invasive medical imaging modality which plays a vital role in diagnosing various diseases. Despite this, many rural and remote communities do not have access to ultrasound scans due to the lack of local experts trained to perform them. To address this challenge, we built a mixed reality and haptics-based tele-ultrasound system to enable an expert to precisely guide a novice remotely in carrying out an ultrasound exam. The precision and flexibility of our solution makes it more practical than existing tele-ultrasound solutions. We tested the system in Skidegate on the islands of Haida Gwaii, BC, Canada, with the experts positioned 754 km away at the University of British Columbia, Vancouver, Canada. We performed 11 scans with 10 novices and 2 experts. The experts were tasked with acquiring 5 target images and measurements in the epigastric region. The novices of various backgrounds and ages were all inexperienced in mixed reality and were not required to have prior ultrasound experience. The captured images were evaluated by two radiologists who were not present for the tests. These results are discussed along with new insights into the human computer interaction in such a system. We show that human teleoperation is feasible and can achieve high performance for completing remote ultrasound procedures, even at a large distance and with completely novice followers., Comment: 8 pages, 10 figures, submitted to IEEE VR 2025

Hyperspectral Imaging (HSI) for fluorescence-guided brain tumor resection enables visualization of differences between tissues that are not distinguishable to humans. This augmentation can maximize brain tumor resection, improving patient outcomes. However, much of the processing in HSI uses simplified linear methods that are unable to capture the non-linear, wavelength-dependent phenomena that must be modeled for accurate recovery of fluorophore abundances. We therefore propose two deep learning models for correction and unmixing, which can account for the nonlinear effects and produce more accurate estimates of abundances. Both models use an autoencoder-like architecture to process the captured spectra. One is trained with protoporphyrin IX (PpIX) concentration labels. The other undergoes semi-supervised training, first learning hyperspectral unmixing self-supervised and then learning to correct fluorescence emission spectra for heterogeneous optical and geometric properties using a reference white-light reflectance spectrum in a few-shot manner. The models were evaluated against phantom and pig brain data with known PpIX concentration; the supervised model achieved Pearson correlation coefficients (R values) between the known and computed PpIX concentrations of 0.997 and 0.990, respectively, whereas the classical approach achieved only 0.93 and 0.82. The semi-supervised approach's R values were 0.98 and 0.91, respectively. On human data, the semi-supervised model gives qualitatively more realistic results than the classical method, better removing bright spots of specular reflectance and reducing the variance in PpIX abundance over biopsies that should be relatively homogeneous. These results show promise for using deep learning to improve HSI in fluorescence-guided neurosurgery., Comment: 20 pages, 8 figures, 3 tables - Under Review

Through spectral unmixing, hyperspectral imaging (HSI) in fluorescence-guided brain tumor surgery has enabled detection and classification of tumor regions invisible to the human eye. Prior unmixing work has focused on determining a minimal set of viable fluorophore spectra known to be present in the brain and effectively reconstructing human data without overfitting. With these endmembers, non-negative least squares regression (NNLS) was used to compute the abundances. However, HSI images are heterogeneous, so one small set of endmember spectra may not fit all pixels well. Additionally, NNLS is the maximum likelihood estimator only if the measurement is normally distributed, and it does not enforce sparsity, which leads to overfitting and unphysical results. Here, we analyzed 555666 HSI fluorescence spectra from 891 ex vivo measurements of patients with brain tumors to show that a Poisson distribution models the measured data 82% better than a Gaussian in terms of the Kullback-Leibler divergence and that the endmember abundance vectors are sparse. With this knowledge, we introduce (1) a library of 9 endmember spectra, (2) a sparse, non-negative Poisson regression algorithm to perform physics-informed unmixing with this library without overfitting, and (3) a highly realistic spectral measurement simulation with known endmember abundances. The new unmixing method was then tested on the human and simulated data and compared to four other candidate methods. It outperforms previous methods with 25% lower error in the computed abundances on the simulated data than NNLS, lower reconstruction error on human data, beUer sparsity, and 31 times faster runtime than state-of-the-art Poisson regression. This method and library of endmember spectra can enable more accurate spectral unmixing to beUer aid the surgeon during brain tumor resection., Comment: 17 pages, 4 tables, 6 figures; Under review

Complete resection of malignant gliomas is hampered by the difficulty in distinguishing tumor cells at the infiltration zone. Fluorescence guidance with 5-ALA assists in reaching this goal. Using hyperspectral imaging, previous work characterized five fluorophores' emission spectra in most human brain tumors. In this paper, the effectiveness of these five spectra was explored for different tumor and tissue classification tasks in 184 patients (891 hyperspectral measurements) harboring low- (n=30) and high-grade gliomas (n=115), non-glial primary brain tumors (n=19), radiation necrosis (n=2), miscellaneous (n=10) and metastases (n=8). Four machine learning models were trained to classify tumor type, grade, glioma margins and IDH mutation. Using random forests and multi-layer perceptrons, the classifiers achieved average test accuracies of 74-82%, 79%, 81%, and 93% respectively. All five fluorophore abundances varied between tumor margin types and tumor grades (p < 0.01). For tissue type, at least four of the five fluorophore abundances were found to be significantly different (p < 0.01) between all classes. These results demonstrate the fluorophores' differing abundances in different tissue classes, as well as the value of the five fluorophores as potential optical biomarkers, opening new opportunities for intraoperative classification systems in fluorescence-guided neurosurgery., Comment: 22 pages, 8 figures

Exoplanets are abundant in our galaxy and yet characterizing them remains a technical challenge. Solar System planets provide an opportunity to test the practical limitations of exoplanet observations with high signal-to-noise data that we cannot access for exoplanets. However, data on Solar System planets differ from exoplanets in that Solar System planets are spatially resolved while exoplanets are unresolved point-sources. We present a novel instrument designed to observe Solar System planets as though they are exoplanets, the Planet as Exoplanet Analog Spectrograph (PEAS). PEAS consists of a dedicated 0.5-m telescope and off-the-shelf optics, located at Lick Observatory. PEAS uses an integrating sphere to disk-integrate light from the Solar System planets, producing spatially mixed light more similar to the spectra we can obtain from exoplanets. This paper describes the general system design and early results of the PEAS instrument., Comment: SPIE Astronomical Telescopes & Instrumentation 2020 Proceedings (11447-153), 9 pages, 9 figures

Solving the atomic structure of metallic clusters is fundamental to understanding their optical, electronic, and chemical properties. We report the structure of Au$_{\text{146}}$(p-MBA)$_{\text{57}}$ at subatomic resolution (0.85 {\AA}) using electron diffraction (MicroED) and atomic resolution by X-ray diffraction. The 146 gold atoms may be decomposed into two constituent sets consisting of 119 core and 27 peripheral atoms. The core atoms are organized in a twinned FCC structure whereas the surface gold atoms follow a C$_{2}$ rotational symmetry about an axis bisecting the twinning plane. The protective layer of 57 p-MBAs fully encloses the cluster and comprises bridging, monomeric, and dimeric staple motifs. Au$_{\text{146}}$(p-MBA)$_{\text{57}}$ is the largest cluster observed exhibiting a bulk-like FCC structure as well as the smallest gold particle exhibiting a stacking fault., Comment: S. Vergara, D. A. Lukes, and M. W. Martynowycz contributed equally to this work. Correspondence to: guc9@pitt.edu, tgonen@ucla.edu, and robert.whetten@utsa.edu