Search

Many Augmented Reality (AR) applications require the alignment of virtual objects to the real world; this is particularly important in medical AR scenarios where medical imaging information may be displayed directly on a patient and is used to identify the exact locations of specific anatomical structures within the body. For optical see-through AR, alignment accuracy depends both on the optical parameters of the AR display as well as the visualization parameters of the virtual model. In this paper, we explore how different static visualization techniques influence users’ ability to perform perception-based alignment in AR for breast reconstruction surgery, where surgeons must accurately identify the locations of several perforator blood vessels while planning the procedure. We conducted a pilot study in which four subjects used four different visualization techniques with varying degrees of opaqueness and brightness as well as outline contrast to align virtual replicas of the relevant anatomy to their 3D-printed counterparts. We collected quantitative scores on spatial alignment accuracy using an external tracking system and qualitative scores on user preference and perceived performance. Results indicate that the highest source of alignment error was along the depth dimension, with users consistently overestimating depth when aligning the virtual renderings. The majority of subjects preferred visualization techniques rendered with lower levels of opaqueness and brightness as well as higher outline contrast, which were also found to support more accurate alignment.

Medical augmented reality (AR) applications where virtual renderings are aligned with the real world allow to visualize internal anatomy of the patient to a medical caregiver wearing an AR headset. Accurate alignment of virtual and real content is important for applications where the virtual rendering is used to guide the medical procedure such as a surgery. Compared to 2D AR applications, where the alignment accuracy can be directly measured on the 2D screen, 3D medical AR applications require alignment measurements using phantoms and external tracking systems. In this paper we present an approach for landmark-based alignment, validation and accuracy measurement of a 3D AR overlay of medical images on the real-world subject. This is done by performing an initial MRI of a subject’s head, an AR alignment task of the virtual rendering of the head MRI data to the subject’s real-world head using virtual fiducials, and a second MRI scan to test the accuracy of the AR alignment task. We have performed these 3D medical AR alignment measurements on seven volunteers using a MagicLeap AR head-mounted display. Across all seven volunteers we measured an alignment accuracy of $4.7 \pm 2.6$ mm. These results suggest that such an AR application can be a valuable tool for guiding non-invasive transcranial magnetic brain stimulation treatment. The presented MRI-based accuracy validation will furthermore be an important versatile tool to establish the safety of medical AR techniques. Index Terms: Mixed / augmented reality; Visualization design and evaluation methods

Depression affects more than 16 million American adults and more than half do not respond to medication. Transcranial magnetic stimulation (TMS) is an important anti-depressant treatment that targets specific brain circuits responsible for mood and behavior. TMS efficacy and risk is strongly linked to correct TMS coil placement and can be significantly improved by accurate neuronavigation. In this paper, we present tools for the development of a novel mixed reality neuronavigation setup that allows the TMS operator to view the patient's brain anatomy directly overlaid on the head. This is performed by integrating patient tracking and visualization of brain magnetic resonance imaging (MRI) to provide a streamlined visualization of the patient's anatomy in a single immersive environment.

One quarter of women who undergo breast lumpectomy to treat early-stage breast cancer in the United States undergo a repeat surgery due to concerns that residual tumor was left behind. This has led to a significant increase in women choosing mastectomy operations in the United States. We have developed a mixed-reality system that projects a 3D “hologram” of images from a breast MRI onto a patient using the Microsoft HoloLens. The goal of this system is to reduce the number of repeated surgeries by improving surgeons’ ability to determine tumor extent. We are conducting a pilot study in patients with palpable tumors that tests a surgeon’s ability to accurately identify the tumor location via mixed-reality visualization during surgical planning. Although early results are promising, it is critical but not straightforward to align holograms to the breast and to account for tissue deformations. More work is needed to improve the registration and holographic display at arm’s-length working distance. Nonetheless, first results from breast cancer surgeries have shown that mixed-reality guidance can indeed provide information about tumor location, and that this exciting new use for AR has the potential to improve the lives of many patients.

We present results of experiments with a haptic feedback device that imparts tangential deformations to the fingertips for displaying changes in force experienced by an MR-compatible optically-instrumented biopsy needle. The display is actuated using multiple layers of MR-compatible electroactive polymers stretched in a plastic frame. Users can use the device to sense events such as membrane puncture, as the needle is driven through a tissue phantom, with 98.9% reliability.

Magnetic resonance imaging (MRI) offers many benefits, including unsurpassed soft-tissue characterization and the ability to combine detection and biopsy into a single procedure. However, limited patient access in the narrow scanner bore requires tedious iterative positioning or use of robotic assistants that isolate the physician from the patient. As an alternative, we present a teleoperation technology for percutaneous procedures to meet the needs of interventional radiologists and overcome challenges imposed by the MR environment. The technology is demonstrated for a 1-DOF needle insertion procedure. The technology uses rolling diaphragms, a clutch, and a cable-capstan drive to propel the needle while relaying forces and motions to the operator. The system demonstrates excellent position tracking (< 0.7° error in the unloaded case) and reliably transmits changes in force. During needle teleoperation, users were able to detect light membrane punctures and differentiate spring stiffnesses nearly as accurately as by hand manipulation.

An MR-compatible biopsy needle stylet is instrumented with optical fibers that provide information about contact conditions between the needle tip and organs or hard tissues such as bone or tumors. This information is rendered via a haptic display that uses ultrasonic motors to convey directional cues to users. Lateral haptic cues at the fingertips improve the targeting accuracy and success rate in penetrating a prostate phantom. Although the original intent was for haptic cues to match the direction of contact forces and needle bending, more consistent results were obtained by using the cues as steering guidance (opposite to contact forces); accordingly this convention was adopted for the user experiments reported.

This paper presents calibration and user test results of a 3-D tip-force sensing needle with haptic feedback. The needle is a modified MRI-compatible biopsy needle with embedded fiber Bragg grating (FBG) sensors for strain detection. After calibration, the needle is interrogated at 2 kHz, and dynamic forces are displayed remotely with a voice coil actuator. The needle is tested in a single-axis master/slave system, with the voice coil haptic display at the master, and the needle at the slave end. Tissue phantoms with embedded membranes were used to determine the ability of the tip-force sensors to provide real-time haptic feedback as compared to external sensors at the needle base during needle insertion via the master/slave system. Subjects were able to determine the position of the embedded membranes with significantly better accuracy using FBG tip feedback than with base feedback using a commercial force/torque sensor (p = 0.045) or with no added haptic feedback (p = 0.0024).

We describe an instrumented biopsy needle that provides physicians the capability to sense interaction forces directly at the tip of the needle's inner stylet. The sensors consist of optical fiber Bragg gratings (FBGs), and are unaffected by electromagnetic fields; hence the needle is suitable for MR-guided procedures. In comparison to previous instrumented needles that measure bending strains, the new design has additional sensors and a series of micro-machined holes at the tip. The holes increase strain sensitivity to axial forces, without significantly reducing the stiffness or strength. Axial loads of 10 mN can be detected with flat response from 0-200 Hz. A comparison of the dynamic forces measured with the needle's sensors and those obtained using an external force/torque sensor at the base shows that the enhanced tip sensitivity is particularly noticeable when there is significant friction along the needle sheath.

A side optical actuation method is presented for a slender MR-compatible biopsy needle. The needle includes an active region with a shape memory alloy (SMA) wire actuator, where the wire generates a contraction force when optically heated by laser light delivered though optical fibers, producing needle tip bending. A prototype, with multiple side heating regions, demonstrates twice as fast an initial response compared to fiber tip heating when 0.8 W of optical power is applied. A single-ended optical sensor with a gold reflector is also presented to measure the curvature independently of temperature as a function of optical transmission loss. Preliminary tests with the sensor prototype demonstrate approximately linear response and a repeatable signal, independent of the bending history.