Search

Modern surgical scenarios typically provide surgeons with additional information through fusion of video and other imaging modalities. To provide this information, the tools and devices used in surgery must be registered together with interventional guidance equipment and surgical navigation systems. In this work, we focus explicitly on registering ultrasound with a stereo camera system using photoacoustic markers. Previous work has shown that photoacoustic markers can be used in this registration task to achieve target registration errors lower than the current available systems. Photoacoustic markers are defined as a set of non-collinear laser spots projected onto some surface. They can be simultaneously visualized by a stereo camera system and an ultrasound transducer because of the photoacoustic effect. In more recent work, the three-dimensional ultrasound volume was replaced by images from a single ultrasound image pose from a convex array transducer. The feasibility of this approach was demonstrated, but the accuracy was lacking due to the physical limitations of the convex array transducer. In this work, we propose the use of a bi-plane transrectal ultrasound transducer. The main advantage of using this type of transducer is that the ultrasound elements are no longer restricted to a single plane. While this development would be limited to prostate applications, liver and kidney applications are also feasible if a suitable transducer is built. This work is demonstrated in two experiments, one without photoacoustic sources and one with. The resulting target registration error for these experiments were 1.07mm±0.35mm and 1.27mm±0.47mm respectively, both of which are better than current available navigation systems.

A requirement to reconstruct photoacoustic (PA) image is to have a synchronized channel data acquisition with laser firing. Unfortunately, most clinical ultrasound (US) systems don’t offer an interface to obtain synchronized channel data. To broaden the impact of clinical PA imaging, we propose a PA image reconstruction algorithm utilizing US B-mode image, which is readily available from clinical scanners. US B-mode image involves a series of signal processing including beamforming, followed by envelope detection, and end with log compression. Yet, it will be defocused when PA signals are input due to incorrect delay function. Our approach is to reverse the order of image processing steps and recover the original US post-beamformed radio-frequency (RF) data, in which a synthetic aperture based PA rebeamforming algorithm can be further applied. Taking B-mode image as the input, we firstly recovered US postbeamformed RF data by applying log decompression and convoluting an acoustic impulse response to combine carrier frequency information. Then, the US post-beamformed RF data is utilized as pre-beamformed RF data for the adaptive PA beamforming algorithm, and the new delay function is applied by taking into account that the focus depth in US beamforming is at the half depth of the PA case. The feasibility of the proposed method was validated through simulation, and was experimentally demonstrated using an acoustic point source. The point source was successfully beamformed from a US B-mode image, and the full with at the half maximum of the point improved 3.97 times. Comparing this result to the ground-truth reconstruction using channel data, the FWHM was slightly degraded with 1.28 times caused by information loss during envelope detection and convolution of the RF information.

Photoacoustic imaging (PA) is becoming a promising modality for pre-clinical and clinical application by providing functional information with high penetration depth. In particular, PA computed tomography (PACT) aims to visualize the photoacoustic source distribution by scanning ultrasound transducers around a surface of the structure. Placing transducers around the subject or rotating the subject with fixing transducer position are two major approaches to scan in circular arc trajectory, but both are not flexible and have their drawbacks. To resolve the problem, we propose a new scanning method based on the robotic tracking technique. High flexibility of the scanning geometry is available because the trajectory of the tracked transducer generated by robot motion will be regarded as the scanning path. A simulation study of proposed method is conducted, assuming a 6cm array ultrasound transducer was used as receivers. To replicate the scenario that the probe is moved by robot, the probe is placed at two positions across a designated rotation angle, and received signals at both positions are used to generate an image. Compared to the result without robot motion, the lateral resolution of the target drastically improved due to aperture extension. In addition, the effect of error in one of the pose in lateral and axial direction was also simulated. Finally, an experiment was conducted to demonstrate the feasibility of the system. The result indicates that the ultrasound calibration is required to the system, and the robotically tracked PACT has a huge potential to be a new scanning strategy.

As thermal imaging attempts to estimate very small tissue motion (on the order of tens of microns), it can be negatively influenced by signal decorrelation. Patient's breathing and cardiac cycle generate shifts in the RF signal patterns. Other sources of movement could be found outside the patient's body, like transducer slippage or small vibrations due to environment factors like electronic noise. Here, we build upon a robust displacement estimation method for ultrasound elastography and we investigate an iterative motion compensation algorithm, which can detect and remove non-heat induced tissue motion at every step of the ablation procedure. The validation experiments are performed on laboratory induced ablation lesions in ex-vivo tissue. The ultrasound probe is either held by the operator's hand or supported by a robotic arm. We demonstrate the ability to detect and remove non-heat induced tissue motion in both settings. We show that removing extraneous motion helps unmask the effects of heating. Our strain estimation curves closely mirror the temperature changes within the tissue. While previous results in the area of motion compensation were reported for experiments lasting less than 10 seconds, our algorithm was tested on experiments that lasted close to 20 minutes.

We investigated a novel needle visualization using the PA effect to enhance needle-tip tracking. An optical fiber and laser source are used to generate acoustic waves inside the needle with the PA effect. Acoustic waves are generated along the needle. Some amount of acoustic energy leaks into the surrounding material. The leakage of acoustic waves is captured by a conventional US transducer and US channel data collection system. Then, the collected data are converted to a PA image. The needle-tip can be visualized more clearly in this PA image than a general US brightness mode image.

Photoacoustic (PA) imaging is becoming an important tool for various clinical and pre-clinical applications. Acquiring pre-beamformed channel ultrasound data is essential to reconstruct PA images. Accessing these pre-beamformed channel data requires custom hardware to allow parallel beam-forming, and is available for only few research ultrasound platforms. However, post-beamformed radio frequency (RF) data is readily available in real-time and in several clinical and research ultrasound platforms. To broaden the impact of clinical PA imaging, our goal is to devise new PA reconstruction approach based on these post-beamformed RF data. In this paper, we propose to generate PA image by using a single receive focus beamformed RF data. These beamformed RF data are considered as pre-beamformed input data to a synthetic aperture beamforming algorithm, where the focal point per received RF line is a virtual element. The image resolution is determined by the fixed focusing depth as well as the aperture size used in fixed focusing. In addition, the signal-to-noise (SNR) improvement is expected because beamforming is performed twice with different noise distribution. The performance of the proposed method is analyzed through simulation, the practical feasibility is validated experimentally. The results indicate that the post-beamformed RF data has potential to be re-beamformed to a PA image using the proposed synthetic aperture beamformer.

Targeted contrast agents can improve the sensitivity of imaging systems for cancer detection and monitoring the treatment. In order to accurately detect contrast agent concentration from photoacoustic images, we developed a decomposition algorithm to separate photoacoustic absorption spectrum into components from individual absorbers. In this study, we evaluated novel prostate-specific membrane antigen (PSMA) targeted agents for imaging prostate cancer. Three agents were synthesized through conjugating PSMA-targeting urea with optical dyes ICG, IRDye800CW and ATTO740 respectively. In our preliminary PA study, dyes were injected in a thin wall plastic tube embedded in water tank. The tube was illuminated with pulsed laser light using a tunable Q-switch ND-YAG laser. PA signal along with the B-mode ultrasound images were detected with a diagnostic ultrasound probe in orthogonal mode. PA spectrums of each dye at 0.5 to 20 μM concentrations were estimated using the maximum PA signal extracted from images which are obtained at illumination wavelengths of 700nm-850nm. Subsequently, we developed nonnegative linear least square optimization method along with localized regularization to solve the spectral unmixing. The algorithm was tested by imaging mixture of those dyes. The concentration of each dye was estimated with about 20% error on average from almost all mixtures albeit the small separation between dyes spectrums.

Fusion of video and other imaging modalities is common in modern surgical scenarios to provide surgeons with additional information. Doing so requires the use of interventional guidance equipment and surgical navigation systems to register the tools and devices used in surgery with each other. In this work, we focus explicitly on registering ultrasound with a stereocamera system using photoacoustic markers. Previous work has shown that photoacoustic markers can be used to register three-dimensional ultrasound with video resulting in target registration errors lower than the current available systems. Photoacoustic markers are non-collinear laser spots projected onto some surface. They can be simultaneously visualized by a stereocamera system and in an ultra-sound volume because of the photoacoustic effect. This work replaces the three-dimensional ultrasound volume with images from a single ultrasound image pose. While an ultrasound volume provides more information than an ultrasound image, it has its disadvantages such as higher cost and slower acquisition rate. However, in general, it is difficult to register two-dimensional with three-dimensional spatial data. We propose the use of photoacoustic markers viewed by a convex array ultrasound transducer. Each photoacoustic markers wavefront provides information on its elevational position, resulting in three-dimensional spatial data. This development enhances this methods practicality as convex array transducers are more common in surgical practice than three-dimensional transducers. This work is demonstrated on a synthetic phantom. The resulting target registration error for this experiment was 2.47mm and the standard deviations was 1.29mm, which is comparable to current available systems.

Acquisition of ultrasound (US) pre-beamformed radio-frequency (RF) data is essential in photoacoustic (PA) imaging research. Moreover, 3D PA imaging can provide volumetric information for a target of interest. However, existing 3D PA systems require specifically designed motion stages, an ultrasound scanner and a data acquisition system to collect 3D pre-beamformed RF data. These systems are incompatible with clinical ultrasound systems and are difficult to reconfigure and generalize to other PA research. To overcome these limitations, we proposed and developed a new software framework for spatially-tracked pre-beamformed RF data acquisition with a conventional 2D ultrasound transducer and external tracking device. We upgraded our previous software framework using task-classes of OpenIGTLinkMUSiiC2.0 and MUSiiCToolkit 2.0. We also improved our MUSiiCToolKit 2.0 by adding MUSiiCNotes 2.0, a collection of specific task-classes for US research. MUSiiC-DAQServer2.0, MUSiiC-TrackerServer and MUSiiCSync are the main modules of our software framework. Spatially-tracked 2D PA frames are collected efficiently using this software framework for 3D PA research and imaging. The software modules of our software framework are based on the concept of network distributed modules and can simultaneously support multiple-client connections via TCP/IP network. In addition, the collected 2D PA frames are compatible with other MUSiiCToolKit 2.0 modules such as MUSiiC-Beamform, MUSiiC -BMode and MUSiiC - ImageViewer modules. These aspects of our software framework allow us to easily reconfigure and customize our system to other PA or US research.

Fusion of video and other imaging modalities is common in modern surgical procedures to provide surgeons with additional information that can provide precise surgical guidance. An example of such uses interventional guidance equipment and surgical navigation systems to register the tools and devices used in surgery with each other. In this work, we focus explicitly on registering three-dimensional ultrasound with a stereocamera system. These surgical navigation systems often use optical or electromagnetic trackers. However, both of these tracking systems have various drawbacks leading to target registration errors of approximately 3mm. Previous work has shown that photoacoustic markers can be used to register three-dimensional ultrasound with video resulting in target registration errors which are much lower than the current state of the art. This work extends this idea by generating multiple photoacoustic markers concurrently as opposed to the sequential method used in the previous work. This development greatly enhances the acquisition time by a factor equal to the number of concurrently generated photoacoustic markers. This work is demonstrated on a synthetic phantom and an ex vivo porcine kidney phantom. The resulting target registration errors for these experiments ranged from 840 to 1360 μm and standard deviations from 370 to 640 μm.

Photoacoustic (PA) imaging is an emerging medical imaging modality that relies on the absorption of optical energy and the subsequent emission of acoustic waves that are detected with a conventional ultrasound probe. PA images are susceptible to background noise artifacts that reduce the signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR). We investigated spatial-angular compounding of PA images to enhance these image qualities. Spatial-angular compounding was implemented by averaging multiple PA images acquired as an ultrasound probe was rotated about the elevational axis with the laser beam and PA target fixed in the same location. An external tracking system was used to provide the position and orientation (i.e. pose) information of each PA image. Based on this pose information, frames in similar elevational planes were filtered from the acquired image data and compounded using one of two methods. One method registered overlapping signals between frames prior to compounding (using the pose information), while the second method omitted this spatial registration step. These two methods were applied to pre-beamformed RF, beamformed RF, and envelope-detected data, resulting in six different compounding pipelines. Compounded PA images with similar lateral resolution to a single reference image had factors of 1.1 - 1.6, 2.0 - 11.1, and 2.0 - 11.1 improvements in contrast, CNR, and SNR, respectively, when compared to the reference image. These improvements depended on the amount of relative motion between the reference image and the images that were compounded. The inclusion of spatial registration prior to compounding preserved lateral resolution and signal location when the relative rotations about the elevation axis were 3.5° or less for images that were within an elevational distance of 2.5 mm from the reference image, particularly when the method was applied to the enveloped-detected data. Results indicate that spatial-angular compounding has the potential to improve image quality for a variety of photoacoustic imaging applications.

In recent years, various methods have been developed to improve ultrasound based interventional tool tracking. However, none of them has yet provided a solution that effectively solves the tool visualization and mid-plane localization accuracy problem and fully meets the clinical requirements. Our previous work has demonstrated a new active ultrasound pattern injection system (AUSPIS), which integrates active ultrasound transducers with the interventional tool, actively monitors the beacon signals and transmits ultrasound pulses back to the US probe with the correct timing. Ex vivo and in vivo experiments have proved that AUSPIS greatly improved tool visualization, and provided tool-tip localization accuracy of less than 300 μm. In the previous work, the active elements were made of piezoelectric materials. However, in some applications the high driving voltage of the piezoelectric element raises safety concerns. In addition, the metallic electrical wires connecting the piezoelectric element may also cause artifacts in CT and MR imaging. This work explicitly focuses on an all-optical active ultrasound element approach to overcome these problems. In this approach, the active ultrasound element is composed of two optical fibers - one for transmission and one for reception. The transmission fiber delivers a laser beam from a pulsed laser diode and excites a photoacoustic target to generate ultrasound pulses. The reception fiber is a Fabry–Perot hydrophone. We have made a prototype catheter and performed phantom experiments. Catheter tip localization, mid-plan detection and arbitrary pattern injection functions have been demonstrated using the all-optical AUSPIS.

The concern with interstitial ablative therapy for a treatment of hepatic tumors has been growing. In spite of advances in these therapies, there are several technical challenges due to tissue deformation and target motion: localization of the tumor and monitoring for ablator's tip and thermal dose in heated tissue. In the previous work, a steerable acoustic ablator, called ACUSITT, for targeting of ablation tip accurately into tumor area has been developed. However, real-time monitoring techniques for providing image feedback of the ablation tip positioning and thermal dose deposited in the tissue by heating are still needed. In this paper, a new software framework for real-time monitoring ablative therapy during pre- and intra-operation is presented. The software framework provides ultrasound Brightness Mode (B-Mode) image and elastography simultaneously and with real-time. A position of ablator's tip and a region of heated tissue are monitored on B-Mode image, because the image represents tissue morphology. Furthermore, ultrasound elasticity image is used for finding a boundary and region of tumor on pre-ablation, and monitoring thermal dose in tissue during ablation. By providing B-Mode image and elastography at the same time, reliable information for monitoring thermal therapy can be offered.

Surgical robots provide many advantages for surgery, including minimal invasiveness, precise motion, high dexterity, and crisp stereovision. One limitation of current robotic procedures, compared to open surgery, is the loss of haptic information for such purposes as palpation, which can be very important in minimally invasive tumor resection. Numerous studies have reported the use of real-time ultrasound elastography, in conjunction with conventional B-mode ultrasound, to differentiate malignant from benign lesions. Several groups (including our own) have reported integration of ultrasound with the da Vinci robot, and ultrasound elastography is a very promising image guidance method for robotassisted procedures that will further enable the role of robots in interventions where precise knowledge of sub-surface anatomical features is crucial. We present a novel robot-assisted real-time ultrasound elastography system for minimally invasive robot-assisted interventions. Our system combines a da Vinci surgical robot with a non-clinical experimental software interface, a robotically articulated laparoscopic ultrasound probe, and our GPU-based elastography system. Elasticity and B-mode ultrasound images are displayed as picture-in-picture overlays in the da Vinci console. Our system minimizes dependence on human performance factors by incorporating computer-assisted motion control that automatically generates the tissue palpation required for elastography imaging, while leaving high-level control in the hands of the user. In addition to ensuring consistent strain imaging, the elastography assistance mode avoids the cognitive burden of tedious manual palpation. Preliminary tests of the system with an elasticity phantom demonstrate the ability to differentiate simulated lesions of varied stiffness and to clearly delineate lesion boundaries.

Acquisition of pre-beamformed data is essential in advanced imaging research studies such as adaptive beamforming, synthetic aperture imaging, and photoacoustic imaging. Ultrasonix Co. has developed such a data acquisition device for pre-beamformed data known as the SONIX-DAQ, but data can only be downloaded and processed offline rather than streamed in real-time. In this work, we developed a software framework to extend the functionality of the SONIX-DAQ for streaming and processing data in near real-time. As an example, we applied this functionality to our previous work of visualizing photoacoustic images of prostate brachytherapy seeds. In this paper, we present our software framework, applying it to a real-time photoacoustic imaging system, including real-time data collection and data-processing software modules for brachytherapy treatment.

Registration is a key technology in image-guided navigation systems. By aligning pre-operative images with the intra-operative setting these systems provide visual feedback that improves the physician's understanding of the spatial relationships between anatomical structures and surgical tools. Most often the alignment is obtained using fiducials. Another option is to replace the use of fiducials with intra-operative imaging. Two dimensional ultrasound (US) is a widely available intra-operative non-ionizing imaging modality. To utilize this modality for registration one must first perform spatial calibration of the US. In this work we describe the implementation of three spatial calibration methods as part of the image-guided surgery toolkit (IGSTK). The implementation follows the IGSTK calibration framework, separating algorithmic aspects from user interaction aspects of the calibration. Our calibration framework includes three methods. The first is a phantom-less method using a tracked pointer tool in addition to the tracked US, the second method uses a cross-wire phantom, and the third method is based on the use of a plane phantom.

The localization of brachytherapy seeds in relation to the prostate is a key step in intraoperative treatment planning (ITP) for improving outcomes in prostate cancer patients treated with low dose rate prostate brachytherapy. Transrectal ultrasound (TRUS) has traditionally been the modality of choice to guide the prostate brachytherapy procedure due to its relatively low cost and apparent ease of use. However, TRUS is unable to visualize seeds well, precluding ITP and producing suboptimal results. While other modalities such as X-ray and magnetic resonance imaging have been investigated to localize seeds in relation to the prostate, photoacoustic imaging has become an emerging and promising modality to solve this challenge. Moreover, photoacoustic imaging may be more practical in the clinical setting compared to other methods since it adds little additional equipment to the ultrasound system already adopted in procedure today, reducing cost and simplifying engineering steps. In this paper, we demonstrate the latest efforts of localizing prostate brachytherapy seeds using photoacoustic imaging, including visualization of multiple seeds in actual prostate tissue. Although there are still several challenges to be met before photoacoustic imaging can be used in the operating room, we are pleased to present the current progress in this effort.

Many recent studies have demonstrated the efficacy of interstitial ablative approaches for the treatment of hepatic tumors. Despite these promising results, current systems remain highly dependent on operator skill, and cannot treat many tumors because there is little control of the size and shape of the zone of necrosis, and no control over ablator trajectory within tissue once insertion has taken place. Additionally, tissue deformation and target motion make it extremely difficult to place the ablator device precisely into the target. Irregularly shaped target volumes typically require multiple insertions and several overlapping (thermal) lesions, which are even more challenging to accomplish in a precise, predictable, and timely manner without causing excessive damage to surrounding normal tissues. In answer to these problems, we have developed a steerable acoustic ablator called the ACUSITT with the ability of directional energy delivery to precisely shape the applied thermal dose . In this paper, we address image guidance for this device, proposing an innovative method for accurate tracking and tool registration with spatially-registered intra-operative three-dimensional US volumes, without relying on an external tracking device. This method is applied to guid-ance of the flexible, snake-like, lightweight, and inexpensive ACUSITT to facilitate precise placement of its ablator tip within the liver, with ablation monitoring via strain imaging. Recent advancements in interstitial high-power ultrasound applicators enable controllable and penetrating heating patterns which can be dynamically altered. This paper summarizes the design and development of the first synergistic system that integrates a novel steerable interstitial acoustic ablation device with a novel trackerless 3DUS guidance strategy.

Overlay performance and control requirements have become crucial for achieving high yield and reducing rework process. Increasing discrepancy between hardware solutions and overlay requirements, especially in sub-40nm dynamic random access memory (DRAM) devices, motivates us to study process budgeting techniques and reasonable validation methods. In this paper, we introduce a SMEM (Statistical process Margin Estimation Method) to design the DRAM cell architecture which considers critical dimension (CD) and overlay variations in the perspectives of both cell architecture and manufacturability. We also proposed the method to determine overlay specifications. Using the methodologies, we obtained successfully optimized sub-40 DRAM cells which accurately estimated process tolerances and determined overlay specifications for all layers.

Applications that require overlay measurement between layers separated by absorbing interlayer films (such as α- carbon) pose significant challenges for sub-50nm processes. In this paper scatterometry methods are investigated as an alternative to meet these stringent overlay metrology requirements. In this article, a spectroscopic Diffraction Based Overlay (DBO) measurement technique is used where registration errors are extracted from specially designed diffraction targets. DBO measurements are performed on detailed set of wafers with varying α-carbon (ACL) thicknesses. The correlation in overlay values between wafers with varying ACL thicknesses will be discussed. The total measurement uncertainty (TMU) requirements for these layers are discussed and the DBO TMU results from sub-50nm samples are reviewed.

Simulation and experimental study results are reported to solve align/overlay problem in dark hard mask process in lithography. For simulation part, an in-house simulator, which is based on rigorous coupled wave analysis and Fourier optics method of high NA imaging, is used. According to the simulation and experiment study, image quality of alignment and overlay marks can be optimized by choosing hard mask and sub-film thickness carefully for a given process condition. In addition, it is important to keep the specification of film thickness uniformity within a certain limit. Simulation results are confirmed by experiment using the state of art memory process in Samsung semiconductor R&D facility.

The introduction of ArF lithography technology is needed for on-chip linewidth variation(OCV) control less than 10nm in 90nm logic transistor development. Since conventional KrF lithography increased the burdens of mask fabrication and photo process due to excessive optical proximity correction(OPC), ArF lithography is more required to improve pattern feasibility in terms of line edge roughness(LER), corner rounding and contact overlapping margin than before. In this paper, we investigated two major components of OCV, that is, proximity and uniformity using ArF lithography. For a tighter CD control, the proximity can be corrected by hybrid OPC method, which is a combination of rule-based and model-based OPC. The uniformity can be effectively improved by several methods such as lithography-friendly layout formation, optimal substrate condition, decrease in MEEF and tuning of the resist process. In conclusion, by using ArF lithography we could obtain the satisfactory OCV control less than 10nm and reasonable process latitude simultaneously for 90nm logic gate under the condition of well-controlled proximity and uniformity.

For 100nm-level patterning using optical lithography, high NA system and various RETs such as PSM, off-axis illumination and OPC are obviously required. In particular, assistant feature (AF)-OPC is indispensable to overcome narrow depth of focus (DOF) caused by iso-dense bias and to compensate for linearity difference under the given OAI condition. Previously we reported the application of AF-OPC in DRAM process with 120nm design rule. The extraction of OPC rule and the feasibility of AF-OPC were successfully confirmed by experimental method in real process. In this paper, more comprehensive and aggressive AF-OPC rule is investigated. The old rule is modified in order to obtain larger common DOF. TO avoid dead zone that means discontinuity between dense line and semi-dense line, we apply a comprehensive rule such as insertion of AF between the neighboring main patterns as many as possible. As a result, the discontinuity of OPC application, which is used with or without AF in the boundary region, is effectively minimized. Also, polygon-shaped AF is used to improve DOF of special main pattern. And then, the mask specification and the behavior of isolated line pattern are predicted in case of very high NA KrF and ArF lithography by simulation result. Considering 100nm design rule, the decrease of common DOF is expected to be severer than now. Finally, the optimum AF-OPC rules such as AF size, space and shape are available and shown in case of very high NA KrF and ArF lithography.