Search

We have developed some liver posture estimation methods for achieving a liver surgical navigation system that can support surgeons who needs to know very precise information about vessels in the liver. Those methods use 3D liver models scanned from patients and 2D images scanned by depth cameras for estimating the liver posture as accurately as possible. Since a new posture estimation method using simple and high-speed image hashing algorithm was developed last year, we are trying to improve the method in accuracy and applicability for the real-time liver posture tracking. In this paper, we examine how deep learning methods can be used for liver posture estimation and its tracking over the 2D images scanned from depth cameras. We study how can a multi-layer perceptron neural network learn and estimate the liver rotation expressed in quaternion form. The real-time surgical navigation system should be efficiently implemented by combining multiple estimation methods including the deep learning method.

As we have been developing a liver surgical navigation system that uses a scanned 3D liver model of the patient, we have pointed the necessity of the efficient posture estimation from depth images to be matched with the posture of a virtual liver rendered on a computer. We have previously used the technique of calculating the sum of squared errors using whole pixels contained in the depth images, and have compared over nine images simultaneously using the technique for the posture estimation. However, it requires a massive number of calculation repetition and that could be a barrier to the real time navigation through the surgery. In this paper, we propose a new method for comparing depth images and estimating postures by hashing the depth images vertically and horizontally. The liver postures can be identified by calculating 25 dimensional integer hashed values with the accuracy around 90%. By combining the previous posture estimation techniques with this method, the efficiency of the navigation system should be increased for the practical use in the real time surgical environment.

Liver surgery is typically performed to dissect part of the liver to remove a malignant tumor. The role of technology is to assist the surgeon to swiftly navigate to the area of interest. Our work involved the development of a liver surgery navigation system in which a steepest descent liver tracking algorithm is used to accurately track the real liver with a virtual liver. We recently demonstrated that our digital potential function was globally stable at the point at which the virtual liver coincided with its real counterpart. The same stability was achieved for several actual surgeries using 3D printed viscoelastic liver in an operating room with two light-emitting diode (LED) shadowless lamps. Increasing the number of lamps improved the stability of depth-depth matching in the steepest descent algorithm because the lamps did not emit in the infrared wavelength region unlike the depth cameras. Furthermore, the use of the characteristic uneven shape of the liver has greatly improved liver tracking accuracy. The complex and asymmetric shape of the upper part of the liver during surgery plays a key role in the liver in depth and depth matching. In this study, we experimentally investigated the stability of a virtual liver dissection configuration following a real liver in an operating room equipped with two LED shadowless lamps. As a result, deeply incised livers have superior depth-depth matching stability. In addition, even when using occlusion to simulate actual surgery, the convergence stability of the experimental performance is improved.

We are developing a liver surgical navigation system that uses a 3D liver model of the patient. As the liver contains several types of blood vessels, surgeons can sterically check the treating positions using the navigation system. For better navigation, the 3D liver model should be continuously and precisely matched with the patient’s liver state, and the liver location and orientation should be calculated in real time. The location and orientation is estimated by comparing two depth images of the liver; one is captured from the patient’s body using a depth camera, and one is rendered from the pre-scanned 3D liver model using OpenGL. The tracking is performed using gradient descent optimization algorithm. In this paper, we measure how the tracking agreement rate will change while the liver location and orientation are changing, so that we can understand the tracking performance, accuracy and the pitfall range of the navigation system.

We have developed the liver surgical navigation system composed of three subsystems: the liver position and orientation estimator, the surgical knife position estimator and the liver surgical navigator. These subsystems work separately for estimating the liver position and the knife position, and the liver surgical navigation system will use those positions to navigate surgeons accurately. The liver position is estimated by comparing two depth images; an image come from a depth camera targeting the liver and an image rendered by OpenGL using the Polyhedrons with STL-format data previously scanned from a patient. The knife position is estimated by tracking markers put at the top of the knife. The surgical navigation system holds precise data such as positions of vessels or the surgical steps, and show appropriate navigation data to the surgeons. In this paper, we describe the overview of this system and how we integrated these subsystems into the liver surgery supporting system.