Search

Previously, we identified 8 objective suturing performance metrics highly predictive of urinary continence recovery after robotic-assisted radical prostatectomy. Here, we aimed to test the feasibility of providing tailored feedback based upon these clinically relevant metrics and explore the impact on the acquisition of robotic suturing skills.Training surgeons were recruited and randomized to a feedback group or a control group. Both groups completed a baseline, midterm and final dry laboratory vesicourethral anastomosis (VUA) and underwent 4 intervening training sessions each, consisting of 3 suturing exercises. Eight performance metrics were recorded during each exercise: 4 automated performance metrics (derived from kinematic and system events data of the da Vinci® Robotic System) representing efficiency and console manipulation competency, and 4 suturing technical skill scores. The feedback group received tailored feedback (a visual diagram+verbal instructions+video examples) based on these metrics after each session. Generalized linear mixed model was used to compare metric improvement (Δ) from baseline to the midterm and final VUA.Twenty-three participants were randomized to the feedback group (11) or the control group (12). Demographic data and baseline VUA metrics were comparable between groups. The feedback group showed greater improvement than the control group in aggregate suturing scores at midterm (mean Δ feedback group 4.5 vs Δ control group 1.1) and final VUA (Δ feedback group 5.3 vs Δ control group 4.9). The feedback group also showed greater improvement in the majority of the included metrics at midterm and final VUA.Tailored feedback based on specific, clinically relevant performance metrics is feasible and may expedite the acquisition of robotic suturing skills.

Purpose: We attempt to understand the relationship between surgeon technical skills, cognitive workload, and errors during a simulated robotic dissection task. Materials and Methods: Participant surgeons performed a robotic surgery dissection exercise. Participants were grouped based on surgical experience. Technical skills were evaluated utilizing the validated Global Evaluative Assessment of Robotic Skills (GEARS) assessment tool. The dissection task was evaluated for errors during active dissection or passive retraction maneuvers. We quantified cognitive workload of surgeon participants as an index of cognitive activity (ICA), derived from task-evoked pupillary response metrics; ICA ranged 0 to 1, with 1 representing maximum ICA. Generalized estimating equation (GEE) was used for all modelings to establish relationships between surgeon technical skills, cognitive workload, and errors. Results: We found a strong association between technical skills as measured by multiple GEARS domains (depth perception, force sensitivity, and robotic control) and passive errors, with higher GEARS scores associated with a lower relative risk of errors (all p

Objective To automate surgeon skills evaluation using robotic instrument kinematic data. Additionally, to implement an unsupervised mislabeling detection algorithm to identify potentially mislabeled samples that can be removed to improve model performance. Methods Video recordings and instrument kinematic data were derived from suturing exercises completed on the Mimic FlexVR robotic simulator. A structured human consensus-building process was developed to determine Robotic Anastomosis Competency Evaluation technical scores across 3 human graders. A 2-layer long short-term memory–based classification model used instrument kinematic data to automate suturing skills assessment. An unsupervised label analyzer (NoiseRank) was used to identify potential mislabeling of skills data. Performance of the long short-term memory model’s technical skill score prediction was measured by best area under the curve over the training runs. NoiseRank outputted a ranked list of rated skills assessments based on likelihood of mislabeling. Results 22 surgeons performed 226 suturing attempts, which were broken down into 1,404 individual skill assessment points. Automation of needle entry angle, needle driving, and needle withdrawal technical skill scores performed better (area under the curve 0.698–0.705) than needle positioning (0.532) at baseline using all available data. Potential mislabels were subsequently identified by NoiseRank and removed, improving model performance across all domains (area under the curve 0.551–0.766). Conclusion Using ground truth labels from human graders and robotic instrument kinematic data, machine learning models have automated assessment of detailed suturing technical skills with good performance. Further, an unsupervised mislabeling detection algorithm projected mislabeled data, allowing for their removal and subsequent improvement of model performance.

Background It has been shown that metrics recorded for instrument kinematics during robotic surgery can predict urinary continence outcomes. Objective To evaluate the contributions of patient and treatment factors, surgeon efficiency metrics, and surgeon technical skill scores, especially for vesicourethral anastomosis (VUA), to models predicting urinary continence recovery following robot-assisted radical prostatectomy (RARP). Design, setting, and participants Automated performance metrics (APMs; instrument kinematics and system events) and patient data were collected for RARPs performed from July 2016 to December 2017. Robotic Anastomosis Competency Evaluation (RACE) scores during VUA were manually evaluated. Training datasets included: (1) patient factors; (2) summarized APMs (reported over RARP steps); (3) detailed APMs (reported over suturing phases of VUA); and (4) technical skills (RACE). Feature selection was used to compress the dimensionality of the inputs. Outcome measurements and statistical analysis The study outcome was urinary continence recovery, defined as use of 0 or 1 safety pads per day. Two predictive models (Cox proportional hazards [CoxPH] and deep learning survival analysis [DeepSurv]) were used. Results and limitations Of 115 patients undergoing RARP, 89 (77.4%) recovered their urinary continence and the median recovery time was 166 d (interquartile range [IQR] 82–337). VUAs were performed by 23 surgeons. The median RACE score was 28/30 (IQR 27–29). Among the individual datasets, technical skills (RACE) produced the best models (C index: CoxPH 0.695, DeepSurv: 0.708). Among summary APMs, posterior/anterior VUA yielded superior model performance over other RARP steps (C index 0.543–0.592). Among detailed APMs, metrics for needle driving yielded top-performing models (C index 0.614–0.655) over other suturing phases. DeepSurv models consistently outperformed CoxPH; both approaches performed best when provided with all the datasets. Limitations include feature selection, which may have excluded relevant information but prevented overfitting. Conclusions Technical skills and “needle driving” APMs during VUA were most contributory. The best-performing model used synergistic data from all datasets. Patient summary One of the steps in robot-assisted surgical removal of the prostate involves joining the bladder to the urethra. Detailed information on surgeon performance for this step improved the accuracy of predicting recovery of urinary continence among men undergoing this operation for prostate cancer.

PURPOSE: To create a suturing skills assessment tool that comprehensively defines criteria around relevant sub-skills of suturing and to confirm its validity. MATERIALS AND METHODS: 5 expert surgeons and an educational psychologist participated in a cognitive task analysis (CTA) to deconstruct robotic suturing into an exhaustive list of technical skill domains and sub-skill descriptions. Using the Delphi methodology, each CTA element was systematically reviewed by a multi-institutional panel of 16 surgical educators and implemented in the final product when content validity index (CVI) reached ≥0.80. In the subsequent validation phase, 3 blinded reviewers independently scored 8 training videos and 39 vesicourethral anastomoses (VUA) using EASE; 10 VUA were also scored using Robotic Anastomosis Competency Evaluation (RACE), a previously validated, but simplified suturing assessment tool. Inter-rater reliability was measured with intra-class correlation (ICC) for normally distributed values and prevalence-adjusted bias-adjusted Kappa (PABAK) for skewed distributions. Expert (≥100 prior robotic cases) and trainee (

Objectives: Manually-collected suturing technical skill scores are strong predictors of continence recovery after robotic radical prostatectomy. Herein, we automate suturing technical skill scoring through computer vision (CV) methods as a scalable method to provide feedback. Methods: Twenty-two surgeons completed a suturing exercise three times on the Mimic™ Flex VR simulator. Instrument kinematic data (XYZ coordinates of each instrument and pose) were captured at 30 Hz. After standardized training, three human raters manually video segmented suturing task into four sub-stitch phases (Needle handling, Needle targeting, Needle driving, Needle withdrawal) and labeled the corresponding technical skill domains (Needle positioning, Needle entry, Needle driving, and Needle withdrawal). The CV framework extracted RGB features and optical flow frames using a pre-trained AlexNet. Additional CV strategies including auxiliary supervision (using kinematic data during training only) and attention mechanisms were implemented to improve performance. Results: This study included data from 15 expert surgeons (median caseload 300 [IQR 165–750]) and 7 training surgeons (0 [IQR 0–8]). In all, 226 virtual sutures were captured. Automated assessments for Needle positioning performed best with the simplest approach (1 s video; AUC 0.749). Remaining skill domains exhibited improvements with the implementation of auxiliary supervision and attention mechanisms when deployed separately (AUC 0.604–0.794). All techniques combined produced the best performance, particularly for Needle driving and Needle withdrawal (AUC 0.959 and 0.879, respectively). Conclusions: This study demonstrated the best performance of automated suturing technical skills assessment to date using advanced CV techniques. Future work will determine if a “human in the loop” is necessary to verify surgeon evaluations.

Automated performance metrics (APMs) objectively measure surgeon performance during a robot-assisted radical prostatectomy (RARP). Machine learning (ML) has shown that APMs, especially during the vesico-urethral anastomosis (VUA) of the RARP, are predictive of long-term outcomes such as continence recovery time. This study focuses on APMs during the VUA, specifically on stitch versus sub-stitch levels, to distinguish surgeon experience. During the VUA, APMs, recorded by a systems data recorder (Intuitive Surgical), were reported for each overall stitch (C(total)) and its individual components: needle handling/targeting (C(1)), needle driving (C(2)), and suture cinching (C(3)) (Figure 1A). These metrics were organized into three datasets (GlobalSet [whole stitch], RowSet [independent sub-stitches], and ColumnSet [associated sub-stitches] (Figure 1B) and applied to three ML models (AdaBoost, Gradient Boosting, and Random Forest) in order to solve two classifications tasks: experts (≥ 100 cases) vs. novices (

INTRODUCTION AND OBJECTIVE:Automated performance metrics (APMs), derived from instrument kinematic/systems events data during surgery and captured by a systems data recorder (Intuitive Surgical), a...

INTRODUCTION AND OBJECTIVE: In March 2020, elective surgical procedures in the US were postponed due to the COVID-19 pandemic. The potential degradation of surgical skills during this hiatus has not yet been studied. Our previous work demonstrated that suturing performance is associated with urinary continence recovery after robotic prostatectomy. Herein, we use objective virtual reality (VR) simulation metrics to evaluate suturing technical skill degradation during the COVID-19 shutdown. METHODS: Surgeons completed a robotic VR suturing sponge exercise immediately prior to and after a mandated six-week surgical hiatus that ran from March 20, 2020 to May 5, 2020 at our institution. VR simulator metrics were collected including completion time, economy of motion, instrument collisions, needle drops, and missed targets. Comparisons of parameters before/after the shutdown were performed with either paired Student's t test or Wilcoxon-matched signed rank test. RESULTS: 22 urologic surgeons completed VR sessions preand post- COVID-19 shutdown. Surgeons were classified a priori as novice ( 300 cases, n=9) surgeons. Among all participants, weekly robotic surgery caseload decreased from median (IQR) 2.75 (0.5-6.0) pre- COVID to 0.75 (0-1.5) during the shutdown (p

INTRODUCTION AND OBJECTIVE:One of the major functional outcomes following robot-assisted radical prostatectomy (RARP) is recovery of erectile function (EF). In this study, we aim to evaluate the ef...

Background During robotic surgeries, kinematic metrics objectively quantify surgeon performance. Objective To determine whether clinical factors confound the ability of surgeon performance metrics to anticipate urinary continence recovery after robot-assisted radical prostatectomies (RARPs). Design, setting, and participants Clinical data (patient characteristics, continence recovery, and treatment factors) and surgeon data from RARPs performed between July 2016 and November 2018 were prospectively collected. Surgeon data included 40 automated performance metrics (APMs) derived from robot systems (instrument kinematics and events) and summarized over each standardized RARP step. The data were collected from two high-volume robotic centers in the USA and Germany. Surgeons from both institutions performed RARPs. The inclusion criteria were consecutive RARPs having both clinical and surgeon data. Intervention RARP with curative intent to treat prostate cancer. Outcome measurements and statistical analysis The outcome was 3- and 6-mo urinary continence recovery status. Continence was defined as the use of zero or one safety pad per day. Random forest (SAS HPFOREST) was utilized. Results and limitations A total of 193 RARPs performed by 20 surgeons were included. Of the patients, 56.7% (102/180) and 73.3% (129/176) achieved urinary continence by 3 and 6 mo after RARP, respectively. The model anticipated continence recovery (area under the curve = 0.74, 95% confidence interval [CI] 0.66–0.81 for 3-mo, and area under the curve = 0.67, 95% CI 0.58–0.76 for 6 mo). Clinical factors, including pT stage, confounded APMs during prediction of continence recovery at 3 mo after RARP (Δβ median –13.3%, interquartile range [–28.2% to –6.5%]). After adjusting for clinical factors, 11/20 (55%) top-ranking APMs remained significant and independent predictors (ie, velocity and wrist articulation during the vesicourethral anastomosis). Limitations included heterogeneity of surgeon/patient data between institutions, although it was accounted for during multivariate analysis. Conclusions Clinical factors confound surgeon performance metrics during the prediction of urinary continence recovery after RARP. Nonetheless, many surgeon factors are still independent predictors of early continence recovery. Patient summary Both patient factors and surgeon kinematic metrics, recorded during robotic prostatectomies, impact early urinary continence recovery after robot-assisted radical prostatectomy., Take Home Message In this cohort study, machine learning anticipates urinary continence recovery at 3 mo (area under the curve 0.74) using surgeon performance metrics and clinical data. While clinical factors confound metrics (Δβ median –13.3%, interquartile range [–28.2% to –6.5%]), top-ranked surgeon metrics remain independent predictors.

Introduction: Evaluation of surgical competency has important implications for training new surgeons, accreditation, and improving patient outcomes. A method to specifically evaluate dissection performance does not yet exist. This project aimed to design a tool to assess surgical dissection quality. Methods: Delphi method was used to validate structure and content of the dissection evaluation. A multi-institutional and multi-disciplinary panel of 14 expert surgeons systematically evaluated each element of the dissection tool. Ten blinded reviewers evaluated 46 de-identified videos of pelvic lymph node and seminal vesicle dissections during the robot-assisted radical prostatectomy. Inter-rater variability was calculated using prevalence-adjusted and bias-adjusted kappa. The area under the curve from receiver operating characteristic curve was used to assess discrimination power for overall Dissection Assessment for Robotic Technique (DART) scores as well as domains in discriminating trainees (100). Results: Four rounds of Delphi method achieved language and content validity in 27/28 elements. Use of 3 or 5-point scale remained contested; thus, both scales were evaluated during validation. The 3-point scale showed improved kappa for each domain. Experts demonstrated significantly greater total scores on both scales (3-point, p

Automated performance metrics provide a novel approach to the assessment of surgical performance. Herein, we present a construct validation of automated performance metrics during robotic assisted partial nephrectomy.Automated performance metrics (instrument motion tracking/system events) and synchronized surgical videos from da Vinci® Si systems during robotic assisted partial nephrectomy were recorded using a system data recorder. Each case was segmented into 7 steps: colon mobilization, ureteral identification/dissection, hilar dissection, exposure of tumor within Gerota's fascia, intraoperative ultrasound/tumor scoring, tumor excision, and renorrhaphy. Automated performance metrics from each step were compared between expert (≥150 cases) and trainee (150 cases) surgeons by Mann-Whitney U test (continuous variables) and Pearson's chi-squared test (categorical variables). Clinical outcomes were collected prospectively and correlated to automated performance metrics and R.E.N.A.L. (radius, exophytic/endophytic, nearness of tumor to collecting system, anterior/posterior, location relative to polar line) nephrometry score by Spearman's correlation coefficients (r).A total of 50 robotic assisted partial nephrectomy cases were included for analysis, performed by 7 expert and 10 trainee surgeons. Automated performance metric profiles significantly differed between experts and novices in the initial 5 steps (p0.05). Specifically, experts exhibited faster dominant instrument movement and greater dominant instrument usage (bimanual dexterity) than trainees in select steps (p ≤0.045). Automated performance metrics during tumor excision and renorrhaphy were significantly correlated with R.E.N.A.L. score (r ≥0.364; p ≤0.041). These included metrics related to instrument efficiency, task duration, and dominant instrument use.Experts are more efficient and directed in their movement during robotic assisted partial nephrectomy. Automated performance metrics during key steps correlate with objective measures of tumor complexity and may serve as predictors of clinical outcomes. These data help establish a standardized metric for surgeon assessment and training during robotic assisted partial nephrectomy.

Deconstruction of robotic surgical gestures into semantic vocabulary yields an effective tool for surgical education. In this study we disassembled tissue dissection into basic gestures, created a classification system, and showed its ability to distinguish between experts and novices.Videos of renal hilum preparation during robotic assisted partial nephrectomies were manually reviewed to identify all discrete surgical movements. Identified dissection movements were classified into distinct gestures based on the consensus of 6 expert surgeons. This classification system was then employed to compare expert and novice dissection patterns during the renal hilum preparation.A total of 40 robotic renal hilum preparation videos were reviewed, representing 16 from 6 expert surgeons (100 or more robotic cases) and 24 from 13 novice surgeons (fewer than 100 robotic cases). Overall 9,819 surgical movements were identified, including 5,667 dissection movements and 4,152 supporting movements. Nine distinct dissection gestures were identified and classified into the 3 categories of single blunt dissection (spread, peel/push, hook), single sharp dissection (cold cut, hot cut and burn dissect) and combination gestures (pedicalize, 2-hand spread, and coagulate then cut). Experts completed 5 of 9 dissection gestures more efficiently than novices (p ≤0.033). In consideration of specific anatomical locations, experts used more peel/push and less hot cut while dissecting the renal vein (p0.001), and used more pedicalize while dissecting the renal artery (p0.001).Using this novel dissection gesture classification system, key differences in dissection patterns can be found between experts/novices. This comprehensive classification of dissection gestures may be broadly applied to streamline surgical education.

Our previous work classified a taxonomy of suturing gestures during a vesicourethral anastomosis of robotic radical prostatectomy in association with tissue tears and patient outcomes. Herein, we train deep-learning based computer vision (CV) to automate the identification and classification of suturing gestures for needle driving attempts. Using two independent raters, we manually annotated live suturing video clips to label timepoints and gestures. Identification (2395 videos) and classification (511 videos) datasets were compiled to train CV models to produce two- and five-class label predictions, respectively. Networks were trained on inputs of raw RGB pixels as well as optical flow for each frame. Each model was trained on 80/20 train/test splits. In this study, all models were able to reliably predict either the presence of a gesture (identification, AUC: 0.88) as well as the type of gesture (classification, AUC: 0.87) at significantly above chance levels. For both gesture identification and classification datasets, we observed no effect of recurrent classification model choice (LSTM vs. convLSTM) on performance. Our results demonstrate CV's ability to recognize features that not only can identify the action of suturing but also distinguish between different classifications of suturing gestures. This demonstrates the potential to utilize deep learning CV towards future automation of surgical skill assessment., 5 figures, 2 tables

INTRODUCTION AND OBJECTIVE:The operating room (OR) is a costly area of hospital operations and the surgeon is often identified as a major source of variability affecting OR efficiency. Surgeons aim...

INTRODUCTION AND OBJECTIVE:Previously, our group demonstrated with machine learning that automated performance metrics (APMs; kinematic and events data) during robot-assisted radical prostatectomy ...

Objective To investigate the applications of artificial intelligence (AI) in diagnosis, treatment and outcome predictionin urologic diseases and evaluate its advantages over traditional models and methods. Materials and methods A literature search was performed after PROSPERO registration (CRD42018103701) and in compliance with Preferred Reported Items for Systematic Reviews and Meta-Analyses (PRISMA) methods. Articles between 1994 and 2018 using the search terms "urology", "artificial intelligence", "machine learning" were included and categorized by the application of AI in urology. Review articles, editorial comments, articles with no full-text access, and nonurologic studies were excluded. Results Initial search yielded 231 articles, but after excluding duplicates and following full-text review and examination of article references, only 111 articles were included in the final analysis. AI applications in urology include: utilizing radiomic imaging or ultrasonic echo data to improve or automate cancer detection or outcome prediction, utilizing digitized tissue specimen images to automate detection of cancer on pathology slides, and combining patient clinical data, biomarkers, or gene expression to assist disease diagnosis or outcome prediction. Some studies employed AI to plan brachytherapy and radiation treatments while others used video based or robotic automated performance metrics to objectively evaluate surgical skill. Compared to conventional statistical analysis, 71.8% of studies concluded that AI is superior in diagnosis and outcome prediction. Conclusion AI has been widely adopted in urology. Compared to conventional statistics AI approaches are more accurate in prediction and more explorative for analyzing large data cohorts. With an increasing library of patient data accessible to clinicians, AI may help facilitate evidence-based and individualized patient care.

PURPOSE: In this study, we investigate the ability of automated performance metrics (APMs) and task-evoked pupillary response (TEPR), as objective measures of surgeon performance, to distinguish varying levels of surgeon expertise during generic robotic surgical tasks. Additionally, we evaluate the association between APMs and TEPR. METHODS: Participants completed ten tasks on a da Vinci Xi Surgical System (Intuitive Surgical, Inc.), each representing a surgical skill type: EndoWrist ® manipulation, needle targeting, suturing/knot tying, and excision/dissection. APMs (instrument motion tracking, EndoWrist® articulation, and system events data) and TEPR were recorded by a systems data recorder (Intuitive Surgical, Inc.) and Tobii Pro Glasses 2 (Tobii Technologies, Inc.) respectively. The Kruskal-Wallis test determined significant differences between groups of varying expertise. Spearman’s rank correlation coefficient measured associations between APMs and TEPR. RESULTS: Twenty-six participants were stratified by robotic surgical experience: novice (no prior experience; n=9), intermediate (0.66, p

INTRODUCTION AND OBJECTIVES:We previously created a robotic vesicourethral anastomosis (VUA) tutorial with standardized hand gestures for each stitch based on the most efficient automated performan...

INTRODUCTION AND OBJECTIVES:Urinary continence is an important outcome after a robot-assisted radical prostatectomy (RARP). Survival analysis (time-to-event analysis) models predicting time to urin...

INTRODUCTION AND OBJECTIVES:Automated performance metrics (APMs), validated measures of surgical performance, have not yet been applied to deep learning survival analyses (time-to-event analyses). ...