Search

Abstract Accurate, unbiased, and reproducible assessment of skill is a vital resource for surgeons throughout their career. The objective in this research is to develop and validate algorithms for video-based assessment of intraoperative surgical skill. Algorithms to classify surgical video into expert or novice categories provide a summative assessment of skill, which is useful for evaluating surgeons at discrete time points in their training or certification of surgeons. Using a spatial-temporal neural network architecture, we tested the hypothesis that explicit supervision of spatial attention supervised by instrument tip locations improves the algorithm’s generalizability to unseen dataset. The best performing model had an area under the receiver operating characteristic curve (AUC) of 0.88. Augmenting the network with supervision of spatial attention improved specificity of its predictions (with small changes in sensitivity and AUC) and led to improved measures of discrimination when tested with unseen dataset. Our findings show that explicit supervision of attention learned from images using instrument tip locations can improve performance of algorithms for objective video-based assessment of surgical skill.

BACKGROUND:Surgical tasks are performed in a sequence of steps, and technical skill evaluation includes assessing task flow efficiency. Our objective was to describe differences in task flow for expert and novice surgeons for a basic surgical task. METHODS:We used a hierarchical semantic vocabulary to decompose and annotate maneuvers and gestures for 135 instances of a surgeon's knot performed by 18 surgeons. We compared counts of maneuvers and gestures, and analyzed task flow by skill level. RESULTS:Experts used fewer gestures to perform the task (26.29; 95% CI = 25.21 to 27.38 for experts vs. 31.30; 95% CI = 29.05 to 33.55 for novices) and made fewer errors in gestures than novices (1.00; 95% CI = 0.61 to 1.39 vs. 2.84; 95% CI = 2.3 to 3.37). Transitions among maneuvers, and among gestures within each maneuver for expert trials were more predictable than novice trials. CONCLUSIONS:Activity segments and state flow transitions within a basic surgical task differ by surgical skill level, and can be used to provide targeted feedback to surgical trainees.

OBJECTIVE: To describe the sources of internal company documents used in public health and healthcare research. METHODS: We searched PubMed and Embase for articles using internal company documents to address a research question about a health-related topic. Our primary interest was where authors obtained internal company documents for their research. We also extracted information on type of company, type of research question, type of internal documents, and funding source. RESULTS: Our searches identified 9,305 citations of which 357 were eligible. Scanning of reference lists and consultation with colleagues identified 4 additional articles, resulting in 361 included articles. Most articles examined internal tobacco company documents (325/361; 90%). Articles using documents from pharmaceutical companies (20/361; 6%) were the next most common. Tobacco articles used documents from repositories; pharmaceutical documents were from a range of sources. Most included articles relied upon internal company documents obtained through litigation (350/361; 97%). The research questions posed were primarily about company strategies to promote or position the company and its products (326/361; 90%). Most articles (346/361; 96%) used information from miscellaneous documents such as memos or letters, or from unspecified types of documents. When explicit information about study funding was provided (290/361 articles), the most common source was the US-based National Cancer Institute. We developed an alternative and more sensitive search targeted at identifying additional research articles using internal pharmaceutical company documents, but the search retrieved an impractical number of citations for review. CONCLUSIONS: Internal company documents provide an excellent source of information on health topics (e.g., corporate behavior, study data) exemplified by articles based on tobacco industry documents. Pharmaceutical and other industry documents appear to have been less used for research, indicating a need for funding for this type of research and well-indexed and curated repositories to provide researchers with ready access to the documents.

BACKGROUND: Details about the type of analysis (e.g., intent to treat [ITT]) and definitions (i.e., criteria for including participants in the analysis) are necessary for interpreting a clinical trial's findings. Our objective was to compare the description of types of analyses and criteria for including participants in the publication (i.e., what was reported) with descriptions in the corresponding internal company documents (i.e., what was planned and what was done). Trials were for off-label uses of gabapentin sponsored by Pfizer and Parke-Davis, and documents were obtained through litigation. METHODS AND FINDINGS: For each trial, we compared internal company documents (protocols, statistical analysis plans, and research reports, all unpublished), with publications. One author extracted data and another verified, with a third person verifying discordant items and a sample of the rest. Extracted data included the number of participants randomized and analyzed for efficacy, and types of analyses for efficacy and safety and their definitions (i.e., criteria for including participants in each type of analysis). We identified 21 trials, 11 of which were published randomized controlled trials, and that provided the documents needed for planned comparisons. For three trials, there was disagreement on the number of randomized participants between the research report and publication. Seven types of efficacy analyses were described in the protocols, statistical analysis plans, and publications, including ITT and six others. The protocol or publication described ITT using six different definitions, resulting in frequent disagreements between the two documents (i.e., different numbers of participants were included in the analyses). CONCLUSIONS: Descriptions of analyses conducted did not agree between internal company documents and what was publicly reported. Internal company documents provide extensive documentation of methods planned and used, and trial findings, and should be publicly accessible. Reporting standards for randomized controlled trials should recommend transparent descriptions and definitions of analyses performed and which study participants are excluded.

Abstract This study was designed to determine the effect of a novel simulation-based training curriculum for scleral tunnel construction in manual small incision cataract surgery (MSICS) compared with traditional training. In this multicenter, investigator-masked, randomized clinical trial, resident surgeons within 3 months of matriculation with minimal or no prior experience with MSICS were assigned either to simulation-based training, the Experimental Group (EG), or to conventional training, the Control Group (CG). EG residents were trained to perform scleral tunnel construction using a simulation-based curriculum (HelpMeSee Eye Surgery Simulator), while residents in the CG followed institution-specific curriculum before progressing to live surgery. Surgical videos of the first 20 attempts at tunnel construction were reviewed by masked video raters. The primary outcome was the total number of any of 9 pre-specified errors. On average, the total number of errors was 9.25 (95% CI 0–18.95) in the EG and 17.56 (95% CI 6.63–28.49) in the CG (P = 0.05); the number of major errors was 4.86 (95% CI 0.13–9.59) in the EG and 10.09 (95% CI 4.76–15.41) in the CG (P = 0.02); and the number of minor errors was 4.39 (95% CI 0–9.75) in the EG and 7.47 (95% CI 1.43–13.51) in the CG (P = 0.16). These results support that novice surgeons trained using the novel simulation-based curriculum performed fewer errors in their first 20 attempts at tunnel construction compared to those trained with a conventional curriculum.

OBJECTIVES/GOALS: Surgical training is constrained by duty hour limits, bias, and a trial-and-error learning process. Surgeon skill variation is a healthcare system disparity that can impact patient outcomes. Incorporating validated, standardized assessment tools and machine learning (ML) algorithms may help to standardize and reduce bias in surgeon education. METHODS/STUDY POPULATION: To support assessment tool and ML algorithm development, we are curating an annotated video registry of neurosurgical procedures. Point-of-view video of resident and attending neurosurgeons performing craniotomies is recorded via an eye-tracking headset. A Delphi panel of neurosurgeons will review the video and determine which represent expert versus trainee performance. Neurosurgery attendings will be interviewed to provide descriptions of craniotomies which will be used to develop an assessment rubric. A Delphi panel will determine what rubric components should be maintained. New craniotomy videos will be viewed by attendings in a blinded fashion while completing the assessment rubric. An online feedback platform is being developed allowing residents to prospectively track assessment data. RESULTS/ANTICIPATED RESULTS: We anticipate development of an annotated, institutional video database featuring craniotomies performed by residents and attending neurosurgeons. Using a Delphi approach, we anticipate achieving consensus on which videos reflect expert versus trainee performance. We anticipate development of a novel craniotomy assessment rubric that is both valid and reliable. Our online feedback platform will allow prospective tracking of assessment data from multiple sources and enhanced transparency in the feedback process. The video registry and assessment data will enable development of novel ML algorithms able to recognize craniotomy segments and estimate operator skill. DISCUSSION/SIGNIFICANCE: Building a video registry of procedures, validated assessment tools, and a prototype feedback platform enables a pipeline for ML algorithm development. Together these tools will help to standardize and optimize resident education translating to earlier operative independence, improved patient safety, and reduced bias during surgical training.

To be useful, clinical prediction models (CPMs) must be generalizable to patients in new settings. Evaluating generalizability of CPMs helps identify spurious relationships in data, provides insights on when they fail, and thus, improves the explainability of the CPMs. There are discontinuities in concepts related to generalizability of CPMs in the clinical research and machine learning domains. Specifically, conventional statistical reasons to explain poor generalizability such as inadequate model development for the purposes of generalizability, differences in coding of predictors and outcome between development and external datasets, measurement error, inability to measure some predictors, and missing data, all have differing and often complementary treatments, in the two domains. Much of the current machine learning literature on generalizability of CPMs is in terms of dataset shift of which several types have been described. However, little research exists to synthesize concepts in the two domains. Bridging this conceptual discontinuity in the context of CPMs can facilitate systematic development of CPMs and evaluation of their sensitivity to factors that affect generalizability. We survey generalizability and dataset shift in CPMs from both the clinical research and machine learning perspectives, and describe a unifying framework to analyze generalizability of CPMs and to explain their sensitivity to factors affecting it. Our framework leads to a set of signaling statements that can be used to characterize differences between datasets in terms of factors that affect generalizability of the CPMs.