Value of an objective assessment tool in the operating room

Background: Concerns about the achievement of surgical proficiency during residency are increasing. To objectify surgical skills, the Objective Structured Assessment of Technical Skills (OSATS) was developed and proven valid, feasible and reliable for use in laboratory settings. This study aimed to evaluate the value of this tool for intraoperative use. Methods: Residents were assessed with an OSATS after every procedure they performed as the primary surgeon during a 3-month clinical rotation in gynecological surgery. We mapped individual learning curves (OSATS scores plotted against experience) and established the average procedure-specific learning curve. We used linear mixed models to assess the relation between performance and experience. Results: Nine residents were recruited and 319 OSATS analyzed. Individual learning curves revealed progression beyond 24 of 30 OSATS points for 7 residents. Performance on the average procedure improved with experience, and the OSATS score increased by an average of 1.10 points per assessed procedure (p = 0.008, 95%confidence interval 0.44-1.77). Median OSATS scores ranged from 18 to 30 among the 21 assessors. Conclusion: Intraoperative implementation of OSATS seems to offer important advantages: structured feedback is facilitated, and learning curves enable insight into individual progression. However, doubts have been raised about the objectivity of the tool. Therefore, caution is warranted in using it for graduation and certification. Contexte : La maitrise de la chirurgie acquise au cours de la residence preoccupe de plus en plus. Pour rendre les techniques chirurgicales objectives, on a mis au point revaluation structuree objective des habiletes techniques (Objective Structured Assessment of Technical Skills; OSATS) et on en a demontre la viabilite, la faisabilite et la fiabilite pour utilisation en laboratoire. Cette etude vise a determiner la valeur de cet outil au cours des interventions. Methodes : Les medecins residents ont ete soumis a une evaluation OSATS apres chaque intervention qu'ils ont pratiquee comme chirurgien principal au cours d'un stage clinique de 3 mois en chirurgie gynecologique. Nous avons etabli les courbes d'apprentissage individuelles (courbes des resultats OSATS en fonction de l'experience), ainsi que la courbe d'apprentissage moyenne particuliere a l'intervention. Nous avons utilise des modeles mixtes lineaires pour evaluer le lien entre le rendement et l'experience. Resultats : Nous avons recrute 9 medecins residents et analyse 319 evaluations OSATS. Les courbes d'apprentissage individuelles ont revele un progres superieur a 24 points OSATS sur 30 chez 7 medecins residents. L'execution de l'intervention moyenne s'est amelioree avec l'experience et le resultat de l'evaluation OSATS a augmente en moyenne de 1,10 point par intervention evaluee (p = 0,008, intervalle de confiance a 95 %, 0,44-1,77). Les resultats medians de l'evaluation OSATS ont varie de 18 a 30 chez les 21 evaluateurs. Conclusion : L'application de devaluation OSATS aux interventions semble offrir des avantages importants : les commentaires structures sont facilites et les courbes d'apprentissage donnent une idee de l'evolution de chaque medecin. On a toutefois souleve des doutes quant a l'objectivite de l'outil. La prudence s'impose donc dans l'utilisation de celui-ci aux fins d'obtention du diplome et du certificat.
Nowadays, it is becoming more and more difficult to achieve surgical proficiency. Residents experience less training owing to reduced working hours and a decreased surgical caseload. (1) Additionally, with the [...]