Evaluating the Effectiveness of an Online Journal Club: Experience from the International Urology Journal Club.

Background: The International Urology Journal Club attracts global participation; however, its effectiveness as an educational resource is undetermined.
Objective: To evaluate the effectiveness of the international Twitter-based urology journal club (#urojc) using Kirkpatrick's evaluation model and to quantify the development of #urojc into a robust educational platform.
Design, Setting, and Participants: A 13-item survey was sent to 3127 followers of #urojc. Twitter metrics including number of users, tweets, and impressions during 24 mo of operation were obtained via a third-party service "Symplur." Tweets were qualitatively analyzed to assess for relevance. Kirkpatrick's four-level evaluation model (reaction, learning, behavior, and results) was applied to analyze impact and effectiveness.
Results and Limitations: Survey participation rate was 48% (after accounting for spam and inactive accounts). The reaction (Kirkpatrick level 1) to #urojc was overwhelmingly positive. Consultant urologists were the most active participants while trainees reported highest educational value. As much as 45% of junior consultant urologists and trainees reported a change in practice as a result of #urojc participation. The number of unique users of #urojc almost doubled from yr 1 to 2. The number of tweets per month and author participation increased from yr 1 to 2. The proportion of content-relevant tweets remained stably high at approximately 60%. This study was unable to document any impact on knowledge, behavior, or outcomes of #urojc (Kirkpatrick levels 2-4). Silent observers who do not follow #urojc were not captured.
Conclusions: To our knowledge, this is the first time Kirkpatrick's evaluation model has been applied to a journal club. The reaction to #urojc among survey responders is positive with change in practice also reported. It appears that #urojc achieves level 1 of Kirkpatrick's evaluation model. Further investigation is required for levels 2-4. Twitter-based journal clubs draw robust global participation and have inspired journal clubs in other specialties.
Patient Summary: This report highlights the importance of the International Urology Journal Club as a potential educational resource of consultant urologists and trainees alike. Global participation was a major advantage of this platform. This journal club has inspired other medical specialties to follow suit.
(Copyright © 2019 European Association of Urology. All rights reserved.)