Does the journal impact factor reflect the impact of German medical guideline contributions?

Contributions to medical guidelines, so far, have hardly been considered in the measurement and evaluation of research performance in Germany. We therefore examined 70 high-quality medical guidelines from the years 2017 and 2018 and investigated the type of publications cited by the guidelines, whether the citation rates of articles differ between substantiating guideline recommendations and background text, and whether or not the Journal Impact Factor is correlated with the guidelines’ citation frequency of individual journals. Our study found that the guidelines cited original articles much more than books, reviews, or other guidelines. Slightly less than 10% of the citations came from the 2 years preceding guideline publication, and more than 50% of the references were at least 8 years old. A subsample showed that articles which only provided background information were cited less frequently outside the guidelines than those that substantiated a specific recommendation. Lastly, there was only a weak correlation (0.1 ≤ Tau ≤ 0.35) between the citations counts of individual journals in the guidelines and their respective JIFs despite guideline subject. Our study suggests that the JIF is not an appropriate tool to assess the clinical relevance of medical research.