Assessment of Adverse Events in Protocols, Clinical Study Reports, and Published Papers of Trials of Orlistat: A Document Analysis

Background Little is known about how adverse events are summarised and reported in trials, as detailed information is usually considered confidential. We have acquired clinical study reports (CSRs) from the European Medicines Agency through the Freedom of Information Act. The CSRs describe the results of studies conducted as part of the application for marketing authorisation for the slimming pill orlistat. The purpose of this study was to study how adverse events were summarised and reported in study protocols, CSRs, and published papers of orlistat trials. Methods and Findings We received the CSRs from seven randomised placebo controlled orlistat trials (4,225 participants) submitted by Roche. The CSRs consisted of 8,716 pages and included protocols. Two researchers independently extracted data on adverse events from protocols and CSRs. Corresponding published papers were identified on PubMed and adverse event data were extracted from this source as well. All three sources were compared. Individual adverse events from one trial were summed and compared to the totals in the summary report. None of the protocols or CSRs contained instructions for investigators on how to question participants about adverse events. In CSRs, gastrointestinal adverse events were only coded if the participant reported that they were “bothersome,” a condition that was not specified in the protocol for two of the trials. Serious adverse events were assessed for relationship to the drug by the sponsor, and all adverse events were coded by the sponsor using a glossary that could be updated by the sponsor. The criteria for withdrawal due to adverse events were in one case related to efficacy (high fasting glucose led to withdrawal), which meant that one trial had more withdrawals due to adverse events in the placebo group. Finally, only between 3% and 33% of the total number of investigator-reported adverse events from the trials were reported in the publications because of post hoc filters, though six of seven papers stated that “all adverse events were recorded.” For one trial, we identified an additional 1,318 adverse events that were not listed or mentioned in the CSR itself but could be identified through manually counting individual adverse events reported in an appendix. We discovered that the majority of patients had multiple episodes of the same adverse event that were only counted once, though this was not described in the CSRs. We also discovered that participants treated with orlistat experienced twice as many days with adverse events as participants treated with placebo (22.7 d versus 14.9 d, p-value < 0.0001, Student’s t test). Furthermore, compared with the placebo group, adverse events in the orlistat group were more severe. None of this was stated in the CSR or in the published paper. Our analysis was restricted to one drug tested in the mid-1990s; our results might therefore not be applicable for newer drugs. Conclusions In the orlistat trials, we identified important disparities in the reporting of adverse events between protocols, clinical study reports, and published papers. Reports of these trials seemed to have systematically understated adverse events. Based on these findings, systematic reviews of drugs might be improved by including protocols and CSRs in addition to published articles.
In a document analysis, Jeppe B. Schroll and colleagues seek insights into potential bias and underreporting in documentation of adverse events from the mid-1990s clinical trials of the anti-obesity drug orlistat.
Author Summary Why Was This Study Done? Most drugs have adverse effects, or harms, that may become evident in clinical trials. Pharmaceutical companies seeking to market a new drug must report adverse effects observed in trial participants in the Clinical Study Reports (CSRs), which they provide to regulatory authorities. Additionally, investigators may report harms in published reports of their trials. We sought to understand the accuracy, and potential bias, in harms reporting for trials of orlistat, a slimming drug from Roche approved in Europe in 1998 and still marketed in Europe today. What Did The Researchers Do And Find? Using a Freedom of Information Act request to the European Medicines Agency (EMA), we obtained CSRs describing seven clinical trials of orlistat. We studied protocol instructions to investigators for reporting harms, the actual reporting of harms in individual CSR records versus summaries, and the final reporting of harms in published papers. We found that protocol instructions to trial investigators had the potential to dilute the appearance of drug-associated harms. Between 3% and 33% of the total adverse effects from CSR summaries were described in published papers. In one trial, we counted adverse events individually and found that both the number of adverse effects and the number of days with adverse effects in participants taking the drug were understated in the corresponding publication. What Do These Findings Mean? The reporting of trials of orlistat in the 1990s understated harms in the summarised results submitted to the EMA for drug approval and in the published papers. Based on the characteristics of harms observed and reported in these trials, we suggest that reports of harms include duration of adverse effects. We also suggest that systematic reviews of drugs might be improved by including protocols and CSRs in addition to published articles.