Industry sponsorship bias in research findings: a network meta-analysis of LDL cholesterol reduction in randomised trials of statins
BMJ 2014; 349 doi: https://doi.org/10.1136/bmj.g5741 (Published 03 October 2014) Cite this as: BMJ 2014;349:g5741
- Huseyin Naci, research fellow; fellow in pharmaceutical policy research12,
- Sofia Dias, research fellow3,
- A E Ades, professor3
- 1LSE Health, London School of Economics and Political Science, London, UK
- 2Drug Policy Research Group, Department of Population Medicine, Harvard Medical School, Boston MA, USA
- 3School of Social and Community Medicine, University of Bristol, Bristol, UK
- Correspondence to: H Naci h.naci{at}lse.ac.uk
- Accepted 26 August 2014
Abstract
Objective To explore the risk of industry sponsorship bias in a systematically identified set of placebo controlled and active comparator trials of statins.
Design Systematic review and network meta-analysis.
Eligibility Open label and double blind randomised controlled trials comparing one statin with another at any dose or with control (placebo, diet, or usual care) for adults with, or at risk of developing, cardiovascular disease. Only trials that lasted longer than four weeks with more than 50 participants per trial arm were included. Two investigators assessed study eligibility.
Data sources Bibliographic databases and reference lists of relevant articles published between 1 January 1985 and 10 March 2013.
Data extraction One investigator extracted data and another confirmed accuracy.
Main outcome measure Mean absolute change from baseline concentration of low density lipoprotein (LDL) cholesterol.
Data synthesis Study level outcomes from randomised trials were combined using random effects network meta-analyses.
Results We included 183 randomised controlled trials of statins, 103 of which were two-armed or multi-armed active comparator trials. When all of the existing randomised evidence was synthesised in network meta-analyses, there were clear differences in the LDL cholesterol lowering effects of individual statins at different doses. In general, higher doses resulted in higher reductions in baseline LDL cholesterol levels. Of a total of 146 industry sponsored trials, 64 were placebo controlled (43.8%). The corresponding number for the non-industry sponsored trials was 16 (43.2%). Of the 35 unique comparisons available in 37 non-industry sponsored trials, 31 were also available in industry sponsored trials. There were no systematic differences in magnitude between the LDL cholesterol lowering effects of individual statins observed in industry sponsored versus non-industry sponsored trials. In industry sponsored trials, the mean change from baseline LDL cholesterol level was on average 1.77 mg/dL (95% credible interval −11.12 to 7.66) lower than the change observed in non-industry sponsored trials. There was no detectable inconsistency in the evidence network.
Conclusions Our analysis shows that the findings obtained from industry sponsored statin trials seem similar in magnitude as those in non-industry sources. There are actual differences in the effectiveness of individual statins at various doses that explain previously observed discrepancies between industry and non-industry sponsored trials.
Introduction
There is concern that the vast majority of published medical research findings may be false, irrespective of study design.1 Although randomised trials are typically placed at the top of evidence hierarchies,2 causal inferences from such trials can be jeopardised by limitations in methodological quality in their design, conduct, analysis, and reporting, leading to underestimation or overestimation of the true intervention effect, that is, bias.3 4 5 6 7 An important source of potential bias relates to the influence of pharmaceutical industry sponsorship on the trial findings. Previous reviews showed that industry sponsored research is more likely to favour the product developed by the company sponsoring the research than research funded by other sources.8 9 10
Statins, a lipid lowering class of drugs, are an interesting case study for exploring industry sponsorship bias. They are among the most widely prescribed classes of drugs due to their benefits in prolonging survival by reducing the risk of coronary and cerebrovascular events. Clinical practice guidelines in the United States and United Kingdom have considerably lowered the threshold for receiving statins, potentially increasing the number of patients eligible to receive the drugs by millions.11 12 13 Not all statins are the same: recent comparative analyses synthesising the placebo controlled and active comparator randomised trials of individual statins established some statin dose combinations as more efficacious and tolerable than others.14 15 16 17 If present, industry sponsorship bias may render the findings of such comparative assessments invalid.18 Indeed, head-to-head comparisons of statins were shown to be more likely to report results and conclusions favouring the sponsor’s product compared with the comparator drug, suggesting the presence of industry sponsorship bias.19
There are three potential explanations for the observed relationship between industry sponsorship and trial outcomes. First, pharmaceutical companies may introduce bias into individual randomised controlled trials. Second, pharmaceutical companies may ask the “right” questions and preferentially sponsor or report trials on a drug that they expect to be superior to its alternatives.8 This phenomenon, commonly termed design bias, refers to planning or reporting trials in such a way that their findings likely favour the sponsored product.20 Third, it is possible that industry sponsored trials are unbiased: the effectiveness of different drugs at different doses may indeed vary, making it difficult to distinguish between a genuine industry bias and actual differences in treatment effectiveness.
Empirical evidence for an industry sponsorship bias would show that industry sponsored trials produce systematically different findings compared with those obtained from identical (or at least comparable) trials funded by non-industry sources.21 However, such comparability has not been established: the existing body of literature did not take into account the important differences in drug interventions, patient populations, and dosing regimens across industry and non-industry sponsored studies. In addition, the analyses demonstrated a statistical association between industry sponsorship and trial conclusions, rather than differences in the magnitude of effect sizes between industry sponsored trials and non-industry sponsored trials.22 Currently, it is not clear by which mechanism industry sponsorship bias would operate, and how it would influence a complex evidence network including a large number of placebo controlled and active comparator trials.
By accounting for dose and treatment differences across trials, network meta-analysis provides a methodological framework for evaluating industry sponsorship bias.23 Using statins as an example, the objective of our study was to explore the methodological quality and risk of industry sponsorship bias in a systematically identified set of placebo controlled and active comparator trials of statins.
Methods
Study identification and selection
We searched Medline, Embase, Cochrane Database of Systematic Reviews, and Cochrane Central Register of Controlled Trials between 1 January 1985 and 10 March 2013 using the search terms atorvastatin, fluvastatin, simvastatin, lovastatin, pravastatin, rosuvastatin, cholesterol, cardiovascular disease, and hydroxymethylglutaryl-CoA reductase inhibitors/therapeutic use. We supplemented electronic database searches by manually cross checking reference lists from published trials and review articles. Two researchers independently performed abstract, title, and full text screening. A third researcher approved study selection.
As described previously,14 15 16 17 we included open label and double blind randomised controlled trials comparing one statin with another at any dose or with control (placebo, diet, or usual care) for adults with, or at risk of developing, cardiovascular disease. We included both fixed dose and titration trials of atorvastatin, fluvastatin, lovastatin, pravastatin, rosuvastatin, and simvastatin if they had more than 50 participants per trial arm, lasted longer than four weeks, and reported changes from baseline low density lipoprotein (LDL) cholesterol concentration. In accordance with predefined criteria,24 we excluded trials conducted in patients with renal insufficiency.
The primary outcome of interest was the difference in mean change from baseline in serum LDL cholesterol concentration between two comparator treatments for a given dose (change from baseline in the treatment group minus that in the control group). We extracted information on trial and patient population characteristics, and the outcome of interest. We extracted the total number of randomly assigned participants unless only a subsample of patients were eligible for statin therapy; this was the case in some of the large trials, which measured LDL cholesterol levels only in a subsample of the original trial population. One researcher extracted data and another independently checked for accuracy.
Categorisation of potential bias
We explored two sources of bias. First was quality related bias, which we defined as the potential underestimation or overestimation of the true intervention effect due to flaws in the methodological attributes of randomised controlled trials.3 Second, we explored industry sponsorship bias, which could occur when a pharmaceutical sponsor favoured its own drug in placebo controlled or active comparator trials or the highest dose of its own drug in dose comparator trials.23
To evaluate quality related bias, we extracted data on the methodological quality of included randomised trials at the study level.3 Our assessment of methodological quality included published trial protocols when such documents were cited in the original trial publications. Using the risk of bias tool developed by the Cochrane Collaboration,25 we collected information on blinding (that is, did the investigators blind trial participants and researchers from knowledge of which treatment a trial participant received?); random sequence generation (that is, were the methods for allocation sequence reported to determine whether it produced comparable groups?); allocation concealment (that is, were the methods used to conceal the allocation sequence reported to determine whether group allocations could have been foreseen before or during randomisation?); blinding of outcome assessment (that is, did trial investigators blind outcome assessment from knowledge of which intervention a participant received?); indications of incomplete outcome data (that is, did the investigators report completeness of outcome data for LDL cholesterol lowering, including attrition and exclusions from the analysis?); and indications of selective reporting (that is, did the investigators fail to report tolerability and harm outcomes commonly reported in randomised trials of statins (such as withdrawals due to adverse events, creatine kinase elevations, hepatic transaminase elevations, or myalgia), and were there deviations in trial outcomes from published protocols?). On each methodological attribute, we gave studies a rating of high, low, or uncertain quality. For each trial, we assigned one point for each item with “high quality” to calculate the overall methodological quality score, ranging from 0 (worst methodological quality) to 6 (best methodological quality). Using this information, we evaluated the distribution of methodological quality of different comparisons across the evidence network (for example, explored whether trials of simvastatin v control had on average higher or lower methodological quality than those of atorvastatin v rosuvastatin).
We then extracted information on the funding sources of included trials. Two potential funding sources were industry and non-industry (such as government, academic institutions, other not for profit research organisations). Industry sources included any private, for profit, pharmaceutical company involved in research and development, manufacturing, or marketing of statins. Most of the included trial publications contained a statement delineating the funding sources; in cases where sponsorship information was not clearly documented, we checked the author affiliations and categorised the studies with industry affiliated authors as industry sponsored. Trials with industry, government, and/or academic institution co-sponsorship were categorised as industry sponsored (unless the trial investigators included a statement that the funding body had no involvement in trial design, conduct, analysis, or reporting).
To explore the presence of design bias, we compared the percentage of industry sponsored trials with placebo controls with the corresponding percentage in non-industry sponsored trials. In addition, we compared the type of drug and dose comparisons between industry and non-industry sponsored trials. We considered industry sponsored trials where investigators included a statement that the funding body had no involvement in trial design, conduct, analysis, or reporting as industry sponsored for the evaluation of design bias.
Finally, we identified which statin (at a specific dose) was favoured in a given trial. For the exploration of industry sponsorship bias, we had two a priori hypotheses (scenarios 1 and 2). Our first hypothesis (scenario 1) was that a pharmaceutical sponsor would favour its own drug in placebo controlled or active comparator trials of statins or the highest dose of its own drug.26 Accordingly, we labelled one arm per trial as potentially biased in the industry sponsored trials of statins. In scenario 2, we hypothesised that a sponsor would favour its own drug in placebo controlled or active comparator trials of statins, but that it would not differentiate between different doses of its own drug in dose comparator trials. Accordingly, we labelled one arm per trial as potentially biased in the industry sponsored trials of statins unless a given trial compared the same statin at different doses.
Statistical analysis
We used a network meta-analytic approach to examine the methodological quality and risk of industry sponsorship bias in the randomised trials of statins. By taking into account actual differences in the effectiveness of individual statins at different doses, network meta-analysis provides an analytic framework for the exploration of bias.23 We evaluated both the methodological quality and risk of bias on the basis of the mean change from baseline in serum LDL cholesterol levels between two comparator treatments for a given dose (change from baseline in the treatment group minus that in the control group).
To determine the dose comparative effects of individual statins, we simultaneously combined the direct within-trial comparisons between two treatments (such as atorvastatin at 10 mg/day v control) with indirect comparisons from trials that had one treatment in common (such as atorvastatin ≤10 mg/day v control and simvastatin ≤10 mg/day v control).27 28 29 When combining direct and indirect evidence, the likelihood for each trial was based on the mean LDL cholesterol change from baseline in each treatment arm and its variance. The information on any multi-arm trial was used in the network meta-analysis just once, thereby correctly informing all treatment contrasts while also correctly taking into account the between-arm correlations in multi-arm trials.30 This analysis preserved the within-trial randomised treatment comparison of each trial while combining all available comparisons between treatments across trials. We combined study-level relative treatment effects using bayesian Markov chain Monte Carlo methods in WinBUGS version 1.4.3.31 We used the model developed by Dias and colleagues for the National Institute of Health and Care Excellence (NICE) Decision Support Unit,30 32 33 assuming that the mean change from baseline in LDL cholesterol per trial arm had a normal distribution. We used the identity function to link the relative effects across trials making different comparisons, while taking into account the correlations between treatment effects within multi-arm trials. All analyses were dose-specific and explored the effects of individual statins at different doses separately. Accordingly, we did not assume any trends and treatment effects at different doses were not “nested” within an overall treatment effect for each statin.
Our models adopted both fixed and random effects for dose comparative treatment effects.34 The fixed effect model assumed that there was no between-study heterogeneity. The random effects model took into account potential heterogeneity by assuming that the trial-specific treatment effects were drawn from a distribution, with a mean that was specific for each treatment, and a common variance shared by all treatments. Our bayesian models adopted non-informative prior distributions for treatment effects (normal (0, 10 000)) and the between-trial variance (uniform (0, 100)) in random effects analyses. We tested the sensitivity of our analyses to prior distributions by examining whether the effect sizes and credibility intervals changed after using more informative priors for the between-trial variance in random effects models (uniform (0, 10)).
To qualitatively evaluate the sensitivity of our base case analyses to study-level methodological quality attributes, we carried out six sets of separate network meta-analyses by excluding studies with low quality on the following methodological items: (1) blinding of patients and investigators; (2) random sequence generation; (3) allocation concealment; (4) blinding of outcome assessment; (5) indications of incomplete outcome data; and (6) indications of selective reporting. We subsequently plotted side by side the treatment effect sizes and 95% credible intervals obtained from each of the six separate analyses and visually inspected any potential discrepancies.
To evaluate potential industry sponsorship bias, we carried out meta-regression analyses.26 35 36 We performed all meta-regression analyses by allowing for a common treatment bias interaction for each statin dose compared with control.37 In addition to assuming that each trial estimated a study-specific bias, which came from a common bias distribution (that is, random effects model for bias), in a separate analysis we also assumed that all studies estimated the same bias parameter (that is, fixed effect model for bias).
To evaluate inconsistency between direct and indirect evidence, we first qualitatively evaluated the consistency of relative treatment effects obtained from an analysis of active comparator trials (that is, direct evidence) with those obtained from an analysis combining both placebo controlled and active comparator trials (that is, mixed evidence). We plotted and visually inspected the relative treatment effects obtained from two sets of analyses (that is, direct and mixed estimates) and found no discernible evidence of inconsistency. As we previously reported,16 there was also no evidence that the potential heterogeneity and inconsistency across the evidence base could be explained by baseline mean age, baseline mean LDL cholesterol concentration, or publication year.
Our analyses employed a long burn-in period (50 000 iterations) and follow-up period (80 000-100 000 iterations) to allow for convergence. We visually inspected trace plots for key parameters for each analysis to assess convergence in terms of stability. We formally evaluated the goodness of fit of our models using the total residual deviance (posterior mean of the deviance under a given model minus the deviance for the saturated model) along with the deviance information criterion (sum of the posterior mean of the residual deviance and the effective number of parameters).32 Because of better fit, we only reported the results obtained from the fixed effect model for bias and random effects model for treatment effects; the total residual deviance for the fixed effect model for bias and random effects model for treatment effects was 228.20 as compared with 229.20 for the random effects model for both bias and treatment effects.
Results
Characteristics of eligible trials
We included 183 randomised controlled trials of statins (fig 1⇓). Most of the eligible trials included active comparators: there were 80 two-armed placebo controlled trials as compared to 103 two-armed or multi-armed active comparator trials (see table 1 in web appendix). There were 53 trials conducted among individuals with established coronary heart disease; 41 trials investigated primary prevention (nine of which were among individuals with diabetes); 10 included patients with acute coronary syndrome; four included patients with metabolic syndrome; and three were among patients with heart failure. The remaining 72 trials included individuals with hypercholesterolaemia with or without established coronary heart disease.
Fig 1 Flow diagram of study identification and selection
Figure 2⇓ shows the network of eligible statin and dose comparisons for the base case network meta-analysis. Atorvastatin and rosuvastatin at various doses were compared most often (n=61), followed by pravastatin and placebo (n=35), simvastatin and placebo (n=18), and atorvastatin and placebo (n=24). There were no comparisons between fluvastatin and rosuvastatin or between lovastatin and rosuvastatin at any dose formulation.
Fig 2 Network of available comparisons. Size of node is proportional to number of trial participants, and thickness of line connecting nodes is proportional to number of participants randomised in trials directly comparing the two treatments
Methodological quality of included trials
Out of a total of 183 trials, 112 were double blind (rated as high quality) while 58 were open label and two were single blind (rated as low quality). Blinding was not clear for the remaining 11 trials. Fifty four of the included trials were rated as high quality in terms of random sequence generation; the corresponding numbers were 24 for allocation concealment; 70 for blinding of outcome assessment; 95 for incomplete outcome data; and 127 for selective reporting. A large number of trials were considered to have unclear methodological quality: 94 trials for random sequence generation; 77 for allocation concealment; 46 for blinding of outcome assessment; 48 for incomplete outcome data; and 11 for selective reporting.
On average, the included set of trials had between two and three methodological attributes with high quality (average methodological quality score across the included set of trials was 2.65). Fourteen trials did not have any methodological attributes with high quality; 38 trials had one; 37 had two, 38 had three; 30 had four; and 14 had five. Only 11 trials were rated as having high quality on all six methodological quality attributes.
Distribution of methodological quality in the evidence network
Figure 3⇓ shows the available comparisons between individual statins and control. The majority of comparisons had an average methodological score between two and three. On average, placebo controlled trials of rosuvastatin and simvastatin had higher methodological quality than other comparisons (average methodological quality score for both comparisons >3). Trials comparing atorvastatin and fluvastatin (two trials, average methodological quality score 1.50); atorvastatin and rosuvastatin (34 trials, score 1.97); atorvastatin and simvastatin (24 trials, score 1.79); fluvastatin and pravastatin (one trial, score 1.00); fluvastatin and simvastatin (two trials, score 1.30); lovastatin and simvastatin (three trials, score 1.66); and pravastatin and simvastatin (10 trials, score 1.80) were rated as having very low methodological quality.
Fig 3 Network of available comparisons between individual statins and control. Size of node is proportional to number of trial participants, and thickness of line connecting nodes is proportional to number of participants randomised in trials directly comparing the two treatments
Influence of methodological quality attributes on dose comparative effects
Figure 4⇓ shows the sensitivity of the dose comparative effects of individual statins to different trial-level methodological quality attributes. Overall, the results based on high quality trials seemed similar to all other trials. When the findings of six separate network meta-analyses were visually inspected side by side, the magnitude of the LDL cholesterol lowering effect of individual statins was qualitatively similar. The effect size tended to get smaller in sensitivity analyses that included only trials with high methodological quality attributes. In cases where there were few data points available (for example, fluvastatin at all dose formulations, lovastatin at ≤10 mg/day, and pravastatin at >40 mg/day), there was considerable uncertainty (wide 95% credible intervals) around the observed treatment effects in sensitivity analyses. In all analyses, 95% credible intervals greatly overlapped across base case and sensitivity analyses.
Fig 4 Sensitivity of network meta-analysis findings to methodological quality attributes of randomised controlled trials of statins. In addition to the findings of the base case analysis (blue circles), findings of separate sensitivity analyses are shown. These excluded trials with low quality on the following methodological quality attributes: blinding (white circles), random sequence generation (blue squares), allocation concealment (white squares), blinding of outcome assessment (blue triangles), incomplete outcome data (white triangles), and selective reporting (blue diamonds). Figure shows mean (95% credible interval) change from baseline low density lipoprotein (LDL) cholesterol concentration with different statins and doses compared with control; lower (more negative) values favour statin treatment
Evaluation of industry sponsorship bias
A total of 146 randomised trials were sponsored by the pharmaceutical industry, and 37 trials were sponsored by governmental agencies, academic research centres, and other not-for-profit organisations. Of the 146 industry sponsored trials, 64 were placebo controlled (43.8%). The corresponding number for the non-industry sponsored trials was 16 (43.2%). There were 33 multi-armed industry sponsored trials, compared with six multi-armed non-industry sponsored trials. Of a total of 59 statin and dose comparisons available in the 37 non-industry sponsored trials (including multiple comparisons in multi-armed trials), 35 were unique comparisons, and 31 of the unique comparisons were also available in the industry sponsored trials (see table 2 in web appendix).
There were no systematic differences between the findings obtained from industry sponsored and non-industry sponsored trials (fig 5⇓). There were differences in the LDL cholesterol lowering effects of individual statins at different doses. In general, higher doses resulted in greater reductions in baseline LDL cholesterol levels (fig 5). While the network meta-analysis of industry sponsored trials estimated a larger treatment effect for some statin dose combinations as compared with the network meta-analysis of non-industry sponsored trials, the opposite was true in other cases. There was generally greater uncertainty around the cholesterol lowering effects of individual statins in non-industry sponsored trials, reflecting the smaller number of randomised controlled trials that were funded by non-industry sources. Overall, point estimates and 95% credible intervals greatly overlapped for all statin dose combinations included in both sets of network meta-analyses.
Fig 5 Dose comparative effects of statins on serum low density lipoprotein (LDL) cholesterol concentration in industry sponsored clinical trials versus non-industry sponsored trials. Findings from industry sponsored trials are shown in white and findings from non-industry sponsored trials are shown in blue. Estimates shown are mean (95% credible interval) change from baseline serum LDL cholesterol concentration compared with control
Figure 6⇓ shows the magnitude of mean industry sponsorship bias, as obtained from the meta-regression analysis. There was no clear evidence of industry sponsorship bias: the mean change from baseline LDL cholesterol levels was on average 1.77 mg/dL with very wide credible intervals (−11.12 to 7.66) lower in scenario 1 and 1.66 mg/dL (−11.27 to 7.91) in scenario 2.
Fig 6 Meta-regression analysis results: evaluation of industry sponsorship bias in the randomised controlled trials of statins. In scenario 1 a pharmaceutical sponsor would favour its own drug in placebo controlled or active comparator trials of statins or the highest dose of its own drug; in scenario 2 a sponsor would favour its own drug in placebo controlled or active comparator trials of statins, but it would not differentiate between different doses of its own drug in dose comparator trials. Figure shows the extent to which mean (95% credible interval) change from baseline low density lipoprotein (LDL) cholesterol concentration was exaggerated in trials with industry favoured statins; lower (more negative) values suggest greater bias.
Discussion
The Cochrane Collaboration defines bias as a systematic deviation from the truth, in results or inferences of studies.25 In other words, bias refers to systematic error, suggesting that multiple replications of the same trial would reach the wrong answer on average. Although randomised research designs are largely immune to many biases that affect weaker forms of evidence obtained from nonrandomised studies, important deficits in the design, conduct, analysis, and reporting may still lead to bias in randomised controlled trials.3 Consistent with the literature evaluating the risk of bias due to reported design characteristics of randomised trials,38 39 our study did not detect any effect of trial-level methodological flaws on objective laboratory outcomes such as reduction in serum LDL cholesterol concentration.
There is now a long history of studies evaluating industry sponsorship bias. These studies unequivocally conclude that industry sponsorship biases study outcomes in favour of the sponsoring company’s product.8 9 10 40 41 42 Previous reviews have outlined a number of potential mechanisms by which industry sponsors can influence the outcome of a study, including how the trial is designed, conducted, analysed, and reported. Given the conclusive evidence that the methodological quality of industry sponsored trials is at least as good as (and often better than) those sponsored by non-industry sources,40 the observed discrepancies may be attributable to factors that cannot be explained by standard tools for assessment of risk of bias.
Unlike the findings of previous studies,19 our analysis did not find empirical evidence of industry sponsorship bias in a systematically identified set of placebo controlled and active comparator trials of statins. First, the mean change from baseline LDL cholesterol levels achieved in industry sponsored trials closely paralleled in magnitude the reductions observed in non-industry sponsored trials: there was no evidence that pharmaceutical sponsors favoured their own drugs in placebo controlled or active comparator trials of statins. There were actual differences in the effectiveness of individual statins at different doses that,16 when taken into account, explained previously observed differences between industry and non-industry sponsored trials. Second, a similar share of industry and non-industry sponsored statin trials was placebo controlled. In addition, out of a total of 35 unique statin dose comparisons available in non-industry sponsored trials, 31 were also available in industry sponsored trials.
Our assessment of industry sponsorship bias differed from previous reviews in the literature in three important ways. First, we paid particular attention to the comparability of patient populations, interventions, and doses across the identified set of trials.21 Our review adopted mean reduction in LDL cholesterol concentration from baseline as the primary outcome of interest for two important reasons. First, mean LDL cholesterol reduction from baseline was the most commonly reported outcome in the randomised controlled trials of statins. Second, LDL cholesterol reduction is an objective laboratory outcome, which is largely immune to biases resulting from deficits in methodological quality in randomised controlled trials.38 39 It would not be possible to distinguish industry sponsorship bias from other design related biases if our models were based on other, more subjective outcomes such as myocardial infarctions.
Second, we explored the mechanism of potential bias. It is difficult to determine how industry biases from multiple funding sources would influence a complex body of evidence, with different pharmaceutical companies sponsoring different products.22 We tested the hypothesis that industry sponsorship bias would occur when a pharmaceutical sponsor favoured its own product in placebo controlled or active comparator trials, or the highest dose of its own drug in a trial comparing multiple doses. Based on these analyses, there was no evidence that the pharmaceutical companies intentionally favoured their products in the randomised controlled trials of statins.
Third, our analyses took into account actual differences in effectiveness between individual statins at different doses, and investigated intervention effect sizes. Previous reviews showed that there was a statistical association between industry sponsorship and positive results.8 10 40 Positive results observed in earlier studies referred to a dichotomous outcome of either reporting statistically significant findings or positive conclusions that were favourable for the sponsoring company. However, the difference in the frequency of either statistically significant findings or favourable conclusions observed between industry sponsored and non-industry sponsored trials should not be referred to as bias. Bias is the underestimation or overestimation of true intervention effects. Therefore, any bias assessment would need to evaluate the magnitude of the intervention effect (as well as its uncertainty). Using network meta-analyses, we took into account actual effectiveness differences between individual statins at different doses and evaluated the magnitude of mean LDL cholesterol reduction from baseline achieved with individual statins in industry and non-industry sponsored trials, and showed that the magnitude of effect was not systematically different—either qualitatively or quantitatively—in the two sets of trials.
Generalisability of findings
An important consideration is whether our findings are generalisable to the assessment of industry sponsorship bias in other therapeutic areas. Certainly, there are unique aspects of the statin evidence base that complicate comparisons with other therapeutic areas. First, Lathyris and colleagues previously suggested that the more favourable results of industry sponsored trials might be due to design issues, in particular the choice of comparators with dosing regimens that are either inactive or suboptimal.43 Unlike other therapeutic areas where the influence of dose on treatment effectiveness may not be widely known outside of industry researchers, statin drugs have a widely established dose response relationship with higher doses resulting in greater cholesterol lowering effects. It is possible that this relationship might have made it difficult for pharmaceutical companies to introduce overt design biases in randomised controlled trials using dose as a design factor.
Second, previous studies have shown that each pharmaceutical company generates a clinical research agenda that is strongly focused on its own products, while comparisons involving interventions from different companies are uncommon.43 However, this was not the case in the clinical literature evaluating the effectiveness of statin therapy. There were a large number of trials where pharmaceutical companies compared their products against licensed regimens belonging to other companies.
Potential industry sponsorship bias is among many intertwined bias domains in the literature. For example, problematic industry practices in terms of deliberate delays in publication and withholding data result in significant publication biases.44 45 In recent years, pharmaceutical companies have attempted to prevent studies with unfavourable results from being published, and to publish positive studies with positive results more than once in high profile cases.46 47 Perpetuating such publication biases, pharmaceutical companies have been shown to selectively report favourable outcomes while deliberately hiding information on others such as harm endpoints.48 Such practices, which do not seem to be prevalent in the statin literature,14 may partly explain the previous findings of bias in favour of outcomes of research funded by industry.46
Bias in an individual trial versus a collection of trials
Pharmaceutical companies almost exclusively sponsor trials that have favourable conclusions for its products. Perhaps unsurprisingly, some companies selectively compare their products against others that they deem strategic competitors. For example, all of the direct head-to-head comparisons between rosuvastatin, which entered the US market in 2003 (sponsoring company, AstraZeneca), and atorvastatin, which was approved by the Food and Drug Administration in 1996 (sponsoring company, Pfizer), were funded by AstraZeneca. In a similar fashion, Pfizer sponsored the majority of head-to-head trials of atorvastatin and earlier statins. This seems consistent with the recent concern voiced by Light and Lexchin, postulating that much of the research and development activities of pharmaceutical companies are driven by strategic marketing and sales motives.49 The industry’s tendency to trial its products against weaker alternatives may skew the nature of the available evidence and introduce potential asymmetries; however, doing so does not mean that the trial results themselves are biased. In the case of statins, findings obtained from industry sponsored trials seem similar in magnitude as those from non-industry sources,
Over the past quarter century, every randomised trial provided a piece of a puzzle that, when taken together, provided a relatively symmetric and balanced evidence network for statins (despite the small number of trials available for some statins such as fluvastatin and lovastatin). Unlike other therapeutic fields,50 51 the randomised controlled trial evidence base of statins included a large number of head-to-head trials providing information on a range of doses. Nonetheless, there is a risk of deriving biased conclusions about the comparative effects of individual statins. For instance, a meta-analysis failing to take into account dose differences between individual statins would produce biased results. A better approach is to synthesise the relevant evidence network, which includes all placebo controlled and active comparator trials and provide direct and indirect information for all treatment contrasts of interest at different doses.14 15 16 17
Limitations
The findings from our analysis should be considered in light of its limitations. First, it is possible that our assessment of the methodological attributes of included trials could underestimate their quality. Such assessments have an important shortcoming: when applied to published accounts of randomised trials, it becomes very challenging to distinguish between factors that were missing (such as adequate concealment of treatment allocation) and those that were simply not reported.
Second, we were unable to base our evaluation of industry sponsorship bias on other objective and clinically meaningful outcomes such as all-cause mortality. As trials reporting such outcomes were almost exclusively funded by the industry, there was an inadequate number of non-industry sponsored trials for a meaningful comparison between the two sets of trials. In a similar fashion, there were few prospectively designed, direct head-to-head trials of individual statins evaluating all-cause mortality.
Third, as with any meta-regression analysis, our primary statistical assessment of industry sponsorship bias had low power to detect bias and resulted in wide credible intervals.52 Nevertheless, our null finding is unlikely to be attributable to the low power in the statistical analysis. The side by side comparison of the LDL cholesterol lowering effects of individual statins obtained from industry and non-industry sponsored trials provides compelling evidence that there is no systematic difference in the LDL cholesterol reduction from baseline achieved in the two sets of trials evaluating the same sets of statins at the same doses.
Finally, our analyses share the limitations of the studies included in our review. Many of the randomised controlled trials included in our study did not use an analysis of covariance model (ANCOVA) and instead modelled the changes in LDL cholesterol levels without adjusting for baseline values. Such analytic approaches have been shown to produce biased results in cases where baseline values in different treatment arms are not balanced, which is particularly the case in small studies.53 54 In accordance with predefined criteria, we excluded randomised trials with <50 patients per arm to guard against small study biases, including potential imbalances in baseline levels between treatment arms.
Conclusion
Despite the previously observed relationship between industry sponsorship and trial outcomes, we found no evidence of industry sponsorship bias in a systematically identified set of placebo controlled and active comparator trials of statins. There are actual effectiveness differences among different statins at different doses, which should be considered in future evaluations of sponsorship bias. Our findings highlight the need for further research into the empirical basis for industry sponsorship biases.
What is already known on this subject?
Research sponsored by the pharmaceutical industry is more likely to favour the product developed by the company sponsoring the research than research funded by other sources
Head-to-head comparisons of statins are more likely to report findings favouring the sponsor’s product over the comparator drug
It is possible that the effectiveness of different drugs at different doses varies, making it difficult to distinguish between a genuine industry bias and actual differences in treatment effectiveness
What this study adds
Using statins as a case study, our network meta-analysis did not find empirical evidence of industry sponsorship bias in a systematically identified set of placebo controlled and active comparator trials
There are differences in the effectiveness of individual statins at different doses that, when taken into account, explain previously observed differences between industry sponsored and non-industry sponsored trials
Notes
Cite this as: BMJ 2014;349:g5741
Footnotes
We thank Bernice Tsoi and Harleen Toor from McMaster University, Canada, for their contribution to the systematic review.
Contributors: HN conceived the study. SD and AEA provided guidance during study design, protocol development, data analysis and interpretation, and manuscript preparation. Bernice Tsoi, Harleen Toor, and HN identified and selected studies for inclusion in the review. Bernice Tsoi, Harleen Toor, and HN extracted data from included trials. HN analysed and interpreted the data. HN, SD, and AEA interpreted the findings. HN drafted the paper and SD and AEA made revisions on subsequent drafts. All authors read and approved the final version of the paper. HN is the guarantor.
Funding: The authors received no funding for this study.
Competing interests: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years, no other relationships or activities that could appear to have influenced the submitted work. HN received research support for a project on asthma from GlaxoSmithKline in the past three years. SD and AEA are employed by the University of Bristol and have previously received funds from the UK National Institute for Health and Care Excellence (NICE).
Ethical approval: Not required.
Transparency declaration: HN affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
Data sharing statement: Raw data used in the analysis will be made available upon request.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.