Creating and using real-world evidence to answer questions about clinical effectiveness

New forms of evidence are needed to complement evidence generated from randomised controlled trials (RCTs). Real-World Evidence (RWE) is a potential new form of evidence, but remains undefined. This paper sets to fill that gap by defining RWE as the output from a rigorous research process which: (1) includes a clear a priori statement of a hypothesis to be tested or research question to be answered; (2) defines the data sources that will be used and critically appraises their strengths and weaknesses; and (3) applies appropriate methods, including advanced analytics. These elements should be set out in advance of the study commencing, ideally in a published protocol. The strengths of RWE studies are that they are more inclusive than RCTs and can enable an evidence base to be developed around real-world effectiveness and to start to address the complications of managing other real-world problems such as multimorbidity. Computerised medical record systems and big data provide a rich source of data for RWE studies. However, guidance is needed to help assess the rigour of RWE studies so that the strength of recommendations based on their output can be determined. Additionally, RWE advanced analytics methods need better categorisation and validation. We predict that the core role of RCTs will shift towards assessing safety and achieving regulatory compliance. RWE studies, notwithstanding their limitations, may become established as the best vehicle to assess efficacy.


INTRODUCTION
New ways are needed to generate evidence in addition to conducting randomised controlled trials (RCTs). 1, 2This is particularly so as medicine struggles to understand how to apply an evidence base developed for single conditions to an ageing population with multimorbidity. 3Studies using Real-World Evidence (RWE) are a potential next step (Box 1).

RWE:
• Is a form of evidence (along with RCTs, health economics studies, etc.) generated to answer a question or test a hypothesis • Is derived from primary or secondary real-world data (RWD) sources, often data from computerised medical record (CMR) systems • Includes appropriate and rigorous design and analyses, generally set out in a protocol in advance of conducting the study • Provides evidence about patient populations diseases, medicines, and health care that will inform clinical practice • Generates further research questions Patients, providers and payers all need evidence beyond that generated in clinical trials: • Patients want to know to what extent health interventions work in the real world for patients like them; • Providers of health care want to make more informed recommendations for patients; • Commissioners and payers need to know how interventions work in their populations in the real world; • Society needs to be informed of any long-term unexpected complications of therapies.
RWE is enabled by the ubiquity of health care information technology systems and the resultant increase in health care data that is born digital, as well as by advances in analytics.
The move towards greater use of RWE is also driven by the general shift in health care from a transactional, fee-forservice model to a more value-based system with payments based on outcomes. 4n summary, RWE is a form of evidence generated by applying a research question or testing a hypothesis using real-world data (RWD), and applying a method that generally includes advanced analytics to generate evidence (Figure 1).

REAL-WORLD DATA (RWD)
RWD is either primary -collected specifically for the RWE study -or secondary use of data collected primarily for another purpose (Box 2).All these data have strengths and limitations 5 that need to be set out in full.to areas where case definition and outcomes might be readily measured from routine data.RWE is able to gain insights from complex patients with multimorbidities.The way an intervention is implemented in the 'real world' is not necessarily the same as the way that intervention was planned to be.
Data from CMR systems and administrative claims records are a mainstay of such studies.Importantly, records can be used to rapidly identify any unexpected harms and ensure patient safety.And, because of their observational nature, there is no need for subjects of such studies to specially attend clinics or clinical research centres beyond that is required for their routine care.

LIMITATIONS OF RWE
The primary limitation of RWE is that as it is a new and emerging approach, there is no agreed upon guidance for RWE methodologies and evaluation.There is uncertainty about Computerised medical record (CMR) systems have been a key enabler of RWE investigations, particularly in health systems where individuals have a unique ID and can be followed over time. 5,6However, access to RWD is not always straightforward -and laws and policies to protect privacy can be a greater limitation on the ability to carry out RWE research than conducting analyses.

Design of RWE studies
The design used in an RWE study needs to have rigour appropriate to the disciplines it is drawn from.Table 1 sets out the possible elements of an RWE observational study.They should have written, published protocols, and peerreviewed outputs.

REAL-WORLD ANALYTICS
Advanced analytic methodologies are used to mine RWD and provide evidence for populations and subgroups.Whereas trials and conventional evidence inform how populations might respond to treatment, RWE looks to provide evidence for subgroups and eventually for individuals.

Strengths of RWE
RWE studies are likely to cost much less than RCTs, and retrospective RWE investigations can be completed much faster than RCTs -and the same may be true of some taken from hypothesis to creating evidence and uncertainty about the relevance of trial results to an ageing population with multimorbidity.RCTs in the therapeutic world are shortmainly because that is all that regulators require.As a result, rare complications of therapies may not be picked up in a trial that lasts 6-12 months.This is a very important function of RWE or post-marketing studies, i.e. the detection of unusual but significant drug-related issues -either on target or off target.It is it not just the size of the numerator that is important but the possibility of tracking large numbers of patients over long periods that provides such useful information.Whilst RWE will most often not prove causality, it does enable a hypothesis to be raised and explored further.The potential advantages of using RWE are that costs are potentially much lower and retrospective RWE investigations can take place much faster, and the same may be true of some prospective investigations.
Whilst RWD are inherently more messy, their advantage is that RWE studies will include people with multimorbidity, on usual prescribed doses, and standard patterns of adherence.Also, adverse effects of medication may not always be detected in the initial trials.

CALL FOR THE ESTABLISHMENT OF RWE CENTRES
Notwithstanding these limitations, studies using RWE are probably here to stay; the urgent need is for methods to assess the quality of these studies.Unless we develop robust ways of working, RWE might be dismissed as just another attempt to create new ways to conduct comparative effectiveness research. 20Research funders should be sponsoring the development of RWE centres to develop methodologies and conduct RWE studies.Prospective RWE studies could focus on stratified medicine. 21These studies should look to understand which groups of patients therapeutic interventions fit with trial evidence and for which groups they are more or less effective in real-world use.Retrospective studies can rapidly answer questions about adherence, thresholds at which treatments are implemented, unanticipated safety issues, and other important questions.They should inform health services about the cost-effectiveness of different care pathway options; currently, they are often dealt with piecemeal.
sample size, what represents a valid and reliable method, how to compare care pathways, especially in people with complex health needs, the best method for economic evaluation, and how to reduce bias.A recent debate around possible harms of screening for lung cancer reflects this uncertainty. 7uidance from the Enhancing the Quality and Transparency of health Research network (http://www.equator-network.org/), notably Consolidated Standards of Reporting Trials, informs whether an RCT is of quality.However, despite all attempts to reduce the risk of bias, there are concerns that funding source may bias trial results, 8,9 though there is not a consensus about this. 10, 11 The International Society for Pharmacoeconomics and Outcomes Research has to some extent sought to address this gap, 12 though this does not address the study of RWE per se.
The RWD used in RWE studies are messy and often incomplete; sophisticated statistical methods must be used to rigorously gain accurate RWE insights.Further, apparently easy to define conditions can actually be more complexdiabetes provides a good example of this. 13Pitfalls in case selection are all too easy and can only be detected if coding lists used in analysis are published. 14Case finding if it is to be reliable between different brands of computer system -let alone international -needs to be much more ontological 15,16 (i.e.transparent models that link clinical concept to coding lists used to identify cases, outcomes, confounders, etc.). 17

RWE IS COMPLEMENTARY TO RCT EVIDENCE
Whilst the RCT remains the gold standard for testing hypotheses about treatment, other forms of evidence are needed, as RCTs fail to answer questions of benefit for those excluded from trials.RCTs will always be needed and form an important part of the licencing of medications; it is important that they are demonstrated to be effective in the core target group who will receive them.However, a number of factors suggest that whilst RCTs will always be required, a first response of 'Let's do a trial' to test a hypothesis, particularly about the effectiveness of treatments in clinical practice, should be accompanied by a careful evaluation as to whether an RCT is the best option.Downsides of RCTs (Figure 2) include high costs, the time  24 There are enormous opportunities for the informatics community if these predictions about the future research direction are correct.The future research landscape will be one in which RCTs are required for regulation and for approval of therapies, devices, and other interventions.RCTs are needed primarily to demonstrate comparative efficacy and safety in a controlled environment.RWE studies will become the norm for demonstrating effectiveness and cost-effectiveness, particularly in the context of an ageing population with multimorbidity.

SUMMARY
RWE studies are needed to properly assess benefit/risk of treatments when used in standard clinical practice.CMR can facilitate concurrent safety monitoring and data collection without direct patient contact and enable large randomised study populations to be identified for pragmatic RCTs.Informaticians or investigators with a detailed understanding of the strengths and weaknesses of routine data -how to infer meaning and to avoid pitfalls -should lead such study teams.Ensuring privacy and ethical use of patient data is also essential for the success of these projects.Analysis of health data needs to take place within a secure environment.There needs to be a high level of trust alongside professionalism in protecting privacy.

Figure 1. RWE is generated by applying a question or testing a hypothesis using RWD and applying analytical techniques set out in the study design
RWE studies might use a million.Exclusion criteria for RCTs are often extensive and limit generalisablity; RWE studies can use data that are statistically generalisable.RWE studies may particularly lend themselves

Table 1 . The design of RWE studies Element of RWE study Design of an RWE observational study Subjects
Range of treatment pathways• Comprehensive use of computerised medical records, across the care pathway • Need to infer meaning from 'messy' routine data • Includes those eligible for care from the health system • Clinical topics where case finding and outcomes are measured using routine data • Large number of cases • Case definition will be that of usual practice • Wide range of other therapies and co-morbidities • Few inclusion/exclusion criteriaSetting• Routine care •

Figure 2. The case for increasing the use of RWE
Simon de Lusignan is the lead academic and Neil Munro is a professor supporting the RWE centre based at University of Surrey funded by Eli Lilly and Company (http://www.surrey.ac.uk/fhms/research/researchgroups/real_world_evidence_ centre_fhms.htm).
22,23 New et al. sum up the current situation and the need for RWE: RCTs are the gold standard but additional information is needed about the risk/benefit profile of new treatments in real-world practice, and especially evidence providing