You are here

Article Text

Article menu

Download PDFPDF

XML
Research article
The Heimdall framework for supporting characterisation of learning health systems
  1. Scott McLachlan,
  2. Henry W. W. Potts,
  3. Kudakwashe Dube,
  4. Derek Buchanan,
  5. Stephen Lean,
  6. Thomas Gallagher,
  7. Owen Johnson,
  8. Bridget Daley,
  9. William Marsh and
  10. Norman Fenton
  1. Queen Mary University of London, London, UK
  2. University College London, London, UK
  3. Massey University, Auckland, New Zealand
  4. Massey University, Auckland, New Zealand
  5. Massey University, Auckland, New Zealand
  6. University of Montana, Missoula, MT
  7. University of Leeds, Leeds, UK
  8. Blizzard Institute, Queen Mary University of London, London, UK
  9. Queen Mary University of London, London, UK
  10. Queen Mary University of London, London, UK
  1. Author address for correspondence: Scott McLachlan Queen Mary University of London (QMUL) London E1 4FZ, UK s.mclachlan{at}qmul.ac.uk

Abstract

Background There are many proposed benefits of using learning health systems (LHSs), including improved patient outcomes. There has been little adoption of LHS in practice due to challenges and barriers that limit adoption of new data-driven technologies in healthcare. We have identified a more fundamental explanation: the majority of developments in LHS are not identified as LHS. The absence of a unifying namespace and framework brings a lack of consistency in how LHS is identified and classified. As a result, the LHS ‘community’ is fragmented, with groups working on similar systems being unaware of each other’s work. This leads to duplication and the lack of a critical mass of researchers necessary to address barriers to adoption.

Objective To find a way to support easy identification and classification of research works within the domain of LHS.

Method A qualitative meta-narrative study focusing on works that self-identified as LHS was used for two purposes. First, to find existing standard definitions and frameworks using these to create a new unifying framework. Second, seeking whether it was possible to classify those LHS solutions within the new framework.

Results The study found that with apparently limited awareness, all current LHS works fall within nine primary archetypes. These findings were used to develop a unifying framework for LHS to classify works as LHS, and reduce diversity and fragmentation within the domain.

Conclusions Our finding brings clarification where there has been limited awareness for LHS among researchers. We believe our framework is simple and may help researchers to classify works in the LHS domain. This framework may enable realisation of the critical mass necessary to bring more substantial collaboration and funding to LHS. Ongoing research will seek to establish the framework’s effect on the LHS domain.

  • electronic health records
  • learning health systems
  • learning healthcare systems
  • precision medicine

Commons license http://creativecommons.org/licenses/by/4.0/

https://doi.org/10.14236/jhi.v25i2.996

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    INTRODUCTION

    Learning health systems (subsequently referred to as LHS) are defined by the Institute of Medicine (IoM) as systems in which alignment of scientific and cultural tools lead to knowledge generation to improve healthcare as a result of daily practice.1 Since LHS was conceptualised in 2007, they have been the focus of increasing research attention.2–6 The opportunity and promise of LHS have resulted in texts presenting collections of LHS-specific research1,7 creation of a new journal, Learning Health Systems,8 and new courses of academic study.9,10 We believe it could be the most significant development in healthcare since the advent of evidence based medicine (EBM) and electronic health records (EHRs) that support EBM.14–16 EHR has existed for more than 40 years11–13 and organisations that implemented EHR discovered reductions in costs, clinical testing and patterns of repeated and sometimes unneeded prescriptions. Enhanced co-ordination and communication between clinicians were seen to improve the quality of patient care.12,17–20

    Despite the benefits, early EHR systems were considered expensive, focused on information gathering rather than improving healthcare.20 Development lacked clinical input, existed as multiple stand-alone systems, experienced slow adoption, suffered from trust and data quality issues, claims that systems increase or exacerbate risk for errors, and concerns over patient privacy and security.18,20 All of these issues are still seen as unresolved barriers to adoption of EHR.18,21–27 Despite this, EHRs are the foundation for LHS. Efforts towards LHS, coupled with proposed changes to legislation, policy and the ethics of how clinicians engage with clinical datasets suggest an entirely new dimension to EHR. One in which they are used collectively as ‘big data’ and focused using individual patient’s attributes to identify causes and optimal treatments strategies for disease.

    Descriptions developed in IoM reports are the basis of most author definitions and descriptions of LHS.28–33 Medical information systems that can be predictive, preventative, personalised and participatory represent the core principles of 4P medicine.34 According to the IoM, these systems have the potential to identify groups at greatest risk of complications for purposes of targeting interventions.7 In parallel, maturing technologies such as large datasets, machine learning, and enhanced processing power further enable the concept of LHS.2,34,35

    The IoM promoted EBM as the primary driver for LHS,7 yet their definition fails to describe LHS attributes which contribute to quality, safety, efficiency and effectiveness of patient care.36 The four fundamental attributes37 listed in Figure 1 provide tangible metrics to compare and contrast LHS efforts. These attributes were not included in the IoM definition, but are widely found in LHS. They clarify the use of large EHR datasets as the source of knowledge for LHS achieving the goal of improving quality in individual patient care.

    Figure 1
    Figure 1 Four elements of an LHS

    There are numerous examples of proposed benefits of LHS. For clinicians, these include assessing which laboratory or imaging tests may be more diagnostic given a patient’s presenting symptomology38; reducing risk from prescribing errors31 and increasing awareness of pharmacogenetics.39 It is claimed that patients would benefit from advanced knowledge developed from the experiences and diagnoses of past patients, which saves time and reduces costs.38,40,41 A learning healthcare organisation culture supports EBM, while successful integration of research into practice is what enables it.42 The ability to use technology to record, compare, contrast and present information in almost real-time enhances the input, analysis and decision phases of the learning lifecycle. Alternatively, it is said that the financial burden to implement and support health technology43–45 along with a persistent need for data and systems standardisation,45–48 interoperability46,49,50 and integration49,51 have all acted as barriers to broad LHS adoption.

    In our group’s recent letter to the editor of this journal,52 we demonstrated the lack of awareness and barriers for researchers to appropriately identify their efforts as LHS solutions. We believe that this results from a number of significant problems in the domain. The lack of adequate classification and standardisation results in groups working on comparable systems not identifying their efforts as LHS, and may be the cause of unnecessary duplication of efforts and the observable lack of collaboration. The absence of a unifying framework means the domain is yet to generate a necessary critical mass, limiting efforts to resolve barriers and challenges to the adoption of LHS and constraining funding availability. We were only able to identify one primary article that attempted to consolidate and analyse the current state of knowledge in LHS.36 We extend that effort drawing on a larger collection of works to establish a unifying framework for LHS.

    METHOD

    Our study sought first to define a comprehensive framework and taxonomy for LHS, and then to demonstrate its application to self-identified literature from the LHS domain. The literature search used identical plain language search terms as Foley and Vale36: ‘LHS’ and ‘learning healthcare system’ drawing articles from Scopia, Science Direct, PubMed, EBSCOhost, DOAJ and Elsevier (n = 1083). These works were all authored in the decade since the IoM’s initial LHS report.7 Figure 2 shows this literature, by year, in orange (the drop in 2017 is due to reporting only up to July); contrasted with literature identified by Foley and Vale36 shown in grey. The leap in publications in 2011 followed the first53 and second54 meetings of the Committee on the Learning Healthcare System in America, both proceedings published during 2011. More than 50% of LHS publications were generated since 2013.

    Figure 2
    Figure 2 LHS publications by year

    We undertook this review following the identification of seminal sources approach of meta-narrative reviewing.27,55 An initial read of abstracts was used to reject duplicates and papers not related to the central topic, including those using the search term in context of learning in the academic or education sense.56,57 This reduced the collection source pool (n = 542). Conclusions and methods were reviewed, seeking to reject papers that did not present or propose an LHS; for example, those exploring the medicolegal, ethical or societal aspects of LHS.31,58,59 The resulting core pool (n = 230) was then comprehensively reviewed. Of these, 53% proposed a potential solution compared to 47% that presented an existing solution.

    We used content and thematic analysis60 to recognise and classify LHS uses while also identifying common barriers and thematic concepts for investigation. Formal concept analysis61 was used to identify the frequency and interrelationships between the identified concepts. The elements of both analysis methods were identified inductively during the first full reading of the core pool of literature and used to develop spreadsheets for analysis. A second full reading was performed to data mine the literature and populate spreadsheets for further analysis during framework development. Table 1 provides examples of attributes identified for the formal concept analysis.

    View this table:
    Table 1 LHS formal concept analysis attributes

    RESULTS

    Taxonomy for LHS

    We found only three papers36,62,63 that proposed classification systems for LHS. Surveillance and Comparative Effectiveness Research were the only types common to all three. Figure 3 unifies the knowledge identified from all three papers into a taxonomy of nine LHS classification types, indicating abbreviations (initials) and the primary reference for each.

    Figure 3
    Figure 3 LHS taxonomy

    Cohort identification (CI) seeks patients with similar attributes, used to determine the feasibility of studies and quantify numbers of potential patients that may be helped.62 CI is also the first operational step of most other LHS types.

    Positive deviance (PD) uses outcome data to benchmark clinical care. PD identifies elements of safer, more effective, timely and patient-centred care, recognising beneficial behaviours for incorporation into another clinician’s practice.64 PD can also identify common traits of patients benefiting from a treatment, using these to identify others who may benefit from the same intervention.62

    Negative deviance (ND) identifies instances of sub-optimal care outcomes. ND presupposes some particular clinical behaviour negatively impacted patient care and the resulting outcome. The clinician critically evaluates the care provided, investigating causes for sub-optimal results.63

    Predictive patient risk modelling (PPRM) uses patterns discovered in patient datasets to identify cohorts at higher risk for future adverse events. PPRM can use routine health data to identify ‘triple fail’ events; where treatment fails to improve patient care experiences, population health, or lower healthcare costs.65

    Predictive care risk and outcome model (PCROM) algorithms identify situations of high risk for ‘unsafe’, ‘delayed’ or ‘inefficient’ care, providing estimates of the effectiveness of different interventions.36 Geisinger Health Systems have incorporated PCROM approaches into clinical software, identifying when spikes in hospital activity or patient non-attendance may occur.66

    Clinical decision support systems (CDSSs) are active knowledge systems where two or more characteristics of the patient are matched to computerised knowledge bases with algorithms generating patient-specific treatment recommendations.67–69

    Comparative effectiveness research (CER) compares interventions and outcomes within an EHR dataset to determine the most effective treatment, using a method considered more efficient than randomised control trials.36 CER isolates patients with similar attributes to the current patient, returning knowledge on treatments that deliver optimum health outcomes.70

    Intelligent assistance (IA) uses data sources to automate routine processes such as prepopulating pathology orders and clinical notes, or summarising patient case notes prior to consultations.36

    Surveillance (S) monitors EHR data for outbreaks of disease (e.g., measles) or treatment issues (e.g., contaminated medicines or increased frequency for post-surgical infections). Examples observed include health and demographic surveillance systems used in sub-Saharan Africa.71

    The Heimdall-integrated LHS framework

    Just as the Norse God Heimdall was said to be the son of nine mothers, we started from our nine LHS classifications to develop the integrated LHS framework in Figure 4. The diagram’s conical structure demonstrates the use of technology (large datasets and processing systems) to record, store, index and present information that flows into and improves the learning processes used in EBM, focusing clinical practice towards delivery of precision medicine (PM). This enables the learning healthcare organisation to engage in decisions individualised to match unique patient characteristics.

    Figure 4
    Figure 4 The LHS unifying framework

    PM results from enhancing the generalised population health approach using attributes in the EHR to constrain analysis for diagnosis and treatment options to cohort predominately matching the presenting patient’s profile. As the clinician enters attributes about the current patient, the speed and accuracy of decisions increase as illustrated by arrows in Figure 4. LHS draws knowledge from a reducing cohort whose attributes predominately match the current patient as illustrated by the circular design. In examining a cross-section of Figure 4, the larger white circle represents the entire population used to select the most effective common treatment. The light grey circle reduces that population to those who share some basic attributes with the patient that would normally be identified in the slower learning organisation approach of EBM. The inner dark circle further reduces the population to a significant cohort with clinical, genetic and socioeconomic attributes predominately matching the patient at the centre. The interrelationship between the Heimdall framework and our taxonomy is shown down the right side of the conical portion of the diagram. While LHS is technology solutions, the majority operate in the context of either the treatment provider’s learning organisation, or the clinician’s primary patient-facing role.

    Within this framework, we incorporate the concept of a clinical lifecycle as shown in Figure 5 and adapted from multiple works in this review.63,66,70 The right side of the diagram represents largely clinician-driven aspects, while the left side identifies those aspects where LHS technology delivers improvements. The more challenging barriers to LHS regarding data quality, interoperability and standardisation all result from activities on the right side of the lifecycle.

    Figure 5
    Figure 5 An example of the clinical lifecycle

    This cycle is repeated, both for surveilling the proposed transformation and to seek further items of knowledge.63,66,70 LHS engenders a close relationship between care, research and knowledge translation, aimed at providing a platform for integrating various data to better understand patients.72 LHS is commonly described using an iterative lifecycle similar to many other EBM processes where: (a) patient data is collected by clinicians; (b) aligned, transformed and amalgamated into larger data sets; (c) a problem is defined and (d) analysis performed; (e) with evidence data returned and (f) made into new knowledge, that is, (g) used to transform clinical practice.

    The taxonomy and Heimdall framework provided a toolkit for characterising LHS literature in terms of the following thematic and analytical aspects to assess whether the LHS demonstrates:

    • Taxonomic consistency – conforms to the taxonomy.

    • Patient focused – ensures personalised health care, known as PM.

    • Technology usage – uses health IT with big data, machine learning algorithms and automation.

    • Decision support – near real-time support to clinicians, bringing recent scientific advances, machine learning and EBM together at point-of-care.

    • Application of LHS – goes beyond selecting ‘most likely treatment for a population’, to selecting the ‘most applicable treatment for an individual’.

    • Barriers and further observations – challenges limiting implementation.

    What follows is a survey and discussion using the Heimdall framework to identify and resolve key questions from within the domain of LHS. Answers to these questions were resolved through applying the framework to the broader literature cohort.

    Validation of the LHS taxonomy

    To validate our taxonomy, proposed or presented solutions were reviewed and classified using the taxonomic descriptions. Some proposed solutions were incompletely described, but we found that the intention of the authors was always clear from the information presented. All LHS solutions conformed easily to one of our identified taxonomic types. We believe this validates the taxonomy as we have presented. Our validation of the taxonomy also found that CER were the most prevalent type of LHS presented in the literature, as identified in Table 2. CER was followed at some distance by stand-alone CI, although it should be noted that this classification type had not been identified by either Deeny and Steventon63 or Foley and Vale36.

    View this table:
    Table 2 Distribution of LHS solutions (per 100 publications)

    DISCUSSION

    A number of themes emerged during this analysis and are discussed in the following section, which includes topical analysis to investigate their effect on the LHS domain.

    Patient focused

    While clinicians argue they always practiced patient-focused medicine, normal clinical practice follows population medicine-based EBM.73 PM extends diagnostic practices with profiling techniques and therapies tailored to the individual.74,75 Patient focus is a key dimension that LHS improve.36 PM approaches can be retrospective, as in CER, and prospective, when genotyping for treatment selection.74,75 Patient-centred care encourages data use in optimising care for individuals.76 Aggregated patient records enable LHS to identify cohorts similar to the patient.77 LHS is an efficient tool for integrating PM into practice. As the clinician enters attributes about the patient, the LHS refines a cohort of prior similar patients. It assesses the treatments they received to recommend one most likely to produce an optimal outcome.

    Technology usage

    The focus for health IT has shifted from issues of adoption to identifying how to best use technology to improve healthcare delivery and outcomes.78 This shift is significant in creating LHS and elevates issues in EMR/EHR interoperability, data standardisation and quality that must be resolved if LHS is to be truly practical and ubiquitous.78–80 Health IT’s ability to improve healthcare service delivery quality and efficiency is recognised.81,82 Enabling necessary data flow and integration of data sources are key abilities technology can deliver, representing core requirements to enabling LHS.72,83 Integration of learning into technology is observed in every LHS solution reviewed. Technology is fundamental to LHS. As EBM evolves from paper-based roots, clinicians and healthcare providers will realise benefits from coupling technology to Learning Health Organisations, thus realising LHS.79,84

    Decision support

    Healthcare providers evolved from considering health IT as a billing and documentation facilitator, to contemplating its active participation and capabilities to answer complex questions in care delivery.85 LHS brings opportunities for improving speed and efficiency of clinical decision support.83,85,86 LHS solutions are context-sensitive, incorporating CI and risk modelling in real time to identify interventions for improving individual patient outcomes.87,88 LHS has potential to rapidly perform retrospective comparative effectiveness trials, evaluating treatment options against each other where they have been provided to similar patients.88,89 In contrast with randomised clinical trials, LHS is considered safer, and engenders greater confidence in accuracy of the treatment choice.88,89

    Application of LHS

    Clinical epidemiology is an example of learning healthcare. EBM evolved from clinical epidemiology: statistically identifying the optimal treatment which becomes best practice for that condition.90,91 Conversely, the focus for PM is selecting from available interventions the treatment that will best serve the individual. The primary driver towards population medicine was economic: maximising benefit while minimising cost, harm and waste.90 While meant for benefiting the individual patient, population medicine has disadvantages in that the individual’s best interests may conflict with those of the population and it is difficult to reconcile the two.76,90 Individual patients may be denied higher priced precision interventions in favour of lower cost population-optimised interventions.90 The Heimdall framework demonstrates that LHS focus healthcare using population medicine and EBM directly onto the presenting patient. While EBM selects one treatment for all patients, LHS produces a cohort with attributes similar to the presenting patient, identifying the treatment most likely to be effective for this individual patient. LHS in this way delivers PM.

    Barriers and further observations

    Most authors discuss barriers to implementation. The most common are cost,32,92,93 data interoperability and standardisation,94–96 poor data quality and integrity,63,97,98 informed consent and ethics review complications,99–101 privacy and security issues70,95 and slow technology adoption.95,102,103 These issues are seen in the same context for adopting EHR/EMR. This suggests LHS is inheriting problems from the EHR/EMR on which they depend.

    CONCLUSION

    LHS represents a significant improvement on the present learning organisation, evidence-based practice and population medicine approaches. LHS improves the focus on the individual patient, bringing efficient and expedient PM solutions. LHS is a significant evolution to EBM, and the natural next step in realising the benefits that were expected from implementing EHR/EMR. However, the lack of taxonomy for classifying and describing LHS may be a significant reason for fragmented and duplicated research and solutions in LHS that has impeded widespread adoption. Many authors presenting solutions fail to identify them as LHS. Our research has presented a taxonomy and framework to address this problem, and may help address the challenges in realising all that LHS promise.

    References

    1. 1.↵
      IoM. Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care. Washington,DC: Institute of Medicine, 2011.Available from: https://www.ncbi.nlm.nih.gov/books/NBK83569/pdf/Bookshelf_NBK83569.pdf
    2. 2.↵
      1. Friedman C,
      2. Rubin J,
      3. Brown J,
      4. Buntin M,
      5. Corn M,
      6. Etheredge L, et al
      . Toward a science of learning systems: a research agenda for the high-functioning learning health system. Journal of American Medical Informatics Association 2015;2243-50
      OpenUrlCrossRefPubMed
    3. 3.
      OECD. Data-Driven Innovation: Big Data for Growth and Well-Being. Paris, France: OECD Publishing, 2015.
    4. 4.
      1. Hanna N
      . Mastering Digital Transformation: Towards a Smarter Society, Economy, City and Nation. Bingley, UK: Emerald Group Publishing, 2016.
    5. 5.
      1. Rubin J,
      2. Friedman C
      . Weaving together a healthcare improvement tapestry: learning health system brings together health IT data stakeholders to share knowledge and improve health. Journal of AHIMA 2014;85(5):38-43
      OpenUrl
    6. 6.↵
      1. Lim S,
      2. Keung C,
      3. Zhao L,
      4. Curcin V,
      5. Ethier J,
      6. Burgun A, et al
      . TRANSFoRm: Implementing a Learning Healthcare System in Europe through embedding clinical research into clinical practice. International Conference on System Sciences (HICSS) 2015Hawaii
    7. 7.↵
      IoMOlsen L, Aisner D, McGinnis M. Foreward. The Learning Healthcare System: Workshop Summary. Washington, DC: Institute of Medicine, 2007.
    8. 8.↵
      WileyLearning health systems2017. Available from: http://onlinelibrary.wiley.com/journal/10.1002/. Accessed 13 September 2017.
    9. 9.↵
      UCLPartnersLearning health systems2016. Available from: https://uclpartners.com/events/learning-health-systems/. Accessed 13 September 2017.
    10. 10.↵
      UCLHealth informatics2017. Available from: http://www.ucl.ac.uk/health-informatics. Accessed 13 September 2017.
    11. 11.↵
      1. Basden A,
      2. Clark E
      . Data integrity in a general practice computer system (CLINICS). International Journal Bio-Medical Computing 1980;11511-9. doi:10.1016/0020-7101(80)90017-3
      OpenUrlCrossRefPubMed
    12. 12.↵
      1. Yucha C,
      2. Reigeluth C
      . The use of computers in nursing education, practice and administration. Computer Education 1983;7(4):223-6. doi:10.1016/0360-1315(83)90011-8
      OpenUrlCrossRef
    13. 13.↵
      1. Sengstack P,
      2. Boicey C
      . Mastering Informatics: A Healthcare Handbook for Success. Indianapolis, IN: Sigma Theta Tau, 2015.pmid:25921891
    14. 14.↵
      1. Sur R,
      2. Dahm P
      . History of evidence based medicine. Indian Journal of Urology 2011;27(4):487-9. doi:10.4103/0970-1591.91438pmid:22279315; pmcid:PMC3263217
      OpenUrlCrossRefPubMed
    15. 15.
      1. Smith G
      . Electronic Health Record. Los Angeles, CA: California Department of Alcohol and Drug Programs, 1980.
    16. 16.↵
      1. Gans D,
      2. Kralewski J,
      3. Hammons T,
      4. Dowd B
      . Medical groups’ adoption of of electronic health records and information systems. Health Affairs 2005;24(5):1323-33. doi:10.1377/hlthaff.24.5.1323pmid:16162580
      OpenUrlAbstract/FREE Full Text
    17. 17.↵
      1. Rumball-Smith J,
      2. Shekelle P,
      3. Bates D
      . Using the Electronic Health record to understand and minimise overuse. Journal of the American Medical Association 2017;317(3):257-8. doi:10.1001/jama.2016.18609pmid:28114561
      OpenUrlCrossRefPubMed
    18. 18.↵
      1. Hudis C
      . Comment on Health Care Competition. Alexandria, VA: American Society of Clinical Oncology, ASCO Website, 2014.
    19. 19.
      1. Klimt C,
      2. Becker S,
      3. Fox BS,
      4. Ensminger F
      . A transcribed emergency record at minimum cost. Annals of Emergency Medicine 1983;12(9):556-8. doi:10.1016/S0196-0644(83)80297-2
      OpenUrlCrossRefPubMedWeb of Science
    20. 20.↵
      1. Wyatt J
      . Clinical data systems, part 1: data and medical records. The Lancet 1994;3441543-7. doi:10.1016/S0140-6736(94)90353-0
      OpenUrlCrossRefPubMedWeb of Science
    21. 21.↵
      1. Menachemi N,
      2. Collum T
      . Benefits and drawbacks of electronic health record systems. Risk Management and Healthcare Policy 2011;447-55. doi:10.2147/RMHP.S12985pmid:22312227; pmcid:PMC3270933
      OpenUrlCrossRefPubMed
    22. 22.
      1. Cresswell K,
      2. Mozaffar H,
      3. Lee L,
      4. Williams R,
      5. Sheikh A
      . Safety risks associated with the lack of integration and interfacing of hospital health information technologies: a qualitative study of hospital electronic prescribing systems in England. British Medical Journal of Quality and Safety 2017;26530-41
      OpenUrl
    23. 23.
      1. Furukawa M,
      2. King J,
      3. Patel V,
      4. Hsiao CJ,
      5. Adler-Milstein J,
      6. Jha AK
      . Despite substantial progress in EHR adoption, health information exchange and patient engagement remain low in office settings. Health Affairs 2014;33(9):1672-9. doi:10.1377/hlthaff.2014.0445pmid:25104827
      OpenUrlAbstract/FREE Full Text
    24. 24.
      1. Tange H
      . How to approach the structuring of the medical record? Towards a model for flexible access to free text medical data. International Journal of Biomedical Computing 1996;4227-34. doi:10.1016/0020-7101(96)01178-6
      OpenUrlCrossRefPubMed
    25. 25.
      1. Roos L,
      2. Sharp S,
      3. Wadja A
      . Assessing data quality: a computerised approach. Social Science and Medicine 1989;28(2):175-82. doi:10.1016/0277-9536(89)90145-7
      OpenUrlCrossRefPubMedWeb of Science
    26. 26.
      1. Koppel R,
      2. Metlay JP,
      3. Cohen A,
      4. Abaluck B,
      5. Localio AR,
      6. Kimmel SE, et al
      . Role of computerized physician order entry systems in facilitating medication errors. Journal of the American Medical Association 2005;293(10):1197-203. doi:10.1001/jama.293.10.1197pmid:15755942
      OpenUrlCrossRefPubMedWeb of Science
    27. 27.↵
      1. Greenhalgh T,
      2. Potts HW,
      3. Wong G,
      4. Bark P,
      5. Swinglehurst D
      . Tensions and paradoxes in electronic patient record research: a systematic literature review using the meta-narrative method. The Millbank Quarterly 2009;87(4):729-88. doi:10.1111/j.1468-0009.2009.00578.xpmid:20021585; pmcid:PMC2888022
      OpenUrlCrossRefPubMedWeb of Science
    28. 28.↵
      1. Grumbach K,
      2. Lucey C,
      3. Johnston C
      . Transforming from centres of learning to learning health systems: the challenge for academic health centres. Journal of the American Medical Association 2014;311(11):1109-10. doi:10.1001/jama.2014.705pmid:24643597
      OpenUrlCrossRefPubMed
    29. 29.
      1. Bhandari R,
      2. Feinstein AB,
      3. Huestis SE,
      4. Krane EJ,
      5. Dunn AL,
      6. Cohen LL, et al
      . Pediatric-collaborative health outcomes information registry (Peds-CHOIR): a learning health system to guide pediatric pain research and treatment. PAIN 2017;157(9):2033-44. doi:10.1097/j.pain.0000000000000609pmid:27280328; pmcid:PMC4988911
      OpenUrlCrossRefPubMed
    30. 30.
      1. Crawford L,
      2. Matczak GJ,
      3. Moore EM,
      4. Haydar RA,
      5. Coderre PT
      . Patient-centered drug development and the Learning Health System. Learning Health Systems 2017;1(3):1-7
      OpenUrl
    31. 31.↵
      1. Faden R,
      2. Kas NE,
      3. Goodman SN,
      4. Provost P,
      5. Tunis S,
      6. Beuchamp TL
      . An ethics framework for a Learning Health Care System: a departure from traditional research ethics and clinical ethics. Ethical Oversight of Learning Health Care Systems, Hastings Centre Report Special Report 2013
    32. 32.↵
      SCOAP CollaborativeWriting Group for the SCOAP CollaborativeKwon S, Florence M, Grigas P, Horton M, Horvath K, Johnson M, et al. Creating a learning healthcare system in surgery: Washington State’s Surgical Care and Outcomes Assessment Program (SCOAP) at 5 years. Surgery 2011;151(2):146-52pmcid:PMC4208432
      OpenUrl
    33. 33.↵
      1. Gold M,
      2. Hossain M,
      3. Mangum A
      . Consumer engagement in Health IT: distinguishing rhetoric from reality. eGEMS 2015;3(1):1190. doi:10.13063/2327-9214.1190pmid:26665120; pmcid:PMC4672873
      OpenUrlCrossRefPubMed
    34. 34.↵
      1. Flores M,
      2. Glusman G,
      3. Brogaard K,
      4. Price ND,
      5. Hood L
      . P4 Medicine: how systems medicine will transform the healthcare sector and society. Personalised Medicine 2013;10(6):565-76. doi:10.2217/pme.13.57pmid:25342952; pmcid:PMC4204402
      OpenUrlCrossRefPubMed
    35. 35.↵
      1. Friedman C,
      2. Allee NJ,
      3. Delaney BC,
      4. Flynn AJ,
      5. Silverstein JC,
      6. Sullivan K, et al
      . The science of learning health systems: foundations for a new journal. Learning Health Systems 2016;1
    36. 36.↵
      1. Foley T,
      2. Vale L
      . What role for learning health systems in quality improvement within healthcare providers?. Learning Health Systems 2017;1(4
    37. 37.↵
      1. Forrest C,
      2. Margolis P,
      3. Seid M,
      4. Colletti RB
      . PEDSnet: how a prototype pediatric learning health system is being expanded into a national network. Health Affairs 2014;33(7):1171-7. doi:10.1377/hlthaff.2014.0127pmid:25006143
      OpenUrlAbstract/FREE Full Text
    38. 38.↵
      1. Everson J
      . The implications and impact of 3 approaches to health information exchange: community, enterprise, and vendor-mediated health information exchange. Learning Health Systems 2016;1e10021
      OpenUrl
    39. 39.↵
      1. Etheredge L
      . A rapid-learning health system. Health Affairs 2007;26(2):107-18. doi:10.1377/hlthaff.26.2.w107pmid:17259191
      OpenUrlAbstract/FREE Full Text
    40. 40.↵
      1. Curcin V
      . Embedding data provenance into the learning health system to facilitate reproducible research. Learning Health Systems 2017;1(2):e10019
      OpenUrl
    41. 41.↵
      1. Faden R,
      2. Kass N,
      3. Whicher D,
      4. Stewart W,
      5. Tunis S
      . Ethics and Informed Consent for Comparative Effectiveness Research with prospective electronic clinical data. Medical Care 2013;51(8):S53-7. doi:10.1097/MLR.0b013e31829b1e4b
      OpenUrlCrossRefPubMedWeb of Science
    42. 42.↵
      1. Kitson A,
      2. Harvey G,
      3. McCormack B
      . Enabling the implementation of evidence based practice: a conceptual framework. Quality in Health Care 1998;7149-58. doi:10.1136/qshc.7.3.149pmid:10185141; pmcid:PMC2483604
      OpenUrlAbstract
    43. 43.↵
      1. Cherry B,
      2. Carter M,
      3. Owen D,
      4. Lockhart C
      . Factors affecting electronic health record adoption in long-term care facilities. Journal for Healthcare Quality 2008;30(2):37-47. doi:10.1111/j.1945-1474.2008.tb01133.xpmid:18411891
      OpenUrlCrossRefPubMed
    44. 44.
      1. Jamoom E,
      2. Patel V,
      3. Furukawa MF,
      4. King J
      . EHR adopters vs. non-adopters: Impacts of, barriers to, and federal initiatives for EHR adoption. Healthcare Amsterdam 2014;2(1):33-9. doi:10.1016/j.hjdsi.2013.12.004pmid:26250087; pmcid:PMC4878018
      OpenUrlCrossRefPubMed
    45. 45.↵
      1. Nguyen L,
      2. Bellucci E,
      3. Nguyen L
      . Electronic health records implementation: an evaluation of information system impact and contingency factors. International Journal of Medical Informatics 2014;83779-96. doi:10.1016/j.ijmedinf.2014.06.011
      OpenUrlCrossRefPubMed
    46. 46.↵
      1. Tang P,
      2. Ash JS,
      3. Bates DW,
      4. Overhage JM,
      5. Sands DZ
      . Personal health records: definitions, benefits and strategies for overcoming barriers to adoption. Journal of the American Medical Informatics Association 2006;13(2):121-6. doi:10.1197/jamia.M2025pmid:16357345; pmcid:PMC1447551
      OpenUrlCrossRefPubMed
    47. 47.
      1. Schumaker R,
      2. Reganti K
      . Implementation of Electronic Health Record (EHR) system in the heathcare industry. International Journal of Privacy and Health Information Management 2014;2(2):57-71. doi:10.4018/ijphim.2014070104
      OpenUrlCrossRef
    48. 48.↵
      1. Ludwick D,
      2. Doucette J
      . Adopting electronic medical records in primary care: lessons learned from health information systems implementation experience in seven countries. International Journal of Medical Informatics 2009;7822-31. doi:10.1016/j.ijmedinf.2008.06.005
      OpenUrlCrossRefPubMedWeb of Science
    49. 49.↵
      1. Kaye R,
      2. Kokia E,
      3. Shalev V,
      4. Idar D,
      5. Chinitz D
      . Barriers and success factors in health information technology: a practitioners perspective. Journal of Management and Marketing in Healthcare 2010;3(2):163-75. doi:10.1179/175330310X12736577732764
      OpenUrlCrossRef
    50. 50.↵
      1. McGinn C,
      2. Grenier S,
      3. Duplantie J,
      4. Shaw N,
      5. Sicotte C,
      6. Mathieu L, et al
      . Comparison of user groups’ perspectives of barriers and facilitators to implementing electronic health records: a systematic review. BMC Medicine 2011;9(46):46. doi:10.1186/1741-7015-9-46
      OpenUrlCrossRefPubMed
    51. 51.↵
      1. Lluch M
      . Healthcare professionals’ organisational barriers to health information technologies: a literature review. International Journal of Medical Informatics 2011;80849-62. doi:10.1016/j.ijmedinf.2011.09.005
      OpenUrlCrossRefPubMed
    52. 52.↵
      1. Mclachlan S,
      2. Dube K,
      3. Buchanan D,
      4. Lean S,
      5. Johnson O,
      6. Potts H, et al
      . Learning Health Systems: the research community awareness challenge. BCS Journal of Innovation in Health Informatics 2018;25(1):38-40. doi:10.14236/jhi.v25i1.981
      OpenUrlAbstract/FREE Full Text
    53. 53.↵
      NASEMThe learning healthcare system in America2011. Available from: http://www.nationalacademies.org/hmd/Activities/Quality/LearningHealthCare/2011-JAN-11.aspx. January 2011.
    54. 54.↵
      NASEMThe learning healthcare system in America2011. Available from: http://www.nationalacademies.org/hmd/Activities/Quality/LearningHealthCare/2011-MAY-05.aspx. May 2011.
    55. 55.↵
      1. Wong G,
      2. Greenhalgh T,
      3. Westhorp G,
      4. Buckingham J,
      5. Pawson R
      . RAMESES publication standards: realist syntheses. BMC Medicine 2013;11(1):21. doi:10.1111/jan.12095doi:10.1186/1741-7015-11-21
      OpenUrlCrossRefPubMed
    56. 56.↵
      1. Bellack J
      . Creating a continuously learning health system through technology: a call to action. Journal of Nursing Education 2016;55(1):3-5. doi:10.3928/01484834-20151214-01pmid:26812374
      OpenUrlCrossRefPubMed
    57. 57.↵
      1. Yoder-Wise P
      . The continuously learning health system: recommendations for the Josiah Macy Jr. Foundation. Journal of Continuing Education in Nursing 2015;46(9):379-80. doi:10.3928/00220124-20150821-10pmid:26352036
      OpenUrlCrossRefPubMed
    58. 58.↵
      1. Bernstein J,
      2. Friedman C,
      3. Jacobsen P,
      4. Rubin J
      . Ensuring public health’s future in a national-scale learning health system. American Journal of Preventative Medicine 2015;48(4):480-7. doi:10.1016/j.amepre.2014.11.013pmid:25700654
      OpenUrlCrossRefPubMed
    59. 59.↵
      1. Brody H,
      2. Miller F
      . The research-clinical practice distinction, learning health system and relationships. Hastings Centre Report 2013;43(5):41-7. doi:10.1002/hast.199pmid:24092591
      OpenUrlCrossRefPubMed
    60. 60.↵
      1. Vaismoradi M,
      2. Turunen H,
      3. Bondas T
      . Content analysis and thematic analysis: implications for conducting a qualitative descriptive study. Nursing and Health Sciences 2013;15(3):398-405. doi:10.1111/nhs.12048pmid:23480423
      OpenUrlCrossRefPubMed
    61. 61.↵
      1. Stumme G
      Staab S, Studer R. Formal concept analysis. Handbook on Ontologies. Berlin, Germany: Springer, 2009.177-99. doi:10.1007/978-3-540-92673-3_8
      OpenUrlCrossRef
    62. 62.↵
      1. Friedman C,
      2. Wong A,
      3. Blumenthal D
      . Achieving a nationwide learning health system. Science Transitional Medicine 2010;2(57):1-3. doi:10.1126/scitranslmed.3001456
      OpenUrlCrossRef
    63. 63.↵
      1. Deeny S,
      2. Steventon A
      . Making sense of the shadows: Priorities for creating a learning healthcare system based on routinely collected data. BMJ Quality Safety 2015;24505-15. doi:10.1136/bmjqs-2015-004278pmid:26065466; pmcid:PMC4515981
      OpenUrlAbstract/FREE Full Text
    64. 64.↵
      1. Bradley E,
      2. Curry LA,
      3. Ramanadhan S,
      4. Rowe L,
      5. Nembhard IM,
      6. Krumholz HM
      . Research in action: using positive deviance to improve quality of health care. BMC Implementation Science 2009;425
      OpenUrl
    65. 65.↵
      1. Lewis G,
      2. Kirkham H,
      3. Vaithianathan R
      . How health systems could avert ‘triple fail’ events that are harmful, are costly, and result in poor patient satisfaction. Health Affairs 2013;32(4):669-76. doi:10.1377/hlthaff.2012.1350pmid:23569046
      OpenUrlAbstract/FREE Full Text
    66. 66.↵
      1. Foley T,
      2. Fairmichael F
      . Site visit to Geisinger Health Systems. The Learning Healthcare Project 2015. Available from: http://www.learninghealthcareproject.org/publication/6/63/site-visit-to-geisinger-health-system. August 2017.
    67. 67.↵
      1. Garg A,
      2. Adhikari NK,
      3. Mcdonald H,
      4. Rosas-Arellano M,
      5. Devereaux PJ,
      6. Beyene J,
      7. Hayes RB
      . Effects of computerised clinical decision suport systems on practitioner performance and patient outcomes. Journal of the American Medical Association 2005;293(10):1223-38. doi:10.1001/jama.293.10.1223
      OpenUrlCrossRefPubMedWeb of Science
    68. 68.
      1. Wyatt J,
      2. Spiegelhalter D
      . Field trials of medical decision-aids: potential problems and solutions. Fifteenth Annual Symposium on Computer Applications in Medical Care. Washington, DC: American Medical Informatics Association, 1991.
    69. 69.↵
      1. Kawamoto K,
      2. Houlihan CA,
      3. Balas EA,
      4. Lobach DF
      . Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. British Medical Journal 2005;330765-8. doi:10.1136/bmj.38398.500764.8Fpmid:15767266; pmcid:PMC555881
      OpenUrlAbstract/FREE Full Text
    70. 70.↵
      1. Abernethy A,
      2. Ahmad A,
      3. Zafar SY,
      4. Wheeler JL,
      5. Reese JB,
      6. Lyerly HK
      . Electronic patient-reported data capture as a foundation of rapid learning cancer care. Medical Care 2010;48(6):S32-8. doi:10.1097/MLR.0b013e3181db53a4pmid:20473201
      OpenUrlCrossRefPubMedWeb of Science
    71. 71.↵
      1. Ye Y,
      2. Wamukoya M,
      3. Ezeh A,
      4. Emina JB,
      5. Sankoh O
      . Health and Demographic Surveillance Systems: a step towards full civil registration and vital statistics in sub-Saharan Africa?. BMC Public Health 2012;12741. doi:10.1186/1471-2458-12-741
      OpenUrlCrossRefPubMed
    72. 72.↵
      1. Ethier J
      . Integrating Resources for Translational Research: A Unified Approach for Learning Health Systems. Paris, France: Universite Pierre et Marie Curie, 2016.
    73. 73.↵
      1. Sacristan J
      . Patient-centered medicine and patient-oriented research: improving health outcomes for individual patients. BMC Medical Informatics and Decision Making 2013;136. doi:10.1186/1472-6947-13-6
      OpenUrlCrossRef
    74. 74.↵
      1. Mirnezami R,
      2. Nicholson J,
      3. Darzi A
      . Preparing for precision medicine. New England Journal of Medicine 2012;366(6):489-91. doi:10.1056/NEJMp1114866pmid:22256780
      OpenUrlCrossRefPubMedWeb of Science
    75. 75.↵
      1. Jameson J,
      2. Longo D
      . Precision medicine—personalized, problematic and promising. New England Journal of Medicine 2015;372(23):2229-34. doi:10.1056/NEJMsb1503104pmid:26014593
      OpenUrlCrossRefPubMed
    76. 76.↵
      1. Harwood J,
      2. Butler C,
      3. Page A
      . Patient-centred care and population health: establishing their role in orthapaedic practice. The Journal of Bone and Joint Surgery 2016;98(10):e40. doi:10.2106/JBJS.15.00752pmid:27194502
      OpenUrlCrossRefPubMed
    77. 77.↵
      1. Yu P,
      2. Artz D,
      3. Warner J
      . Electronic health records (EHRs): supporting ASCO’s vision of cancer care. ASCO Educational Book 2014;2014225-31
      OpenUrl
    78. 78.↵
      1. Cresswell,
      2. Bates D,
      3. Sheikh A
      . Ten key considerations for the successful optimisation of large-scale health information technology. Journal of the American Medical Informatics Association 2017;24(1):182-7
      OpenUrlCrossRefPubMed
    79. 79.↵
      1. Rioth M,
      2. Warner J,
      3. Savani BN,
      4. Jagasia M
      . Next generation long term transplant clinics: improving resource utilisation and the wuality of care through health information technology. Bone Marrow Transplantation 2016;51(1):34-40. doi:10.1038/bmt.2015.210pmid:26367235; pmcid:PMC4915825
      OpenUrlCrossRefPubMed
    80. 80.↵
      1. Pritchard D,
      2. Moeckel F,
      3. Villa MS,
      4. Housman LT,
      5. McCarty CA,
      6. McLeod HL
      . Strategies for integrating personalised medicine into healthcare practice. Personalised Medicine 2017;14(2):141-52
      OpenUrl
    81. 81.↵
      1. Chaudhry B,
      2. Wang J,
      3. Wu S,
      4. Maglione M,
      5. Mojica W,
      6. Roth E, et al
      . Systematic review: impact of health information technology on quality, efficiency and cost of medical care. Annals of Internal Medicine 2006;144(10):742-52. doi:10.7326/0003-4819-144-10-200605160-00125pmid:16702590
      OpenUrlCrossRefPubMedWeb of Science
    82. 82.↵
      1. Buntin M,
      2. Burke MF,
      3. Hoaglin MC,
      4. Blumenthal D
      . The benefits of health information technology: a review of the recent literature shows predominately positive results. Health Affairs 2011;30(3):464-71. doi:10.1377/hlthaff.2011.0178pmid:21383365
      OpenUrlAbstract/FREE Full Text
    83. 83.↵
      1. Yu P
      . Knowledge bases, clinical decision support systems, and rapid learning in oncology. Journal of Oncology Practice 2015;11(2):e206-11. doi:10.1200/JOP.2014.000620pmid:25715002
      OpenUrlAbstract/FREE Full Text
    84. 84.↵
      1. Zalon M
      . Technology and continuously learning health systems. The Journal of Continuing Education in Nursing 2016;47(6):243-5. doi:10.3928/00220124-20160518-01pmid:27232218
      OpenUrlCrossRefPubMed
    85. 85.↵
      1. Chorpita B,
      2. Daleiden E,
      3. Bernstein A
      . At the intersection of Health Information Technology and Decision Support: measurement feedback systems... and beyond. Administration and Policy in Mental Health and Mental Health Services Research 2016;43(3):471-7. doi:10.1007/s10488-015-0702-5; pmcid:PMC5300738
      OpenUrlCrossRef
    86. 86.↵
      1. Etheredge L
      . Rapid learning: a breakthrough Agenda. Health Affairs 2014;33(7):1155-62. doi:10.1377/hlthaff.2014.0043pmid:25006141
      OpenUrlAbstract/FREE Full Text
    87. 87.↵
      1. Fihn S,
      2. Francis J,
      3. Clancy C,
      4. Nielson C,
      5. Nelson K,
      6. Rumsfeld J, et al
      . Insights fron advanced analytics at the Veterans Affairs Health Administration. Health Affairs 2014;33(7):1203-11. doi:10.1377/hlthaff.2014.0054pmid:25006147
      OpenUrlAbstract/FREE Full Text
    88. 88.↵
      1. Angus D
      . Fusing randomised trials with big data: the key to self-learning health care systems. Journal of the American Medical Association 2015;314(8):767-8. doi:10.1001/jama.2015.7762pmid:26305643
      OpenUrlCrossRefPubMed
    89. 89.↵
      1. Dreyer N
      . Making observational studies count: shaping the future of comparitive effectiveness research. Epidemiology 2011;22(3):295-7. doi:10.1097/EDE.0b013e3182126569pmid:21464648
      OpenUrlCrossRefPubMed
    90. 90.↵
      1. Muir Gray J
      . The shift in personalised and population medicine. The Lancet 2013;382(9888):200-1
      OpenUrl
    91. 91.↵
      1. LeFevre M
      . From authority- to evidence-based medicine: are clinical practice guidelines moving us forwards or backwards?. Annals of Family Medicine 2017;15(5):410-12. doi:10.1370/afm.2141pmid:28893809
      OpenUrlFREE Full Text
    92. 92.↵
      1. Boonstra A,
      2. Broekhuis M
      . Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Health Services Research 2010;10(1):231
      OpenUrl
    93. 93.↵
      1. Hillstead R,
      2. Bigelow J,
      3. Bower A,
      4. Girosi F,
      5. Meili R,
      6. Scoville R, et al
      . Can electronic medical record systems transform health care? Potential health benefits, savings and costs. Health Affairs 2005;24(5):1103-17. doi:10.1377/hlthaff.24.5.1103pmid:16162551
      OpenUrlAbstract/FREE Full Text
    94. 94.↵
      AMAShared electronic medical records2016. Available from: https://myhealthrecord.gov.au/internet/mhr/publishing.nsf/content/home. Accessed 3 September 2017.
    95. 95.↵
      1. Davidson A
      . Creating value: unifying silos into public health business intelligence. eGEMS 2015;2(4):1172
      OpenUrl
    96. 96.↵
      1. Denaxas S,
      2. Friedman CP,
      3. Geissbuhler A,
      4. Hemingway H,
      5. Kalra D,
      6. Kimura M, et al
      . Discussion of “combining health data uses to ignite health system learning.”. Methods of Information in Medicine 2015;54(6):488-99. doi:10.3414/ME15-12-0004pmid:26538343
      OpenUrlCrossRefPubMed
    97. 97.↵
      1. Wilk A,
      2. Platt P
      . Measuring physicians’ trust: a scoping review with implications for public policy. Social Science and Medicine 2016;16575-81. doi:10.1016/j.socscimed.2016.07.039pmid:27497858
      OpenUrlCrossRefPubMed
    98. 98.↵
      1. Lean S,
      2. Guesgen HW,
      3. Hunter I,
      4. Tretiakov A
      . Working toward automated coding in general practice2016Palmerston North, New Zealand: Massey University
    99. 99.↵
      1. Faden R,
      2. Beauchamp T,
      3. Kass N
      . Informed consent, comparative effectiveness, and learning health care. New England Medical Journal 2014;370(8):766-8
      OpenUrl
    100. 100.
      1. Marsolo K,
      2. Margolis PA,
      3. Forrest CB,
      4. Colletti RB,
      5. Hutton JJ
      . A digital architecture for a network-based learning health system - Integrating chronic care management, quality improvement and research. eGEMS 2015;3(1):1168. doi:10.13063/2327-9214.1168pmid:26357665; pmcid:PMC4562738
      OpenUrlCrossRefPubMed
    101. 101.↵
      1. Nishimura A,
      2. Carey J,
      3. Erwin PJ,
      4. Tilburt JC,
      5. Murad MH,
      6. McCormick JB
      . Improving understanding in the research informed consent process: a systematic review of 54 interventions tested in randomized control trials. BMC Medical Ethics 2013;14(28):28. doi:10.1186/1472-6939-14-28
      OpenUrlCrossRefPubMed
    102. 102.↵
      1. Feeley T,
      2. Sledge GW,
      3. Levit L,
      4. Ganz PA
      . Improving the quality of cancer care in America through health information technology. Journal of the American Medical Informatics Association 2014;21772-5. doi:10.1136/amiajnl-2013-002346pmid:24352553; pmcid:PMC4147622
      OpenUrlCrossRefPubMed
    103. 103.↵
      1. Hughes K,
      2. Ambinder EP,
      3. Hess GP,
      4. Yu PP,
      5. Bernstam EV,
      6. Routbort MJ, et al
      . Identifying health information technology needs of oncologists to facilitate the adoption of genomic medicine: recommendations from the 2016 American society of clinical oncology omics and precision oncology workshop. Journal of Clinical Oncology 2017;353153-9
      OpenUrl

    Footnotes

    • Funding Scott McLachlan, William Marsh and Norman Fenton acknowledge support from the EPSRC under project EP/P009964/1: PAMBAYESIAN: Patient Managed decision-support using Bayes Networks. Henry W. W. Potts has received consultancy fees from Crystallise and The HELP Trust, and funding from myownteam and Shift.ms. Owen Johnson was supported by ClearPath Connected Health Cities Project.

    • Competing Interests None of the authors identified a competing interest relevant to this research.

    • Contributors William Marsh raised the topic, leading to Scott McLachlan performing the primary research and preparing the first draft of this paper. Kudakwashe Dube proposed formation of the framework, Owen Johnson developed the initial framework which was further enhanced by Derek Buchanan from the clinician’s perspective. Derek Buchanan and Bridget Daley provided review and input based on clinical experience. The initial method was proposed by Scott McLachlan, extended and significantly improved by Henry W. W. Potts, with Stephen Lean, Kudakwashe Dube and Thomas Gallagher providing input to the survey structure and framework. Thomas Gallagher, Kudakwashe Dube, Norman Fenton, Bridget Daley, Stephen Lean and Henry W. W. Potts provided editorial review. Norman Fenton and William Marsh supervised the research. All authors commented on and approved the final draft.