Mixed methods protocol for a realist evaluation of electronic personal health records design features and use to support medication adherence (ePHRma)
======================================================================================================================================================

* Elisavet Andrikopoulou
* Philip J Scott
* Helena Herrera

## Abstract

**Background** National Health Service policy suggests that increasing usage of electronic personal health records (PHR) by patients will result in cost savings and improved public health. Medication adherence means that patients take their prescribed medication as agreed with their doctors. Some of the claimed benefits of PHRs are decreasing healthcare costs and improving medication adherence and patient outcomes.

**Methods** This is a mixed methods convergent study, primarily qualitative. The qualitative and quantitative data collection and analysis will occur in parallel, and then be synthesised. We are interviewing and surveying adults with long-term conditions to identify what are the most important and useful features of their current PHR. The data collection comprises patient demographics, the Medication Adherence Questionnaire, the personality scale Big Five Inventory-2 Extra-Short Form and the WHO Quality of Life-BREF scale. Qualitative data will be analysed using the Framework method.

**Ethics** We have received a favourable ethical opinion from the Health Research Authority/Research Ethics Committee.

*   public health
*   medical informatics
*   primary health care
*   patient care
*   record systems

## Background

The WHO reported that medication adherence in patients with long-term conditions averages to 50% in developed countries.1 Based on the results of a 2017 US survey, roughly 40% of patients who are chronically ill were interested in using technology to assist them with medication, diagnosis, test results and managing their condition in their home environment.2 National Health Service (NHS) England policies published over the past decade such as Personalised Health and Care 2020 (PHC2020)3 and the Five Year Forward View4 specify that the NHS needs to use information and communication technologies (ICT) to reduce healthcare costs and improve healthcare outcomes. The most recent NHS policy document, *The NHS Long Term Plan*,5 focuses on ‘personalised healthcare’ to improve quality of life and public health and aspires that over the next 5 years with the introduction of further ICT, outpatient visits will drop by one-third. The aim is that people with long-term conditions such as diabetes, respiratory or renal problems will have further access to technology designed to help them manage their condition, for example, continuous glucose monitoring for all pregnant patients with type 1 diabetes.

Recent literature suggests that there is limited evidence that the use of health applications can improve patient adherence to prescribed medication, and the quality of the evidence is often questionable.6–8 The impact of ICT on cost, quality and safety of healthcare remains questionable, based on the conflicting evidence of more ‘optimistic’9–11 versus more cautious studies.6 

Medication adherence is ‘*the extent to which a person’s behaviour—taking medication, following a diet, and/or executing lifestyle changes, corresponds with agreed recommendations from a healthcare provider*’12 and is considered a well-known challenge in healthcare,13 acknowledged by PHC2020.3 The ABC taxonomy14 is selected as the conceptual framework for medication adherence in this study, since it is well cited and it is considered more comprehensive than the WHO Five interacting dimensions that affect adherence,12 which will still inform this study as a secondary source.

It is widely suggested in the literature that medication adherence should be measured in conjunction with quality of life in order to define whether a patient’s health is better.15 16 Health-related quality of life (HRQoL) is the ‘*subjective assessment of the impact of disease and treatment across the physical, psychological, social and somatic domains of functioning and well-being*’.13 Literature also suggests that patient personality traits often affect medication adherence.17 18 The Five-Factor Model is an established taxonomy of personality traits.19 According to it, personality can be described in terms of five basic personality trait dimensions: agreeableness, conscientiousness, extraversion, neuroticism and openness to experience.20 21 

The NHS standards for commissioning personal health records (PHR)22 provide guidance on good practice for the development of PHRs in England, but they do not provide enough evidence on how the PHR standards impact public health, nor on what design features should a PHR include nor evidence on how these features impact health outcomes. PHRs are ‘*online systems that include collections of patients’ healthcare and medical data, which utilise health informatics standards to enable patients to share, organize and manage these data according to their own views*’.7 Some of the many claimed benefits of PHRs are the ability of PHR to improve patient outcomes, decrease healthcare costs, allow patients the ability to self-manage their health, empower patients and improve medication adherence.23–25 

### Aim and objectives

The aim of this study is to determine how best computerised PHR features should be designed to help patients get the full benefit of their prescribed medication. It could be hypothesised that health and information technology literacy may be important factors in identifying these essential PHR design features. Health literacy is ‘*the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions*’.26 The Health Education England defines digital literacy as ‘the capabilities that fit someone for living, learning, working, participating and thriving in a digital society’.27 

#### Primary objective

Identify the essential design features of PHRs to improve medication adherence in adults with long-term conditions.

#### Secondary objectives

*   Identify how patient and disease-specific factors mediate the impact of PHRs.
    
    *   Patient specific: personality traits and sociodemographics.
    
    *   Disease specific: progression, severity, intervention type, polypharmacy.

*   Develop a theoretical model that describes the interaction between the PHR design features and the patient and disease-specific factors, to help determine what works for whom in what circumstances.

## Methods

In the absence of a comprehensive checklist for our mixed methods study design, we have adopted the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement28 given that our study is observational. The STROBE statement covers cohort, case–control and cross-sectional studies and we will follow the relevant statement’s sections about the cohort (for our qualitative data) and cross-sectional (for our quantitative data) studies.

### Study design

This study will follow the mixed methods research methodology, based on a pragmatist perspective, which is considered appropriate in the literature for medical informatics.29 30 The convergent mixed methods design31 will be used. The convergent design (or parallel or concurrent32 33) is parallel and both qualitative and quantitative data are gathered at the same time, are analysed separately and then synthesised. The purpose of this design is to gather complementary data for the topic and aims to mitigate the pitfalls of solely quantitative or qualitative approaches, by illustrating quantitative and qualitative results together and examining relationships between variables. The results can then be merged and compared to answer the study’s aims and objectives.31 34 In this study, the qualitative data have been identified as dominant,34 as the area is largely under-researched and complex. The quantitative data will supplement and indicate the extent with which findings from the qualitative work are present. Figure 1 illustrates the different stages of the research and a more detailed explanation follows.

![Figure 1](http://informatics.bmj.com/https://informatics.bmj.com/content/bmjhci/27/1/e100046/F1.medium.gif)

[Figure 1](http://informatics.bmj.com/content/27/1/e100046/F1)

Figure 1 
Study design. BFI-2-XS, Big Five Inventory-2 Extra-Short Form; MAQ, Medication Adherence Questionnaire; WHOQOL-BREF, WHO Quality of Life-BREF.

This study’s evaluation is based on the Medical Research Council guidance on developing and evaluating complex interventions35 that emphasises the importance of assessing the effectiveness and the outcomes of complex interventions contextually in interventions that are currently implemented and cannot be reversed.

This study will use the realist evaluation framework36 as its overarching theoretical approach to identify the optimal configurations of context, mechanism and outcomes in the use of PHRs for medicine optimisation. A realist evaluation includes a contextual evaluation (in what circumstances),37 a process evaluation (what works for whom)38 and outcomes evaluation (how). This holistic evaluation approach makes realist evaluation framework an ideal candidate to deal with the uncertainty and the complexity of interventions in medical informatics.39 40 

The evaluation36 will begin with the construction of an initial programme theory (table 1), informed by the literature review,7 8 which identifies the outcomes that are derived from the intervention and what mechanisms are in place to generate these outcomes as well as what context will affect them. The context–mechanism–outcome (CMO) configuration will be used as analytical approach, to identify mechanisms and contextual factors that are associated with the variation in outcomes.36 The study will iteratively revise the programme theory complying with the realist evaluation principles.

View this table:
[Table 1](http://informatics.bmj.com/content/27/1/e100046/T1)

Table 1 
Initial programme theory (acronyms explained and further details provided in the Measures section)

### Sampling and recruitment

Based on the study’s intention to directly relate the data sets, as well as to avoid influencing our ability to converge the results, we will use the same individuals to participate in both the quantitative and the qualitative strands of the study.34 The primary data of this study are qualitative; thus the sampling strategy is based on the qualitative component of the study, which will be merged with the quantitative data strand. A purposive convenient sample34 of participants will be recruited from volunteers through a variety of methods. The ‘gold standard’ for a purposive sample is to achieve saturation, which is impossible to predict. Therefore, based on the aims and objectives of this study and the type of sampling scheme selected, a typical recommendation is medium sample size, roughly 30–40 participants to achieve theoretical saturation.34 

Posters will be displayed in general practitioner surgeries, pharmacies in Hampshire and University of Portsmouth communal areas. Recruitment flyers will be electronically distributed by social media and via groups and forums of patients who are chronically ill, such as Diabetes UK. A project website will be created that will also make available the flyer and information sheets. Furthermore, more volunteers will be recruited via charities and other universities and colleges in South East England.

### Participants

An electronic consent form and demographic information including their age and gender will be collected prior to the interviews and survey. The recruitment will occur in two stages. The first stage will not specify a long-term condition; rather we will interview anyone who complies with the inclusion criteria and consents. When we reach 15 (40%) of the intended participants, we will select the three majority long-term conditions, as was advised by the initial patient and public involvement (PPI) focus group. The participant recruitment will finish in August 2019.

#### Inclusion criteria

*   Participant is willing and able to provide informed consent for participation in the study.

*   Male or female, aged 18 years or above.

*   Participant has a long-term condition and is treated outside hospital.

*   Participant is able to self-administer his/her medication.

*   Participant is frequently (at least once a week) using a PHR to manage in full or partially his/her medication.

*   Participant is able to communicate freely.

#### Exclusion criteria

*   Participants who are pregnant or terminally ill or considered vulnerable adults and patients with cancer.

*   Adults with medically serious problems that are not classified as long-term conditions.

*   Patients require assistance with taking their medication.

*   Patients unable to communicate or unable to self-manage their medication.

*   Inpatients or patients living in care homes.

### Measures

This study will gather qualitative and quantitative data from the same population during an online/telephone interview, and will collect extra quantitative data via an online survey.

#### Quantitative data

The Medication Adherence Questionnaire (MAQ) scale will be used to measure medication adherence, which is a free-to-use four-item scale and has sensitivity (81%), specificity (44%) and Cronbach’s alpha reliability of 0.61.41 The MAQ scale was preferred over the more reliable and widely used 8-item Morisky Medication Adherence Scale (MMAS-8)42 43 or the 13-item Self-Efficacy for Appropriate Medication Use Scale.44 Both instruments were regarded as too long by our PPI group, additionally MMAS-8 is a licence product and this study is not resourced for this expenditure.

The WHO Quality of Life-BREF (WHOQOL-BREF)45 scale will be used to measure HRQoL, which is a free licence-based valuable scale for measuring quality of life in multiple chronic conditions, scoring 0.92 internal consistency reliability and has validity similar to the 36-Item Short Form Survey (SF-36) scale.45 The WHOQOL-BREF was preferred over other well-cited measures of HRQoL which include the SF-36,46 EuroQoL EQ-5D47 and the WHOQOL-100,45 purely for brevity following the PPI advice.

The shortest available form of the Big Five Inventory (BFI) questionnaire that is used to measure personality traits, namely the Big Five Inventory-2 Extra-Short Form (BFI-2-XS), was selected for this study. Although the BFI-2-XS has a lower psychometric scope48 than the complete BFI questionnaire,49 it appears to be adequate for this study based on PPI advice.

The sociodemographic information that we intend to gather is age group, gender, education, domestic status and employment status. We will also derive the overall participant’s digital and health literacy from the interviews. There are a number of ways to measure digital and health literacy,50 51 however, we have chosen not to use further scales, based on the PPI advice regarding potential participation fatigue.52 We will simply infer the degree of health and digital literacy as high, medium or low, based on generic categorisations in the literature.27 51 Health literacy will be inferred from the questions 1, 4 and 5, and digital literacy from the questions 1 and 5 of the interview (see online supplementary file 1), as well as the employment and education status of the participant.27 50 The guideline that we use to infer the health and digital literacy is provided in the online supplementary file 2.

### Supplementary data

[[bmjhci-2019-100046supp001.pdf]](pending:yes) 

### Supplementary data

[[bmjhci-2019-100046supp002.xlsx]](pending:yes) 

#### Qualitative data

Qualitative dimensions of the concepts of effectiveness of PHR design features with medication adherence will be assessed with a semistructured interview (see online supplementary file 1). Questions that infer the medication adherence and quality of life are included in the interview, to strengthen our methodological and analytical approach.34 

### Analysis

The qualitative data analysis will use the Framework method,53 which will enable themes to be developed inductively (from the interviews) but also incorporating themes identified in the literature review.7 The Framework method is systematic, thorough, data driven but also flexible and enables visual representation of the data.53 54 There are step-by-step guides to the application of the Framework method and this study intends to follow the guidelines of Gale *et al*.54 

We will use NVivo V.12 for coding. To achieve coding inter-rater reliability, a subset of files (25%) will be coded by an independent researcher and then the themes will be compared and the inter-rater reliability score will be calculated.54 After coding the 50% of the transcripts, a framework will be developed in an iterative manner, to apply to the rest of the transcripts. The framework will include codes that are grouped together and are clearly defined. The interpretation of the data will occur in conjunction with the quantitative strand during data merging.

We will use Microsoft Excel 2016 for descriptive statistics; namely mean, minimum, maximum, SD and coefficient of variance.

Joint displays will be used in a fashion similar to Vaughan Dickson *et al*,55 so as the quantitative and qualitative results will be analysed thematically based on the CMO configuration to identify the essential design features of PHRs to improve medication adherence in adults with long-term conditions, taking into consideration the different personality types, adherence levels and quality of life opinions of the participants.

### Patient and public involvement

A PPI focus group with eight participants took place at the University of Portsmouth in June 2018. This group’s suggestions are taken into consideration by the research team. The study protocol has been also reviewed and approved by this PPI group. PPI will be involved throughout this research, in order to minimise bias, increase transparency and ensure approval.

### Limitations

This study is limited to the design features of a PHR and not the overall design. There are no explicit questions in the interview guide regarding usability or interface design. Another limitation is that the sample size of this study is limited to 30–40 participants. The dominant data of this study are the qualitative and it is currently acceptable in broad literature to interview approximately 40 people.34 56 Due to time constraints a higher number would be unrealistic.

## Acknowledgments

The authors acknowledge the participants of the PPI group for their insightful feedback and time.

## Footnotes

*   Twitter @Prayance

*   Contributors EA drafted the manuscript, and is the guarantor of the protocol. PJS revised the manuscript multiple times for methodological and intellectual content. HH revised the manuscript twice for methodological, conceptual and intellectual content from a pharmaceutical’s perspective.

*   Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

*   Competing interests None declared.

*   Patient consent for publication Not required.

*   Ethics approval This research has been approved by the NHS North East-Newcastle & North Tyneside 2 Research Ethics Committee (10 September 2018). The findings will be published in peer-reviewed journals and presented at national and international scientific conferences.

*   Provenance and peer review Not commissioned; externally peer reviewed.

*   Data availability statement There are no data in this work

*   Received May 16, 2019.
*   Revision received October 10, 2019.
*   Accepted December 17, 2019.


*   © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

[http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/).

## References

1.  Who. Defining adherence, 2003: 1–28. [www.who.int/chp/knowledge/publications/adherence\_report/en/](http://www.who.int/chp/knowledge/publications/adherence_report/en/). (accessed 25 Oct 2017).
    
    

2.  Deloitte. 2017 global health care sector outlook, 2017. Available: [https://www2.deloitte.com/content/dam/Deloitte/global/Documents/Life-Sciences-Health-Care/gx-lshc-2015-health-care-outlook-infographic.pdf](https://www2.deloitte.com/content/dam/Deloitte/global/Documents/Life-Sciences-Health-Care/gx-lshc-2015-health-care-outlook-infographic.pdf) [Accessed 10 Nov 2018].
    
    

3.  National Information Board,Department of Health. Personalised health and care 2020: a framework for action, 2014. Available: [https://www.gov.uk/government/publications/personalised-health-and-care-2020/using-data-and-technology-to-transform-outcomes-for-patients-and-citizens](https://www.gov.uk/government/publications/personalised-health-and-care-2020/using-data-and-technology-to-transform-outcomes-for-patients-and-citizens) [Accessed 11 Feb 2018].
    
    

4.  Public Health England - Trust Development Authority. NHS England care quality Commission health education England monitor. five year forward view, 2014. Available: [https://www.england.nhs.uk/wp-content/uploads/2014/10/5yfv-web.pdf](https://www.england.nhs.uk/wp-content/uploads/2014/10/5yfv-web.pdf) 
    
    

5.  NHS. The NHS long term plan, 2019. Available: [www.longtermplan.nhs.uk](http://www.longtermplan.nhs.uk) [Accessed 2 May 2019].
    
    

6.   Scott P . Exploiting the information revolution: call for independent evaluation of the latest English national experiment. J Innov Health Inform 2015;22:244–9.[doi:10.14236/jhi.v22i1.139](http://dx.doi.org/10.14236/jhi.v22i1.139) 
    
    

7.   Andrikopoulou E , Scott PJ , Herrera H . Important design features of personal health records to improve medication adherence for patients with long-term conditions: protocol for a systematic literature review. JMIR Res Protoc 2018;7:e159.[doi:10.2196/resprot.9778](http://dx.doi.org/10.2196/resprot.9778) 
    
    

8.   Andrikopoulou E , Scott P , Herrera H , et al . What are the important design features of personal health records to improve medication adherence for patients with long-term conditions? A systematic literature review. BMJ Open 2019;9:e028628.[doi:10.1136/bmjopen-2018-028628](http://dx.doi.org/10.1136/bmjopen-2018-028628) 
    
    

9.   Hillestad R , Bigelow J , Bower A , et al . Can electronic medical record systems transform health care? potential health benefits, savings, and costs. Health Aff 2005;24:1103–17.[doi:10.1377/hlthaff.24.5.1103](http://dx.doi.org/10.1377/hlthaff.24.5.1103) 
    
    [Abstract/FREE Full Text](http://informatics.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiaGVhbHRoYWZmIjtzOjU6InJlc2lkIjtzOjk6IjI0LzUvMTEwMyI7czo0OiJhdG9tIjtzOjI1OiIvYm1qaGNpLzI3LzEvZTEwMDA0Ni5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

10.  George TP , Hopla DL . Advantages of personal health records. Nurs Crit Care 2015;10:10–12.[doi:10.1097/01.CCN.0000472852.70431.1b](http://dx.doi.org/10.1097/01.CCN.0000472852.70431.1b) 
    
    

11.  Tang PC , Ash JS , Bates DW , et al . Personal health records: definitions, benefits, and strategies for overcoming barriers to adoption. J Am Med Inform Assoc 2006;13:121–6.[doi:10.1197/jamia.M2025](http://dx.doi.org/10.1197/jamia.M2025) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1197/jamia.M2025&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=16357345&link_type=MED&atom=%2Fbmjhci%2F27%2F1%2Fe100046.atom) 

12.  De Geest S , Sabaté E . Adherence to long-term therapies: evidence for action. Eur J Cardiovasc Nurs 2003;2:323.[doi:10.1016/S1474-5151(03)00091-4](http://dx.doi.org/10.1016/S1474-5151(03)00091-4) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/S1474-5151(03)00091-4&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=14667488&link_type=MED&atom=%2Fbmjhci%2F27%2F1%2Fe100046.atom) 

13.  Revicki DA , Osoba D , Fairclough D , et al . Recommendations on health-related quality of life research to support labeling and promotional claims in the United States. Qual Life Res 2000;9:887–900.[doi:10.1023/A:1008996223999](http://dx.doi.org/10.1023/A:1008996223999) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1023/A:1008996223999&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=11284208&link_type=MED&atom=%2Fbmjhci%2F27%2F1%2Fe100046.atom) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000166905000001&link_type=ISI) 

14.  Vrijens B , De Geest S , Hughes DA , et al . A new taxonomy for describing and defining adherence to medications. Br J Clin Pharmacol 2012;73:691–705.[doi:10.1111/j.1365-2125.2012.04167.x](http://dx.doi.org/10.1111/j.1365-2125.2012.04167.x) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1111/j.1365-2125.2011.04167.x&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=22486599&link_type=MED&atom=%2Fbmjhci%2F27%2F1%2Fe100046.atom) 

15.  Haynes RB , Mattson ME , Chobanian A . Management of patient compliance in the treatment of hypertension. Report of the NHLBI Working group. Hypertension 1982;4:415–23.[doi:10.1161/01.hyp.4.3.415](http://dx.doi.org/10.1161/01.hyp.4.3.415) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1161/01.HYP.4.3.415&link_type=DOI) 

16.  Haynes RB , Ackloo E , Sahota N , et al . Interventions for enhancing medication adherence. Cochrane Database Syst Rev 2008:CD000011.[doi:10.1002/14651858.CD000011.pub3](http://dx.doi.org/10.1002/14651858.CD000011.pub3) 
    
    

17.  Axelsson M , Brink E , Lundgren J , et al . The influence of personality traits on reported adherence to medication in individuals with chronic disease: an epidemiological study in West Sweden. PLoS One 2011;6:e18241–7.[doi:10.1371/journal.pone.0018241](http://dx.doi.org/10.1371/journal.pone.0018241) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1371/journal.pone.0018241&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=21464898&link_type=MED&atom=%2Fbmjhci%2F27%2F1%2Fe100046.atom) 

18.  Gellad WF , Grenard JL , Marcum ZA . A systematic review of barriers to medication adherence in the elderly: looking beyond cost and regimen complexity. Am J Geriatr Pharmacother 2011;9:11–23.[doi:10.1016/j.amjopharm.2011.02.004](http://dx.doi.org/10.1016/j.amjopharm.2011.02.004) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/j.amjopharm.2011.02.004&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=21459305&link_type=MED&atom=%2Fbmjhci%2F27%2F1%2Fe100046.atom) 

19.  Costa PT , McCrae RR . The revised Neo personality inventory (NEO-PI-R), 2008.
    
    

20.  McCrae RR , John OP . An introduction to the five-factor model and its applications. J Pers 1992;60:175–215.[doi:10.1111/j.1467-6494.1992.tb00970.x](http://dx.doi.org/10.1111/j.1467-6494.1992.tb00970.x) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1111/j.1467-6494.1992.tb00970.x&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=1635039&link_type=MED&atom=%2Fbmjhci%2F27%2F1%2Fe100046.atom) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=A1992JB39500001&link_type=ISI) 

21.  Costa Jr. PT , McCrae RR . Domains and facets: hierarchical personality assessment using the revised Neo personality inventory. J Pers Assess 1995;64:21–50.[doi:10.1207/s15327752jpa6401\_2](http://dx.doi.org/10.1207/s15327752jpa6401_2) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1207/s15327752jpa6401_2&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=16367732&link_type=MED&atom=%2Fbmjhci%2F27%2F1%2Fe100046.atom) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=A1995QE82600002&link_type=ISI) 

22. NHS Digital. Standards for commissioning or developing personal health records, 2018. Available: [https://digital.nhs.uk/services/personal-health-records-adoption-service/personal-health-records-adoption-toolkit/developing-a-personal-health-record/standards-for-developing-personal-health-records](https://digital.nhs.uk/services/personal-health-records-adoption-service/personal-health-records-adoption-toolkit/developing-a-personal-health-record/standards-for-developing-personal-health-records) [Accessed 1 Feb 2019].
    
    

23.  Chrischilles EA , Hourcade JP , Doucette W , et al . Personal health records: a randomized trial of effects on elder medication safety. J Am Med Inform Assoc 2014;21:679–86.[doi:10.1136/amiajnl-2013-002284](http://dx.doi.org/10.1136/amiajnl-2013-002284) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1136/amiajnl-2013-002284&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=24326536&link_type=MED&atom=%2Fbmjhci%2F27%2F1%2Fe100046.atom) 

24.  Heart T , Ben-Assuli O , Shabtai I . A review of PHR, EMR and EHR integration: a more personalized healthcare and public health policy. Health Policy Technol 2017;6:20–5.[doi:10.1016/j.hlpt.2016.08.002](http://dx.doi.org/10.1016/j.hlpt.2016.08.002) 
    
    

25. Institute of Medicine. Health it and patient safety. Washington, D.C.: National Academies Press, 2012.
    
    

26. 1.  
    2.   Selden CR , 
    3.   Zorn M , 
    4.   Ratzan SC  
    
     Ratzan SC , Parker R . Introduction. In:  Selden CR ,  Zorn M ,  Ratzan SC  , eds. National library of medicine current bibliographies in medicine: health literacy. NLM PUB. No. CBM 2000-1. Bethesda, MD: National Institutes of Health: U.S. Department of Health and Human Services, 2000.
    
    

27. Health Education England. A health and care digital capabilities framework, 2018. Available: [https://hee.nhs.uk/sites/default/files/documents/Digital Literacy Capability Framework 2018.pdf](https://hee.nhs.uk/sites/default/files/documents/Digital%20Literacy%20Capability%20Framework%202018.pdf) 
    
    

28.  von Elm E , Altman DG , Egger M , et al . The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med 2007;4:e296–7.[doi:10.1371/journal.pmed.0040296](http://dx.doi.org/10.1371/journal.pmed.0040296) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1371/journal.pmed.0040296&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=17941714&link_type=MED&atom=%2Fbmjhci%2F27%2F1%2Fe100046.atom) 

29.  Piller Y . Pragmatism as a valid philosophical paradigm in social research 2015.
    
    

30.  Scott PJ , Briggs JS . A Pragmatist argument for mixed methodology in medical informatics. J Mix Methods Res 2009;3:223–41.[doi:10.1177/1558689809334209](http://dx.doi.org/10.1177/1558689809334209) 
    
    

31.  Robson C , McCartan K . Real world research : a resource for users of social research methods in applied settings. Chichester, West Sussex, United Kingdom: Wiley, 2016.
    
    

32.  Bryman A . Social research methods. Oxford, United Kingdom: Oxford University Press, 2016.
    
    

33.  Teddlie C , Tashakkori A . Foundations of mixed methods research: integrating quantitative and qualitative approaches in the social and behavioral sciences. Los Angeles, [Calif.] ; London: SAGE, 2009.
    
    

34.  Creswell JW , Plano Clark VL . Designing and conducting mixed methods research. Los Angeles: SAGE, 2017.
    
    

35.  Craig P , Dieppe P , Macintyre S , et al . Developing and evaluating complex interventions: the new medical Research Council guidance. Int J Nurs Stud 2013;50:587–92.[doi:10.1016/j.ijnurstu.2012.09.010](http://dx.doi.org/10.1016/j.ijnurstu.2012.09.010) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/j.ijnurstu.2012.09.010&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=23159157&link_type=MED&atom=%2Fbmjhci%2F27%2F1%2Fe100046.atom) 

36.  Pawson R . Evidence-based policy: a realist perspective. London: SAGE, 2006.
    
    

37.  Parry GJ , Carson-Stevens A , Luff DF , et al . Recommendations for evaluation of health care improvement initiatives. Acad Pediatr 2013;13:S23–30.[doi:10.1016/j.acap.2013.04.007](http://dx.doi.org/10.1016/j.acap.2013.04.007) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/j.acap.2013.04.007&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=24268081&link_type=MED&atom=%2Fbmjhci%2F27%2F1%2Fe100046.atom) 

38.  Craig P , Dieppe P , Macintyre S , et al . Developing and evaluating complex interventions: the new medical Research Council guidance. BMJ 2008;337:a1655–83.[doi:10.1136/bmj.a1655](http://dx.doi.org/10.1136/bmj.a1655) 
    
    [FREE Full Text](http://informatics.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzMzcvc2VwMjlfMS9hMTY1NSI7czo0OiJhdG9tIjtzOjI1OiIvYm1qaGNpLzI3LzEvZTEwMDA0Ni5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

39.  Nurjono M , Shrestha P , Lee A , et al . Realist evaluation of a complex integrated care programme: protocol for a mixed methods study. BMJ Open 2018;8:e017111.[doi:10.1136/bmjopen-2017-017111](http://dx.doi.org/10.1136/bmjopen-2017-017111) 
    
    

40.  Scott P . Mixed methods: a paradigm for holistic evaluation of health it. in: evidence-based health informatics: promoting safety and efficiency through scientific methods and ethical policy. IOS Press, 2016: 102–13.
    
    

41.  Morisky DE , Ang A , Krousel-Wood M , et al . Predictive validity of a medication adherence measure in an outpatient setting. J Clin Hypertens 2008;10:348–54.[doi:10.1111/j.1751-7176.2008.07572.x](http://dx.doi.org/10.1111/j.1751-7176.2008.07572.x) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1111/j.1751-7176.2008.07572.x&link_type=DOI) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000261099700003&link_type=ISI) 

42.  Lam WY , Fresco P . Medication adherence measures: an overview. Biomed Res Int 2015;2015:1–12.[doi:10.1155/2015/217047](http://dx.doi.org/10.1155/2015/217047) 
    
    

43.  Wong MCS , Liu J , Zhou S , et al . The association between multimorbidity and poor adherence with cardiovascular medications. Int J Cardiol 2014;177:477–82.[doi:10.1016/j.ijcard.2014.09.103](http://dx.doi.org/10.1016/j.ijcard.2014.09.103) 
    
    

44.  Risser J , Jacobson TA , Kripalani S . Development and psychometric evaluation of the self-efficacy for appropriate medication use scale (SEAMS) in low-literacy patients with chronic disease. J Nurs Meas 2007;15:203–19.[doi:10.1891/106137407783095757](http://dx.doi.org/10.1891/106137407783095757) 
    
    [Abstract/FREE Full Text](http://informatics.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Njoic2dyam5tIjtzOjU6InJlc2lkIjtzOjg6IjE1LzMvMjAzIjtzOjQ6ImF0b20iO3M6MjU6Ii9ibWpoY2kvMjcvMS9lMTAwMDQ2LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

45.  Skevington SM , McCrate FM . Expecting a good quality of life in health: assessing people with diverse diseases and conditions using the WHOQOL-BREF. Heal Expect 2012;15:49–62.[doi:10.1111/j.1369-7625.2010.00650.x](http://dx.doi.org/10.1111/j.1369-7625.2010.00650.x) 
    
    

46. RAND. 36-Item short form survey instrument (SF-36), 2016. Available: [http://www.rand.org/health/surveys_tools/mos/36-item-short-form.html](http://www.rand.org/health/surveys_tools/mos/36-item-short-form.html) 
    
    

47.  Weldring T , Smith SMS . Patient-reported outcomes (pros) and patient-reported outcome measures (PROMs). Health Serv Insights 2013;6:61–8.[doi:10.4137/HSI.S11093](http://dx.doi.org/10.4137/HSI.S11093) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.4137/HSI.S11093&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=25114561&link_type=MED&atom=%2Fbmjhci%2F27%2F1%2Fe100046.atom) 

48.  Soto CJ , John OP . Short and extra-short forms of the big five Inventory–2: the BFI-2-S and BFI-2-XS. J Res Pers 2017;68:69–81.[doi:10.1016/j.jrp.2017.02.004](http://dx.doi.org/10.1016/j.jrp.2017.02.004) 
    
    

49.  Benet-Martínez V , John OP . Los Cinco Grandes across cultures and ethnic groups: Multitrait-multimethod analyses of the big five in Spanish and English. J Pers Soc Psychol 1998;75:729–50.[doi:10.1037/0022-3514.75.3.729](http://dx.doi.org/10.1037/0022-3514.75.3.729) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1037/0022-3514.75.3.729&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=9781409&link_type=MED&atom=%2Fbmjhci%2F27%2F1%2Fe100046.atom) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000076224900011&link_type=ISI) 

50.  Flaherty MG , Flaherty MG . Health literacy: an overview of an emerging field. SLIS Student Res J 2011;1:4.
    
    

51.  ECORYS UK . Digital skills for the UK economy, 2016. Available: [https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment\_data/file/492889/DCMSDigitalSkillsReportJan2016.pdf](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/492889/DCMSDigitalSkillsReportJan2016.pdf) 
    
    

52.  Galesic M , Bosnjak M . Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opin Q 2009;73:349–60.[doi:10.1093/poq/nfp031](http://dx.doi.org/10.1093/poq/nfp031) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1093/poq/nfp031&link_type=DOI) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000266968000007&link_type=ISI) 

53. 1.  
    2.   Huberman AM , 
    3.   Miles MB  
    
     Ritchie J , Spencer L . Qualitative data analysis for applied policy research. In:  Huberman AM ,  Miles MB  , eds. The qualitative researcher’s companion. Thousand Oaks, California: SAGE Publications, Inc, 2002: 305–29.
    
    

54.  Gale NK , Heath G , Cameron E , et al . Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol 2013;13:117.[doi:10.1186/1471-2288-13-117](http://dx.doi.org/10.1186/1471-2288-13-117) 
    
    

55.  Vaughan Dickson V , Lee CS , Riegel B . How do cognitive function and knowledge affect heart failure self-care? J Mix Methods Res 2011;5:167–89.[doi:10.1177/1558689811402355](http://dx.doi.org/10.1177/1558689811402355) 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1177/1558689811402355&link_type=DOI) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000290434200005&link_type=ISI) 

56. 1.  
    2.   Tashakkori A , 
    3.   Charles T  
    
     Collins KMT . Advanced Sampling Designs in Mixed Research: Current Practices and Emerging Trends in the Social and Behavioral Sciences. In:  Tashakkori A ,  Charles T  , eds. SAGE Handbook of Mixed Methods in Social & Behavioral Research. SAGE Publications, 2010: 353–78.