Digital literacy knowledge and needs of pharmacy staff: A systematic review
===========================================================================

* Katie MacLure
* Derek Stewart

## Abstract

**Objective** To explore the digital literacy knowledge and needs of pharmacy staff including pharmacists, graduate (pre-registration) pharmacists, pharmacy technicians, dispensing assistants and medicine counter assistants.

**Methods** A systematic review was conducted following a pre-published protocol. Two reviewers systematically performed the reproducible search, followed by independent screening of titles/abstracts then full papers, before critical appraisal and data extraction. Full articles matching the search terms were eligible for inclusion. Exclusions were recorded with reasons. Kirkpatrick’s 4 level model of training evaluation (reaction, learning, behaviour and results) was applied as an analytical framework.

**Results** Screening reduced the initial 86 papers to 5 for full review. Settings included hospital and community pharmacy plus education in Australia, Canada and the US. No studies of pharmacy staff other than pharmacists were identified. Main findings indicate that pharmacy staff lack digital literacy knowledge with minimal research evidenced at each level of Kirkpatrick’s model.

**Conclusions** As a society, we acknowledge that technology is an important part of everyday life impacting on the efficiency and effectiveness of working practices but, in pharmacy, do we take cognisance, ‘that technology can change the nature of work faster than people can change their skills’? It seems that pharmacy has embraced technology without recognised occupational standards, definition of baseline skills or related personal development plans. There is little evidence that digital literacy has been integrated into pharmacy staff training, which remains an under-researched area.

*   digital literacy
*   Kirkpatrick’s 4 level model
*   pharmacy education
*   systematic review
*   training

## INTRODUCTION

### Background

Pharmacy staff across all practice settings are reliant on information technology (IT).1–4 Pharmacists, graduate (pre-registration) pharmacists, pharmacy technicians, dispensing assistants and medicine counter assistants use widely available office, retail and management information systems alongside dedicated pharmacy management and electronic health (ehealth) applications in a range of community, hospital and other pharmacy settings. The abilities of pharmacy staff to use these applications at home and at work, also known as digital literacy or digital competence or e-skills, depend on personal experience and related education and training.5–8 The British Computer Society defines digital literacy as, ‘being able to make use of technologies to participate in and contribute to modern social, cultural, political and economic life’.9 A similar definition of digital literacy is adopted in the US, ‘the ability to use information and communication technologies to find, evaluate, create, and communicate information; it requires both technical and cognitive skills’.10 Both definitions are grounded in historical and conceptual definitions of digital literacies.11

IT facilitates the provision of core pharmacy services in the UK in collaboration with other healthcare professionals with similar examples worldwide.12–15 In the US, digital literacy also forms the basis for pharmacy led health literacy as a tool for improving public health and patient outcomes.16

Collaborative working in health has been viewed as both beneficial to patients and a more efficient use of health professionals’ skills since long before the advent of ehealth.17–19 Health strategists worldwide promote the adoption of IT and ehealth to support patient care through collaborative working, which is tracked globally by World Health Organization.1–3,12,13,20–22 Both the adoption of ehealth and standards of digital literacy at home and in the workplace are key themes of interest at the international level.23–25

United Nations Educational Scientific and Cultural Organisation identifies digital literacy as both a ‘life skill’ and ‘gate skill’ because ‘it targets all areas of contemporary existence’.26 The European Commission Information Society promotes and tracks citizens’ and member states’ digital engagement.27,28 Similarly, the European Parliament promotes digital literacy for lifelong learning along with a recommendation for ‘better identification of occupational needs’.29

In the US, a government initiative to create a ‘digital nation’ recognised the role of digital literacy in promoting inclusion.23 A government commissioned report into digital literacy in Australia concluded that ‘both citizen and worker will need to be digitally literate for the digital economy to work effectively’ while a report from New Zealand argues ‘that technology can change the nature of work faster than people can change their skills’.24,25

In the UK, a range of strategic principles, national competency frameworks for training, core skills and digital literacies for the general public, and recently more specific targets for the health sector, have been developed by the government, advisory and professional bodies.30–36

Pharmacy students in the UK undertake the General Pharmaceutical Council accredited and regulated Master of Pharmacy course.5 This undergraduate university course is followed by a pre-registration year, based in practice culminating in an end of year written examination. Training for pharmacy technicians, dispensing assistants and medicines counter assistants is similarly accredited and regulated but undertaken as a combination of practical experience, college and open learning.6 Specific mention of the digital literacies required to facilitate pharmacy staff’s collaborative healthcare role is not evident in the UK curricula for initial training or their continuing professional development (CPD).37,38 The Accreditation Council for Pharmacy Education (ACPE) in America added health informatics to its standards for pharmacist training in 2007. The focus was initially on basic computer skills and then on ability to find appropriate relevant evidence base for practice.39 Pharmacy program accreditation in Canada, Australia and New Zealand make explicit mention of the need to prepare students to make best ‘use of information technology in pharmacy and more widely in health care’.40,41 While digital literacy may be covered to an extent in some initial training programmes, there is limited evidence that it features in CPD for existing members of pharmacy staff.

In summary, despite the increasing adoption of IT and ehealth to support the role of pharmacy staff, there is a paucity of research exploring their perceptions and digital literacy knowledge and related training. This review identifies evidence of perceived levels of digital literacy amongst pharmacy staff and their related training experiences and future needs.

### Objective

To explore the digital literacy knowledge and needs of pharmacy staff.

### Review question

This review asks, ‘What are the digital literacy knowledge and needs of pharmacy staff?’ to summarise existing evidence of pharmacy staff perceptions and measures of:

1.  levels of digital literacy knowledge;

2.  inclusion of digital literacy in pharmacy training;

3.  specific digital literacy training experiences;

4.  digital literacy training needs.

## METHODS

### Design

This systematic review followed the Centre for Review and Dissemination guidance for healthcare reviews (PROSPERO Protocol 2013:CRD42013005503) in seeking to ‘identify, evaluate and summarise the findings of all relevant individual studies’ and to ‘demonstrate where knowledge is lacking… to guide future research’.42,43 The theoretical framework adopted for analysis was Kirkpatrick’s 4 level model (reaction, learning, behaviour, results) for evaluating training programmes.44,45 Kirkpatrick likens level 1 (reaction) to a ‘measure of customer satisfaction’ with level 2 (learning) a ‘measure of knowledge acquired, skills improved or attitudes changed due to training’. Level 3 (behaviour) measures ‘the extent to which participants change their on-the-job behaviour’ while level 4 (results) looks for wider impact in organisational terms. Although Kirkpatrick’s model has been criticised for over-simplification and a lack of contextual consideration, its pre-eminence as a training evaluation tool has been acknowledged over several decades.46,47

### Eligibility criteria

This review considered English language studies that related to any aspect of digital literacy or computer skills training for any member of pharmacy staff in all pharmacy settings with no geographical or date restrictions applied.

## SEARCH STRATEGY

A three-step search strategy was utilised in this review. An initial limited search of MEDLINE and CINAHL was undertaken followed by analysis of the text words contained in the title and abstract, and of the index terms used to describe the article. A second search using all identified keywords and index terms was then undertaken across all included databases (Figure 1). Thirdly, the reference list of all identified reports and articles was searched for additional studies. Titles of papers were independently screened by two reviewers followed by abstracts then full papers. The search string, database returns and exclusions are shown in an adapted PRISMA diagram.48

![Figure 1](http://informatics.bmj.com/https://informatics.bmj.com/content/bmjhci/23/3/560/F1.medium.gif)

[Figure 1](http://informatics.bmj.com/content/23/3/560/F1)

Figure 1 Adapted PRISMA flowchart showing the search strategy and returns47

### Assessment of methodological quality

To reduce risk of bias, papers selected for critical appraisal were assessed independently by two reviewers for methodological quality before inclusion using a standardised critical appraisal tool adapted to suit all study types.49

### Data extraction

Data were extracted using a bespoke data extraction tool (Table 3). The data extraction was performed independently by two reviewers before cross-checking to minimise errors and reduce risk of bias.

### Data synthesis

Findings were narratively explored through Kirkpatrick’s 4 level model for evaluating training programmes by focusing on evidence of reaction, learning, behaviour and results.

## RESULTS

### Study selection

Systematic application of the search strategy returned 86 published papers, which after independent screening of titles, abstracts and full papers was reduced to 5 (Figure 1 and Table 1).

View this table:
[Table 1](http://informatics.bmj.com/content/23/3/560/T1)

Table 1 The ‘when, who and what’ of the five papers included prior to critical appraisal

There were no studies featuring pharmacy staff other than pharmacists and no unpublished studies identified.

### Quality assessment

Each study was independently reviewed for quality by two of the research team. Details of the clarity of the research question, appropriateness of the design, description of context, population, sampling, data collection and analysis along with results, limitations and conclusions are provided in Table 2. It also details reasons for the exclusion of one study on quality grounds, while four were taken forward for data extraction.

View this table:
[Table 2](http://informatics.bmj.com/content/23/3/560/T2)

Table 2 The critical appraisal tool adapted from Mays N et al49

### Study characteristics

The data extraction table (Table 3) provides summarised study characteristics and contextual information. In brief, one of the studies was a survey conducted to establish baseline computer skills of hospital pharmacists in Canada prior to an educational intervention.50 Another surveyed allied health professionals (AHPs), including pharmacists, in Australia about their use of electronic evidence resources.53 A further Australian study, based on community pharmacists, combined pre-intervention focus groups with a post-educational intervention evaluative survey.51 The final study used mixed methods to review informatics content, including computer and digital literacy skills, in pharmacy education by mapping syllabi returned by schools of pharmacy against the US ACPE Standards.52

View this table:
[Table 3](http://informatics.bmj.com/content/23/3/560/T3)

Table 3 Data extraction of four papers reviewed

### Pharmacy staff digital literacy level

Balen and Jewesson50 concluded there was not yet ‘a standard definition of computer literacy and valid dimensions of computer competency for pharmacy practice’. Based on a literature review, Bearman et al51 found ‘there was little or no information regarding community pharmacists’ skills and knowledge levels or how they currently employ internet technologies’. Through focus groups, they identified a ‘wide variety of technical knowledge and skills.’ Where access was available, participants most commonly used the internet at work for email and to search pharmacy-related topics, such as medicines or patient information, with a small proportion contributing to a pharmacy message forum. They were ‘less familiar with local health-specific portals or websites.’ Lack of familiarity or expertise and difficulty finding relevant information online were noted issues.

Balen and Jewesson50 found pharmacists were likely to have both home and work access to computers. Work use included information management, internet searching and email, drug distribution systems, patient care systems but minimal use of spreadsheets, statistical or presentation software. They concluded hospital pharmacists were ‘computer literate’ and ‘not anxious’ about using IT.

Fox et al52 identified ‘confusion within the academy/profession between pharmacy informatics and drug information practice’ and low compliance with ACPE Standards 2007 on pharmacy informatics competencies. Three progressive levels of pharmacy informatics competency were detailed under headings of terminology, systematic approaches, benefits and constraints. Fox et al52 concluded that pharmacists ‘must utilize information technology and automation’ implying but not specifying levels of digital literacy.

In a convenience sample survey of AHP, Gosling and Westbrook53 found pharmacists were the highest users of an online evidence system. Two of 25 closed questions in the survey related to database searching and computer skills. Across all AHPs, nearly three quarters reported their computer skills as good, very good or excellent with pharmacists rated most able to find online information.

### Digital literacy in pharmacy training

The study by Fox et al52 focused on searching pharmacy syllabi for elements of informatics training. They cite Flynn55 in asserting ‘few pharmacy programs provide formal pharmacy informatics’. However, they elaborated the role of the Institute of Medicine (IOM) who recognise ‘utilizing the tools and techniques of informatics’ as a core competency for all clinical healthcare professionals. This is further evidenced by the initiatives around educational provision by the American Medical Informatics Association (AMIA) and International Medical Informatics Association (IMIA) whose recommendations were adopted by the American Society of Health-Systems Pharmacists (ASHP).

Balen and Jewesson50 noted that ‘informatics is not a formal component of the core undergraduate or graduate programs’ at their local university and ‘remains an uncommon component of most pharmacy and medical school curricula’ in North America. Findings from Bearman et al’s51 literature review showed that ‘many [community pharmacists] have not been educated in internet use for professional practice’. Although Gosling and Westbrook53 found a ‘marked difference between professions use’ of an online evidence system, there is no clear way to relate this back to pharmacy or other allied health professional training.

### Digital literacy training experiences

Balen and Jewesson50 found that 79% of pharmacists (*n* =106) who responded to their survey ‘had received no formal computer training’. Following their educational intervention, Bearman et al51 received positive feedback from participants about improved searching skills and more effective searching while ‘almost half of the 93 respondents reported a change in practice’. While declaring informatics a new discipline for pharmacy, Fox et al52 emphasised the intricate link between IT and pharmacy informatics explaining ‘IT tools provide the infrastructure for information management to support pharmacy informatics’. In findings across all AHPs, Gosling and Westbrook53 identified social, organisational and professional support, along with general computer skills training, as important facilitators influencing the use of technologies in pharmacy practice.

### Digital literacy training needs

Each of the included studies indicated participants want or need more digital literacy related training. Balen and Jewesson50 found that 77% (*n* =106) in need of ‘general computer skills upgrading’ ranking medical database and Internet search as priority areas. Access to Internet related education was viewed as a priority for community pharmacists by Bearman et al,51 while Gosling and Westbrook53 found ‘general training aimed at improving computer skills more important… than specific system-based training’. Finally, Fox et al52 recommended a set of foundational core competencies, ‘based on themes extracted from course syllabi and from personal experience’, and encouraged pharmacy educators to ‘look to informatics in other disciplines, such as medicine and nursing, for guidance’.

### Applying the analytical framework

In terms of Kirkpatrick’s 4 level model, the pre-training survey of computer skills conducted by Kirkpatrick and Kirkpatrick45 and Balen and Jewesson50 evidenced baseline evaluation recognised as a preliminary activity for level 2 (learning). Similarly, Gosling and Westbrook53 conducted a survey which included measures of baseline skills (level 2: learning). Although good computer skills were shown to be associated with the use of technology by pharmacists (level 3: behaviour), it is not clear from the findings presented whether pharmacists were included in the AHPs who received training. Bearman et al51 reported pharmacists ‘were highly positive about the learning experience’ (level 1: reaction), in particular the ‘flexible delivery of the course,’ while online resource identification attracted most comments. Findings also reported ‘specific changes in practice’ (level 3: behaviour) around ‘use of new websites, more effective searching, a change to regular use of specific resources.’ The emphasis in the article by Fox et al52 was on syllabus content: which informatics skills (level 2: learning) are taught in pharmacy education. Their conclusions and recommendations aspired to levels 3 (behaviour) and 4 (results) in urging pharmacy programs ‘to prepare future pharmacists to approach their professional practice as drug safety experts and medication knowledge workers who must utilize information technology and automation in order to create a safer, more effective medication-use system’.

## DISCUSSION

The evolving role of pharmacy within the collaborative healthcare team is increasingly reliant on a range of ehealth technologies and digital literacy. This review set out to summarise the best available existing evidence of pharmacy staff perceptions and measures of their levels of digital literacy knowledge, the inclusion of digital literacy in their pharmacy training, specific digital literacy training experiences, and their digital literacy training needs. Very limited research was identified about pharmacists, while no studies were found in relation to other pharmacy staff.

The need for better identification of citizen and workforce skills for the digital age is a matter of increasing focus worldwide, but there is little evidence of its impact on pharmacy education or pharmacy practice. While pharmacy programmes in America demonstrated a lack of compliance with the ACPE standards 2007, there was even less evidence of digital literacies in pharmacy programmes outside the US. So, while digital literacy is acknowledged as an important lifelong and work-based skill, this is not readily evidenced in initial pharmacy staff training nor CPD. Further research might also explain whether higher digital literacy levels amongst hospital pharmacists compared with their community-based colleagues is due to their realm of daily practice, postgraduate training or multidisciplinary team working.

The limited evidence found of digital literacy training experience was, however, positive. Improved search and basic computer skills with social, organisational and professional support were shown to facilitate the use of technologies in pharmacy. Yet, the majority of pharmacy staff had received no digital literacy training. The US leads in viewing digital literacy in pharmacy as a pathway to engaging the community in health literacy with the potential to improve social welfare, inclusion and individual health and well being.

All studies indicated that pharmacists want or need more digital literacy training, but their recommendations lack baseline data and are not current, quantifiable, measurable or specific. A key finding of this review is the lack of digital literacy research amongst not only pharmacists but all pharmacy staff as pharmacy assistants, technicians and medicine counter assistants all use technology in daily pharmacy practice. The core competencies recommended for pharmacy informatics are founded on digital literacy and so may offer a starting point for further research, which should be broadened to include all pharmacy staff. As the role of pharmacists and therefore all pharmacy staff continues to grow in the UK, the digital literacy levels for current and future pharmacists and staff must keep pace with technological change.

In launching the US 2012 Digital Government Strategy, President Obama said, ‘I want us to ask ourselves every day, how are we using technology to make a real difference in people’s lives’.56 As the role of pharmacy in healthcare continues to expand, there are calls for enhanced workforce skills, most recently from the UK Academy of Medical Royal Colleges who emphasise the need for ‘enhanced informatics skills in healthcare professionals so that the significant benefits that technology can enable are realised’.57

### Strengths and weaknesses

It is a strength of this review that it demonstrates the lack of research conducted around digital literacy of pharmacy staff, but it is also its main weakness. With so few studies on which to base the review, findings must be treated with caution. Although not a weakness of this review, the survey-or evaluation-based studies used self-reporting, which is recognised for its inherent bias. The use of standardised tools in this review applied independently by more than one reviewer reduced the risk of bias for both inclusion and reporting.

## CONCLUSION

In conclusion, although all pharmacy staff are reliant on technology in their daily practice, there is a lack of evidence of their specific and measurable digital literacy knowledge levels, training experience and needs. As a society, we acknowledge that technology is an important part of everyday life impacting on the efficiency and effectiveness of working practices but, in pharmacy, do we take cognisance, ‘that technology can change the nature of work faster than people can change their skills’?23 It seems pharmacy has embraced technology without recognised occupational standards, definition of baseline skills or related personal development plans. This review recommends future research should be focused on establishing what digital literacy knowledge is needed and how training should be designed, delivered and evaluated for all pharmacy staff at all levels and career stages.

## Acknowledgments

The lead author gratefully acknowledges the expert input of the late Yash Kumarasamy and Professor Alison Strath to the study design and review protocol.

## Footnotes

*   Competing interests The authors have no competing interests to declare.

*   Funding No dedicated funding was received for this review, but it is part of a larger research programme funded by NHS Education for Scotland. The authors gratefully acknowledge their support.

*   © 2016 The Author(s). Published by BCS, The Chartered Institute for IT under Creative

Commons license http://creativecommons.org/licenses/by/4.0/

## References

1.  1.Darzi A. High Quality Care For All: NHS Next Stage Review Final Report 2008. Accessed 29 February 2012 DH_085825
    
    

2.  2.Scottish Government. eHealth Strategy 2011–2017. 2011. Edinburgh: Scottish Government.
    
    

3.  3.Department of Health. A Report from the NHS Future Forum: Education and Training 2011. London: Department of Health.
    
    

4.  4.Crown J. Review of Prescribing, Supply and Administration of Medicines 1999;NHS Executive: Department of Health. Final report
    
    

5.  5.General Pharmaceutical Council. Future Pharmacists: Standards for Initial Education and Training of Pharmacists 2011. London: GPHC. Accessed 01 August 2016. Available from: [www.pharmacyregulation.org](http://www.pharmacyregulation.org)
    
    

6.  6.General Pharmaceutical Council. Accreditation and Recognition of Pharmacy Technician Level 3 Knowledge-Based and Competence-Based Qualifications, Scottish Qualifications Authority. 2011. London: GPhC. Accessed 01 August 2016. Available from: [www.pharmacyregulation.org](http://www.pharmacyregulation.org)
    
    

7.  7.Beetham H, McGill L, Littlejohn A. Thriving in the 21st Century: the Report of the LLiDA Project (Learning Literacies for the Digital Age): Provision in the UK 2009. Accessed 01 August 2016 Available from: [http://www.webarchive.org.uk/wayback/archive/20140614091049/](http://www.webarchive.org.uk/wayback/archive/20140614091049/); [http://www.jisc.ac.uk/whatwedo/projects/elearningllida.aspx](http://www.jisc.ac.uk/whatwedo/projects/elearningllida.aspx)
    
    

8.  8.MacLure K, Stewart D. Self-Reported Digital Literacy of the Pharmacy Workforce in North East Scotland. Pharmacy 2015;3(4):182–196. [http://www.mdpi.com/2226-4787/3/4/182](http://www.mdpi.com/2226-4787/3/4/182). doi:10.3390/pharmacy3040182
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.3390/pharmacy3040182&link_type=DOI) 

9.  9.British Computer Society. Digital Literacy for Life programme. Accessed 01 August 2016 Available from: [http://dlfl.bcs.org/](http://dlfl.bcs.org/)
    
    

10. 10.American Library Association 2013. Digital Literacy, Libraries, and Public Policy: Report of the Office for Information Technology Policy’s Digital Literacy Task Force. Accessed 01 August 2016 Available from: [http://www.districtdispatch.org/wp-content/uploads/2013/01/2012\_OITP\_digilitreport\_1\_22\_13.pdf](http://www.districtdispatch.org/wp-content/uploads/2013/01/2012\_OITP\_digilitreport_1_22_13.pdf)
    
    

11. 11.1.  Lankshear C, 
    2.  Knobel M
    
    . Digital literacies: Concepts, Policies and Practices. 2008;30: New York: Peter Lang.
    
    

12. 12.CMS Advisory Group. Establishing Effective Therapeutic Partnerships: A Generic Framework to Underpin the Chronic Medication Service Element of the Community Pharmacy Contract. 2009. Edinburgh: The Scottish Government.
    
    

13. 13.McElenay JC. Connected health and the pharmacist. IJPP 2011;19(1):3–4Available from: [http://onlinelibrary.wiley.com/doi/10.1111/j.2042-7174.2010.00088.x/pdf](http://onlinelibrary.wiley.com/doi/10.1111/j.2042-7174.2010.00088.x/pdf). [http://dx.doi.org/10.1111/j.2042-7174.2010.00088.x](http://dx.doi.org/10.1111/j.2042-7174.2010.00088.x)
    
    

14. 14.Australian Government Department of Health. eHealth and pharmacists 2013. Accessed 18 October 2013
    
    

15. 15.Ministry of Health, British Columbia. eHealth – Faster, safer, better healthcare: Benefits for Health Professionals. Accessed 01 August 2016 Available from: [http://www.health.gov.bc.ca/ehealth/professionals.html](http://www.health.gov.bc.ca/ehealth/professionals.html)
    
    

16. 16.Agency for Healthcare Research and Quality. Is Our Pharmacy Meeting Patients’ Needs? Pharmacy Health Literacy Assessment Tool User’s Guide. Rockville:Agency for Healthcare Research and Quality. 2007. Accessed 01 August 2016 [http://www.ahrq.gov/professionals/quality-patient-safety/pharmhealthlit/pharmlit/index.html](http://www.ahrq.gov/professionals/quality-patient-safety/pharmhealthlit/pharmlit/index.html)
    
    

17. 17.Department of Health. Working for Patients. 2007. London: HMSO.
    
    

18. 18.Nolan M. Towards an ethos of interdisciplinary practice. BMJ 1995;311(7000):305. Accessed 01 August 2016. Available from: [http://www.bmj.com/content/311/7000/305](http://www.bmj.com/content/311/7000/305). doi:10.1136/bmj.311.7000.305
    
    [FREE Full Text](http://informatics.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTEvNzAwMC8zMDUiO3M6NDoiYXRvbSI7czoyMToiL2JtamhjaS8yMy8zLzU2MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

19. 19.Hepler CD, Strand LM. Opportunities and responsibilities in pharmaceutical care. American Journal of Hospital Pharmacy 1990;47(3):533–543.
    
    [Abstract](http://informatics.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpocCI7czo1OiJyZXNpZCI7czo4OiI0Ny8zLzUzMyI7czo0OiJhdG9tIjtzOjIxOiIvYm1qaGNpLzIzLzMvNTYwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

20. 20.Scottish Government, NHS Scotland. eHealth Citizen eHealth Survey. Accessed 12 January 2012 Available online from: [http://www.ehealth.scot.nhs.uk/?page\_id=143](http://www.ehealth.scot.nhs.uk/?page_id=143)
    
    

21. 21.European Commission. eHealth Action Plan 2012-2020 public consultation. 2011. Accessed 01 August 2016 Available from: [https://joinup.ec.europa.eu/community/epractice/news/uk-ehealth-survey-enhance-healthcare-scotland](https://joinup.ec.europa.eu/community/epractice/news/uk-ehealth-survey-enhance-healthcare-scotland)
    
    

22. 22.World Health Organisation. eHealth Country Profiles. 2011. Geneva: WHO.
    
    

23. 23.NTIA United States Department of Commerce. Exploring the Digital Nation: America’s Emerging Online Experience. 2013. Accessed 01 August 2016 Available from: [http://www.ntia.doc.gov/report/2013/exploringdigital-nation-americas-emerging-online-experience](http://www.ntia.doc.gov/report/2013/exploringdigital-nation-americas-emerging-online-experience)
    
    

24. 24.Innovation & Business Skills Australia. Scoping Study – identifying digital literacy skills: cybercitizen and e-employee in the 21st century. 2010. Accessed 01 August 2016 Available from: [https://www.ibsa.org.au/sites/default/files/media/Investigating%20Digital%20Literacy%20Skills.pdf](https://www.ibsa.org.au/sites/default/files/media/Investigating%20Digital%20Literacy%20Skills.pdf)
    
    

25. 25.New Zealand Ministry of Education. Digital Information Literacy: Supported Development of Capability in Tertiary Environments. 2010. Accessed 10 October 2013 Available from: [https://www.educationcounts.govt.nz/publications/e-Learning/80624](https://www.educationcounts.govt.nz/publications/e-Learning/80624)
    
    

26. 26.UNESCO Institute for Technologies in Education. Policy Brief: Digital Literacy in Education. 2011. Accessed 01 August 2016 Available from: [http://iite.unesco.org/publications/3214688/](http://iite.unesco.org/publications/3214688/)
    
    

27. 27.European Commission – Information Society. Accessed 01 August 2016 Available from: [http://ec.europa.eu/information\_society/index\_en.htm](http://ec.europa.eu/information_society/index_en.htm)
    
    

28. 28.European Commission – Digital Agenda for Europe (website). Accessed 01 August 2016 Available from: [https://digital-agenda-data.eu/](https://digital-agenda-data.eu/)
    
    

29. 29.European Parliament. Recommendation 2006/962/EC of the European Parliament and of the Council of 18 December 2006 on key competences for lifelong learning [Official Journal L 394 of 30.12.2006]. Accessed 01 August 2016 Available from: [http://europa.eu/legislation\_summaries/education\_training\_youth/lifelong\_learning/c11090\_en.htm](http://europa.eu/legislation\_summaries/education\_training_youth/lifelong_learning/c11090_en.htm)
    
    

30. 30.Royal Pharmaceutical Society & Royal College of General Practitioners. Joint Statement: Breaking Down the Barriers – How Community Pharmacists and GPs can Work Together to Improve Patient Care. 2011. London: RPS & RCGP.
    
    

31. 31.British Computer Society – The Chartered Institute for IT. Adult Education and Further Education. Accessed 01 August 2016 Available from: [http://www.bcs.org/category/15758](http://www.bcs.org/category/15758)
    
    

32. 32.Royal Pharmaceutical Society. Empowering Pharmacists to Improve Pharmaceutical Care and Medicines Safety Through Information Technology: Information Technology Strategic Principles. 2011. Edinburgh: RPS.
    
    

33. 33.Academy of Medical Royal Colleges & the Scottish Government. E Health Competency Framework: Defining the role of the expert clinician. Accessed 01 August 2016 Available from: [http://www.aomrc.org.uk/publications/reports-guidance/ehealth-competency-framework-0611/](http://www.aomrc.org.uk/publications/reports-guidance/ehealth-competency-framework-0611/)
    
    

34. 34.e-Skills UK. National Occupational Standards for IT Users. Accessed 01 August 2016 Available from: [https://www.e-skills.com/standards-and-qualifications/national-occupational-standards-nos](https://www.e-skills.com/standards-and-qualifications/national-occupational-standards-nos)
    
    

35. 35.Skills for Health. First ever UK-wide core skills and training framework for health. Accessed 01 August 2016 Available online at: [http://www.skillsforhealth.org.uk/services/item/146-core-skills-training-framework](http://www.skillsforhealth.org.uk/services/item/146-core-skills-training-framework)
    
    

36. 36.Department of Health. The NHS Knowledge and Skills Framework and the Development Review Process: Final Draft 2004;Crown Copyright.
    
    

37. 37.Royal Pharmaceutical Society. Continuing Professional Development section (website). Accessed 01 August 2016 Available from: [http://www.rpharms.com/development/continuing-professional-development.asp](http://www.rpharms.com/development/continuing-professional-development.asp)
    
    

38. 38.NHS Education for Scotland Pharmacy. National UK Centre for Continuing Pharmaceutical Education in Scotland (website). Accessed 01 August 2016 Available from: [http://www.nes.scot.nhs.uk/education-and-training/by-discipline/pharmacy.aspx](http://www.nes.scot.nhs.uk/education-and-training/by-discipline/pharmacy.aspx)
    
    

39. 39.Accreditation Council for Pharmacy Education (website). Accessed 01 August 2016 Available from: [https://www.acpe-accredit.org/default.asp](https://www.acpe-accredit.org/default.asp)
    
    

40. 40.Canadian Council for Accreditation of Pharmacy Programs (website). Accessed 01 August 2016 Available from: [http://ccapp-accredit.ca/find-anaccredited-program/](http://ccapp-accredit.ca/find-anaccredited-program/)
    
    

41. 41.Accreditation Standards for Pharmacy Degree Programmes in Australia and New Zealand (website). Accessed 01 August 2016 [http://pharmacycouncil.org.au](http://pharmacycouncil.org.au)
    
    

42. 42.MacLure K, Stewart D, Strath A, Kumarasamy Y. Pharmacy staff’s digital literacy training experiences and needs: a systematic review protocol. PROSPERO 2013:CRD42013005503. Available from: [www.crd.york.ac.uk/PROSPERO/display\_record.asp?ID=CRD42013005503](http://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42013005503)
    
    

43. 43.Ackers J, Aguiar-Ibáñez R, Baba-Akbari Sari A, Beynon S, Booth A, Burch J. Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care 2009;Centre for Reviews and Dissemination, University of York.
    
    

44. 44.Kirkpatrick D. Great Ideas Revisited. Techniques for Evaluating Training Programs. Revisiting Kirkpatrick’s Four-Level Model. Training and Development Journal 1996;50(1):54–59.
    
    

45. 45.Kirkpatrick DL, Kirkpatrick JD. Implementing the Four Levels: A Practical Guide for Effective Evaluation of Training Programs. 2007. San Francisco: Berrett-Koehler Store.
    
    

46. 46.Bates R. A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning 2004;27:341–347Available from: [https://aetcnec.ucsf.edu/sites/aetcnec.ucsf.edu/files/A%20critical%20analysis%20of%20evaluation%.20practice\_0.pdf](https://aetcnec.ucsf.edu/sites/aetcnec.ucsf.edu/files/A%20critical%20analysis%20of%20evaluation%.20practice_0.pdf). doi:10.1016/j.evalprogplan.2004.04.011
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/j.evalprogplan.2004.04.011&link_type=DOI) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000222724100010&link_type=ISI) 

47. 47.Employment Security Department, Washington State. Evaluating Training Programs: Kirkpatrick’s 4 Levels. Accessed 01 August 2016 Available from: [www.wa.gov/esd/training/toolbox/tg\_kirkpatrick.htm](http://www.wa.gov/esd/training/toolbox/tg_kirkpatrick.htm)
    
    

48. 48.Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of Internal Medicine 2009;151(4):264–269. doi:10.7326/0003-4819-151-4-200908180-00135. pmid:19622511
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.7326/0003-4819-151-4-200908180-00135&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=19622511&link_type=MED&atom=%2Fbmjhci%2F23%2F3%2F560.atom) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000269038900006&link_type=ISI) 

49. 49.1.  Fulop N, 
    2.  Allen P, 
    3.  Clarke A, 
    4.  Black N
    
    Mays N, Roberts E, Popay JFulop N, Allen P, Clarke A, Black N. Synthesising research evidence. Methods for Studying the Delivery and Organisation of Health Services 2001. London: Routledge.
    
    

50. 50.Balen RM, Jewesson PJ. Pharmacist computer skills and needs assessment survey. JMIR 2004;6(1):e11Available from: [http://www.jmir.org/2004/1/e11/](http://www.jmir.org/2004/1/e11/). doi:10.2196/jmir.6.1.e11. pmid:15111277. pmid:PMC1550586
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.2196/jmir.6.1.e11&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=PMC1550586&link_type=MED&atom=%2Fbmjhci%2F23%2F3%2F560.atom) 

51. 51.Bearman M, Bessell T, Gogler J, McPhee W. Educating Australian pharmacists about the use of online information in community pharmacy practice. IJPP 2005;13(2):109–115. Accessed 01 August 2016. doi:10.1211/0022357055876
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1211/0022357055876&link_type=DOI) 

52. 52.Fox BI, Karcher RB, Flynn A, Mitchell S. Pharmacy informatics syllabi in doctor of pharmacy programs in the US. AJPE 2008; 72(4):89. Accessed 01 August 2016 doi:10.5688/aj720489. pmid:19002287
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.5688/aj720489&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=19002287&link_type=MED&atom=%2Fbmjhci%2F23%2F3%2F560.atom) 

53. 53.Gosling AS, Westbrook JI. Allied health professionals’ use of online evidence: a survey of 790 staff working in the Australian public hospital system. International Journal of Medical Informatics 2004;73(4):391–401. Accessed 01 August 2016. doi:10.1016/j.ijmedinf.2003.12.017
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/j.ijmedinf.2003.12.017&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=15135758&link_type=MED&atom=%2Fbmjhci%2F23%2F3%2F560.atom) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000221685100008&link_type=ISI) 

54. 54.Gour N, Srivastava D. Knowledge of computer among healthcare professionals of India: a key toward e-health. Telemed J E Health 2010. 16(9):957–962. Accessed 01 August 2016, Available from: [http://www.ncbi.nlm.nih.gov/pubmed/21091287](http://www.ncbi.nlm.nih.gov/pubmed/21091287)
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=21091287&link_type=MED&atom=%2Fbmjhci%2F23%2F3%2F560.atom) 

55. 55.Flynn AJ. The current state of pharmacy informatics education in professional programs at US colleges of pharmacy. American Journal of Pharmaceutical Education 2005;69(4):Article 66. Accessed 01 August 2016. Available from: [http://www.ajpe.org/doi/abs/10.5688/aj690466](http://www.ajpe.org/doi/abs/10.5688/aj690466). doi:10.5688/aj690466
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.5688/aj690466&link_type=DOI) 

56. 56.US Government. Digital Government: Building a 21st Century Platform to Better Serve the American People. 2012. Accessed 01 August 2016. Available from: [http://www.whitehouse.gov/sites/default/files/omb/egov/digital-government/digital-government.html](http://www.whitehouse.gov/sites/default/files/omb/egov/digital-government/digital-government.html)
    
    

57. 57.Academy of Royal Medical Colleges. i-care: Information, Communication and Technology in the NHS. 2013. Accessed 01 August 2016 Available from: [http://www.aomrc.org.uk/publications/reports-guidance/icare-ict-in-the-nhs-1013/](http://www.aomrc.org.uk/publications/reports-guidance/icare-ict-in-the-nhs-1013/)