Clinical decision support (CDS) has demonstrated the ability to shape health care provider behavior towards more evidence based clinical practice by improving diagnosis, treatment, and preventative care services [1], [2], [3], [4], [5], [6]. CDS is typically integrated into the electronic health record (EHR) and functions to bring key pieces of evidence or best practice guidelines to the point of care. These tools stand to improve the American healthcare system where on average it takes five
This was a qualitative observational study done at the University of Wisconsin, a large academic health care center. “Think Aloud” testing was completed with 4 participants. The tool was revised based on these results before “Near Live” testing was conducted with 8 participants. Different participants were recruited for each type of testing, as is typically the case, to minimize the time commitment required from each of these busy health care providers. Both “Think Aloud” and “Near Live”
Participants were primarily medical doctors, along with one nurse practitioner and one physician assistant. (Table 1) Our sample was 42% female with an average age of 47.1, 18.7 years of post-graduate practice, and 7.7 years of experience using an EHR. Average SUS was 85.6 during “Think Aloud” testing and 81.3 during “Near Live” testing. SUS scores range from 0 to 100, with 100 being a perfect score [17]. An SUS score of 68 is considered average [18]. Those raw scores would correspond to the
“Think Aloud” and “Near Live” usability testing of these two CDS tools generated unique insights and lessons generalizable to all forms of CDS. Participant commentary was consistent across participants and across the two tools. This is the first study to evaluate generalizable lessons learned from “Think Aloud” and “Near Live” usability testing of complex clinical decision support tools. Previous studies documented usability findings particular to the tools studied and the relative percentages
“Think Aloud” and “Near Live” usability testing provide CDS tool designers with complementary insights that when combined provide a more robust understanding of CDS usability. Participant comments made during “Think Aloud” testing about visibility, content, understand-ability and navigation primarily helped to improve the tools ease of use. Participant comments and observed behavior during “Near Live” testing, which mimics a real clinical encounter, were more helpful for eliciting key barriers
The authors declare that they have no competing interests.
This project was funded by the National Institutes of Health, National institute of Allergy and Infectious Diseases, under grant #5R01 AI108680-03. The funding body had no role in the design of the study or the collection, analysis, or interpretation of data.
Written informed consent was obtained from all participants. The Institutional Review Boards at both institutions approved the research protocol.
Not applicable.
Safiya Richardson, Rachel Mishuris and Alexander O’Connell analyzed and interpreted the qualitative data. David Feldstein, Rachel Hess, Paul Smith and Lauren McCullagh conducted usability testing with participants. Safiya Richardson, Thomas McGinn and Devin Mann were major contributors in writing the manuscript. All authors read and approved the final manuscript.
Datasets analyzed during this study are available from the corresponding author on reasonable request. SUMMARY POINTS Already Known Low provider adoption continues to be a significant barrier to realizing the potential of clinical decision support. Efficiency, usefulness, information content, user interface, and workflow have been reported by clinicians to be the keys to effective decision support. This Study Has Added: “Think Aloud” and “Near Live” usability testing of these two CDS tools generated
Not applicable.
Several of our themes have been mentioned previously in the literature,24,49 the most recognizable being automation, which has long been cited as a key source of physician burnout related to EHRs but which remains a challenge as the EHR becomes increasingly capable yet complex. In our review of usability testing studies involving only end-users, in addition to automation, themes commonly related to workflow,18 level of interruption,37 content,41 functionality,51 and usefulness.39 This literature supports our findings that end-users are more likely to identify issues relating to workflow and automation.
Finally, we showed in a previously published qualitative study that efficiency and ease of use are two key facilitators for the adoption of CDSS.8 Even though the tools were developed iteratively with a testing process involving clinicians, we probably should have invested more time to involve end-users through an extensive user-experience process to optimise these aspects and improve user acceptance and adoption.30 The effectiveness of the COMPASS intervention to reduce overall antimicrobial use remains inconclusive.
