Assessing performance of an Electronic Health Record (EHR) using Cognitive Task Analysis

Abstract

Background

Many Electronic Health Record (EHR) systems fail to provide user-friendly interfaces due to the lack of systematic consideration of human-centered computing issues. Such interfaces can be improved to provide easy to use, easy to learn, and error-resistant EHR systems to the users.

Objective

To evaluate the usability of an EHR system and suggest areas of improvement in the user interface.

Methods

The user interface of the AHLTA (Armed Forces Health Longitudinal Technology Application) was analyzed using the Cognitive Task Analysis (CTA) method called GOMS (Goals, Operators, Methods, and Selection rules) and an associated technique called KLM (Keystroke Level Model). The GOMS method was used to evaluate the AHLTA user interface by classifying each step of a given task into Mental (Internal) or Physical (External) operators. This analysis was performed by two analysts independently and the inter-rater reliability was computed to verify the reliability of the GOMS method. Further evaluation was performed using KLM to estimate the execution time required to perform the given task through application of its standard set of operators.

Results

The results are based on the analysis of 14 prototypical tasks performed by AHLTA users. The results show that on average a user needs to go through 106 steps to complete a task. To perform all 14 tasks, they would spend about 22 min (independent of system response time) for data entry, of which 11 min are spent on more effortful mental operators. The inter-rater reliability analysis performed for all 14 tasks was 0.8 (kappa), indicating good reliability of the method.

Conclusion

This paper empirically reveals and identifies the following finding related to the performance of AHLTA: (1) large number of average total steps to complete common tasks, (2) high average execution time and (3) large percentage of mental operators. The user interface can be improved by reducing (a) the total number of steps and (b) the percentage of mental effort, required for the tasks.

Introduction

Many health information system projects fail due to lack of systematic consideration of human-centered computing issues such as usability, workflow, organizational change, and process reengineering [1]. This paper evaluates the usability of the AHLTA (Armed forces Health Longitudinal Technology Application) EHR (Electronic Health Record) system that is used by the US military which connects 412 medical clinics, 414 dental clinics and 65 military hospitals. According to Dr. Casscells, assistant secretary of defense for health affairs, AHLTA is difficult to learn, and once you have learned it, it is cumbersome and difficult to navigate [2]. Clinician users of AHLTA also report seeing fewer patients and having longer workdays, largely because of the extra time needed to use the system [3]. Two major factors that lead to sluggish performance of this EHR system are complexity of the GUI (Graphical User Interface) and system response time [4]. This paper analyzes the complexity of the GUI of AHLTA independent of its system response time.

The usability and usefulness of EHR can be evaluated through UFuRT (User, Function, Representation, Task) analysis [5], [6], [7], which is a process for the design and evaluation of work-centered products through User, Function, Representation, and Task analysis. User analysis identifies the categories of users; Functional analysis identifies the ontology of a given work domain; Representational analysis identifies an appropriate information display format for a given task, and Task Analysis identifies the procedures and actions to be carried out for a given task goals by using specific representations. Task analysis of AHLTA was the main focus of this study. Specifically, we used a Cognitive Task Analysis called GOMS (Goals, Operators, Methods, and Selection rules) to analyze a set of prototypical tasks of AHLTA users. In addition, we applied an associated GOMS method – KLM (Keystroke Level Model) to evaluate the execution time required for the given task [8].

• Cognitive Task Analysis (CTA) is a core methodology used in cognitive science to study human performance in both laboratory and real-world settings [9]. It can uncover the underlying knowledge, skills and structures of task performance by characterizing the decision making and reasoning skills and information processing needs of subjects as they perform tasks [10]. Compared to other methods for system design and assessment, CTA is superior for its predominance in analyzing both physical and mental procedures in a task. There are various methods to conduct CTA. In this paper, we use GOMS – a goal–subgoal structured model to assess the cognitive complexity and task performance in AHLTA.

• Distributed cognition: Distributed cognition is the theoretical development in the distributed system approach [11], [12], [13]. Recent studies by Patel et al. extended the distributed cognition approach to the medical domain [14]. Distributed cognition emphasizes the inherently social and collaborative nature of cognition and also characterizes the mediating effects of technology or other artifacts on cognition [10], [13]. The paper focuses on analyzing how human cognition factors are distributed across the task performance in AHLTA system, using the Extended Hierarchical Task Analysis method developed by Chung et al. [15]. In this study, distributed cognition is constrained and characterized as a process of coordinating distributed internal (mental) and external (physical) representations and factors in the interaction between physicians and the EHR system. In the task performance, physicians perform mental actions (i.e., retrieving information from memory, making a choice) and also physical actions (i.e., scanning a patient list, writing a note). External representations can minimize the difficulty of a task by supporting recognition-based memory or perceptual judgments rather than recall [16].