Article Text
PDF
Abstract
Background Health risk assessment tools compute an individual’s risk of developing a disease. Routine use of such tools by primary care physicians (PCPs) is potentially useful in chronic disease prevention. We sought physicians’ awareness and perceptions of the usefulness, usability and feasibility of performing assessments with computer-based risk assessment tools in primary care settings.
Methods Focus groups and usability testing with a computer-based risk assessment tool were conducted with PCPs from both university-affiliated and community-based practices. Analysis was derived from grounded theory methodology.
Results PCPs (n = 30) were aware of several risk assessment tools although only select tools were used routinely. The decision to use a tool depended on how use impacted practice workflow and whether the tool had credibility. Participants felt that embedding tools in the electronic medical records (EMRs) system might allow for health information from the medical record to auto-populate into the tool. User comprehension of risk could also be improved with computer-based interfaces that present risk in different formats.
Conclusions In this study, PCPs chose to use certain tools more regularly because of usability and credibility. Despite there being differences in the particular tools a clinical practice used, there was general appreciation for the usefulness of tools for different clinical situations. Participants characterised particular features of an ideal tool, feeling strongly that embedding risk assessment tools in the EMR would maximise accessibility and use of the tool for chronic disease management. However, appropriate practice workflow integration and features that facilitate patient understanding at point-of-care are also essential.
Where this study fits in
This is a qualitative study determining the perspectives of primary care physicians on health risk assessment tools.
Participants observed that computerisation of primary care practices offers scope to increase the routine use of health risk assessment tools.
EMRs can facilitate the administration of risk assessment tools as well as convey risk to patients in a meaningful manner.
The design and functionality of computer-based risk assessment tools should be expanded to improve the ease of use, display of results, and integration with clinical workflow.
- chronic disease
- disease management
- primary health care
- primary prevention
- risk assessment
- risk reduction behaviour
Commons license http://creativecommons.org/licenses/by/4.0/
Statistics from Altmetric.com
- chronic disease
- disease management
- primary health care
- primary prevention
- risk assessment
- risk reduction behaviour
Background
A major component of primary care practice is the identification and counselling of individuals at increased risk of chronic disease.1 Assessing and monitoring risk may be facilitated by various strategies including use of health risk assessment tools.2 Using predictive models based on epidemiological data, health risk assessment tools compute risk of developing a disease, and the resulting risk estimate is conveyed in numerical, text or visual formats.3,4 These estimates are used in a wide variety of contexts and for purposes including behaviour counselling,5,6 screening for health issues7–9 and decision making.10,11 The advantage of using these tools is that the computed risk information is individualised with a patient’s own risk factors, thereby making such information more meaningful. To leverage patient health information available in the medical record, some electronic medical record (EMR) systems have begun to incorporate risk assessment tools either within the EMR or link to external websites with risk assessment tools.12–14
Despite availability and potential utility, there are few published studies on the use of risk assessment in modern, computerised primary care practices. Previous studies have focused on themes such as physicians’ willingness to adopt computerbased health risk assessment tools,15 tool implementation,16 physicians’ understanding of risk scores17,18 and different modes of communicating risk estimates.19 However, none have used a mixed methods design to examine the role of primary care physicians (PCPs) in performing risk assessments with tools.
Methods
The objectives of this study were to investigate current practices for assessing risk, awareness and use of risk assessment tools in primary care, and to assess PCPs’ perspectives regarding the usefulness, usability and feasibility of implementing computer- based health risk assessment tools into routine clinical practice. Data collection and analysis followed a mixed methods approach employing focus groups and usability testing.
Participating PCPs were recruited from four settings: university-affiliated clinics (two sites) and community-based practices (three sites), in Toronto, Ontario and Edmonton, Alberta. All participants completed a brief questionnaire to collect demographic characteristics, information on awareness of common risk assessment tools and information on perceived usefulness of risk assessment tools at point-of-care (rated on a 5-point Likert scale).
Five focus groups (n = 25 participants) were held at primary care clinics. Focus groups were moderated by the study investigator with a semi-structured interview guide that was pilot-tested, and were held for 1 h. The guide consisted of open-ended questions to facilitate discussion of participants’ awareness of risk assessment tools, and views on their usefulness, usability and feasibility of routinely using them in clinical practice.
A usability study was conducted with five PCPs from each of the study settings who were not part of the focus groups. The objective of usability testing was to simulate completion of a risk assessment on a desktop computer in a PCP’s office using a computer-based tool at point-of-care. Using an opensource EMR program called Open Source Clinical Application and Resource (OSCAR),13 a mock patient chart was created. The participant was prompted by the study investigator to retrieve the patient’s information and use the Framingham Coronary Heart Disease and Stroke Risk Assessment Tool embedded in the EMR. The participant was asked to ‘think aloud’ as tasks were performed, describing their thought process and experience while using the EMR and the risk assessment tool.20 Each usability test lasted approximately half an hour.
Sampling
A selective sampling method, called snowball chain sampling, was used to identify physicians for the focus groups and usability study.21 At each location, PCPs were contacted to participate, and also inform colleagues about this study. Letters of invitation were sent to those who expressed interest in participating. A target sample of five to eight PCPs for each focus group and usability study were recruited. Informed consent was obtained from each participant and honoraria were provided. This study was approved by Research Ethics Boards from each of the universities and hospitals where the participating physicians practiced.
Data analysis
A convergent parallel mixed methods design was used whereby focus group and the usability study data were collected concurrently. During analysis, data from focus groups and usability testing were triangulated for comparison and corroboration of themes, strengthening the credibility of findings, and where there was overlap, emerging themes were pooled together.22–24 Grounded theory principles, derived from Strauss and Corbin,25 were employed in the study design, and analysis was conducted using NVivo software (QSR International). Focus groups and usability testing sessions were digitally recorded and transcribed. Data were analysed separately as they were captured, following the tenets of the constant comparative approach. Coding was guided by the study objectives and was performed in three stages. The open coding stage entailed tagging and categorizing transcript text into themes. Contextual and causal links between themes were made during the axial coding stage. Selective coding involved developing a framework unifying themes around a core concept. Redundancy indicated that themes were saturated.26 Analysis was completed independently by two investigators and consensus over differences was reached with help of a third collaborator. Memos were made following interviews and coding sessions to record thought process and were used to highlight issues of potential bias.
Results
Sample description
Demographic characteristics of participants are presented in Table 1. 77% of participants were women, the median age of participants was 37 years (range = 27–66 years), and the median number of years since graduating from medical school was nine (range = 1–44 years). Nearly, all participants had support from allied health professionals in their practices. The practices in this study were transitioning to, or already using an EMR system. Results are presented as six categories that align with the study objectives, and quotations from study participants exemplifying themes are included.
Current practices for assessing and communicating risk
Current risk assessment practices and strategies for integrating risk assessment tools into clinical routines were discussed. Themes emerged around who initiates risk discussions and how they occur, the frequency of risk assessment, and how risk is communicated with patients. Many participants explained that both physicians as well as patients are responsible for driving discussions about risk. While discussions of risk are integral to the annual physical exam, such questions may also be patient initiated. As an example, a PCP commented that a “[patient might] say something like, ‘My great-grandmother had breast cancer…What’s my risk of actually having breast cancer?’”. PCPs explained that risk assessments are performed on a daily basis. Nevertheless, they felt that risk communication is often a challenge, in large part because of varied levels of patient health literacy and numeracy: “The thing I still struggle with after 35 years of practice is conveying risk to people in language that actually has some meaning…like, how do you actually convey to people what is really meant by risk?”. Using a risk assessment tool, however, was identified as a possible method to assist in conveying quantitative risk to patients: “With our Framingham tool you get…, a yellow bar. The average person’s 10-year risk is…two little yellow bars, and your [the patient’s] average risk is…maybe five little yellow bars’”.
Awareness of specific risk assessment tools
All participants knew of the Framingham tool for cardiovascular disease risk27 and the FRAX fracture risk assessment tool.28 One-third of participants were also aware of the Gail model-based Breast Cancer Risk Assessment.29 Participants used a variety of tools including those that were based on paper, smartphone, web and stand-alone computer software. Participants had personal preferences for a tool and the medium used depending on how the information was displayed. Responses to the questionnaire indicated that not all participants were in favour of computer-based tools (on a 5-point Likert scale, mean = 3.0, SD = 1.0).
Perceived benefits and shortcomings of risk assessment tools
Using risk assessment tools was felt to be beneficial for initiating discussion, engaging patients in risk discussions, and guiding both physicians and patients around decision-making. Tools could be used with patients to “[initiate] that discussion in terms of what their risk factors are”, and engage in discussions about how lifestyle choices impact disease risk: “the patient can see very clearly, [and say] ‘Wow, smoking contributes a lot to cardiovascular risk, I do actually want to think about smoking cessation’”. Risk assessment tools were also seen as “a nice guide for physicians” that could assist in making decisions about whether advanced investigations are necessary: “I think FRAX [is] useful because [it] can cut down on unnecessary investigations”.
Specific concerns were expressed about using tools in practice: 1) new issues raised by the physician as a result of risk assessment may lead to unfocused discussions with patients; 2) balancing the patient’s desire for information about the risk of a particular disease with knowledge that there are no effective treatments and 3) lack of evidence underpinning the risk assessment tool and its credibility. Some participants, who worried about the impact of using risk assessment tools on clinical workflow, indicated that “it might bring up a lot more other issues that they [patients] weren’t originally aware of and the discussion might actually… be less directed”. Others felt differently, that “it usually stops a lot of the meandering dialogue that you’d otherwise engage in”. One participant questioned the value of personalised risk assessments for patients without a specific follow-up process or course of action: “Some people would like to know what their risk is, but does the fact that modern medicine has no specific approach to the condition…, does that negate why you are screening?” Another concern was that tools may not be based on credible evidence: “the underlying idea here is that we need to recognise the limitations of a score when we run it, and Framingham’s nice because there’s a lot of weight behind it”.
Expectations of an ideal risk assessment tool
Participants outlined the attributes of an ideal risk assessment tool, noting that integration with the EMR system was important, that the interface should be user-friendly, and the tool should be easily accessible. The complexity of the risk algorithms underlying many risk assessment tools necessitates the use of a computer, and many felt that integration with the EMR system was needed: “if there was a way of incorporating [patient’s EMR data] and saying that certain levels of morbidity trigger certain things to happen that might be useful?” Regarding user interface, several participants expressed that user interaction and flexibility in the display of information were important reasons to use computer-based tools: “I think about a patient of mine...who obsesses about numbers, his blood pressure numbers...Somebody like that would be ecstatic about a tool that he could kind of manipulate and be able to say, ‘If I could get my blood pressure down to this and my weight up to that’, this is what would happen”. Participants felt that the number of steps to complete a risk assessment should be minimised to a few clicks.
Feasibility of implementing risk assessment with computer-based tools in routine practice
Questions around implementation of risk assessment tools in clinical practice generated several suggestions for how risk assessment could be completed. For instance, “You could even have…patients fill in something on a tablet in the waiting room that populates our electronic record and creates a score and helps focus your time on modifiable risk factors”. Another idea that one participant envisioned was that “it wouldn’t necessarily be myself that would need to do [a risk calculation], it could be...our nurse practitioner…another allied health care worker”.
Participants were cognizant of limitations of computerbased tools, including tool interoperability between different EMR systems: “With respect to implementation, the issue will be that there are different systems that people are using, right? So, you know, Nightingale will be different from Practice Solutions” (in reference to two EMR system brands). Privacy was of concern, especially when transferring patient information to third-party risk assessment tools. In addition, a procedure for updating the risk algorithm with up-to-date information was raised as an issue: “what about the 42 installs that you just did yesterday”?
Usability of computer-based tools
Some participants were concerned about engaging with patients while looking back and forth at the computer screen. “If I’m typing on a computer and not really engaging the patient then they’re not going to take responsibility for their own health”. Many PCPs thought that risk assessment results should be printable so that patients could use them as a reminder of their health goals: “if we give them a piece of paper then they’ll actually go home and make the diet and exercise changes.…I find that a patient really likes to take home something. They can put this on their fridge”. One of the difficulties experienced during usability testing was that information could not be easily transferred from the patient record to the tool: “I’ll probably have to minimise one window whilst I have open the lab documents on the other [because] this is not…super-friendly about transposing the data”. Most participants emphasised the importance of a feature that could automatically load data into data fields, known as auto-populating.
Discussion
In this study, we used two methods to capture PCPs’ perspectives on computer-based health risk assessment tools. We found that participants in our sample perform risk assessments often, and are familiar with some risk assessment tools including computer-based programs. It was acknowledged, however, that risk assessment and communication remain challenging tasks and computer system integration is critical to the expanded use of risk assessment tools.
With respect to awareness of specific risk assessment tools, three tools were consistently mentioned by participants in this study, namely, the Framingham tool, the Gail model-based Breast Cancer Risk Assessment Tool and the FRAX tool. However, several reviews indicate that there are hundreds of tools available based on several published risk algorithms (e.g. Levy et al for cancer risk2; Rubin et al for fracture risk30). It is also noticeable that few diabetes risk tools were mentioned by participants despite the availability of several (e.g. Buijsse et al31). Our findings indicate that the discrepancy between awareness and availability of tools could be due to: perceived benefits of using a tool, beliefs about how well-validated a tool is, whether a tool is referred to in clinical practice guidelines, and concerns about implementation (whether a computer-based platform is available and if the tool is already integrated with the EMR system). Those tools which are not incorporated into guidelines are likely to be viewed as having little actionable utility.
PCPs’ opinions that risk assessment tools are helpful to communicate numeric risk information to patients resonates with previous research on using a risk assessment tool as a means to show how changing risk factors can affect risk, and using visual formats can help put numbers in perspective.32 Some focus group participants suggested that being able to show the risk assessment results on the computer screen and manipulating values in real time could help patients understand how their own risk is affected by risk factors. However, it was also expressed by usability testing participants that using computer-based tools could detract from patient engagement if the physician is busy trying to enter values into the system during an appointment, especially for tools requiring specific clinical data. This may indicate the necessity to distinguish tools used for the purpose of patient engagement, which require a quick, qualitative result at pointof- care, from more complex tools that require comprehensive health data inputs to produce a precise quantitative result and may have interfaces less suited to patient engagement. Many of the views expressed by PCPs were related to issues of implementation and usability such as impact of tool use on clinical work flow. Sposito et al,33 in a study on using cardiovascular disease risk assessment tools, found that respondents expressed concerns about the amount of time taken to use the Framingham tool. Yet, Halas et al6 found that the tool could be well integrated into practice without impeding clinical workflow.
The prevailing view of participants in our study was that risk assessment tools serve a number of beneficial purposes, however, evidence from other studies has been mixed. Saver et al34 evaluated how patients responded to personalised risk information presented with the UK Prospective Diabetes Study diabetes risk assessment tool and found it had little effect on patient attitudes, as some participants could not understand the information. A review of tools to identify women with increased risk of fractures found that there were no studies on effectiveness of tools on fracture outcomes.30 However, a three-armed randomised controlled trial evaluating the Framingham tool and HeartAge, two cardiovascular disease tools, found that disease risk was reduced when participants were given risk information.35
While our study used robust qualitative methods, the limitations warrant discussion. Qualitative studies are limited in their generalisability as a result of the interpretive nature of inquiry; however, transferability of findings may be strengthened through appropriate sampling methods and contextualisation of findings. In our study, we sought the opinions of PCPs from both academic and community-based practices in two Canadian provinces. This was done to maximise the source populations and obtain opinions from individuals in different contexts. It should be noted that all provinces in Canada are funded under a single payer, universal health insurance program. The findings here are likely transferable to non- single-payer systems as the identified themes were not related to payment scheme.
Conclusion
Our study is timely as the use of computer-based tools for risk assessment is becoming increasingly common in primary care. Participants in this study were cognizant of the difficulties of communicating risk to patients. Many benefits of risk assessment tools were discussed that prompted the characterisation of an ideal tool. Such a tool would be accessible, computerised or integrated with the EMR system, auto-populated with health data, and have an interactive user interface. While many of these features may be already available for some risk prediction tools, a compelling finding in this study was that participants showed an awareness of the challenges of tool implementation and integration with clinical workflow.
Recommendations arising from this study should help tool designers consider issues impacting tool use. Future qualitative work in this field might employ an ethnographic approach to further understand the impact of tool use on clinical workflow, wherein a researcher immerses him or herself into a situation to collect in-depth, naturalistic observations.36 Additional studies of patient perspectives on risk assessment tools, including around risk communication and interpretation will also be beneficial.
Acknowledgments
This study was supported by the Population Studies Research Network and Cancer Care Ontario. Sharon E. Straus is funded by a Tier 1 Canada Research Chair. Eva Grunfeld is supported by a clinician-scientist award the Ontario Institute for Cancer Research through funding provided by the Government of Ontario. The opinions results and conclusions reported are those of the authors and are independent from the funding sources, and no endorsement by OICR, CCO or the Government of Ontario is intended or should be inferred. The authors wish to thank Melanie Hammond Mobilio for her contributions to data analysis and Donna P. Manca for facilitating focus groups in Alberta, Canada.