Evaluation of ehealth systems and services

We need to move beyond hits and testimonials

For this article the term ehealth includes use of the internet or other electronic media to disseminate health related information or services.1 Many ehealth sites report huge numbers of users and testimonials of their benefits. Unfortunately, similar testimonials can be found for snake oil,2 and pornography sites boast large numbers of hits. Although a systematic review seeking evidence of harm from ehealth websites was inconclusive, few rigorous studies exist that show benefit from ehealth.3

Ehealth evaluations must move beyond testimonials and usage reports, but how much evaluation is enough depends primarily on the user. When creating internet applications, we can easily forget who our users are and what they need. We may forget that a woman with a diagnosis of breast cancer is facing a life threatening disease, must make critical decisions very quickly, and must cope with treatments that alter her quality of life dramatically. One can understand a student who creates a website about exercise that is hard to use, with content that does not match the needs of a user. But to create a website about pancreatic cancer that wastes a patient's or carer's time or misleads or distresses them is unforgivable. The appropriate level of evaluation depends on the needs of the user, the difference we can make and damage we can do. Hence, ehealth developers must first evaluate users' needs,4 then the product's risks and benefits, then its feasibility, and then its acceptability to the user.

This does not mean that we should police the internet, eliminating sites that are not based on these principles.5 However, people who use a website to support a decision about serious illness or injury deserve high quality content and confidence that the site's overall impact is beneficial. This requires testing in carefully designed simulation studies or, preferably, in randomised trials measuring salient outcomes such as health status and quality of life, health related behaviour, or social support.6^–8 Quantitative evaluations should be supplemented by qualitative studies where users explain what they feel when they use the system and how it affects their lives.

Developers need to evaluate usability to help them develop an appropriate interface for different kinds of users. The interface of an ehealth programme intended to keep teenagers from smoking will need to be different from one to help the caregiver of a patient dying of lung cancer. The website about smoking needs to create a tension for change and needs to be fun to use, whereas fast, easy access to key information may be much more important for a caregiver. For caregivers, flashy graphics should be provided only when they facilitate understanding.1 But both will want information that is accurate and up to date.

Cost implications are important but largely absent in ehealth evaluations.9 We should be examining how ehealth programmes affect the use and costs of healthcare relative to other options. How does the ehealth programme change the frequency of visits to doctors compared with distributing pamphlets? What is the cost effectiveness of different ehealth delivery venues—for example, is it practical to expect caregivers of stroke patients to use a public library's computers when they don't even have time to bathe?

Because some studies show that underserved populations benefit substantially from ehealth, evaluations should address the conditions and means by which sociocultural factors should influence the design and do influence the impact of such systems.8 Cultural sensitivity may be more important in ehealth systems tackling primary prevention (where people need to be convinced to change) than in programmes tackling pancreatic cancer (where people are desperate for any help they can get). Finally, evaluations need to determine how people use various ehealth features and how such use relates to health outcomes.8 10 Such evaluations will tell us how to enhance systems11 and may help determine what healthcare functions can be delegated to a website—a fundamental issue facing researchers and policy makers.12

The internet has come far but has a long way to go before it approaches its potential for improving lives. People facing life changing illness deserve the very best systems we can develop. Such systems must be informed by solid evaluation studies that help us understand what features and content work, how they work, for whom, the risks they pose, and the financial impact they have. How much evaluation is enough? The answer will vary by the type of user served. But one thing is certain: testimonials and website hits are no longer sufficient.