Performance of national COVID-19 ‘symptom checkers’: a comparative case simulation study

Introduction

COVID-19 is a new infection in humans. The symptom profile, disease progression and complication rates are still relatively unknown.1 From the available evidence, four broad categories of illness have been postulated. ‘Mild COVID-19’ makes up over 80% of cases and is typically a self-limiting infection similar to the common cold, resolving without intervention. ‘moderate COVID-19’ typically has features of viral pneumonia in the absence of hypoxia, progressing to ‘severe COVID-19’ typically when patients require oxygen therapy. ‘Critical COVID-19’, where ventilatory support is typically required, occurs in less than 5% of cases.2 The rate of disease progression is not fixed: early intervention and various management strategies can reduce the rate of progression to critical illness and death.2–6

While the infection fatality rate is yet to be determined, COVID-19 is associated with a substantive mortality. Over a period of 5 months, COVID-19 has led to more than 300 000 deaths, with more than half these deaths occurring within the last month.7

The risk of mortality is affected by a number of risk factors. Coexisting health problems such as diabetes, heart disease and cancer have been implicated as conferring a higher risk of mortality in COVID-19.8 Age appears to be the most striking and consistent risk factor for COVID-19 related mortality.9 Based on current data, the mortality rate in patients under 50 years of age is thought to be less than 1.1%, rising to around 14% in those over 80 years of age.10

Variation in mortality also seems to exist between countries.11 Initially, this variation was thought to be predominantly related to the method of recording deaths and the total number of tests conducted (ie, the detection of milder cases).12 As the pandemic spreads across the globe, it is becoming increasingly clear that how a country responds to the pandemic impacts the number of deaths their locality will experience.6 11

The national response to the COVID-19 pandemic has many important tenets. On the public health side, infection control initiatives attempt, in part, to mitigate the surge of infections that can accompany new pathogens where there is little circulating immunity. This reduces mortality by preventing the healthcare services from being overwhelmed, thus permitting improved access to medical management for those who need it.6 The clinical response to COVID-19 also centres on access to treatment. To successfully reduce the mortality rate, those patients who are developing more severe disease must be identified.3

Identifying those patients with COVID-19 that require treatment is challenging. First, COVID-19 has a broad range of presentations that can mimic common conditions that rarely require clinical assessment (eg, the common cold).1 Second, there are no clinical signs or symptoms that reliably predict who will progress to severe disease.3 As such, the clinical community is left with a large number of potential cases without any clear symptom indicators for: (1) who has the disease and (2) who is developing more severe disease. The problem is compounded further as more serious, life-threatening conditions (eg, bacterial pneumonia and sepsis) can mimic any stage of COVID-19 disease.13 14

National ‘Symptom Checkers’ have been implemented in many countries in the hope of reducing this burden faced by healthcare services. Symptom checkers are self-assessment tools. The individual—typically online or via computer application—enters their symptoms into a predetermined platform and from there a predetermined algorithm produces an outcome (usually advice). This is a form of self-led triage. It is hoped that such self-directed assessments will enable the identification of potential cases15 and will correctly triage those individuals who would benefit from clinical assessment and/or management into further care.16 For such a hope to be realised, symptom checkers must be able to determine mild conditions from severe conditions.

While self-triage has been used for some years in non-emergency conditions to varying degrees of success,17 self-triage has never before been used in a pandemic setting and as yet the efficacy and safety has not be formally studied. Caution must be exercised as, to date, studies examining symptom checkers have had mixed and disappointing results in general—demonstrating poor diagnostic performance (34%–58%) and questionable triage performance (55%–80%).18 The stakes are high, in that a failure to triage serious medical conditions (such as severe COVID-19, bacterial pneumonia or sepsis) in for further assessment will inevitably lead to delayed treatment and higher mortality.19–22

Here, we test the performance of four nationwide symptom checkers from four nations to ascertain how safe and efficient each symptom checker is in differentiating mild from severe COVID-19 cases, and how well they detect time-sensitive COVID-19 mimickers such as bacterial pneumonia and sepsis.

Methodology

Five countries were initially selected for analysis. Three (Singapore, Japan and Norway) were selected as they maintained low case fatality rates (CFRs) despite a demonstrable surge of cases in the preceding 2 months. Two countries (the UK and the USA) were selected due to concern regarding high CFRs.

Public health guidelines from each country were reviewed. Access was obtained to any available government sponsored online patient-led triage system (Singapore: ‘Singapore COVID-19 Symptom Checker’,23 Japan: ‘Stop COVID-19 Symptom Checker’,24 USA: ‘CDC Coronavirus Symptom Checker’25 and the UK: ‘111 COVID-19 Symptom Checker’26). Whereas the NHS ‘111’ COVID-19 Symptom Checker was and continues to be heavily used (with over 500 000 assessments completed on average each month27), there was no available data as to the usage of the other symptom checkers.

For the purpose of this analysis, data were extracted only from those countries with symptom checkers (Singapore, Japan, UK and USA), in an effort to compare the performance of symptom checkers specifically.

Results

The key baseline population and testing data are presented in table 1. Notably, the highest rate of testing for COVID-19 was by Singapore with the lowest being Japan. The UK had the highest reported physicians per capita, while Japan and Singapore had the lowest. Cases per thousand inhabitants varied greatly, with Singapore and the UK maintaining similar rates. From the available statistics, Singapore had the lowest CFR (<0.1%) and the UK had the highest CFR (13.6%) currently. All population and testing data were extracted from The WHO as of 26 April 2020.

Fifty-two case scenarios were applied to each country’s patient-led triage systems. The results for each scenario are presented in tabulated format (online supplemental data). Singapore had the highest overall referral rate at 88%, and the USA had the lowest at 38% (table 2).

From the cases not referred, the USA and UK triaged a significant number of cases to ‘stay home’ that would typically have required early clinical assessment. The US triage system (CDC Coronavirus Symptom Checker) frequently triaged home case simulations with possible severe COVID-19, bacterial pneumonia and sepsis and triaged possible neutropaenic sepsis to healthcare contact within 24 hours. The UK’s 111 COVID-19 Symptom Checker frequently triaged possible severe COVID-19 and bacterial pneumonia to stay at home with no follow-up and is likely to have delayed treatment for sepsis, severe COVID-19 and neutropaenic sepsis. It is of note that while Japan’s symptom checker generally performed well, our simulation revealed a potential delay to treatment for neutropaenic sepsis. Indeed, all four symptom checkers failed to triage the simulation for neutropaenic sepsis into the ‘emergency department’ (table 3).

Discussion

This case simulation study examined the symptom trackers from four countries. Following application of 52 standardised case simulations to each country’s symptom checker, the percentage of onward referrals were calculated. The low case fatality nations’ (Singapore and Japan) symptom checkers triaged in twice as many cases for direct clinical assessment than the higher case fatality nations (the USA and UK). Of clinical concern was the failure of both the US and UK symptom checkers to triage cases simulating bacterial pneumonia, sepsis and severe COVID-19 on to any healthcare contact.

The upside of symptom checkers, particularly during a pandemic is difficult to ignore. By reducing physical patient contacts, symptom checkers can potentially save valuable resources and avoid further viral transmission. While telephone and telemedicine triaging also protects staff and reduces transmission, such services require more healthcare staff than symptom checkers and hence carries a greater financial and human resource burden.

Evidence to date also suggests the majority of cases of COVID-19 resolve after a short, self-limiting viral illness.1 There are, though, no discriminatory signs or symptoms.2 COVID-19 can present like the common cold or influenza or indeed bacterial pneumonia. COVID-19 can also progress quickly6 35 and can even present with asymptomatic hypoxia.32 Sifting through the mild colds and self-limiting flus and trying to determine who will have a mild course of COVID-19 and also trying not to miss bacterial pneumonia, sepsis and signs of COVID-19 pneumonia is a challenge for even trained clinicians let alone an automated system.

It is here where Singapore’s symptom checker performs well. The checker is presented on a single webpage, more akin to an online risk calculator. There are six inputs required from the patient and one of three outputs generated. The algorithm powering the symptom tracker is not complicated. Age over 65 years, or the presence of any health condition, or duration of symptoms over 4 days triggers the advice to seek medical assessment. Any degree of shortness of breath is triaged directly to the emergency department. The Singapore COVID-19 Symptom Checker, if used by the public, is likely to reduce healthcare contacts by the young, fit patients who are early on in the illness, thus off-loading the healthcare burden to some degree while maintaining a relatively low risk to the public.

The UK ‘111’ symptom checker performs poorly in this regard. The algorithm is complex, attempting to quantify symptoms such as shortness of breath and the overall severity of illness by asking subjective, qualitative questions with multiple choices. The ‘111’ symptom checker seems to take on a much broader clinical role and attempts to triage out cases that would typically be triaged in or out of care based on an actual clinical assessment. For example, a 72 year old person who presents with a 7-day history of fever and cough is triaged by the ‘111’ symptom checker to stay at home with no clinical, nursing or healthcare contact. Faced with such a clinical scenario, clinicians would typically insist on at least basic clinical observations (pulse, temperature, oxygen levels and so on) before considering triaging such a patient to stay at home. The differential in this case includes sepsis, bacterial pneumonia and COVID-19 pneumonia, and while it remains possible that fever can persist for 7 days in mild/moderate COVID-19, complications or alternative diagnoses are much more likely.

The qualifying questions used by the ‘111’ symptom checker to discriminate between severity will have insufficient discriminatory value in such cases. Furthermore, the wording of the question encourages the self-reporting towards lower categories of illness:

Are you so ill that you have stopped doing all of your usual daily activities?

1. ‘Yes - Ive stopped doing everything I usually do’.

2. ‘I feel ill but can do some of my usual activities’.

3. ‘No - I feel well enough to do most of my usual activities’.

(Extracted question from ‘111’ Coronavirus Symptom Checker).

It is the use of absolute and equivocal qualifiers that prevent the severity-qualifying question from achieving any useable clinical triage information: the use of ‘all’ in the question, ‘everything’ in the affirmative answer, and even the negative answer stipulates ‘most’. Our case simulation demonstrated that answering B, the moderately severe answer, still triages patients to self-isolate with no healthcare contact. As such, patients with cough and fever for 7 days would have to be so severely unwell that they are unable to do anything they usually do to be triaged to any clinical contact.

Our case simulation study indicates that both the ‘111 COVID-19 Symptom Checker’ and the ‘CDC Coronavirus Symptom Checker’, if used as the sole initial point of healthcare contact, are likely to delay presentations of serious medical conditions to appropriate care, and as such, are likely to confer an increased risk of morbidity and mortality. Both symptom checkers maintain a high threshold for referring onward to clinical contact, triaging the majority of patients to stay home with no clinical contact. Again, beyond the mortality impact, there is no evidence that such an approach actually reduces healthcare burden. Indeed, beyond the established evidence in pneumonia generally,19–22 there is direct evidence that early correction of hypoxia in COVID-19 prevents progression to mechanical ventilation,5 consistent with basic medical principles. Programming symptom checkers to aggressively triage patients to stay home may well lead to patients presenting to healthcare later, requiring more intensive healthcare to recover, and as such, symptom checkers ‘set’ to keep patients at home may actually increase the burden on intensive care facilities and perpetuate a healthcare crisis.

Symptom checkers are currently being used in the pandemic for two purposes: (1) identifying potential cases for testing/surveillance and (2) identifying ‘unwell’ patients who require medical attention. Both functions are likely to be enhanced by the use of symptom checkers when the intention is to ‘catch’ more patients or reach more cases. That is, when symptom checkers are used to identify more cases than would otherwise be detected and to direct more patients to medical care than would otherwise make healthcare contact, then symptom checkers are merely providing an additional ‘safety-net’, and therefore, in such a healthcare support role, the risk of harm from their use is expected to be relatively minimal. Conversely, if symptom checkers are being used to replace the assessment of patients by trained personnel and are programmed to try and prevent further healthcare contact, then, as our case simulation study highlights, there are real concerns about the potential risk of harm from such an unproven approach.

Considering that the efficacy of symptom checkers have not been established,17 caution would be advisable. Delay in the correction of hypoxia, failure to commence thromboprophylaxis and missing the opportunity for earlier initiation of steroids in the hypoxic patient with COVID-19, are all likely to carry a considerable morbidity and mortality cost.

If we are to accept the lesser option of an automated, self-directed triage system over the standard of care offered by the dynamic, experienced clinical assessment, then we must be mindful of what we are asking of the ‘symptom checker’. Based on our independent case simulation study, symptom checkers do not appear advanced enough to fulfil the ‘stay home’ intent with any sufficient level of safety. They may though be sufficient enough to assist in the improved identification of at risk patients requiring further clinical assessment, and some form of symptom checker may even be able to contribute to the increased ongoing vigilance required for all patients diagnosed with COVID-19. Evidence, though, should be provided before replacing actual clinical contact with an online self-directed triage system.

Conclusion

In this case simulation study, the UK and USA patient-led triage systems (COVID-19 Symptom Checkers) maintained a high disease-severity threshold for onward referral to healthcare assessment. Particular concerns were advising no clinical contact for elderly patients with COVID-19 related symptoms or patients who had developed shortness of breath or any patient with persistent fever. The low CFRcountries (Singapore and Japan) used symptom checkers to reduce clinical demand while maintaining a lower health risk to patients. Our study indicates that while symptom checkers may be of use in the healthcare response to COVID-19, the ‘CDC Coronavirus Symptom Checker’ and the ‘111 COVID-19 Symptom Checker’, if used as the sole point of initial healthcare contact, are likely to confer a tangible risk of delaying the presentation of time-critical acute illnesses. Our results support the recommendation that symptom checkers should be subjected to the same level of evidenced-based quality assurance as other diagnostic tests prior to implementation.