You are here

Article Text

Article menu

Download PDFPDF

XML
Short report
Outcome measurement in speech and language therapy: a digital journey
  1. Kathryn Moyse1,
  2. Pamela Enderby2,
  3. Katie Chadd1,
  4. Kamini Gadhok1,
  5. Mark Bedwell3 and
  6. Patrick Guest3
  1. 1Royal College of Speech and Language Therapists, London, United Kingdom
  2. 2Health Services Research, The University of Sheffield, Sheffield, UK
  3. 3Different Class Solutions Ltd, Keele, United Kingdom
  1. Correspondence to Kathryn Moyse; Kathryn.moyse{at}rcslt.org

Abstract

Background Evidencing the impact of speech and language therapy interventions is challenging. The UK’s professional body for speech and language therapists (SLTs) is supporting a consistent approach to outcome measurement and analysis using Therapy Outcome Measures (TOMs).

Objective To develop a digital solution for collecting TOMs data, evaluate the impact of therapeutic interventions and explore contributing factors to outcome variation across clinical areas.

Method Agile methodology was applied to software development. Organisations were recruited to provide data. Criteria were identified to exemplify outcome variability.

Results A digital tool was developed. 21 organisations provided data on 16 356 individuals. Improvement in at least one domain of TOMs occurred in 77.1% of instances. Data for two clinical areas exemplify the tool’s effectiveness in highlighting the impact of speech and language therapy.

Conclusion This established outcomes data set can be used to evaluate the impact of speech and language therapy, and explore variation in outcomes.

  • information systems
  • information management
  • health care
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjhci-2019-100085

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Speech and language therapists (SLTs), who treat individuals of all ages with a broad range of communication and swallowing needs, face the challenge of providing evidence-based practice and continually improving service delivery. Clinicians recognise the importance of quality improvement, but in practice, achieving this can be challenging due to gaps in research.1 Furthermore, the existence of different service delivery models, including variation in the level, type and frequency of provision,2 may impact on patient outcomes.3 4 Comparing the outcomes of individuals accessing different services and reviewing the factors which affect progress could complement the existing evidence base. Furthermore, collecting data on all individuals would place the information from randomised controlled trials (RCTs) into the broader clinical context. RCTs often exclude individuals with certain personal, demographic or multimorbid clinical attributes in order to maximise internal validity5 yet such attributes describe the majority of patients.

    Increasingly, the value of applying ‘real-world’ data to support quality services, is recognised, with the role of information and digital technology frequently highlighted in national policies.6–10 Nevertheless, there are numerous barriers for SLTs in the UK, including the absence of tools to support data collection and analytics. In recognition of this, the Royal College of Speech and Language Therapists (RCSLT) is undertaking a programme of work on outcome measurement, data collection and analysis with the aim of supporting SLTs with delivering quality services.

    To promote a more consistent approach to outcome measurement across the profession the RCSLT undertook a review of 63 candidate measures.11 Following a synthesis and Delphi consensus approach,12 Therapy Outcome Measures for Rehabilitation Professionals (TOMs)13–16 was identified as an appropriate existing measure that satisfied key criteria and was selected for the project.17

    Aims

    1. To develop a digital solution to support collection and analysis of outcomes data to evaluate the overall impact of speech and language therapy interventions.

    2. To evaluate the impact of speech and language therapy for individuals presenting with the same speech, language, communication or swallowing diagnosis, but with different underlying medical conditions.

    3. To evaluate the impact of speech and language therapy for individuals presenting with the same speech, language, communication or swallowing diagnosis receiving care from different services.

    Methods

    A basic prototype tool for data collection was developed by Different Class Solutions Ltd, who took an Agile approach18 to develop the RCSLT Online Outcome Tool (ROOT).19 Agile methodology prioritises the production and delivery of working software in consultation with end users. End users (‘pilot sites’) were recruited through the RCSLT membership. To ensure the pilot group represented the profession, applicants who were experienced in using TOMs were profiled against a range of criteria, including organisational funding, clinical speciality and geographical region. Sites were consulted and suggested changes to the initial ROOT prototype. Since the initial pilot period, the number of sites involved has expanded.

    To develop, test and use the ROOT, sites provided de-personalised data on all individuals referred for speech and language therapy intervention. There were no recruitment/inclusion criteria. A privacy-friendly approach was adopted to minimise the amount of personal data collected by the ROOT, including age, gender, diagnoses and TOMs ratings at the beginning and end of an episode of care. Advice regarding ethics permission for use of this anonymised audit data for this project was sought and deemed not to be necessary. Sites were required to seek approval from their employing authorities to be involved and followed local processes about informing individuals about how information was being used.

    The General Data Protection Regulation20 came into force during the course of this project. The RCSLT sought support from national bodies, including the Information Commissioner’s Office,21 to ensure that the guidance provided for the ROOT was in line with best practice.

    To demonstrate the ROOT’s utility, data collected by sites over 36 months is presented. Specific examples are provided on clinical areas where a high volume of data has been collected by multiple sites. To evaluate the outcomes of individuals presenting with the same communication or swallowing diagnosis, but with different underlying medical conditions, a clinical area was selected where it can exist due to multiple underlying aetiologies. To evaluate the outcomes of individuals presenting with the same communication or swallowing diagnosis receiving care from different services, a minimum of six completed episodes of care per site for the specified clinical area was required.

    Results

    Twelve speech and language therapy teams/services from across the UK volunteered to co-produce and pilot the ROOT.

    The ROOT facilitates the collection and analysis of outcome data along with other information, such as International Classification of Diseases (ICD-10) codes.22 Two data collection approaches were developed. Clinicians can enter data directly into the ROOT, or, where data is collected in local electronic systems, this can be uploaded to the ROOT in bulk. Sites determined that they would need tailored reports for different purposes and audiences (eg, SLTs, commissioners, service managers, individuals accessing services and the professional body). Thus, different levels and style of analysis were developed with the ability to refine outcomes data through applying a range of filters as required.

    Twenty-one services were recruited (table 1). The sites spanned the UK and were predominantly National Health Service funded. The sites contributed data for between 5 and 8780 episodes of care over the time period.

    View this table:
    Table 1

    Summary of sites involved in the 36 month data collection period (1 June 2016 to 31 May 2019)

    Collectively, they have provided data on 16 356 individuals who have received a total of 17 865 episodes of care for a broad range of communication and swallowing needs (table 2).

    View this table:
    Table 2

    Number of episodes of care delivered by speech and language therapists from 21 sites over 36 months (1 June 2016 to 31 May 2019)

    Across an episode of care, an increase of 0.5 or more on the TOMs is a clinically significant change.15 Table 3 indicates that 22.9% of individuals did not make a clinically significant improvement on any domain of the TOMs and 26.5% improved in four domains, 16.2% improve in one domain, 18% improve in two domains, 13.8% improve in three domains and 2.6% improve in five domains, but it should be recognised that carer well-being is only recorded when appropriate. Overall, improvement in one or more domains occurred in 77.1% of episodes of care.

    View this table:
    Table 3

    Number and percentage of episodes of care in which a clinically significant (positive) change occurred across zero to five domains of the TOMs

    To evaluate the outcomes of individuals presenting with the same communication or swallowing diagnosis, but with different underlying medical conditions, aphasia was identified as a clinical area fulfilling the criteria. Figure 1 compares 469 people with aphasia as a consequence of stroke with 483 people with aphasia associated with other neurological incidents. It indicates a higher proportion of individuals with stroke-related aphasia make clinically significant improvements across each of the domains compared with those with non-stroke-related aphasia.

    Figure 1
    Figure 1

    Proportion of episodes of care showing clinically significant improvement, maintenance or decline between the start and end of Therapy Outcome Measures ratings for each domain (1a-1e) for 469 individuals with stroke-related aphasia (507 episodes of care) and 483 individuals with non-stroke-related aphasia (531 episodes of care). Data from nine sites collected from 1 June 2016 to 31 May 2019.

    To evaluate the outcomes of individuals presenting with the same communication or swallowing diagnosis receiving care from different services, developmental language disorder (DLD) was identified as a clinical area fulfilling the criteria.

    Table 4 presents data for individuals with DLD from three sites working with this client group, and the ROOT total for DLD. It indicates that the average change in TOMs domains in the ROOT total is clinically significant across all domains. However, outcomes from individual services are varied.

    View this table:
    Table 4

    The average change in TOMs scores, across each domain (impairment, activity, participation, well-being and carer well-being)

    Discussion

    The ROOT provides UK SLTs with a means of collecting and analysing outcomes data. Developing a digital solution meeting the requirements of SLTs working across a range of clinical groups, settings and with access to different information systems has been challenging. However, co-producing the ROOT with SLTs and using an Agile approach has been supportive in developing a user-friendly, intuitive tool that can assist SLTs with utilising their outcomes data.

    The expanding outcomes data set is supporting the speech and language therapy profession to evaluate its overall impact within the local context. Table 2 highlights the range of clinical areas SLTs support, indicating dysphagia outcomes are most frequently collected. However, the data set will be skewed as individual sites contributed differing amounts of data (table 1) since caseload size was not controlled for, so this is not necessarily the most prevalent disorder encountered by SLTs.

    The results show that positive, clinically significant improvement in one or more domains occurs in 77.1% of episodes of care (table 3) and is most often seen in four domains of the TOMs (26.5%). This is not surprising as speech and language therapy is a profession contributing to rehabilitation and enablement, which is broader than reducing the disorder alone23 24 and may reflect the holistic and personalised care provided by SLTs. Interventions include providing strategies to improve communication or swallowing, enhance participation socially, educationally and in employment, along with attending to the well-being of the individual and their family. As interventions will not always target all domains of the TOMs, it is not unexpected that there is rarely change in five domains (2.4%). Table 3 also indicates that 22.9% of individuals do not show positive change on the TOMs over an episode of care. Further inspection of the data would provide information on the types of individuals who do not make improvement, including individuals with progressive conditions, for whom maintenance of function or carefully managed decline is the expected outcome.

    The ROOT is beginning to provide insight into the impact of speech and language therapy for individuals with different underlying medical conditions (figure 1) and receiving care from different sites (table 4). The data presented illustrates variation in outcomes between aetiology and service delivery. We recognise this is only descriptive and indicative, and requires further investigation in order to establish significance.

    Figure 1 illustrates trends in the outcomes for individuals with stroke-related aphasia compared with non-stroke-related aphasia. Across all five domains of the TOMs, a higher proportion of individuals with stroke-related aphasia show improvement post-intervention. Speculation on the reasons for this include better and more immediate access to services (such as dedicated stroke units and rehabilitation teams) and possibly a lower level of complexity of needs (such as those experienced following traumatic brain injury, for instance, cognitive impairments). Only a few small studies have made comparisons between stroke and non-stroke aphasia25 26 and this data provides an opportunity to complement these studies with ‘real world’ data.

    Outcomes for similar clinical groups can be compared through benchmarking, which ‘as a component of Quality Management, offers a continuous process by which an organisation can measure and compare its outcomes overtime with peer organisations and use the findings to inform management decision making’.27 Table 4 indicates some variation in outcomes between different sites for a given communication or swallowing diagnosis, reflected elsewhere in the literature.28 29 The ROOT total shows a clinically significant increase in every domain. Sites 1 and 2 are broadly in line with this average, while Site 3 achieves beyond this. Potentially, this is related to the service provision offered, different referral patterns or variation in the severity of impairment at therapy commencement. The reasons for this variation will need to be explored, which then could be considered by other services to support quality management.

    This innovative and ambitious project has demonstrated the utility of ‘Big Data’30 and has equipped the profession with robust data to evidence its impact and use in national influencing. Outcomes data can be valuable in looking at ‘real world’ change across a range of contexts, without applying stringent participant criteria and should be used to complement and facilitate interpretation of the existing evidence base. Nonetheless, we acknowledge there are some methodological limitations. Outcomes are not compared with control groups, nor are variables controlled for within-groups, thus, isolating potential agents of change is limited in comparison to RCTs. Yet, the data is advantageous in other respects. For example, it contains information on individuals that tend to be excluded from RCTs.

    While the ROOT has a notable volume of data, wider implementation of the ROOT would increase its validity. The main barrier to implementation has been delays in approval due to uncertainty around the new data protection legislation. To support prospective sites, an Information Governance Pack was developed31 summarising key information about the ROOT in relation to data processing, online security and risk mitigation. As more SLT services use the ROOT, we can be more confident in assumptions about the data being representative of the range of services provided by SLTs across different clinical groups and settings.

    The data in the ROOT is already beginning to demonstrate its value and case studies are emerging which document use of the ROOT to highlight the impact of speech and language therapy to senior directors, commissioners/funders and evaluate where interventions are having most impact and identify areas for improvement.32–34 Future work should focus on application of the ROOT data to answer clinical uncertainties and for quality management. Furthermore, as the volume of data increases, it may provide insight into complex clinical questions currently unanswered by the traditional research methods.

    This project illustrates that the implementation of a digital solution is about more than introducing new technology. This is part of a larger system change, and working collaboratively to identify what data to gather, how to gather it and make use of the information, along with consideration of the practicalities and minimising the barriers to implementation, is essential. We have shown that our profession can use digital innovation to collectively demonstrate the impact of speech and language therapy, with the potential to inform the way we deliver care and improve outcomes.

    Acknowledgments

    We thank the RCSLT Outcomes Programme Steering Group and the speech and language therapy services involved in the work for their participation and contributions.

    References

    1. ↵
      1. Lynch EA,
      2. Chesworth BM,
      3. Connell LA
      . Implementation-The missing link in the research translation pipeline: is it any wonder no one ever implements evidence-based practice? Neurorehabil Neural Repair 2018;32:751–61.doi:10.1177/1545968318777844pmid:http://www.ncbi.nlm.nih.gov/pubmed/29848171
      OpenUrlPubMed
    2. ↵
      1. Bercow J
      . The Bercow report: a review of services for children and young people (0-19) with speech, language and communication needs, 2008. Available: https://dera.ioe.ac.uk/8405/7/7771-dcsf-bercow_Redacted.pdf [Accessed 4 Nov 2019].
    3. ↵
      1. Roulstone S,
      2. Enderby P
      . The end of an affair: discharging clients from speech-language pathology. Int J Speech Lang Pathol 2010;12:292–5.doi:10.3109/17549501003770277pmid:http://www.ncbi.nlm.nih.gov/pubmed/20590508
      OpenUrlPubMed
    4. ↵
      1. Roulstone S,
      2. Wren Y,
      3. Bakopoulou I, et al
      . Interventions for children with speech, language and communication needs: an exploration of current practice. Child Lang Teach Ther 2012;28:325–41.doi:10.1177/0265659012456385
      OpenUrl
    5. ↵
      1. Van Spall HGC,
      2. Toren A,
      3. Kiss A, et al
      . Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA 2007;297:1233–40.doi:10.1001/jama.297.11.1233pmid:http://www.ncbi.nlm.nih.gov/pubmed/17374817
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. NHS England
      . The NHS long term plan, 2019. Available: https://www.longtermplan.nhs.uk/wp-content/uploads/2019/08/nhs-long-term-plan-version-1.2.pdf [Accessed 4 Nov 2019].
    7. ↵
      1. NHS England
      . A digital framework for allied health professionals, 2019. Available: https://www.england.nhs.uk/wp-content/uploads/2019/04/a-digital-framework-for-allied-health-professionals.pdf [Accessed 4 Nov 2019].
    8. ↵
      1. Scottish Government
      . Scotland's digital health and care strategy: enabling, connecting and empowering, 2018. Available: https://www.gov.scot/publications/scotlands-digital-health-care-strategy-enabling-connecting-empowering/ [Accessed 4 Nov 2019].
    9. ↵
      1. Welsh Government
      . Informed health and care: a digital health and social care strategy for Wales, 2015. Available: https://gov.wales/sites/default/files/publications/2019-03/informed-health-and-care-a-digital-health-and-social-care-strategy-for-wales.pdf [Accessed 4 Nov 2019].
    10. ↵
      1. Health and Social Care Board
      . eHealth and care strategy for Northern Ireland, 2016. Available: https://www.health-ni.gov.uk/sites/default/files/publications/dhssps/interactive-ehealth-strategy.pdf [Accessed 4 Nov 2019].
    11. ↵
      1. Powell G,
      2. Lowenthal D
      . Outcomes and outcome measures. RCSLT Bulletin 2014;749:22–4.
      OpenUrl
    12. ↵
      1. Fink A,
      2. Kosecoff J,
      3. Chassin M, et al
      . Consensus methods: characteristics and guidelines for use. Am J Public Health 1984;74:979–83.doi:10.2105/AJPH.74.9.979pmid:http://www.ncbi.nlm.nih.gov/pubmed/6380323
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Enderby P,
      2. John A
      . Therapy outcome measures: speech-language pathology. San Diego, London: Singular Publishing Group, Inc, 1997.
    14. ↵
      1. Enderby P,
      2. John A,
      3. Petheram B
      . Therapy outcome measures for rehabilitation professionals: speech and language therapy, physiotherapy, occupational therapy, rehabilitation nursing & hearing therapists. 2nd edn. Chichester: John Wiley & Sons Ltd, 2006.
    15. ↵
      1. Enderby P,
      2. John A
      . Therapy outcome measures for rehabilitation professionals. 3rd edn. Croydon: J & R Press Ltd, 2015.
    16. ↵
      1. Enderby P,
      2. John A
      . User guide: therapy outcome measures. Croydon: J & R Press Ltd, 2019.
    17. ↵
      1. Enderby P,
      2. Moyse K
      . International classification of Functioning—An approach to outcome measurement. Perspect ASHA Spec Interest Groups 2018;3:99–108.doi:10.1044/persp3.SIG17.99
      OpenUrl
    18. ↵
      1. Cockburn A,
      2. Highsmith J
      . Agile software development, the people factor. Computer 2001;34:131–3.doi:10.1109/2.963450
      OpenUrlCrossRef
    19. ↵
      1. Royal College of Speech and Language Therapists,
      . Different class solutions Ltd. RCSLT online outcome tool (ROOT), 2019. Available: www.rcslt-root.org [Accessed 4 Nov 2019].
    20. ↵
      Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General data protection regulation).
    21. ↵
      1. Information Commissioner’s Office
      . About the ICO. Available: https://ico.org.uk/about-the-ico/ [Accessed 4 Nov 2019].
    22. ↵
      1. data.gov.uk
      . NHS classifications ICD-10. Available: https://data.gov.uk/dataset/add8ff17-b45e-4169-a826-c5f634f3cccb/nhs-classifications-icd-10 [Accessed 4 Nov 2019].
    23. ↵
      1. Wade DT,
      2. Collen FM,
      3. Robb GF, et al
      . Physiotherapy intervention late after stroke and mobility. BMJ 1992;304:609–13.doi:10.1136/bmj.304.6827.609pmid:http://www.ncbi.nlm.nih.gov/pubmed/1559090
      OpenUrlAbstract/FREE Full Text
    24. ↵
      1. King’s Fund
      . Intermediate Care: Models in Practice. King’s Fund Publishing, 1999. Available: https://www.kingsfund.org.uk/publications/intermediate-care [Accessed 4 Nov 2019].
    25. ↵
      1. Vukovic M,
      2. Vuksanovic J,
      3. Vukovic I
      . Comparison of the recovery patterns of language and cognitive functions in patients with post-traumatic language processing deficits and in patients with aphasia following a stroke. J Commun Disord 2008;41:531–52.doi:10.1016/j.jcomdis.2008.04.001pmid:http://www.ncbi.nlm.nih.gov/pubmed/18571195
      OpenUrlCrossRefPubMed
    26. ↵
      1. Davie GL,
      2. Hutcheson KA,
      3. Barringer DA, et al
      . Aphasia in patients after brain tumour resection. Aphasiology 2009;23:1196–206.doi:10.1080/02687030802436900
      OpenUrlCrossRefWeb of Science
    27. ↵
      1. Benson HR
      . An introduction to benchmarking in healthcare. Radiol Manage 1994;16:35–9.pmid:http://www.ncbi.nlm.nih.gov/pubmed/10139084
      OpenUrlPubMed
    28. ↵
      1. Enderby PM,
      2. John A
      . Therapy outcome measures in speech and language therapy: comparing performance between different providers. Int J Lang Commun Disord 1999;34:417–29.doi:10.1080/136828299247360pmid:http://www.ncbi.nlm.nih.gov/pubmed/10884909
      OpenUrlPubMed
    29. ↵
      1. John A,
      2. Enderby P,
      3. Hughes A, et al
      . Benchmarking can facilitate the sharing of information on outcomes of care. Int J Lang Commun Disord 2001;36 Suppl:385–90.doi:10.3109/13682820109177916pmid:http://www.ncbi.nlm.nih.gov/pubmed/11340817
      OpenUrlPubMed
    30. ↵
      1. Diebold FX
      . On the Origin(s) and Development of the Term 'Big Data'. SSRN Journal 2012.doi:10.2139/ssrn.2152421
    31. ↵
      1. Royal College of Speech and Language Therapists
      . Different class solutions Ltd. RCSLT online outcome tool (ROOT) information governance pack, 2019. Available: https://rcslt-root.org/Files/Documents/ROOT%20Information%20Governance%20Pack%20v2.5%20July%202019.pdf [Accessed 4 Nov 2019].
    32. ↵
      1. Enderby P,
      2. John A
      . Therapy outcome measures for rehabilitation professionals. 4th edn. Croydon: J & R Press Ltd, in press.
    33. ↵
      1. Moyse K,
      2. Koziel S
      . The value of outcome measurement in speech and language therapy services. RCSLT study day, 2018. Available: https://www.rcslt.org/past-events/rcslt-study-day-and-agm-2018#section-3 [Accessed 4 Nov 2019].
    34. ↵
      1. Gadhok K,
      2. Moyse K,
      3. Farrell J
      . Are you ROOT ready? The value of the RCSLT online outcome tool. RCSLT Webinar, 2018. Available: https://www.youtube.com/watch?v=OE5pN2eezM4&feature=youtu.be [Accessed 4 Nov 2019].

    Footnotes

    • Contributors KM, PE, KC, KG, MB and PG contributed to the planning, design and implementation of the study; design and development of the data collection and analytics tool; analysis and interpretation of the data and writing the manuscript. PE developed Therapy Outcome Measures for Rehabilitation Professionals.

    • Funding The RCSLT funded the development of the ROOT.

    • Competing interests MB and PG are directors of Different Class Solutions Ltd. PE is a co-author of Therapy Outcome Measures for Rehabilitation Professionals.

    • Patient consent for publication Not required.

    • Ethics approval Details of governance contained within paper. Advice regarding ethics permission was sought and deemed not to be necessary for this piece of work.

    • Provenance and peer review Not commissioned; externally peer reviewed.