Article Text
PDF
Abstract
Background Treat-to-target (T2T) is a therapeutic strategy currently being studied for its application in systemic lupus erythematosus (SLE). Patients and rheumatologists have little support in making the best treatment decision in the context of a T2T strategy, thus, the use of information technology for systematically processing data and supporting information and knowledge may improve routine decision-making practices, helping to deliver value-based care.
Objective To design and develop an online Clinical Decision Support Systems (CDSS) tool “SLE-T2T”, and test its usability for the implementation of a T2T strategy in the management of patients with SLE.
Methods A prototype of a CDSS was conceived as a web-based application with the task of generating appropriate treatment advice based on entered patients’ data. Once developed, a System Usability Score (SUS) questionnaire was implemented to test whether the eHealth tool was user-friendly, comprehensible, easy-to-deliver and workflow-oriented. Data from the participants’ comments were synthesised, and the elements in need for improvement were identified.
Results The beta version web-based system was developed based on the interim usability and acceptance evaluation. 7 participants completed the SUS survey. The median SUS score of SLE-T2T was 79 (scale 0 to 100), categorising the application as ‘good’ and indicating the need for minor improvements to the design.
Conclusions SLE-T2T is the first eHealth tool to be designed for the management of SLE patients in a T2T context. The SUS score and unstructured feedback showed high acceptance of this digital instrument for its future use in a clinical trial.
- outcome assessment, health care
- medical informatics
- disease management
- decision support systems, clinical
Data availability statement
Data sharing not applicable as no datasets generated and/or analysed for this study. All data relevant to the study are included in the article or uploaded as supplementary information.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- outcome assessment, health care
- medical informatics
- disease management
- decision support systems, clinical
WHAT IS ALREADY KNOWN ON THIS TOPIC
The treat-to-target (T2T) strategy is being studied as a therapeutic approach for managing patients with systemic lupus erythematosus (SLE).
Information technology offers the potential to improve decision-making in clinical practice, delivering value-based care.
WHAT THIS STUDY ADDS
The study presents the design and development of the ‘SLE-T2T’ a web-based Clinical Decision Support System (CDSS)—the first eHealth tool tailored for managing patients with SLE within a T2T framework.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
The ‘SLE-T2T’ CDSS tool could influence policy discussions on incorporating information technology in rheumatology care and promoting evidence-based and patient-centric approaches in managing SLE.
This study may pave the way for further research and validation, encouraging the adoption of digital tools like ‘SLE-T2T’ in routine clinical practice.
Introduction
Systemic lupus erythematosus (SLE) is a chronic, multisystemic and complex autoimmune disease, characterised by multiple manifestations and affecting predominantly woman of childbearing age.1 2 Even when receiving the best possible care, SLE may be associated with damage accrual due to disease activity, comorbidities and the side effects of therapy (in particular, glucocorticoids), which negatively impacts patients’ health-related quality of life (HRQoL).3 The treatment in SLE should, thus, aim for controlling the symptoms and disease activity while minimising the side effects and drug toxicity, ensuring survival, preventing organ damage and optimising HRQoL.4 Formulating such a treatment plan for SLE is challenging due to the heterogeneity in its clinical presentation, disease course and prognosis. Clinicians from different medical specialisations may be involved in the management of patients with SLE and need to handle a vast amount of information to make clinical decisions that are difficult to capture in a single instrument.5 It has been postulated that to achieve this, a treat-to-target (T2T) strategy would be beneficial. The essence of such a strategy can be summarised as setting a therapeutic target, intervening, assessing whether the target has been met, and adjusting therapy if it has not.4 6 7 While endorsed by experts on SLE, the T2T strategy has not been formally proven effective and appears to be implemented only to a limited degree by practitioners.
Electronic health (eHealth) and mobile health (mHealth) are becoming prominent components of healthcare and represent an innovative tool to support practitioners in clinical decision-making.8 Computerised Clinical Decision Support Systems (CDSS) represents a type of eHealth tool that compiles great volume of available data and helps clinicians to sift through it effectively and reliably.9 CDSS have also shown increasing adherence to clinical guidelines, which traditionally have been shown to be difficult to implement in practice, increasing confidence in making decisions and improving prescribing behaviour.10
Hence, we aimed to develop SLE-T2T, a CDSS web-based eHealth tool that could help physicians in their decision-making process, in the context of a T2T approach for patients with SLE. We also aimed to evaluate the feasibility and usability of the first prototype, determining whether the CDSS is user-friendly, comprehensible, easy-to-deliver and workflow-oriented.
Methods
System design and development
The creation process of web-based applications is composed of three phases. For the first phase, the design, SLE-T2T was conceived with an ‘user-centred design’ and with a specific task: to generate appropriate treatment advices based on entered patients’ data. A general sketch of the programme was made, and general consensus was achieved with regards to the desired functionalities. To develop the clinical decision support functionality, European League Against Rheumatism (EULAR) recommendations for the management of SLE and international evidence-based guidelines were reviewed.11 A knowledge-based system was generated, capable of form rule statements from the data collected in the input, similar to first-order logic, knowledge-based systems capture the data inputted and create a rule according to the pre-established conditions in logical system.12 For SLE-T2T, the rules made from literature and guidelines were organised in the form of ‘IF/THEN’ statements in a prespecified decision table. The input was categorised according to disease activity state into: (a) remission, (b) mild disease activity and (c) moderate/severe disease activity, measured by the clinical Systemic Lupus Erythematosus Disease Activity Index 2000 (cSLEDAI-2K) and Physician Global Assessment (PGA). Patient’s medication was also taken into account, and categorised into: (1) use of antimalarials (yes/no), (2) use or immunosuppressives (yes/no and duration) and (3) use of glucocorticoids (yes/no and dosage). The result from the input combining the diverse categories generates a rule, tied to a predesigned set of general recommendations, and shown in the system as an output, so the health professional can make a decision according to that result. Table 1 exemplifies one scenario of rule-based ‘IF/THEN’ statements in a portion of the prespecified decision table.
Example of one scenario from the extended-entry decision table, where remission is the target and remission is not achieved
For the second phase, the development phase, in partnership with the Medical Informatics department of the University of Amsterdam, a beta version of SLE-T2T was developed using a free integrated development environment, and based on Javascript, HTML and CSS programming languages and framework, to be used in web browsers. There was an iterative process of development, with close cooperation between clinicians and developers of the application. After the development, the system was made available temporarily for the participants in the evaluation phase.
System evaluation
The third phase was the testing. During this phase, safety, validation and verification analyses were performed (data not shown) looking at Sommerville’s dependable programming guidelines,13 all aspects inherent to the development phase. The CDSS was also electronically tested to verify that recommendation results matched the prespecified decision tree. Once the beta version of SLE-T2T was ready, the system was tested in terms of usability, which refers to the effectiveness, efficiency and user satisfaction rating of a product in a specific environment by a specific user for a specific purpose.14 A System Usability Score (SUS) survey 15 16 was chosen as the usability test tool, widely adopted in this type of products for usability evaluations given its simplicity and advantages: (1) short questionnaire, quick to answer; (2) versatile for the evaluation of websites, software, mobile devices and medical systems; (3) the final score is interpreted based on a well-established reference standard;17 (4) is suitable even when applied to small samples (N<14) and (5) it has excellent reliability (0.85).16 18 The SUS contains 10 questions based on the Likert five-point scale; questions 1, 3, 5, 7 and 9 are positive and questions 2, 4, 6, 8 and 10 are negative. The 10 questions are closely related and are employed for the comprehensive evaluation of a product. A higher SUS score indicates better product usability. Furthermore, the SUS was coupled with unstructured feedback about areas of improvement, collected from the participants using the ‘think aloud’ method.17
Participants’ recruitment and data collection
The recruitment was based on a convenience sampling method, through invitations to researchers, clinicians and related healthcare personnel to participate in the evaluation of the SLE-T2T website as independent users (not related to the development of the website). Once the participants agreed to take part in the evaluation, consent was obtained and they were invited to a 20–30 min video call to navigate through the page selecting the appropriate options according to a hypothetical clinical case, while describing aloud their overall perception as users; this was followed by the completion of the SUS survey about their experiences with the website. The questionnaire was sent via email and completed via personal computers and mobile terminals.
Statistical analysis
For descriptive statistical analysis, basic information about the participants was collected, including gender, age, education and profession, followed by the calculation of the SUS scores for each of the participants, and the mean SUS score, as described by the author,15 16 using SPSS V.25. Qualitative data were collected through unstructured feedback, was analysed by first, creating an individual list of problems identified by each participant, to then group the duplicate problems between individuals and categorise them in terms of system strengths, anticipated barriers and design recommendations.
System refinement
The SUS score and/or unstructured feedback from the participants in the evaluation phase will enable to identify the necessary elements in need for improvement in the beta version of the CDSS, based on these, a set of criteria for software revision will be defined and the software version will be modified accordingly to reach a final version for later implementation in a pilot study.
Results
System overview
SLE-T2T web-based system was developed. The processing of the system takes place on the user’s computer, and, since no data is stored, the architecture of this decision support system is essentially composed of: (1) an input scheme consisting in the diverse set of index and scores existing for the measurement of SLE disease activity (cSLEDAI-2K, SLEDAI-2K, PGA score) as well as the used medication; (2) a rule-based interface that collects and processes patients’ data and (3) an output dashboard with the generated set of recommendations tailored for the patients’ clinical state and aiming to reach a pre-established target of treatment, based on the T2T strategy. Figures 1 and 2 depict a comprehensive view of the system architecture.
Overview of SLE-T2T CDSS tool architecture. cSLEDAI-2k, Clinical Systemic Lupus Erythematosus Disease Activity Index 2000; PGA: Physician Global Assessment; SLEDAI-2K, SystemicLupus Erythematosus Disease Activity Index 2000.
Desktop view screenshots of the SLE-T2T web-based application (Amsterdam UMC, all rights reserved). (A) Home page. (B) Sequence of screenshots following the evaluation process, as follow: 1. SLEDAI-2K checklist; 2. PGA visual scale from 0 to 3; 3. patient’s current medication list, divided in antimalarial, immunosuppressive therapy (including biologics) and glucocorticoids (prednisolone dosage); 4. target selection page, among remission and LLDAS; 5. output page, describing the recommendations. LLDAS: Lupus Low Disease Activity State; PGA: Physician Global Assessment; SLEDAI-2K, Systemic Lupus Erythematosus Disease Activity Index 2000.
System Usability Scores
A total of seven participants completed the SUS questionnaire for this research. The participants included rheumatologist specialised in the management of patients with SLE and clinical researchers in the field of rheumatology. The mean usability rating given by the participants was 79, on a scale of 0 (worst) to 100 (best), categorising the application as ‘good’ (in the adjectives and acceptability categories associated with SUS scores), indicating the need for minor improvements to the design. Table 2 depicts the distribution of answers for the SUS rating.
System usability average scores given by the participants and SUS final score
Qualitative analysis
The qualitative data were obtained through unstructured feedback from the participants during the evaluation calls and their comments in the SUS form and classified the eHealth tool as practical and simple to use. In terms of the system strengths, participants perceived the web-based application as an advantage, simple and intelligible as exemplified below:
I think the website is well-made and provides an easy to use SLEDAI-2K score form… for physicians who do not see patients with SLE that often, an easy to use SLEDAI-2K calculator and general treatment advices might be very useful (Clinician—Researcher in the field of SLE).
I really like that the advice is (a little) personalised (Rheumatologist).
Easy to use. It could save me some time in the daily practice…(Rheumatologist)
Some of the anticipated barriers were related to the migration of the data inputted and the advice generated to the electronic record environment:
Overall easy to use. How to implement into EPIC? Would be great if we can see changes in scores in a figure in EPIC during follow up (Rheumatologist).
Based on this, a ‘summary table’ was added and can be seen as the user input data through the whole evaluation process. Once completed, it appears at the output screen, below the recommendations. This summary table can be easily copied into electronic records to keep track of the patient evaluation.
On the other hand, the participants identified the lack of patient opinion as a barrier to know the patient’s preference when it comes to the target selection:
It would be of great value to add PROMS/patient opinion about T2T to this project, as discussed (Rheumatologist)
In spite of this, SLE-T2T is intended for healthcare professionals as users, thus, including the collection of patient-reported outcome measures (PROMs) from the patients, at this stage, was not possible. We have suggested that during the clinical evaluation, the HCPs discuss together with the patient the selection of a treatment target. Based also on this comments, the record of PROMs manually will be included during the subsequent study, to further understand the patients’ need in a T2T context.
Finally, in terms of design recommendations, most of the participants agreed that more visual aid will help to sift through the page easily.
For Physician global assessment (PGA) scale would be helpful to indicate which side of the scale is good/bad in a more visual way. Make tables for remission and LLDAS goals next to each other so it is easier to compare what the differences are (Clinician—Researcher in the field of SLE).
In this sense, the graphical design of the SLEDAI-2K table and PGA visual scale were modified and made more eye catching, which translated into an easier way to navigate the site and fill in the required data.
The participants also reported some clarifications needed in the prototype web-based application, these in terms of grammatical typos, definition and specification of cut-off levels for some measurements, which were applied to the beta version of the e-health tool.
Discussion
This study evaluated the performance and usability of ‘SLE-T2T’, a CDSS created to assist clinician in the management of patients with SLE in the context of a T2T strategy. Although well established in the software and development sector, usability testing is less commonplace within the healthcare context. Nonetheless, it has been gradually implemented in various areas where specific CDSS are developed for the improvement of clinical management. Schaaf et al19 have carried out similar assessment process for a CDSS in the field of rare diseases. Using ‘think-aloud’ protocols in combination with SUS, testing the usability of CDSS, allowed them to reach system improvements in design, user interface and user experience (UX). More recently, in the field of rheumatology, Rheuma Care Manager (RCM)—a CDSS tool to support the management of rheumatoid arthritis applying T2T—was similarly evaluated in terms of accuracy, effectiveness, usability and acceptance.20 RCM usability (SUS) was rated as good and was well accepted, showing that CDSS usage could support physicians by decreasing assessment deviations and increasing treatment decision confidence.20
In the context of SLE, eHealth technologies for the management of SLE are still a relatively new and unexplored topic, with potential for future investigation and development of such tools. Current eHealth tools for SLE are limited to educational tools, patient-reporting system, disease activity calculators and interactive online communities.21 These have been described as of poor quality and limited functionality, and the literature examining this area is scarce.21
Our development and first evaluation process of a CDSS for T2T in SLE involved a small number of users who were used to paper-based indices to measure disease activity state in SLE. Conventions of usability testing support our small sample,22 and the overall testing process was highly beneficial to the design and development for several reasons: participants had a wide age range and experience in secondary and tertiary care, and since the testing occurred early in development, it allowed us to identify the needed changes in design elements to arrive to a final version of the web-based application. The qualitative ‘think aloud’ method provided us with specific data and suggestions that we were able to integrate to improve the tool, especially related to UX and technical aspects.
Although there is a growing need and desire for eHealth technologies, the availability of apps designed specifically for SLE and the evidence for their efficacy are still limited. Accelerating the shift from traditional healthcare models to digital solutions remains a challenge faced by patients, their physicians and healthcare systems.23 SLE-T2T CDSS represents a first step to tackle this unmet need. In the future, comprehensive multidisciplinary partnerships between clinical researchers, patients and app developers are critical to continue shifting digital health.
Conclusion
SLE-T2T CDSS is the first eHealth tool to be designed for the management of patients with SLE in a T2T context. The SUS score and unstructured feedback showed high acceptance of this digital instrument, and clinicians strongly supported the implementation of this kind of eHealth tools in the outpatient care setting. A CDSS specifically designed to support the T2T strategy in SLE appears to be both needed and likely to come with significant benefits. The final version reached after the improvements identified through the participants will be used for implementation in a larger T2T pilot study.
Data availability statement
Data sharing not applicable as no datasets generated and/or analysed for this study. All data relevant to the study are included in the article or uploaded as supplementary information.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants and was approved by Medisch Ethische Toetsingscommissie AMCMETC number 2022.0449. Participants gave informed consent to participate in the study before taking part.
Footnotes
Contributors ARPS and RvV conceived the presented idea. ARPS developed the theory and proposal. RvV, AV and MWPT-A-S verified the methods and feasibility, and supervised the findings of this work. MGG, MA, PD, MH, and EJ carried out the development phase and conceived the mhealth tool. HJJ encouraged and supervised MGG, MA, PD, MH and EJ work. ARPS performed the experimental testing and wrote the manuscript with support from MWPT-A-S, RvV, AV and MGG. ARPS is responsible for the overall content as the guarantor.
Funding The work of ARPS is supported by the European Union’s Horizon 2020 research and innovation program (‘ARCAID’; www.arcaid-h2020.eu; grant agreement number 847551). This project was supported by a research grant from AstraZeneca.
Competing interests RV declares that he has received research support (institutional grants) from BMS, GSK, Lilly and UCB and support for educational programs from Pfizer and Roche. RV declares that he has also received consulting fees from AbbVie, AstraZeneca, Biogen, Biotest, BMS, Galapagos, Gilead, Janssen, Pfizer, Sanofi, Servier, UCB and Vielabio and personal honoraria as a speaker from AbbVie, Galapagos, GSK, Janssen, Pfizer and UCB. AV declares that he has received research support (institutional grants) from GSK and UCB. AV declares that he has also received consulting fees from GSK, AstraZeneca, Roche and personal honoraria as a speaker from GSK.
Provenance and peer review Not commissioned; externally peer reviewed.