The WHAAM application: a tool to support the evidence-based practice in the functional behaviour assessment
===========================================================================================================

* Gianluca Merlo
* Giuseppe Chiazzese
* Manuela Sanches-Ferreira
* Antonella Chifari
* Luciano Seta
* Colin McGee
* Alberto Mirisola
* Isabella Giammusso

## Abstract

**Background** The most recent computing technologies can promote the application of evidence-based practice (EBP) in the field of applied behaviour analysis (ABA).

**Objective** The study describes how the use of technology can simplify the application of EBPs in ABA.

**Methods** The Web Health Application for ADHD Monitoring (WHAAM) application demonstrates this in the following two case studies. We are monitoring dysfunctional behaviours, collecting behavioural data, performing systematic direct observations, creating both visual baseline and intervention charts and evaluating the planned interventions using the TAU-U statistical index.

**Results** Significant positive changes of children’s problem behaviours are observed and recorded. Both the duration of the previously identified behaviour ‘to get out of bed in time’ (*r* = −0.79, TAU-U = −0.58, *p* < 0.05) and the frequency of the behaviour ‘interrupting others’ (*r* = −0.96, TAU-U = −0.82, *p* < 0.01) decreased.

**Conclusions** The WHAAM application is an effective tool to support functional behaviour assessments and it is an example of how technology can support practitioners by facilitating the application of EBPs and increasing the communication among clinical, educational and family environments.

*   ADHD
*   behavioural assessment
*   behavioural interventions
*   information processing
*   effect size
*   eHealth system

## INTRODUCTION

According to Baer *et al*.1 applied behaviour analysis (ABA) is the process of systematically applying interventions based upon the principles of learning theory to improve socially significant behaviours to a meaningful degree and to demonstrate that the interventions employed are directly responsible for the improvement in behaviour. The systematic and rigorous data collection characterising ABA has been associated with technologies in leading positions by facilitating the description and understanding of behaviour patterns and the planning and implementation of behavioural interventions.2 This paper uses the term ‘technology’ with the meaning of computing technologies devoted to assisting evidence-based practice (EBP) of ABA processes. Slocum *et al*.3 suggest how the EBP of ABA defined as ‘a decision making process that integrates (a) the best available evidence with (b) clinical expertise and (c) client values and context’ (p. 44) could become a useful practice for making decision about targeted interventions on individuals. Therefore, the digital gathering of information is a strategic part of supporting the decision-making strategy of professionals in the conceptualisation, intervention and evaluation of a case. In this way, the exchanging and gathering of data through information communication technologies provides excellent opportunities in the research of ABA. By collecting a larger source of relevant data communications through technological support, it can enhance and improve the application of EBP. The technology then becomes an experimental tool for making decisions about interventions on individuals as well as an instrument for facilitating and supporting the monitoring of behaviour, the evaluation of the effectiveness of an intervention and the collaborative activities between all stakeholders (teachers, peers and caregivers). The systematic collection of behavioural data and the sharing of outcomes (both successes and failures) of the behavioural interventions, behavioural features and other valuable information delivers evidence for the entire clinical process. Database collection that provides wide access and the free availability of behavioural data also contributes to the categories of data as suggested by Ortega and Rodríguez4 as useful for supporting the behavioural clinical psychology processes. Based on the potential use of technology to support ABA, this article describes the development and use of the Web Health Application for ADHD Monitoring (WHAAM) to facilitate the application of systematic behavioural interventions and the examination of its effectiveness. In particular, the theoretical background underlying the development of the application will be described as well as two case studies intended to demonstrate its use when supporting effective behavioural interventions.

## THE WHAAM APPLICATION

### Theoretical background

The WHAAM application is based on the concepts and methods of functional behavioural assessment (FBA). The FBA is strongly linked to the above-mentioned ABA, bearing in mind that both are founded on the principle of Skinner’s operant conditioning.5,6 Unlike the first stimulus-response (S-R) proposed by Watson7 at the beginning of behavioural analysis, the operant conditioning studies the observed human behaviour and considers the possibilities between antecedent, the behaviour itself, and its consequence. The old S-R model was therefore replaced by a stimulus, response, stimulus (S-R-S) model. According to this new perspective, the reasons that a behaviour is maintained can not only be derived from its external form or topography but also by discovering the conditions which trigger and maintain it. Carr,8 in a renowned scientific paper about self-injurious behaviour, proposed three general classes of possible functions sustained by behaviour: positive reinforcement, negative reinforcement and sensory or automatic consequences of the behaviour. In the last few decades, these functions have been deeply investigated and some researchers thought to consider new additional functions, such as tangible,9 control10 or access to stereotypy.11 FBA is defined by Gresham *et al*.12 as a collection of methods for gathering information about antecedents, behaviours and consequences in order to determine the reason (function) of the behaviour. It differs from experimental functional analysis because it is easily applied in natural settings (home, school, etc.), it needs less training to use it and it does not require direct manipulation of dependent variables to identify the main function sustained by a problem behaviour. The FBA is therefore a descriptive assessment involving indirect and direct observations and the measurement of the target behaviour.13 Questionnaires, rating scales and interviews are examples of indirect methods because they do not require a direct observation of the target behaviour and third party people, such as relatives or teachers, can administer them. The direct observation of behaviour is instead carried out in a natural setting by descriptive assessments and/or systematic recording. While descriptive methods like antecedent behaviour consequence (ABC) charts and narrative recording provide qualitative information about a targeted behaviour, identifying the variables that may trigger or maintain it,14 systematic direct observation of behaviour provides quantitative information about frequency, intensity or duration of a targeted behaviour during a defined time interval. The WHAAM application15,16 is intended to promote the application of FBA methods to reduce the occurrence of problem behaviours in relation to attention deficit hyperactivity disorder (ADHD) or to replace them with positive ones. According to the Diagnostic and Statistical Manual of Mental Disorders-5,17 ADHD is a persistent pattern of inattention and/or hyperactivity–impulsivity that interferes with functioning or development, has symptoms presenting in two or more settings, and negatively impacts directly on social, academic or occupational functioning. For instance, children with ADHD are often rejected by peers and few are popular with their peers18,19 and they are at greater risk of substance abuse, school dropout, delinquency, academic problems and other psychological disorders.20 In the FBA context, the WHAAM application is a successful example of how the most recent technologies can support FBA by adopting an EBP. In the following section, the system’s functions will be described, a case study will be presented highlighting the statistical relevance of the behavioural interventions carried out and the satisfaction of the users of the technology will be evaluated.

### Overview of the WHAAM application features

Starting from the theoretical background of the FBA, the WHAAM application21 features are inspired by the multimodal treatment of ADHD. The most common approaches to manage ADHD symptoms includes behavioural interventions, the involvement of people included in the major settings of life in psycho-educational activities and the analysis of the medical history of the patient. In particular, the WHAAM application provides users with features for creating a network of stakeholders (parents, teachers and caregivers) close to the child with ADHD. This is to promote their collaboration during the whole functional behaviour process using both the web and the mobile sides of the application. The website allows users to sign in using their personal identification to access the following features: 

*   inclusion of data related to a patient (*i.e.* diagnosis, medical therapy, information about schools attended and crucial life events in his personal history);

*   invitation of caregivers to become members of the child’s private network;

*   use of instant messaging to promote collaboration and sharing data and information between the network’s member;

*   identification and description of a child’s specific behavioural problem (target behaviour) which the network wants to change;

*   planning the collection of baseline data;

*   creating a hypothesis about the main function sustaining the target behaviour;

*   creating an intervention plan, describing in detail the strategies to be applied and who will apply them within the network;

*   the evaluation of the effect size of the intervention, applying statistical algorithms.

The mobile application is available for Android operating systems. It provides the following features: 

*   viewing the case data;

*   viewing the target behaviour descriptions;

*   access to the observation sessions assigned to the logged user;

*   ongoing observation of the targeted behaviour through the recording of the behaviour frequencies or duration;

*   collection of qualitative data about the target behaviour through ABC charts.

The WHAAM application supports the FBA process in six steps. The first step is the creation of a child’s record. In order to let the users choose what they want to disclose, the personal information of the child can be omitted, using only a nickname, gender and the date of birth as the required information. Additional data such as diagnoses, medications, school information and other general events characterizing the child’s life can be added in the section of ‘case data’. This section provides health professionals with information intended to support the understanding of the child target behaviours. According to the Italian Personal Data Protection Code,22 data encryption of personal records and a credentials’ authentication system guarantee the personal data protection from intrusive accesses. During the second step, people who have an important role in the life contexts of the child with ADHD can be invited to become a member of the child’s private networks. Thus, they will become active participants in all the phases of the FBA processes, putting into practice the tasks assigned to them, sharing knowledge and information. The third step is creating the accurate description of a target behaviour in order to start the functional assessment process. To help with the clarification of a target behaviour in a recognised, measurable way, the WHAAM application provides users with a set of predefined common ADHD behaviour definitions. It is however possible and helpful to write a specific behavioural definition in an operational, objective and concise way. It needs to be objective and concise because the reader has to have a very clear picture of the specific behaviour they are to observe. The description includes the place and the setting in which the target behaviour occurs. The fourth step is defining the settings for the data collection phases. The WHAAM application supports an AB single case research design, allowing users to compare baseline and intervention data. AB single-case design generally starts with a baseline (phase A) in order to observe the dependent variable as it appears. Once the baseline is established, the observation continues while implementing the intervention (phase B) in order to compare the two time series and testing the hypothesised change. The fifth step is collecting quantitative and qualitative data using the WHAAM mobile app. The user, who plays the role of observer, gathers the frequency or the duration of a targeted behaviour according to the agreed data collection settings. Every member of the child’s network can contribute qualitative data of a behaviour by filling in ABC charts. Data collected during this step is available to the child’s network directly through the WHAAM web application. Health professionals analyse all the gathered observations to formulate a hypothesis about the main function that is sustaining the target behaviour and then to create an intervention plan intended to reduce or eliminate the inappropriate behaviour. The intervention plan consists of a set of behavioural strategies defined by health professionals and assigned to specific members of the child’s network. During the intervention phase, new data are collected for further comparisons. The final step is in the evaluation of intervention. Health professionals can produce a visual analysis of the gathered data accessing the scatter plots representing the observation sessions. Moreover, the Parker *et al*.23 TAU-U test is automatically calculated by the application to show the effect of the treatment on the occurrences of the target behaviour in the AB designs. TAU-U is a non-parametric statistic, which estimates the treatment effect considering both the non-overlap between phases and the trend change in phase B. This effect is estimated controlling the natural trend of phase A. The use of Kendall TAU permits us to estimate the phases monotonicity without assuming any trend function (*e.g.* linear). TAU-U showed to be a robust statistic with autocorrelated data overcoming some limitations of other nonoverlap techniques. The formula *r* = sin (0.5 πτ) is used to convert TAU-U in Pearson’s *r* effect size index.24 Following Cohen,25 absolute value of *r* between 0 and 0.09, 0.10 and 0.29, 0.30 and 0.49 and greater or equal to 0.50 is evidence of a null, small, medium and large effect size, respectively.

The WHAAM application has been extensively tested in Italy, Portugal and United Kingdom during its development. The next section will describe the methodological constraints surrounding the use of the WHAAM application and two illustrative case studies supported by it.

## CASE STUDIES

### Introducing the case: methodological considerations

The development of the WHAAM application adopted a user-centred design process including the final users from the early stages of developing and prototyping to the real context use of the application in a pilot test across Europe. As mentioned in the previous section, the WHAAM application involves many different user strategies in the management of unwanted behaviours of children with ADHD. In the process of management, each role has different duties and responsibilities and takes advantage of a subset of the features provided by the application. In order to get the best possible results in the FBA process using the new technology, users need to be aware of both the theoretical and methodological assumptions behind the FBA and the opportunities offered by the technology. For these reasons, the authors included a psycho educational content in the application and provides an integrated training course for teachers, health professionals and parents of children with ADHD. As underlined by Sanches-Ferreira *et al*.26 ‘the integrated model of training aims to promote community-based interventions in which every element of the school-family system has a key role in the identification and management of pupils’ problematic behaviours’ (p. 22). The training courses focused on three main learning outcomes: the comprehension of the FBA principles, the understanding of a set of specific behavioural techniques and the use of the WHAAM application to support the behavioural assessment and intervention on children’s undesired behaviours. The cases of Matteo and Anthony, which are described in detail in the next paragraphs, were examined during training courses held in Italy and in Portugal. It is worth noting that the names of children and some identifying details have been changed to protect the privacy of individuals. The authors measured the trainees’ thoughts on the usefulness of training, the usability and the overall satisfaction associated to the use of the WHAAM application during the training courses. In general, the course was considered substantial, while the application was seen as useful and easy to be use.27

### Matteo’s case

Matteo is 17 and lives with his family, has one brother aged 22 and one sister aged 23. He attends an Italian secondary school. His mother is employed in a financial loans agency and his father is an employee of the Ministry of the Interior. The family is characterised by high internal conflict and dynamics of manipulation by the father. Matteo received the first diagnosis of ADHD when he was 11 years old and he is currently being treated with methylphenidate. By the time he was diagnosed, he already had a history of serious scholastic difficulties due to being oppositional toward teachers. He often misses instructions given by them and engages in significant off-task behaviour for most subjects, except math and physical education – in which he excels. Parents are very worried because Matteo is struggling daily with his schoolwork: he is disorganised when planning activities and he usually goes to school unprepared and for this reason collects many negative marks. He appears to be having problems making and keeping friends, especially in the school context. This is because his classmates consider him a preferred student and they cannot stand the fact that the professors never question him as they have an understanding of his condition. Matteo will graduate next year and has never received special education help or an individualised education program. Matteo has never been allowed to fail an assessment, according to his parents who think that teachers prefer not to deal with Matteo’s negative behaviours. The behaviour analysed with the support of the WHAAM application is his difficulty to get out of bed in time to go to school. His mother tries to wake up him initially at 06.50 am. Despite her many subsequent attempts, Matteo ignores her and continues to stay in bed. As a consequence of this, he is often so late that he skips the first hour of classes at school or the school day. In order to identify the function of Matteo’s behaviour, the health professional planned a ‘baseline phase’ in which quantitative data were gathered through five, 40-minute observation sessions taken by the mother using a duration recording system. In the baseline, the mean time needed by Matteo to get out of bed was 25.8 minutes [standard deviation (SD) = 13.62] with a range from 7 to 40 and a strong increasing trend (*r* = 0.95). After a clinical interview with the parents (indirect assessment) and from the analysis of collected baseline data, the health professional hypothesised that the primary function of the Matteo’s challenging behaviour was to escape school; second, it was gaining his mother’s attention. Starting from this assessment data, the intervention was aimed to help Matteo in overcoming his difficulty to get-out-of-bed. The intervention was outlined with parents using the following strategies: 

*   Negative reinforcement: an alarm clock was placed away from Matteo in order to train him to get out of bed. The clock rang every 5 minutes after the planned wake up time. In this procedure, the individual is motivated to exhibit a desired behaviour to avoid the aversive stimuli (the shrill sound of the alarm clock).

*   Extinction: being that Matteo’s behaviour is maintained by positive reinforcement of his mother who cuddled him whenever he got up, the therapist invited the mother to eliminate the connection between the problem behaviour (get-out-of-bed) and the positive reinforcement (mother’s attention).

*   Plan a positive reinforcement program by means of a token economy procedure: a paperboard football was given to Matteo every time he got out of bed by himself. Matteo’s participation in a football tournament was linked to the achievement of gaining 10 paperboard Footballs.

Once the intervention was defined, a new phase of systematic observation started with the same modalities and observer of the baseline. Five new observation sessions were carried out. During the intervention, the mean time employed by Matteo to get out of bed was 12.8 minutes (SD = 16.16), with a range from 0 to 40 and a strong decreasing trend (*r* = −0.81). Based on the data obtained during both the baseline and the intervention (Figure 1), the treatment had a strong effect on the decrease of the duration of the behaviour (*r* = −0.79, TAU-U = −0.58, *p* < 0.05).

![Figure 1](http://informatics.bmj.com/https://informatics.bmj.com/content/bmjhci/25/2/63/F1.medium.gif)

[Figure 1](http://informatics.bmj.com/content/25/2/63/F1)

Figure 1 Comparison of the data gathered for the Matteo’s behaviour ‘get out of bed’ during the baseline (A) and intervention (B) phases

Matteo’s noncompliance and lack of independence in getting ready in the morning and going to school was tackled actively involving his own parents. According to the intervention approach suggested by Kearney *et al*.28 the key components of the parent training were (a) restructuring parent commands to clearer, simpler and more direct ways of giving instructions, (b) establishing a set of routines for the early part of the day and throughout the day, (c) negotiate rewards for compliance and school attendance and punishments for noncompliance and school refusal behaviour and (d) forced the school attendance under certain condition.

### Anthony’s case

Anthony is a 9-year-old boy and the only child of a two-parent family. He is enrolled in the third grade of a local elementary school, in which he attends a special education program. Since the first grade, he has been identified as performing below expected academic achievements. Along with the diagnosis of ADHD, at the age of 7, he was referred to special education services in the second grade. The restrictions of following the educational curriculum were mainly identified as difficulties of concentrating his attention, understanding instructions and controlling his psychomotor activity. The inattentiveness and hyperactivity behaviours were also seen at home, not finishing the home chores and engaging in dangerous activities – engaging in excessively rough play. In the last assessment, it also showed him as having dyspraxia, having difficulties in planning complex sequence of movements. Currently, he is medicated with methylphenidate and improvements were reported in his concentration, his attention and remaining in his seat during classes. Nevertheless, teachers and parents still had great concerns with his lack of social relationships. Peers start to reject him for not following the rules of play – not waiting his turn – and seeking excessive physical contact (pushing and hugging tightly the others). At home, the interaction with his parents was also challenging, always seeking their attention by interrupting their conversations or tasks. Parents react differently to his problematic behaviour – the father strongly reprimands him and the mother makes light of his behaviour and gives him lots of attention – this is a source of constant conflict in the house. The problematic behaviour to be monitored by the parents using the WHAAM application was Anthony interrupting their conversation and making irrelevant and repetitive comments during a mealtime. When confronted with Anthony’s excessive and repetitive talk, parents ask him to stop and sometimes they leave the table. That behaviour tends to be even more noticeable when they have guests for dinner. As a consequence of this, there is often a lack of dialogue between parents during the mealtime (directing their full attention at Anthony) or a discussion between them on each other reactions to Anthony’s behaviour. Together with a health professional, it was decided to collect a baseline of data, where the father registered the frequency of Anthony’s interruptions of their conversation or when he engages in repetitive and excessive talk at the dinner table. Observation sessions of 25 minutes were collected. Additionally, in order to gain a complete pattern of the problematic behaviour a brief description of the antecedent and consequent events was collected for the application. The average of the occurrence of the target behaviour during the four baseline sessions was of 10.5 (SD = 3.10), with a range from 2 to 15 and a substantial increasing trend (*r* = 0.50, Figure 2). From the analysis of the frequencies graph of and the ABC descriptions, it was hypothesised that the main function of Anthony’s behaviour was to seek social attention. Using this interpretation, the intervention was outlined with parents using the following strategies: 

*   Establish rules: both before and during the dinner. Anthony was reminded that he should ask permission to talk. The use of a ‘talking object’ (an object to passes between each family member and is used as a way of identifying who has permission to talk at that moment).

*   Reinforce appropriate behaviour: a period of time was identified to play family games such as board games and/or storytelling, every time Anthony followed the rules.

*   Ignore inappropriate behaviour: when Anthony interrupts anybody that holds the talking object and/or makes repetitive comments, the parents were instructed to keep the conversation going without looking to Anthony.

![Figure 2](http://informatics.bmj.com/https://informatics.bmj.com/content/bmjhci/25/2/63/F2.medium.gif)

[Figure 2](http://informatics.bmj.com/content/25/2/63/F2)

Figure 2 Comparison of the data gathered for the Anthony’s behaviour ‘interrupting others’ during the baseline (A) and intervention (B) phases

Starting with the implementation of these parent strategies, they collected data relative to the intervention phase. Four new observations were executed. At this period, the average of the occurrence of the target behaviour was of 3.25 (SD = 2.06), with a range from 1 to 5 and a strong decreasing trend (*r* = −0.97). As shown in Figure 2, there was a strong effect of the intervention successfully decreasing the frequency of the behaviour (*r* = −0.96, TAU-U = −0.82, *p* < 0.01).

## LIMITATIONS

The WHAAM application is one of the first web-based systems that aim to replicate digitally the complexity of the main processes related to the FBA. Many other systems for behavioural assessment were developed in the last few years.29 While these applications generally provide users with just one feature such as counters, behavioural charts, video recording, the WHAAM application has been designed to support the whole FBA process from the definition of the target behaviour to the evaluation of the efficacy of the behavioural interventions carried out. Moreover, the WHAAM application also includes features designed to bring together the child’s stakeholders and to promote their cooperative work. As can be easily imagined, being a pioneer, the WHAAM application still has important limitations. One of the most important is attributable to the possibility to implement only AB single case research designs. Starting from data gathered in the AB design, practitioners cannot draw conclusions about what happened to the target behaviour during the two observed phases relying solely on the data collected. What would be the natural trend of the behaviour over time? Has the treatment really modified the trend or are the observed changes related to some unknown environmental event? In the described cases of Matteo and Anthony, authors maintained good relations with the children’s health professionals and caregivers so as to verify that the changes observed in the children were long lasting. Future releases of the application might explore the opportunity of implementing more powerful design than the AB such as ABA, ABAB, or multi-element designs. At this moment, the WHAAM application can be used to gather more data of a simple AB design, but the results are aggregated only for two phases at a time. A more general limitation application of single case designs is the difficulty to gather the necessary four to five data points in each phase to calculate the effect-size of the intervention. As observed by Hser *et al*.29 this is one of the reasons why single-case experimental designs are still almost rare. The threshold to the calculation of the TAU-U in the WHAAM application is four observations per phase. But for ethical reasons, health professionals cannot wait long before taking action when they have to face, for instance, self-injurious or aggressive behaviours.

## CONCLUSIONS

The use of FBA has been widely recognised as a best practice approach in developing effective intervention plans for children with challenging behaviours, while identifying how the environment plays a determinant role in strengthening or weakening problematic behaviours,30 described as a symptom of hyperactive–impulsive and distracted behaviours. These are triggered from negative reinforcing environmental stimuli and are maintained by a number of functions. The treatment of ADHD symptoms benefits from a multimodal approach. This includes behavioural interventions along with parent/teachers training programs. This can be used independently or in combination with medications.31 Technology has been plays a prominent role in enhancing the implementation of EBP, this paper offers the web application WHAAM as a tool that supports both a comprehensive approach in the assessment and intervention over challenging behaviours and the recording and monitoring of child–environment interactions. Indeed, anchored in FBA principles, the WHAAM provides a set of potentialities that facilitate a successful implementation of behavioural interventions, supporting 

*   a systematic and rigorous data collection by allowing a real-time recording of the behaviour and the surrounding circumstances through the use of mobile devices;

*   the identification of behavioural patterns, integrating data describing the behavioural definition (including contextual and setting features), the rate of occurrence (frequency and/or duration) and environmental circumstances (antecedents and consequences);

*   a consistent application of behavioural strategies, promoting better communication between people who are significant to the ADHD child in different life contexts, and bringing together the key persons from family and school;

*   an evaluation of the effectiveness of the applied treatments, by calculating the Parker *et al*.23 TAU-U statistical indices on the behavioural data gathered by the child network members.

As such, the possibilities credited to the WHAAM application seem to deliver supportive factors for facilitating the praxis of FBA, and fosters collaborative relationships. However, it is important to note that as highlighted in previous studies,26 the accomplishment of the possibilities depends on the continued existence of supportive factors. The training on FBA principles and the correct use of the WHAAM application are important. It uses active learning methodologies, built on both parents and teachers’ experiences and beliefs. This will play a central role on the effectiveness of the behavioural interventions using this innovative application. Tools such as WHAAM can promote the introduction of the EBP into natural settings such as school and family. The use of such tools for the child can also encourage stakeholders to increase their knowledge, skills and competences through a continuous lifelong learning process. Mobile devices and wearable technology open new perspectives in the design of human behaviour monitoring systems. They will be able to display personal data of individuals which could be helpful to facilitate the self-regulation of their behaviour. Finally, this kind of technology might have a crucial role in increasing the positive communication among clinical, educational and family contexts.

*   Received June 1, 2018.


*   © 2018 The Author(s). Published by BCS, The Chartered Institute for IT under Creative

Commons license http://creativecommons.org/licenses/by/4.0/

## References

1.  1.Baer DM, Wolf MM, Risley TR. Some current dimensions of applied behavior analysis 1. Journal of applied behavior analysis 1968;1(1):91-7. doi:10.1901/jaba.1968.1-91pmid:16795165; pmcid:PMC1310980
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1901/jaba.1968.1-91&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=16795165&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=A1968B883900011&link_type=ISI) 

2.  2.Cohen E, Rozenblat R. Applied, behavioral, analytic and. technological: a current literature review on the use of technologies in behavior analysis. European Journal of Behavior Analysis 2015;16(2):178-87. doi:10.1080/15021149.2015.1085720
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1080/15021149.2015.1085720&link_type=DOI) 

3.  3.Slocum TA, Detrich R, Wilczynski SM, Spencer TD, Lewis T, Wolfe K. The evidence-based practice of applied behavior analysis. The Behavior Analyst 2014;37(1):41-56. doi:10.1007/s40614-014-0005-2pmid:27274958; pmcid:PMC4883454
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1007/s40614-014-0005-2&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=27274958&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 

4.  4.Ortega JV, Rodríguez RM. Guidelines for clinical case reports in behavioral clinical psychology. International Journal of Clinical and Health Psychology 2008;8(3):765-77
    
    

5.  5.Skinner BF. Science and Human Behavior. New York: Simon and Schuster, 1953.
    
    

6.  6.Skinner BF. Operant behavior. American Psychologist 1963;18(8):503. doi:10.1037/h0045185
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1037/h0045185&link_type=DOI) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=A1963CCK4500001&link_type=ISI) 

7.  7.Watson JB. Psychology as the behaviorist views it. Psychological Review 1913;20(2):158. doi:10.1037/h0074428
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1037/h0074428&link_type=DOI) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000200141700005&link_type=ISI) 

8.  8.Carr EG. The motivation of self-injurious behavior: a review of some hypotheses. Psychological Bulletin 1977;84(4):800. doi:10.1037/0033-2909.84.4.800pmid:331382
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1037/0033-2909.84.4.800&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=331382&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=A1977DN71800012&link_type=ISI) 

9.  9.Mace FC, West BJ. Analysis of demand conditions associated with reluctant speech. Journal of Behavior Therapy and Experimental Psychiatry 1986;17(4):285-94. doi:10.1016/0005-7916(86)90065-0
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/0005-7916(86)90065-0&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=3805313&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=A1986F998800008&link_type=ISI) 

10. 10.Bowman LG, Fisher WW, Thompson RH, Piazza CC. On the relation of mands and the function of destructive behavior. Journal of Applied Behavior Analysis 1997;30(2):251-65. doi:10.1901/jaba.1997.30-251pmid:9210305; pmcid:PMC1284044
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1901/jaba.1997.30-251&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=9210305&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 

11. 11.Fisher WW, Lindauer SE, Alterson CJ, Thompson RH. Assessment and treatment of destructive behavior maintained by stereotypic object manipulation. Journal of Applied Behavior Analysis 1998;31(4):513-27. doi:10.1901/jaba.1998.31-513pmid:9891391; pmcid:PMC1284145
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1901/jaba.1998.31-513&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=9891391&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 

12. 12.Gresham FM, Watson TS, Skinner CH. Functional behavioral assessment: principles, procedures, and future direction. School Psychology Review 2001;30(2):156
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000169972900002&link_type=ISI) 

13. 13.Hanley GP. Functional assessment of problem behavior: dispelling myths, overcoming implementation obstacles, and developing new lore. Behavior Analysis in Practice 2012;5(1):54-72. doi:10.1007/BF03391818pmid:23326630; pmcid:PMC3546636
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1007/BF03391818&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=23326630&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 

14. 14.Bijou SW, Peterson RF, Ault MH. A method to integrate descriptive and experimental field studies at the level of data and empirical concepts. Journal of Applied Behaviour Analysis 1968;1(2):175. doi:10.1901/jaba.1968.1-175pmid:16795175; pmcid:PMC1310995
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1901/jaba.1968.1-175&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=16795175&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 

15. 15.Merlo G, Chiazzese G, Di Giuseppe O, Spachos D. Monitoring behaviour at school and in the family: the WHAAM application. TD Tecnologie Didattiche 2016;24(1):47-51
    
    

16. 16.Spachos D, Chifari A, Chiazzese G, Merlo G, Doherty G, Bamidis P. WHAAM: a mobile application for ubiquitous monitoring of ADHD behaviors. 2014 International Conference on Interactive mobile communication technologies and learning (iMCL), IEEE 2014305-9. doi:10.1109/IMCTL.2014.7011153
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1109/IMCTL.2014.7011153&link_type=DOI) 

17. 17.American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (dSM-5) 20135thWashington, DCAuthor
    
    

18. 18.Hoza B, Mrug S, Gerdes AC, Hinshaw SP, Bukowski WM, Gold JA, et al. What aspects of peer relationships are impaired in children with attention-deficit/hyperactivity disorder?. Journal of Consulting and Clinical Psychology 2005;73(3):411. doi:10.1037/0022-006X.73.3.411pmid:15982139
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1037/0022-006X.73.3.411&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=15982139&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000230036200004&link_type=ISI) 

19. 19.Pelham WE Jr., Fabiano GA. Evidence-based psychosocial treatments for attention-deficit/hyperactivity disorder. Journal of Clinical Child & Adolescent Psychology 2008;37(1):184-214. doi:10.1080/15374410701818681pmid:18444058
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1080/15374410701818681&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=18444058&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 

20. 20.Bagwell CL, Schmidt ME, Newcomb AF, Bukowski WM. Friendship and peer rejection as predictors of adult adjustment. New Directions for Child and Adolescent Development 2001;2001(91):25. doi:10.1002/cd.4pmid:11280012
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1002/cd.4&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=11280012&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 

21. 21.1.  Chifari A
    
    Alves S, Bamidis P, Bilbow A, Chiazzese G, Chifari A, Denaro P, et al.Chifari A. WHAAM: A New Online Service Supporting Parents and Teachers of Children with ADHD. London, UK; Thessaloniki, Greece: ADDISS, 2015.
    
    

22. 22.2004January1Data Protection Code – Legislative Decree no. 196/2003, . Available from: [http://194.242.234.211/documents/10160/2012405/Personal+Data+Protection+Code+-+Legislat.+Decree+no.196+of+30+June+2003.pdf](http://194.242.234.211/documents/10160/2012405/Personal+Data+Protection+Code+-+Legislat.+Decree+no.196+of+30+June+2003.pdf)
    
    

23. 23.Parker RI, Vannest KJ, Davis JL, Sauber SB. Combining nonoverlap and trend for single-case research: Tau-u. Behavior Therapy 2011;42(2):284-99. doi:10.1016/j.beth.2010.08.006pmid:21496513
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/j.beth.2010.08.006&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=21496513&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 

24. 24.Walker DA. JMASM9: converting Kendall’s tau for correlational or meta-analytic analyses. Journal of Modern Applied Statistical Methods 2003;2(2):26. doi:10.22237/jmasm/1067646360
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.22237/jmasm/1067646360&link_type=DOI) 

25. 25.Cohen J. Statistical Power Analysis for the Behavioral Sciences. Hilsdale, NJ: Lawrence Earlbaum Associates, 1988.doi:10.1002/bs.3830330104
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1002/bs.3830330104&link_type=DOI) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=A1988L598900003&link_type=ISI) 

26. 26.Sanches-Ferreira M, Chifari A, Silveira-Maia M, Chiazzese G, Santos M. A model for training parents and teachers of children with special needs. TD Tecnologie Didattiche 2016;24(1):20-8
    
    

27. 27.1.  Briesmeister JM, 
    2.  Schaefer CE
    
    Kearney CA, LaSota MT, Lemos-Miller A, Vecchio JBriesmeister JM, Schaefer CE. Parent training in the treatment of school refusal behavior. Handbook of Parent Training: Helping Parents Prevent and Solve Problem Behaviors 20073rdHoboken, NJ: John Wiley & Sons Inc164-93
    
    

28. 28.Alves S, Bamidis P, Bilbow A, Callahan A, Chiazzese G, Chifari A, et al. WHAAM Context Driven Framework. Palermo: Istituto per le Tecnologie didattiche (CNR), 2014.
    
    

29. 29.Hser Y-I, Shen H, Chou C-P, Messer SC, Anglin MD. Analytic approaches for assessing long-term treatment effects examples of empirical applications and findings. Evaluation Review 2001;25(2):233-62. doi:10.1177/0193841X0102500206pmid:11317718
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1177/0193841X0102500206&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=11317718&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000176957000006&link_type=ISI) 

30. 30.Pelham WE Jr., Fabiano GA, Massetti GM. Evidence-based assessment of attention deficit hyperactivity disorder in children and adolescents. Journal of Clinical Child and Adolescent Psychology 2005;34(3):449-76. doi:10.1207/s15374424jccp3403_5pmid:16026214
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1207/s15374424jccp3403_5&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=16026214&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 

31. 31.Ferguson JH. National institutes of health consensus development conference statement: diagnosis and treatment of attention-deficit/hyperactivity disorder (ADHD). Journal of the American Academy of Child & Adolescent Psychiatry 2000;39(2):182-93. doi:10.1097/00004583-200002000-00018
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1097/00004583-200002000-00018&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=10673829&link_type=MED&atom=%2Fbmjhci%2F25%2F2%2F63.atom) 
    
    [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000085031900018&link_type=ISI)