| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Research Reports |
BCM Smits-Engelsman, PhD, is Professor, Motor Control Lab, Division of Motor Control and Neuroplasticity, Department of Biomedical Kinesiology, Katholieke Universiteit Leuven, Leuven, Belgium; Professor, Department of Rehabilitation Medicine, Radboud University of Nijmegen, Nijmegen, the Netherlands; and Head, School of Pediatric Physical Therapy, Avans+, University for Professionals, Breda, the Netherlands
MJ Fiers, MPPT, is a practitioner of pediatric physical therapy in Veldhoven, the Netherlands. She was a master's student at Avans+, University for Professionals, at the time of data collection for this study
SE Henderson, PhD, is Professor, School of Psychology and Human Development, Institute of Education, University of London, London, United Kingdom
L Henderson, PhD, is Professor, School of Psychology, University of Hertfordshire and Imperial College Medical School, London, United Kingdom
Address all correspondence to Dr Smits-Engelsman at: bouwiensmits{at}hotmail.com
Submitted March 1, 2007;
Accepted September 18, 2007
 | Abstract |
|---|
Subjects and Methods: For 9 children who were referred to clinical settings for an assessment of possible movement difficulties, performance on the appropriate age band of the M-ABC was videotaped. The 9 children, one at each age from 4 through 12 years, represented all ages covered by the test. The videotaped performances were rated according to the test instructions by 131 pediatric physical therapists with a range of experience and by an expert rater who developed the Dutch version of the test.
Results: The average agreement between therapists in their classification of the children was very high. The kappa coefficients for the 9 videos ranged from .95 to 1.00.
Discussion and Conclusion: Errors made by the therapists could be classified as those that might be common to all tests and those that are specific to the M-ABC.
| Introduction |
|---|
 TopAbstract Introduction MethodResultsDiscussionConclusion and RecommendationsReferences |
|---|
At present, there is much debate about the best way to operationalize the criteria of the Diagnostic and Statistical Manual of Mental Disorders: DSM-IV5 for a diagnosis of DCD.2,6,7 In particular, the formulation of criterion A, the primary inclusion criterion, is of interest to occupational therapists and physical therapists. Criterion A states: "Performance in daily activities that require motor coordination is substantially below that expected, given the person's chronological age and measured intelligence."5(p147) Because the M-ABC1 contains a standardized assessment that furnishes an indication of the extent to which a child falls behind his age peers, providing norms for children aged 4 through 12 years, it seems well suited to a role in the operationalization of criterion A. In addition, the test is used for children with a range of other conditions, some physical, such as benign joint hypermobility syndrome8 or congenital hypothyroidism,9 and some essentially psychological, such as attention deficit-hyperactivity disorder10 or Asperger syndrome.11
First described in 1972 as the Test of Motor Impairment,12 the M-ABC1 has been revised and restandardized on a regular basis. Originally a British-American instrument, the test has been translated into 6 European languages and 2 Asian languages. Of these, the translation into Dutch by Smits-Engelsman,13 accompanied by a restandardization of the test instrument for the Dutch population, has had the greatest influence on local practice. It has been used extensively in the Netherlands by both occupational therapists and physical therapists. In the primary health care system, for example, Dutch pediatric physical therapists have judged it to be the most frequently used measure for both screening and descriptive purposes.14–17
The M-ABC was conceived as a test of motor impairment. Accordingly, some lack of resolution in the upper bandwidth of performance is tolerable, whereas uncertainties about performance as critical bands of impairment are approached are highly undesirable. These bands are the products of cutoff points at the 5th and 15th percentiles, 0 through 5th indicating children with frank abnormalities and 5th through 15th indicating children at risk, who require careful monitoring. In clinical practice, the results of this classification are used, along with other data, to make decisions about the allocation of resources. The reliability of such a classification, therefore, is very important.
Various estimates of the reliability of data for the M-ABC and its predecessor, the Test of Motor Impairment, are reported in the M-ABC manual.1 Three examples of more recent studies must suffice to demonstrate the range of published estimates. These include data from the individual items of the test as well as the total score. In 2001, Croce et al18 reported good test-retest reliability over all age bands of the test. A total of 106 children between the ages of 5 and 12 years were tested twice, 1 week apart. The kappa coefficient for all groups taken together was .95 and ranged from .92 to .98 for each age band separately.
In a more focused study, involving 138 children aged 4 through 6 years, Chow and Henderson19 reported 3 estimates of reliability, 2 based on raw test scores and 1 based on a child's total score. Test-retest reliability was slightly lower than that reported by Croce et al18 but nevertheless fell within the acceptable range. Intraclass correlation coefficients for the individual test items ranged from .64 to .86, with a mean of .77 for the test as a whole. For intertester reliability, intraclass correlation coefficients ranged from .80 to 1.00, with a mean of .96. Of particular interest in the study of Chow and Henderson19 was the participation of 2 raters with quite different training, one an experienced educational psychologist and the other an inexperienced occupational therapist. When Chow and Henderson19 divided their participants into those whose total scores fell above and those whose total scores fell below the 15th percentile, the agreement between testers was perfect.
Although the above-mentioned studies showed that the M-ABC has, at the very least, moderate to good interrater and test-retest reliability, with the exception of the study of Chow and Henderson,19 those studies have 2 limitations. First, they were conducted either with samples drawn from the standardization sample or with samples of children who were developing typically. Second, they focused on the stability of the item or total scores as opposed to the classification of children. Because 85% of the children with scores falling between the 100th and the 15th percentiles will, by definition, be classified in the "normal" category, for most it would take a great shift in percentile terms for their classification to change. The inherent likelihood of category change is much greater at the lower end of the distribution.
We know of only 2 studies of test-retest reliability in clinical populations. Leemrijse et al16 drew samples from 2 special schools and a school for children who are chronically ill. They were able to test 23 children aged 6 to 8 years, all of whom had movement difficulties, 3 times in succession. The authors concluded that both the total scores and the cluster scores were reliable under these conditions and, therefore, that these scores might be used to detect change over time resulting from intervention. However, this reliability did not extend to the individual test items. Van Waelvelde et al20 examined test-retest reliability in thirty-seven 4- to 5-year-old children chosen by their teachers as "the least motor skilled" in their class. Kappa coefficients for this study ranged from .65 to .93 for the individual items and reached .95 for the total scores.
In addition to studies that focus on populations of children with movement difficulties, there is a pressing need for studies in which the raters are "ordinary" clinicians, who lack a background in research but deliver therapy to children with movement difficulties on a daily basis. In the Netherlands, communication between therapists in research and clinical settings is exceptionally good. Accordingly, we were able to rectify most of these shortcomings in the study reported here.
The aim of this study was to assess the reliability of test results obtained by physical therapists who work in clinical settings, use the M-ABC on a regular basis, and use the results obtained to make decisions about children referred to them. All of the children involved were being assessed to determine whether intervention was recommended.
| Method |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusion and RecommendationsReferences |
|---|
Children.
Nine children with movement difficulties participated in this study. The youngest child was 4 years of age, and the oldest child was 12 years of age, this being the age range for which normalized M-ABC test scores are currently available. Between these 2 children, one child represented each intervening year. The children were selected from a clinical population, having been referred to a pediatric physical therapy center in the Netherlands for assessment.
Table 1 shows the age, sex, and M-ABC scores for each child, along with some brief notes to characterize the range of children in the sample. Three children clearly met the criteria for DCD. A further 3 children had nonmotor problems that had already led to another diagnosis, namely, attention deficit-hyperactivity disorder, pervasive developmental disorder (not otherwise specified), and learning disability. Of the remaining 3 children, one child had hemophilia, a condition sometimes associated with poor motor performance.21 Another child's score fell in the range described by the test authors as "at risk," and the last child's score fell on the borderline separating the normal category from the at-risk category (ie, on the 16th percentile). All parents gave informed consent, and the children assented to their performance being videotaped, with the recordings to be used for either research or educational purposes.
|
The therapists were, in general, well acquainted with the M-ABC. Sixty-three percent administered the test between 1 and 4 times per month. A further 30% used the test more frequently. Most of the therapists (66%) had attended a short training session on the M-ABC (between a half-day and a full day), and 29% had taught themselves by using the manual or by working with a colleague over the learning period.
M-ABC
The Dutch translation of the M-ABC13 was used in this study. The test accommodates 4 age bands from 4 to 12 years of age, with each age band covering the same types of skill. There are 8 items within each age band, divided into 3 major performance areas: manipulative skills (items 1, 2, and 3), ball skills (items 4 and 5), and balance skills (items 6, 7, and 8). Children can score between 0 and 5 on each of the 8 items. The total (motor) impairment score (TIS) is then calculated by summation of the 8 item scores, resulting in a range of scores between 0 and 40, with lower scores indicating better performance. This total is then interpreted by comparison with the norm tables, in which a TIS below the 5th percentile is considered to be indicative of a definite motor problem. Scores between the 5th and 15th percentiles suggest a degree of difficulty that is borderline and requires monitoring. Scores above the 15th percentile are deemed to indicate normality. Evidence of the validity of the M-ABC test is presented in the manual1 and by Barnett and Henderson.22 Three studies18,23,24 provide data on the relationship between the M-ABC and the Bruininks-Oseretsky test, an instrument commonly used in the United States.
Procedure
All 9 children were tested by the second author (MJF) in a quiet room with one or both parents or guardians present. These sessions were videotaped by a technician, positioned so that as much of the child's performance as possible could be seen without distracting the child or the tester.
All video ratings took place over a 6-month period and took about 30 minutes per rating. Because the raters were distributed across the Netherlands, it was not possible to meet each one independently, nor was it possible to control the group size exactly. However, most of the ratings were made in small groups, ranging from a minimum of 5 therapists to a maximum of 15 therapists (median=7). In 65 cases, the group viewed one videotape, and in 66 cases, the group viewed 2 videotapes. The videotapes were selected to ensure that they were viewed approximately the same number of times and that, when 2 videotapes were being rated, each child was being assessed on a different age band of the test. In summary, the number of raters per videotape ranged from 16 to 34, yielding a total of 197 ratings (Tab. 2).
|
After performance on the individual items had been viewed, subscores and total scores were calculated together with percentiles derived from tables provided after the viewing session had been completed. If a therapist participated in scoring of a second videotape, the entire procedure was repeated. All observations and measurements were made independently, and discussion among the therapists within a group was not permitted while they were viewing the videotapes. In this study, all 4 age bands of the M-ABC were used. The participating therapists did not know in advance which age bands would be shown to them.
The author of the Dutch version of the M-ABC (BCMSE) also scored the videotapes of the 9 children independently, thus providing an expert opinion on the scoring of all 32 items of the test.
Data Analysis
For each of the 131 therapists involved, the TIS assigned to each child and the classification of each child in the normal, at-risk, or "definitely impaired" category, according to the guidelines given in the test manual, formed the basis of the data analysis. In order to measure the degree of agreement among testers, multiple kappa statistics designed for use with ordinal responses were computed25 with SAS 8.1 software.* In addition, to provide a measure of the confidence with which the M-ABC test scores could be interpreted, we computed: (1) the standard error of measurement (SEM); (2) the smallest detectable difference (SDD), calculated with the formula 1.96x
2xSEM of TIS; and (3) the 95% confidence interval of the scores for each videotape (child) being assessed. These statistical analyses were followed by a detailed assessment of the types of errors made during the scoring of individual items of the test.
| Results |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusion and RecommendationsReferences |
|---|
Table 3 shows the mean TIS assigned by the raters observing each videotape, the SEM associated with that score, the SDD, and the confidence interval or limits of agreement for each. Overall, these results inspired confidence in the test. Averaged over all 9 videotapes, the SDD was 1.1, ranging from 0.9 to 2.2 points. Examination of the 95% confidence interval of the scores for each videotape (child) revealed only one instance in which the degree of uncertainty was problematic. The child in question was the 7-year-old boy, who was observed by 24 therapists. This child was assigned a mean TIS of 8.1 by the therapists, but the agreement among them was slightly lower than that for the other 8 children (kappa coefficient=.95). A score of 8 would place him in the normal category, according to the test norms. However, if the limits of agreement for this age group are considered, then the range of scores associated with his TIS would be ±2, that is, between 6 and 10; the latter value would place him in the at-risk category.
|
Agreement by Videotape
Another way of interpreting the data was to determine whether there were systematic differences in ratings over the 9 videotapes that might be attributable to either the videotape or the child. As noted above, the children were correctly classified 85% of the time. More detailed analysis of agreement as a function of age showed perfect agreement on the videotapes of the children aged 9, 10, and 12 years. For those aged 8 and 11 years, only one therapist disagreed with the others. For the younger children, aged 4 through 7 years, there was slightly more disagreement, with the videotape of the 7-year-old boy generating the most disagreement (6 of the 24 raters).
Sources of Disagreement Among Raters: Analysis of Error Types
In order to determine the sources of disagreement among therapists or videotapes, all of the differences between the item scores of the therapists and those of the expert were examined individually. Almost all of the errors could be categorized into the following 3 groups.
Missing a procedural fault.
The M-ABC manual specifies faults of procedure that result in a child "failing" a task. For example, in catching a ball, a child is not permitted merely to trap it against his or her clothing, and this action was missed by the observers.
Differences in timing or number of errors between rater and expert.
On a very few occasions, the rater and the expert recorded different times or missed an error in a child's performance on a task. In most cases, a small difference (of 1 second or one error) would not affect a child's scaled score. However, in other cases, it could lead to a change in the scaled score. This situation occurred for the videotapes of the 5-, 6-, and 7-year-old children.
Errors in completing the form.
On a few occasions, testers used the wrong age column (in the correct age band) to calculate the scaled score, failed to calculate a mean value for items involving both limbs, or failed to use the best performance to transform a raw score to a scaled score. Curiously, the most frequently made error was caused by misreading the digital stopwatch. When the display showed 1.06, some therapists recorded this value on the form as 106 seconds rather than the true 66 seconds.
Differences Among Therapists
In order to determine whether there was any systematic association between the number of errors, the type of errors, or both made by the therapists and their clinical experience, we compared the characteristics of all those whose classifications agreed with those of the expert examiner and those whose classifications differed. We found no significant differences between these 2 groups in age, basic training, work experience, number of times they used the test, or how they were trained on the test.
| Discussion |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusion and RecommendationsReferences |
|---|
Reliability and Classification
In clinical settings, it is common practice to use the M-ABC as a means of classifying children into 3 groups: normal, at risk, and definitely impaired. These data then are used in conjunction with clinical judgment to decide whether and how a child will receive intervention. Fundamental to any test used in this way is evidence of the reliability with which such decisions can be made. Previous studies showed that both the interrater reliability and the test-retest reliability of data for the M-ABC are good, provided that the whole range of scores is examined. However, there is a severe shortage of data on the stability of scores falling around the predetermined cutoff points indicating the likelihood of impairment. Although Chow et al26 were able to show that 2 occupational therapists were in perfect agreement with each other on 2 occasions of testing, unfortunately, they restricted their attention to the 15th percentile as a means of dichotomizing children into impaired and not-impaired categories.
Distinctive Features of the Present Study
Features that merit special attention include the following. We included an unusually large and diverse sample of physical therapists as raters. To our knowledge, all previous reliability studies of tests of motor impairment that included therapists as raters were confined to judgments made by occupational therapists. Given that the knowledge base and training of these professionals are very different from those of physical therapists, the present investigation extended the findings of previous studies considerably. By focusing entirely on children with movement difficulties, we required raters to make judgments about performance that were more difficult than if the child being observed had no difficulties (eg, deciding whether a series of hops is continuous can be difficult if a child moves rather slowly and with a heavy footstep). Finally, previous studies tended to focus on younger children, whereas the children in the present study represented every age group covered by the test.
Video Recording and Sources of Error
Close inspection of the types of errors made in judging and scoring performance revealed some interesting findings. Some of these seemed likely to occur in almost any test. Observing and scoring any performance test from a videotape is different from observing a child in daily practice. When therapists examine a child in a clinical setting, they are free to move around and take the best possible viewpoint. Such positioning is not possible when a videotape is used, and this constraint certainly accounted for some of the errors made in the present study. Although videotapes of children being assessed with the M-ABC are invaluable for training purposes and are useful in clinical settings, it should be remembered that there is a cost for using videotapes, too. In the present study, the intermittent occlusion of a child's continuous tracing activity caused by a single viewpoint provides a simple example of a source of disagreement directly attributable to the limitations of video recording.
In addition to the error types that might occur with any test involving the observation of movement, there were a few that seemed to be specific to the M-ABC. Sometimes, the source could be traced to an item description in the manual that was not quite clear; at other times, raters seemed to find the layout of the form confusing (the way in which a score for an item involving both limbs is derived needs clarification). Some of our participants reported that they worked in settings that limited the age range of the children that they assessed. Thus, those who worked primarily with younger children might not have used the older age bands of the test for some time. We did not tell the raters in advance which video they would see. With hindsight, this choice might have been a mistake, as conscientious testers who have not tested a child of a particular age for some time might well prepare themselves by reviewing the test manual and form for that age group.
Differences in Classification
There were very few instances in which therapists disagreed on their classification of a child and, when they did, it was usually possible to locate the source of the error quite easily. Nevertheless, one important lesson can be learned from the present study. With any test that classifies people around specified points, there is always a risk of making more errors close to the cutoff points. It is clear that it is important for clinicians to be acquainted with the factors that lead to misclassification, as the effect of misclassification on a child's future can be considerable. In addition, taking the SDD and the limits of agreement (confidence intervals) into account always helps clinicians judge the stability of a test score. In the present study, we found these values to be in the acceptable range for all age bands of the M-ABC.
There was some variation in the reliability coefficients obtained for the 9 videotapes used in this study. We found no evidence to suggest that this variation was the result of differences in the test items for different age groups of children. Rather, it seemed to be attributable to differences among the particular children chosen to represent the age groups. It is important to note, therefore, that different interrater reliability coefficients and different SDDs could be obtained for another 9 children of comparable ages.
| Conclusion and Recommendations |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusion and RecommendationsReferences |
|---|
| Footnotes |
|---|
* SAS Institute Inc, PO Box 8000, Cary, NC 27513. 
| References |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusion and RecommendationsReferences |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
