|
|
||||||||
Research Reports |
SL Fritz, PT, PhD, is Clinical Assistant Professor, Physical Therapy Program, Department of Exercise Science, University of South Carolina, 1300 Wheat St, Blatt PE Bldg, Columbia, SC 29208 (USA)
SZ George, PT, PhD, is Assistant Professor, Department of Physical Therapy, College of Public Health and Health Professions, Brooks Center for Rehabilitation Studies, University of Florida, Gainesville, Fla
SL Wolf, PT, PhD, FAPTA, is Professor, Center for Rehabilitation Medicine, Emory University School of Medicine, Atlanta, Ga
KE Light, PT, PhD, is Associate Professor, Department of Physical Therapy, College of Public Health and Health Professions, University of Florida
Address all correspondence to Dr Fritz at: sfritz{at}gwm.sc.edu
Submitted March 29, 2006;
Accepted September 25, 2006
| Abstract |
|---|
Subjects and Methods: This study was a secondary analysis of a cohort of subjects (N=46) who participated in CIMT trials. Subjects completed measures at baseline and 4 to 6 months later. Hierarchical regression models determined whether change scores or raw follow-up scores of CIMT outcome measures were predictive of perceived recovery. Receiver operating characteristic (ROC) curves determined cutoff scores for measures that significantly contributed to participants' reports of perceived recovery.
Results: The regression models indicated that raw follow-up MALa scores (ß=0.80, P=.024) and WMFT scores (ß=–0.37, P=.03) contributed to perceived recovery. Proposed cutoff scores for the MALa scores were less than 1.15 (negative likelihood ratio [LR]=0.17) for predicting less than 50% recovery and greater than 2.50 (positive LR=2.75) for predicting 50% or greater recovery. Proposed cutoff scores for follow-up WMFT scores were greater than 34.0 seconds (negative LR=0.24) for predicting less than 50% recovery and less than 11.0 seconds (positive LR=5.96) for predicting 50% or greater recovery.
Discussion and Conclusion: Raw follow-up scores for the MALa and WMFT were better predictors of self-report of recovery in comparison with change scores. These data also serve as a starting point for developing cutoff scores that accurately predict self-report of recovery.
| Introduction |
|---|
|
|
|---|
Constraint-induced movement therapy (CIMT), however, is reported to significantly improve functional use of the UE in 20% to 25% of people with chronic stroke disability.1 The goal of CIMT is to overcome learned nonuse by increasing the functional use of the neurologically weaker UE through massed practice, while restraining the lesser-involved UE.1 Originally tested in an animal model, the results of CIMT studies have demonstrated significant and lasting improvements of UE movement function.1,2,6–14
Changes in function following CIMT are characterized primarily by changes in performance or perceived changes on assessment instruments. These mean change scores may demonstrate statistically significant results, especially when larger samples are used. The importance of statistically significant findings, however, can be unclear. One approach to examine the meaningfulness of scores is by comparing outcomes with an established external criterion15,16 such as validated outcome questionnaires or health care practitioner perceptions of recovery.15,17–22 Another external criterion to consider is the patient's perspective, which is important because the patient's perspective may differ from the practitioner's perspective or from a questionnaire. The patient has been described as the final arbiter of how well or how poorly a particular intervention is working.23 This "patient-centered" paradigm has been advocated for in physical therapist practice guidelines24 and utilized in pain studies25–27; however, this topic has not been extensively investigated in stroke rehabilitation. Given that recovery of UE function following stroke is an individual process, there would appear to be value in determining whether commonly used CIMT outcome measures are associated with patient self-reports of UE recovery.
Outcomes following CIMT have been assessed with multiple outcome measures. Standardized assessment tools such as the Wolf Motor Function Test (WMFT)28,29 and the Motor Activity Log (MAL)30,31 quantify changes following CIMT. They have not been compared, however, with an external criterion based on patient perception of recovery. The purposes of this study were: (1) to determine whether change scores or raw follow-up scores on the MAL and WMFT predicted perceived recovery of the UE as measured by the perceived recovery section of the Stroke Impact Scale and (2) to calculate cutoff values that predicted perceived recovery of the UE for the appropriate MAL and WMFT measures.
| Method |
|---|
|
|
|---|
|
Instrumentation
Predictors
Two main outcome measures, commonly reported in CIMT studies, were used for this study: (1) the amount component of the MAL (MALa), a test of perceived use, and (2) the WMFT, a test of movement capability. The MAL, a commonly used CIMT outcome measure,1,9–11,33 is a 30-question, structured interview in which the participants respond with a number corresponding to a given amount of use or perception of how well they have used their affected arm when away from the laboratory environment. For example, a participant would respond to the question "How much do you use your more affected arm to turn on a light switch?" by choosing the appropriate response from the MALa (Tab. 2). Only the mean of the "amount" section of the MAL was used as an outcome measure for this analysis. The internal consistency of the MAL is relatively stable, and the reproducibility of the MAL is sufficient to detect an individual change of less than 1.0 on the scale.31
|
Covariates
The following variables previously were found to be predictive of follow-up outcomes for CIMT34–36 and, therefore, were included as covariates in the model to account for individual differences of participants prior to the intervention. The UE motor component of the Fugl-Meyer Sensorimotor Assessment was used as a covariate in the MALa model.36 It is a measure of the percentage of recovery of a person following a stroke that provides a quantifiable measurement of motor function and was designed primarily for use in rehabilitation settings. The UE motor component of the Fugl-Meyer test is a measure of a person's ability to move in and out of synergy, reflexes, wrist stability, grasping ability, and coordination.37
For the WMFT model, the ability to open the involved hand was used as a covariate.34 This covariate was defined as the ability to actively release a mass flexion grasp as defined by Fugl-Meyer assessment. This 0 to 2 scale is graded as follows: a score of 0 was given if the individual was unable to release the grasp, a score of 1 was given if the participant could release the mass flexion grasp, and a score of 2 was given if the participant could fully extend the fingers from the starting grasp position.38
External Criterion
The external criterion used for this study was the perceived recovery section of the Stroke Impact Scale39 taken at the follow-up test. The participants rated their perceived recovery of their more affected hand and arm since the stroke. This is rated by the individual on a scale of 0 to 100, with 100 corresponding to full recovery and 0 corresponding to no recovery. The use of a participant-based questionnaire as an external criterion has not been widely reported in stroke rehabilitation, but this method is similar to that implemented in other rehabilitation populations in which the respondent's perception of recovery was used as an external criterion.15,17–20,40 For the purposes of this study, we dichotomized patients into
50% (50% or greater perception of recovery in more-affected UE), and <50% (less than 50% perception of recovery in more-affected UE) groups.
Data Analysis
All data analyses were performed with the SPSS software program, version 12.0.* First, descriptive statistics were generated for each measure. Kolmonorov-Smirnov tests were used to assess whether the data for each measure approximated a normal distribution and whether transformations of the measures of interest were necessary before being included in the regression models. The following data analyses then were performed to address the specific purposes of this study.
Prediction of outcome (purpose 1)
Simultaneous regression models determined whether MALa and WMFT scores contributed to perceived recovery ratings 4 to 6 months after starting CIMT. The first regression model included age, Fugl-Meyer UE motor component score, change in MALa scores over 4 to 6 months, and raw follow-up MALa score. The rationale for including age and the Fugl-Meyer UE motor component score was that these variables were previously found to be predictive of follow-up MALa scores for CIMT.35 The second regression model included the ability to open the involved hand (Fugl-Meyer test item), follow-up (4–6 months) change in WMFT score, and raw follow-up WMFT score. The rationale for including the ability to open the involved hand was that this variable was previously found to be predictive of WMFT scores for CIMT.34
The change and raw MALa and WMFT scores were simultaneously entered into the regression model to determine which measure was most appropriate for determining a clinically meaningful outcome at follow-up. Variance inflation factor (VIF) was reported to assess whether excessive collinearity was a concern by having change and raw scores for the predictor variables in the regression model. The VIF expresses the degree to which collinearity among the predictors degrades the precision of the regression estimate. Theoretically, VIF estimates range from 1 to infinite, with larger numbers being more indicative of collinearity. Practically, a VIF value greater than 10 indicates that the predictor variable has a strong linear association (r>.95) with other predictor variables included in the regression model.41
Calculation of cutoff scores (purpose 2)
Cutoff scores were calculated only for individual measures that contributed to perceived recovery ratings at P<.10. We used a liberal criterion for selecting measures for cutoff scores because we wanted to avoid eliminating potentially useful measures at this stage of the analysis. First, receiver operating characteristic (ROC) curves assessed whether the measure accurately predicted subjects with self-reports of 50% or greater perceived recovery since the time of their stroke.42,43 The ROC curve was generated by considering each individual point of the selected measure as a potential cutoff for distinguishing between subjects who reported 50% or greater perceived recovery and those who did not. The predictive performance of each individual point is then plotted on a graph with the y-axis representing the true positive rate and the x-axis representing the false positive rate. The ROC curve is generated when the individual points are connected, giving an overall estimate of the predictive accuracy of the selected measure.
A common way to summarize findings from an ROC curve is to report the amount of area under the ROC curve (AUC). The expected range of AUC scores is from 0.5 (no better than chance identification of outcome of interest) to 1.0 (perfect identification of outcome of interest).44,45 In this study, the AUC (and corresponding 95% confidence interval [CI]) was calculated with a nonparametric method that did not require normal distribution of perceived recovery ratings.46 The AUC for this study can be interpreted as the probability the selected measure has in identifying CIMT participants who rated perceived recovery at 50% or greater at follow-up.
After the ROC curves were generated, cutoff scores were calculated for each measure that explained more than 50% AUC based on the lower bound of 95% CI. This criterion was selected because it indicated that the measure likely predicted improved status at better than chance rates. Individual points were considered for cutoff scores by visually inspecting the ROC curve to identify points associated with maximal true positive and minimal false positive rates. These candidate points were further investigated by calculating sensitivity (Sn), specificity (Sp), and likelihood ratios (LR) for each potential cutoff score. For each point, Sn was determined by calculating the true positive rate, Sp was determined by calculating the false positive rate, the positive LR was calculated by dividing the Sn by (1 – Sp), and the negative LR was calculated by dividing (1 – Sn) by the Sp.47
The decision on which specific cutoff score was reported in the manuscript was based on the following criteria. First, we wanted to generate a cutoff score that accurately identified participants rating their perception of UE recovery at less than 50%. This cutoff score was determined by selected the individual point that minimized the negative LR, or the ratio of false negatives to true negatives. Therefore, the negative LR that resulted in the smallest value was selected as a cutoff score that could potentially identify participants rating their perception of UE recovery at less than 50%. Second, we wanted to generate a cutoff score that accurately identified participants rating their perception of UE recovery at 50% or greater. This cutoff score was determined by selecting the individual point that maximized the positive LR, or the ratio of true positives to false positives. Therefore, the positive LR that resulted in the largest value was selected as a cutoff score that could potentially identify participants rating their perception of UE recover at 50% or greater. Although these analytical techniques have not been widely reported in the stroke literature, similar techniques have been reported previously to determine cutoff scores in the rehabilitation literature.21,22
| Results |
|---|
|
|
|---|
|
|
|
|
|
| Discussion |
|---|
|
|
|---|
Our results show that if participants scored less than 1.15 on the MALa or more than 34.0 seconds on the WMFT on their follow-up posttest, they were more likely to indicate less than 50% recovery from their stroke. In contrast, if they scored greater than 2.50 on the MALa and less than an average of 11 seconds on the WMFT tasks, they were more likely to have rated 50% or more UE recovery from their stroke. Thus, these data give us a potential indication of the relationship between participants' perceived recovery and outcome measures used for CIMT. While the original intent of this project was to determine a minimal clinically important difference (MCID) for outcome measures following CIMT, our data supported the use of raw scores rather than change scores for predicting perceived recovery. When both the change and follow-up scores were simultaneously entered into the regression model, the follow-up scores were statistically more appropriate. This is a preliminary indication that a threshold score, and not a change score, is an appropriate measurement of improvement when the external criterion is participants' perceived recovery of UE function.
This methodology has not been widely utilized in the stroke literature, so comparisons to other studies are limited. For example, van der Lee et al48 have proposed an MCID for the MAL that was 10% of the total range of the scale (0.5 change in MAL). This MCID was based on clinical expertise and reports of similar outcome measures used in manual therapy interventions for the spine.49 While this proposed MCID for the MAL may have been a good starting ground to demonstrate improvement, measurement properties may have been neglected, as our data suggest that a threshold value may be more appropriate. Furthermore, the MCID proposed by van der Lee et al was not based on an external criterion. More research is needed to determine the meanings of changes in CIMT outcome measures and whether threshold or change scores predict outcome.
Other authors50 have indicated that a threshold score based on the outcome measure's scoring definitions may be more appropriate. Preliminary results from the Extremity Constraint-Induced Therapy Evaluation (ExCITE) trial51 suggest that a numerical change in MAL may indicate an increase in amount or function of the affected hand and arm, but this change may not result in substantial or clinical improvement if the individual still cannot use the hand functionally. A suggested threshold of a 3.0 on the MAL, indicating that individuals are able to use their impaired hand independently, may be an appropriate threshold to determine functional significance.50 That said, while possibly being an appropriate parameter to determine success, this approach does not offer a statistically calculated threshold that is linked to an external criterion based on the patient's perception of overall recovery.
Measuring and interpreting "clinically important change" have some limitations. The magnitude of this score can be influenced by prior functional level. Individuals who have lower initial functional levels may need smaller changes in function to achieve a "clinically important change." Individuals with higher initial function may need a greater change to render it clinically important.15,18 We attempted to control for initial function by including covariates in the regression models. The extent of change that is considered to be minimally important also may differ depending on whether the external criterion is determined by clinician, the caregiver, payer, or the patient.15,17,18 Using the participants' perception of recovery was core to our study. Clinically meaningful progress can be defined by many respondents following an intervention as what is "worthwhile."40 In our study, the participants were the judge of what was a worthwhile improvement, consistent with a patient-centered model.24–26 In such a model, the perception of recovery from stroke is the criterion by which the other CIMT outcome measures are judged and the determination of what is a meaningful change ultimately lies with the participant.
The potential limitations of this preliminary study should be considered when interpreting these results. First, this study was performed with individuals who met strict inclusion and exclusion criteria. While more diverse than most CIMT studies, this sample is not representative of the entire stroke population. Second, individuals were not randomly selected from the community, but were respondents to inquiries for participants.
Third, the external criterion for this study was based strictly on participant perspective. Although we have stated the importance of using the patient's perspective to determine what is a meaningful change, this approach can potentially introduce variability that may confound the responses. For instance, the participants' perceptions could have been influenced by depression, medication, or other behavior changes that we could not account for in our regression models.
Fourth, the reliability of MAL scores has been questioned in recent reports,30,31 and although we used a different version of the MAL, our approach could have affected the study. A shortened version of the MAL (MAL-14) has shown improved reliability and validity over the original measure,30 but that scale was not available at the time of the present study. No matter what form of the MAL is used, the effect that readministering the MAL within 2 weeks of its first application could have resulted in patient bias due to familiarity. Although this possibility has not been systematically studied, repeat baseline MAL testing within 2 weeks has not resulted in improved scores.52
| Conclusions |
|---|
|
|
|---|
| Footnotes |
|---|
This study was supported, in part, by the Office of Research and Development Rehabilitation R&D Service, Brain Rehabilitation Research Center, Department of Veterans Affairs, Gainesville, Fla, and Florida Biomedical Grant BM042 (Principal Investigator: Dr Light).
* SPSS Inc, 233 S Wacker Dr, Chicago, IL 60606. ![]()
| References |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
S.-W. Park, S. L. Wolf, S. Blanton, C. Winstein, and D. S. Nichols-Larsen The EXCITE Trial: Predicting a Clinically Meaningful Motor Activity Log Outcome Neurorehabil Neural Repair, September 1, 2008; 22(5): 486 - 493. [Abstract] [PDF] |
||||
![]() |
S. L Wolf On "Modified constraint-induced therapy..." Page et al. Phys Ther. 2008;88:333-340. Physical Therapy, May 1, 2008; 88(5): 680 - 684. [Full Text] [PDF] |
||||
![]() |
S. L. Wolf, C. J. Winstein, J. P. Miller, S. Blanton, P. C. Clark, and D. Nichols-Larsen Looking in the Rear View Mirror When Conversing With Back Seat Drivers: The EXCITE Trial Revisited Neurorehabil Neural Repair, October 1, 2007; 21(5): 379 - 387. [Abstract] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |