|
|
||||||||
Research Reports |
DU Jette, PT, DSc, is Professor and Director, Physical Therapy Program, Simmons College, 300 The Fenway, Boston, MA 02115 (USA) (diane.jette{at}simmons.edu). Address all correspondence to Dr Jette
LG Portney, PT, DPT, PhD, FAPTA, is Associate Professor and Director, Professional Program in Physical Therapy, MGH Institute of Health Professions, Boston, Mass
Submitted July 5, 2002;
Accepted December 18, 2002
| Abstract |
|---|
Key Words: Physical therapist education Professional development Professionalism
| Introduction |
|---|
|
|
|---|
In 1995, May et al9 introduced a model for evaluation of professional attributes of physical therapist students. They adopted an approach called "ability-based assessment." The approach was based on one used by their colleagues at the University of Wisconsin Medical School and involved the identification of professional behaviors that they believed transcend practice settings and are required for success as a physical therapist. The professional behaviors were called "generic abilities" and observable behaviors, or criteria, were determined to help define different levels of competence for each generic ability.
The initial process of determining generic abilities was accomplished using a Delphi technique with input from clinical instructors selected by May and colleagues.9 Through this consensus process, 10 generic abilities emerged: Commitment to Learning, Interpersonal Skills, Communication Skills, Effective Use of Time and Resources, Use of Constructive Feedback, Problem Solving, Professionalism, Responsibility, Critical Thinking, and Stress Management. Using a nominal group process, May et al9 then followed up with a small, selected group of clinical instructors to develop a list of sample behaviors or criteria to further define the 10 generic abilities. The Delphi technique was used again to categorize the behavioral criteria as beginning, developing, or advanced competence. These 3 levels of behavioral criteria were intended to reflect a progression in the generic abilities that students should demonstrate by: (1) the end of the first year of professional education, (2) the end of the second year of professional education, and (3) the end of the final clinical education experience, respectively.
May and coworkers9 have used the generic abilities and behavioral criteria to implement a process of ability-based assessment of physical therapist students within their institution. Following publication of their study describing the model, we believe many professional education programs began to use the generic abilities and behavioral criteria as a framework for advising students, developing course syllabi and objectives, and evaluating students.
Validation of the measurement of the generic abilities is an important consideration if this framework is to have relevance for designing learning activities and assessing students' professional development. To our knowledge, however, no report describing the validity of using the generic abilities has been published beyond the original consensus study by May et al.9 Construct validation is needed in order to interpret the results of an evaluation tool as a measure of a quality or trait, such as professional behavior, that is not readily defined.10,11 A construct, therefore, is an attribute that cannot be directly observed or measured, but has effects that are determined through an indirect measurement.11 As described by May and colleagues, the generic abilities are examples of possible constructs, and the behavioral criteria are examples of measurable attributes that are influenced by them. Our question was whether those behavioral criteria were indeed influenced, at least in part, by specific constructs that could be identified through statistical analysis. The purpose of our study, therefore, was to examine the validity of using the behavioral criteria described by May and colleagues to measure generic abilities.
| Method |
|---|
|
|
|---|
A total of 303 students enrolled in 4 different classes in the 2 programs were eligible for inclusion. One-hundred eighty-three students participated (60%). We attributed the relatively low participation rate to the many school-related demands on students' time, but we have no data to support this belief. Eighty-seven percent of the participants were women. Sixty-four percent of the participants were 25 years of age or younger, 22% were 26 to 30 years of age, 9% were 31 to 35 years of age, and 4% were over 35 years of age. Eighty-eight percent of participants had received a baccalaureate degree, and 12% were completing their senior year of undergraduate study. Table 1 presents participant characteristics.
|
We developed an assessment tool using the generic abilities and performance criteria established by May et al9 in their first iteration of the abilities-based assessment. Students completed the survey questionnaires anonymously and at their leisure. The instrument included questions on demographic information, including age, sex, and number of completed full-time clinical education experiences. The questionnaire asked students to assess how frequently they performed all the behaviors on a 7-point Likert scale with the anchors "rarely" to "usually."12 Using the stem "In my academic and/or clinical work, I ..., " a total of 152 behaviors were assessed. We adapted some of the language to clarify the criteria for application in either a clinical or academic setting. For example, the original criterion "identifies problems" was edited to "identify problems that need to be addressed." The original criterion "identifies and locates appropriate resources" was restated as "identify and locate appropriate resources to carry out required activities."
We chose to ask students for a self-assessment of the frequency of their behaviors. This approach was based on our own use of the generic abilities with the students in our academic programs and various anecdotal reports from other programs that used the generic abilities as part of their students' self-assessment process. We believe that the ability to assess one's behavior is a prerequisite for determining the need for self-improvement and is a cornerstone of professional education. Furthermore, some authors think that knowledge of self evolves through interaction with others and through the perception of how one is evaluated by others.13
Data Analysis
To explore the construct validity of the generic abilities, data were analyzed using principal components factor analysis with varimax rotation. Factor analysis is a procedure that is used to examine a large set of variables that are believed to represent elements of an abstract construct and to reduce the items to a smaller set of underlying concepts.14 The analysis identifies clusters of variables or factors. Items within a factor are highly correlated with each other, and they are not correlated with items from other factors. Items are identified as belonging to a factor based on their correlation with that cluster of items. Rotation allows the simplest solution among a variety of solutions that may be compatible with the data. Cronbach alpha was used to demonstrate internal consistency of items within each factor. We determined Cronbach alpha to provide additional evidence that the items within a factor were measuring the same underlying construct.
We first attempted to create 10 factors, given that the original study included 10 generic abilities.9 The statistical analysis, however, could not generate 10 factors. Nine-factor and 8-factor solutions also could not be generated. A 7-factor solution was successful.
Factor analysis also was used to obtain factor scores, which could be used for further analysis of the various components of the generic abilities. Through factor analysis, scores are generated for each identified factor, essentially representing a weighted sum of item values within a factor. These scores can then be used as an aggregate value for that cluster of items for each subject. This approach allowed us to reduce the number of variables from the original 152 items without losing the meaningfulness of the original variables.15 We then used the factor scores as dependent variables in 7 separate one-way analyses of variance (ANOVA) to examine the differences in generic abilities across the 3 levels of clinical education experiences. Following the ANOVAs, a Tukey post hoc test was used to determine differences at an alpha level of .05. The Statistical Package for the Social Sciences (SPSS, version 10.1 for Windows*) was used for all analyses.
| Results |
|---|
|
|
|---|
.40) with more than one factor, and, in these cases, the item was assigned to the factor with the larger correlation. Table 2 shows the variance explained by each factor and the eigenvalue and Cronbach alpha associated with each factor. Table 3 lists the full set of items and the factor loadings for each item.
|
|
The ANOVAs demonstrated that there was no difference in factor scores across the levels of clinical education experiences for 3 factors: Professionalism, Interpersonal Skills, and Working Relationships. The remaining 4 factors did show a differences in factor scores across the levels of clinical experience (Figure).
|
Factor 2: Critical Thinking
This factor included 32 items and accounted for 5% of the total variance. Critical Thinking represented the abilities to recognize the need for information, to find and analyze information, and to manage decisions. The ANOVA showed a difference in Critical Thinking overall across levels of progression in the program and between students who had completed no clinical education and those who had completed all of their clinical education experiences (Fig. 1B).
Factor 3: Professional Development
This factor included 36 items related to behaviors such as setting professional goals and assuming a leadership role as well as self-assessment. It accounted for 3.2% of the total variance. The factor scores representing Professional Development differed between students who had no clinical education and those who had completed all clinical experiences (Fig. 1C).
Factor 4: Communication Management
The fourth factor included 26 items related to management of information, time, and resources, and it accounted for 3.0% of the total variance. This factor scores differed between the students who had no clinical education experience and students who had completed some or all clinical education experiences (Fig. 1D).
Factor 5: Personal Balance
This factor included 10 items related to prioritizing and focusing on commitments and effective stress management. It accounted for 2.8% of the total variance. Factor scores differed for students who had completed no clinical education experience and those who had completed some or all clinical education experiences (Fig. 1E).
Factor 6: Interpersonal Skills
This factor included 9 items and accounted for 2.2% of the total variance. The factor was composed of behaviors such as positive attitude, ability to motivate others, listening skills, and demonstrating empathy. No difference in factor scores occurred across levels of clinical education in the program (Fig. 1F).
Factor 7: Working Relationships
The seventh factor contained 6 items related to the ability to give and receive feedback, to demonstrate flexibility, and to work well with others. This factor accounted for 2.2% of the total variance. No difference in frequency of this behavior occurred across levels of clinical education in the program (Fig. 1G).
| Discussion |
|---|
|
|
|---|
The work of May et al9 represented the first time that a framework was widely disseminated and used by physical therapist educators for assessment of students' behaviors. In our experience, before May and colleagues' study, clinical and academic faculty relied primarily on intuitive assessments, often with great hesitancy, because they did not have well-accepted or well-understood labels for what they observed. We believe that the 7 factors identified in the our analysis correspond fairly well to the generic abilities identified by May and colleagues.9 Their study provided the basis for items on our questionnaire. Our analysis, therefore, provides evidence for the construct validity of the generic abilities. By identifying 7 rather than 10 generic abilities, however, our analysis suggests that some redundancy might exist in the original list of abilities and that our list may represent an improvement by identifying unique behaviors only.
Professionalism
The factor that we labeled Professionalism is similar to 2 abilities that May et al9 called Professionalism and Responsibility. This single factor represents the largest proportion of variance in our sample and, therefore, can be considered the most prominent of the generic abilities. In a study of self-described professionalism in nurses, "professionalism" emerged as a factor through factor analysis.13 In an instrument developed to examine medical student performance, professionalism was 1 of the 7 basic traits identified.19 Professionalism, as defined in the field of medicine, includes a number of elements, divided or combined into a variety of different categories by different authors.6,20,21 These elements include: (1) a commitment to service,20 (2) an adherence to ethical standards,6 (3) a demonstration of humanistic values (eg, integrity, honesty, respect for others, compassion and altruism),6,19,21 (4) responsibility and accountability,6,19 and (5) commitment to professional advancement.6 In our study, the behavioral criteria related to Professionalism were similar.
For the students in our sample, the ability that we labeled Professionalism did not differ across the levels of professional education. The 2 programs involved in this study use the generic abilities as a framework for student self-assessment and academic counseling from the start of their professional education, emphasizing the importance of professionalism and interpersonal and collaborative skills. This focus could account for the students' perceptions of their professional behavior.
Critical Thinking
Our Critical Thinking factor is similar to both the Problem Solving and Critical Thinking abilities in the model proposed by May et al.9 In an essay arguing for a normative definition of professionalism, Swick6 included behaviors (eg, dealing with high levels of complexity and uncertainty, exercising independent judgment, making appropriate decisions with incomplete information) similar to those reflecting the Critical Thinking factor that we identified. He also described a need for practitioners to reflect upon their actions and decisions, arguing that the ability to critically reflect is essential to deductive reasoning in solving clinical problems. Hayes and colleagues,8 using qualitative methods, reported on behaviors of physical therapist students that might alert clinical instructors to poor performance. According to that study, inadequate knowledge and skills, which included problem solving and judgment, comprised the largest group of behaviors (43%) described by the clinical instructors.
In a study of occupational therapist students, nearly all those who failed fieldwork experiences lacked the ability to be flexible in their thinking and to devise strategies so that their fieldwork experiences were successful.7 The students had difficulty formulating or accepting alternative solutions to problems or accepting that more than one intervention was appropriate for the same problem. Studies of nursing students have provided similar results. In one study of nursing students, Periard et al22 found an inverse relationship between intolerance for ambiguity and professional attitudes. In a study by Brooks and Shepherd,23 a measure of critical thinking was moderately (r=.45) correlated with a measure of professionalism. In the studies by Periard et al22 and Brooks and Shepherd,23 increases in professionalism were found with increased academic preparation. In our study, Critical Thinking abilities also were greater for students with more clinical education. We believe that building skills in critical thinking is a major focus of physical therapist education. In our opinion, therefore, it was to be expected that students would report a difference in this ability at different levels of professional education.
Professional Development
Professional Development, we believe, is similar to, although broader than, the ability that May et al9 labeled Commitment to Learning. Behaviors that fall into this factor are similar to 2 behaviors Swick6 described as being central to professionalism: a commitment to excellence (which includes the lifelong acquisition of new knowledge and skills) and the need for personal accountability (which implies self-critique, knowing one's limitations, and assuming responsibility for setting professional goals).
In our study, Professional Development behaviors showed a difference between the group that did not complete any clinical education experiences and the group that had completed all of their clinical education experiences. This result suggests that the behaviors related to this ability, such as knowing one's limitations, leadership, and setting professional goals, may fully develop only after the student has experienced the rigors of a final clinical education experience. These are skills that may require prolonged exposure to the professional environment to be understood and developed further.
Communication Management
Our fourth factor, which we labeled Communication Management, was complex and combined many items from the abilities that May et al9 described as Communication Skills and Effective Use of Time and Resources. The importance of communication was demonstrated by Hayes and colleagues,8 who found that 28% of the behaviors that clinicians cited as problems affecting student performance could be categorized as communication issues. Skills related to Communication Management showed a higher frequency in students who had 1 or 2 clinical education experiences than in those who had none; the frequency was not higher in students who had completed their final clinical experience. This finding may have occurred because, in our programs, behaviors related to use of literature, research, and information management are stressed early in the curriculum and reinforced over the course of early clinical education experiences.
Personal Balance
The fifth factor, Personal Balance, encompassed the items in the category described by May et al9 as Stress Management, but our factor is also broader in scope. In the study by Hayes et al,8 a group of behaviors classified as "unprofessional" were cited by clinicians as leading them to question students' competence. Poor stress management was one of the behaviors in this group. Behaviors related to Personal Balance showed a higher frequency in students who had completed at least one clinical education experience than in those who had not completed any clinical education experience. We believe that, as students encounter what we view as rigorous graduate programs (including a demanding clinical education experience), the skills that are part of maintaining Personal Balance (eg, prioritizing commitments, focusing on tasks, finding outlets for stress) must develop rapidly and be maintained for the students to succeed academically.
Interpersonal Skills
Interpersonal Skills is a factor similarly identified both in our analysis and in the abilities described by May et al.9 This factor is supported, in part, by a study in which medical school faculty identified interpersonal skills as being 1 of 7 basic professional traits.19
Working Relationships
We identified a factor we called Working Relationships that encompassed items related to Use of Constructive Feedback as described by May et al,9 but went beyond feedback to include behaviors related to working with others. Interestingly, Gutman et al7 found that occupational therapist students who failed clinical fieldwork showed an inability to evaluate the emotions, motivations, strengths, and weaknesses of themselves and others and had difficulty interpreting feedback. They also found that students with difficulties in the clinical setting tended not to take responsibility for their performance but blamed supervisors for being inattentive, being inaccurate in their critiques, or withholding information.
Behaviors related to Interpersonal Skills and Working Relationships did not differ in frequency across groups with various levels of clinical education. These behaviors included listening, interpreting feedback, being nondefensive, and motivating others. Because 88% of the students in our sample were postbaccalaureate students, we believe that these behaviors may have been developed in school and employment settings before matriculation and are likely to have been addressed within written recommendations provided for a candidate's admission and during the admission interview. It is possible, therefore, that students selected for our programs had high levels of these abilities.
Limitations
Our study represents only one step in the approach to provide evidence for the construct validity of the generic abilities. Limitations to this study included the small sample size given the number of items on which students were asked to rate themselves. In addition, a response rate of 60% seems relatively low, and, because the data were collected anonymously, we do not know how our sample differed from those who did not participate. Moreover, students from only 2 schools were studied.
We also used a cross-sectional design. Because we examined differences in behavior across levels of clinical education among different classes, group effects may be a result of differences in curricula or expectations of performance by academic and clinical faculty for different classes. In addition, some of the items we used in the measurement of student behaviors, although derived from the items suggested by May et al,9 were changed slightly so they could be used in self-assessment questionnaires. Deviation from our unique wording and application of the instrument may result in different interpretations of the generic abilities than that reported here.
Because the exact wording and inclusion of specific behaviors can result in somewhat idiosyncratic findings, factor analysis is subject to the interpretation of the researchers. Items within a factor are highly correlated with each other, but the combinations of items that appear within a factor are not always those expected. For instance, our fourth factor, Communication Management, includes several items related to management of time, organization, and use of resources as well as items related to using literature or instructional materials and modifying communication for specific audiences (Tab. 3). By demonstrating a high correlation among these items, the factor analysis forced us to consider what elements represent the essential nature of this behavior. For example, as a construct, these items all require a form of problem solving and analysis. As we continue to explore professional behaviors and methods for inculcating them in our students, these findings offer a challenge to identify the types of learning activities that will be useful across multiple dimensions.
We used the 1995 version of the performance criteria and the 3 levels of competence described at that time. In 1996, May and colleagues24 repeated the Delphi process with a larger number of clinicians, but they reported on this study in a self-published, nonpeer-reviewed form. As a result, some behavioral criteria were modified, deleted because of redundancy, or recategorized as "postentry level," adding a fourth level of competence to the original categories. Because there were no published data on this revision, we chose to use the original comprehensive list of behavioral criteria available. Further study might include examination of the validity of the revised version as well as the congruence of student self-assessment of their professional behaviors with assessment by others.
We believe that the differences across studies demonstrate the need to further explore the constructs of professional behavior in physical therapy. In addition, the factor analysis derived in our study left 48% of the variance in professional behavior unexplained, suggesting the existence of behaviors that we were unable to identify. Further research is needed to determine how the generic abilities change over time, how we can cultivate the growth of the behaviors that are related to these abilities, how much of each ability depends on the skills that the student brings to the program, and how the program's structure, content, and clinical experiences influence the development of these behaviors. Examining the development of these abilities longitudinally and examining how they might change after graduation may be useful. The ultimate goal of such research is to identify teaching activities and assessment tools related to development of professional behaviors that are meaningful to students and faculty.
| Summary and Conclusions |
|---|
|
|
|---|
| Footnotes |
|---|
This study was approved by the institutional review boards of Simmons College and the MGH Institute of Health Professions.
This article was presented as a platform presentation at the American Physical Therapy Association Combined Sections Meeting, February 2024, 2002, Boston, Mass, and at the Massachusetts and Rhode Island Chapter Meeting, October 19, 2002, Hyannis, Mass.
* SPSS Inc, 233 S Wacker Dr, Chicago, IL 60606. ![]()
| References |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
L. L Swisher, J. W Beckstead, and M. J Bebeau Factor Analysis as a Tool for Survey Analysis Using a Professional Role Orientation Inventory as an Example Physical Therapy, September 1, 2004; 84(9): 784 - 799. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |