Abstract
Background and Purpose The use of standardized outcome measures (OMs) can support clinicians' development of appropriate care plans, guide educators in curricular decisions, and enhance the methodological quality and generalizability of clinical trials. The purposes of this case report are: (1) to describe a framework and process for assessing psychometrics and clinical utility of OMs used poststroke; (2) to describe a consensus process used to develop recommendations for stroke-related OMs in clinical practice, research, and professional (entry-level) physical therapist education; (3) to present examples demonstrating how the recommendations have been utilized to date; and (4) to make suggestions for future efforts.
Case Description A task force of 7 physical therapists with diverse clinical and research expertise in stroke rehabilitation used a 3-stage, modified Delphi consensus process to develop recommendations on OM use. An evidence-based systematic review template and a 4-point rating scheme were used to make recommendations on OM use by care setting and patient acuity, for research, and for inclusion in professional education.
Outcomes An initial list of 77 OMs was developed based on input from numerous professional sources. Screening measures and duplicate measures were eliminated. Fifty-six OMs received full review. Measures spanned the constructs of body structure/function (21), activity (28), and participation (14). Fourteen measures received a rating of “highly recommend.”
Discussion Use of highly recommended OMs may provide a common set of tools enabling comparisons across patients, interventions, settings, and studies. The use of a clearly defined, comprehensive assessment template may facilitate the pooling of data on OMs and contribute to best practice guidelines. Educational recommendations may inform curricular decisions.
Recent evidence-based practice initiatives and the need for accountability in clinical practice have focused attention on the use of standardized outcome measures (OMs) in physical therapy.1–4 Monitoring patient status through the appropriate use of OMs is considered good clinical practice5 and has been suggested to enhance patient care as it contributes to a more thorough examination, assists in the development of a care plan,6 allows physical therapists to quantify observations and compare patient status between examination periods,7,8 facilitates communication between care settings,9 and increases the efficiency of practice.10 From an administrative perspective, appropriate use of OMs has been proposed to help managers measure costs,9 identify hospitalized patients who are “at-risk,”11 enhance reimbursement,12 and compare outcomes between clinicians and settings.11 Because OMs are key to answering study questions,12 researchers have been urged to carefully consider OM choice in order to enhance the methodological quality and clinical relevance of clinical trials.4,9,13,14
Although the benefits of routine use of appropriate standardized OMs abound, widespread use is lacking. In a 2009 survey of 1,000 physical therapists in clinical practice, fewer than half reported using standardized OMs.6 Other studies report similar limited use patterns.5,6,13,15–17 Barriers to consistent OM use include limited time; lack of equipment; therapist perception that patients may have difficulty completing the OMs; physical therapist attitude, knowledge, or skill; lack of financial compensation for measure completion; and poor availability of tools.6,15,17–21 Use of OMs also is lacking in research. A recent systematic review of stroke-related randomized trials showed that just slightly more than half used established OMs.13
Reports on frequency of use have focused on what OMs have been used versus what should be used. Test “batteries” of OMs used poststroke have been reported based on frequency of use.5,10,13,22 Several authors have made recommendations for OMs used poststroke,23–29 but most are limited to specific constructs,23,27,30 lack information about how recommendations were developed,27,28 or recommended multiple measures of the same construct without guidance about choice.24 The Guide to Physical Therapist Practice31 (the Guide) lists 1,373 tests and measures in 24 categories but offers limited guidance about choosing between different measures of the same construct. Several online repositories contain information on OMs, both generic32,33 and stroke specific34,35; however, these resources do not provide recommendations regarding OM choice. Development of recommendations regarding OMs, based on appropriateness versus frequency, has been suggested to have numerous advantages, including allowing comparisons across patients, clinicians, facilities, and interventions.8 Consistent clinical use of recommended OMs could support the development of a dataset that would inform clinical decisions and contribute to the evidence for practice guidelines.8
Thus, the purposes of this case report are: (1) to describe a framework and process for reviewing and assessing psychometrics and clinical utility of OMs used poststroke; (2) to describe a consensus process resulting in recommendations regarding stroke-related OMs for use in clinical settings, research studies, and professional physical therapist education; (3) to present examples demonstrating how the recommendations have been utilized to date; and (4) to offer suggestions for future efforts in consensus-based OM recommendations.
Case Description: Target Setting
The recommendations for the use of OMs poststroke were developed in several stages using both qualitative and quantitative data analyses. As part of the first stage, the American Physical Therapy Association (APTA) Neurology Section Board of Directors (NS BOD) appointed 2 individuals representing the Neurology Section's regional continuing education course, “Neurologic Practice Essentials: A Measurement Toolbox” (Toolbox) (J.E.S.), and the Consensus Conference for Entry-Level Education Guidelines (G.P.Z.) to co-chair the stroke task force. The co-chairs and the NS BOD then selected 5 additional task force members, representing geographic diversity and expertise in clinical, educational, and research areas related to stroke. Table 1 illustrates the backgrounds of the task force members. The NS charged the task force with the following objectives: (1) determine criteria for OM review and recommendation; (2) identify OMs to be reviewed; (3) develop the process for achieving consensus on recommendation; and (4) provide recommendations for use of OMs in clinical practice, professional physical therapist education, and research.
Background Information on the StrokEDGE Task Force Members
Development of the Process
Determine the Criteria for Outcome Measures Review and Recommendation
The task force reviewed the Evidence Database to Guide Effectiveness (EDGE)36 template developed by the APTA's Section on Research as a potential framework for assessing OMs. Although the EDGE template provides a general format, it does not offer a decision-making framework specifically with regard to OMs appropriate to stroke. To ensure that the EDGE template would enable the reviewers to capture all necessary data to make an informed recommendation regarding OM use, the task force held a focus group discussion. During this discussion, task force members were asked to review and discuss the merits of each item on the EDGE template. The group proposed several revisions to the EDGE template in order to meet the specific outcomes of this project. For each proposed addition to the template, a formal discussion was initiated. If the group achieved 100% consensus on a proposed item, it was incorporated into the EDGE template. The resulting modified template was termed the “StrokEDGE” template (Appendix). The StrokEDGE template integrates data from the following areas as it relates to each test: construct, type of measurement, instrument properties, instrument clinical usability, recommendation for use by practice setting and patient acuity, and suitability for professional education and research.
Application of the Process
Review of Outcome Measures
The task force used a critically appraised topic (CAT) approach to review the available literature on OMs. This process includes a structured format to formulate questions, appraise literature, and make recommendations.37 The CAT approach was developed by the McMaster University Occupational Therapy Evidenced-Based Practice Research Group and is a structured way to critically review the essential components of published peer-reviewed articles.38 Using the CAT approach, task force members individually reviewed and evaluated the available literature on OMs in assigned content areas. The task force agreed that the International Classification of Functioning, Disability and Health (ICF)39 model would be used as a framework to characterize the OMs reviewed. The ICF framework has been recommended as a useful tool to capture the constructs of OMs.7,8,22 The task force wanted to include OMs capturing 3 levels of the ICF model: body structure and function, activities, and participation. The ICF model defines function as the physiological and psychological functions of body systems and structure as the anatomical parts of the body. Activity describes the execution of a task or action by an individual, and participation refers to an individual's involvement in a life situation. In cases where an OM captured multiple ICF categories (eg, OMs that measure balance), task force members indicated this in their review.
In order to maximize interrater and intrarater reliability in making recommendations for each of the OMs, a 4-point scoring matrix for clinical recommendations was developed. The scoring criteria were discussed and revised until the task force reached unanimous agreement. A score of 4 indicates the OM has good psychometric properties and clinical utility when used in the stroke population, whereas a score of 1 indicates the OM has poor psychometric properties or clinical utility. Table 2 lists the criteria of the 4-point recommendation system.
Outline of the StrokEDGE Scoring Matrix Used to Make Clinical Recommendations for Outcome Measure Use by Evaluating the Strength of the Outcome Measurement Tools' Psychometric Properties and Utility in the Stroke Populationa
Reviewers also made recommendations on OMs physical therapist students should “learn to administer” or “have knowledge of/be exposed to” during professional education. The task force used A Normative Model of Physical Therapist Professional Education40 and the Entry-Level Neurologic Content (E-L NC) to help inform educational recommendations. The E-L NC curriculum guidelines were developed to assist faculty with curriculum development in the area of neurology. These guidelines emerged from a consensus-reaching process among experts in the field using A Normative Model of Physical Therapist Professional Education and the Guide as a frame of reference. Using a structured and systematic decision-making, consensus-reaching process, participants identified specific and all-inclusive entry-level neurologic content, examples of terminal behavioral objectives for that specific content, examples of instructional objectives to be achieved in the classroom, and examples of instructional objectives to be achieved in clinical practice. Based upon the fact that these documents are intended to guide educators in the integration of essential neurologic content within a physical therapist professional curriculum, the documents were used to inform the task force as they evaluated measurements and made recommendations. One of the task force members (G.P.Z.) was a co-chair of the team that developed the E-L NC and provided guidance in using the Guide as an evidence-based frame of reference for the development of the educational recommendations for this project.
The final area of recommendation was relative to use of OMs research involving patients poststroke. Strong psychometric data were the critical threshold in this area. The task force felt that clinical utility limitations such as time to administer and copyright issues were less critical in the research arena.
Formal Outcome Measures Assessment: A Process of Achieving Consensus on Recommendations
A modified Delphi consensus method was used to reach agreement on the recommendations. Traditionally, the Delphi method uses a series of sequential questionnaires with controlled feedback to seek consensus among a group of experts.41 Lindeman42 suggested that the Delphi method improves objectivity because of the participant's lack of inhibition from the group process. Participation in a Delphi process promotes communication and debate, particularly in an area where empirical evidence is lacking or limited. The task force members believed that the focus on objectivity, communication, and scholarly debate to achieve expert consensus made the Delphi process ideal for accomplishing the task. In this project, in order to achieve consensus on the recommendations, the Delphi approach consisted of 2 rounds of formal assessment using a survey questionnaire approach and 1 final round termed the “Delphi consensus conference call.” To further promote quality and efficiency in the Delphi review process, the task force was divided into working OM content subgroups (gait and balance, upper extremity and sensation, and motor control) based upon members' clinical and research expertise. Each task force member was the primary reviewer for 7 to 9 OMs. Primary reviewers conducted a literature search and completed a StrokEDGE document for each assigned OM.
Single Peer Review Delphi Process
Once the StrokEDGE document was completed by a primary reviewer, the document was sent to a secondary reviewer initiating the first step in the Delphi process, the “single peer review” process. The peer reviewer evaluated the StrokEDGE document to determine agreement with the recommendations in each category. In cases of disagreement, the 2 reviewers discussed the evidence and revised the recommendation, if appropriate, until consensus was achieved. The first round of the Delphi process took approximately 3 months.
Group Delphi Online Survey Review Process
The completed StrokEDGE documents were uploaded to an anonymous online survey site housed on the Seton Hall University server through Academic Survey System and Evaluation Tool (ASSET). Task force participants were asked to critically review all StrokEDGE documents and supporting evidence for each category of OM recommendation and indicate their agreement by a “yes” or “no” response. This process of critical review constituted round 2 of the Delphi process. Based upon prior literature, which suggests that 70% to 80% agreement is considered a reasonable guideline for this type of data analysis, 80% agreement was sought for each recommendation.43
Delphi Consensus Conference Call
For those recommendations reaching less than 80% agreement, the co-chairs (G.P.Z. and J.E.S.) independently conducted an additional review of the literature, proposed a recommendation, and provided written support for the ratings. A summary document of the revised ratings and rationale was sent to task force members. Following review of the document by the task force, a conference call was held to address and discuss the proposed ratings and achieve consensus. Following discussion, members were asked to indicate whether they agreed with the revised recommendation. The final vote resulted in 100% consensus for all OM recommendations. The Figure provides an overview of the task force charges and the process the group developed and used to address them.
Task force charges and 3-stage process developed and used by the StrokEDGE task force. OM=outcome measure, CAT=critically appraised topic, CSM=Combined Sections Meeting.
Outcomes
The task force developed an initial list of 77 potential OMs for review, including those recommended by the APTA Neurology Section's Stroke Special-Interest Group (25) and by the E-L NC (19), OMs included in 2 Web-based repositories of stroke OMs (45),31,32 and OMs included in the Toolbox course (16). Numerous OMs were represented in more than one of these sources. The task force agreed that tools capturing the constructs of language (1), depression (3), perception (8), and cognition (5) would not be reviewed at this time because these tools are used primarily during the screening or systems review components of the examination versus measuring the outcome of intervention. Furthermore, the group eliminated measures where there was overlap in a construct. For example, the Two-, Three-, and Five-Minute Walk Tests were eliminated, and only the Six-Minute Walk Test was included for review. A final list of 56 OMs was selected for detailed review and recommendation. Task force members agreed that if review of the literature uncovered additional OMs that would be appropriate for review, these could be added at a later point. However, no additional measures were identified.
Following the modified 3-round Delphi process, 100% consensus was reached among the 7 task force members for the OMs recommendations in the areas of practice setting and patient acuity (Tab. 3). The list includes measures that capture the ICF domains body structure/function (21), activity (28), and participation (14). Some of the reviewed measures captured multiple ICF domains. Fourteen OMs (25%) received a rating of 4 in at least 2 practice categories (setting, patient acuity). These ratings are highlighted in Table 3.
Reviewed Outcome Measures (OMs) by International Classification of Functioning, Disability and Health (ICF) Category, Task Force Recommendations for OM Use by Practice Setting and Patient Acuity, OMs Recommended for Entry-Level Physical Therapist Education, and OMs Recommended for Research Usea
During the Delphi consensus process, task force reviewers made recommendations for inclusion of OMs in professional physical therapist education by either not recommending inclusion or indicating students should “learn to administer” or “have knowledge of/be exposed to” the OM. As with other recommendations, a standard of 80% agreement was used in the area of educational recommendations. Table 3 illustrates the 14 OMs that the task force recommended physical therapist students learn to administer, as well as the 20 OMs that are recommended for student exposure.
Finally, using this same consensus process, the task force developed OM recommendations for use in studies involving individuals poststroke. Forty-eight measures were recommended for research purposes. These measures span all 3 ICF domains. All measures recommended for research have “good” to “excellent” psychometric properties. Many OMs receiving a recommendation for research are not highly recommended for clinical practice, however, due to longer administration time, equipment required, copyright restrictions, or cost.
Discussion
One of the goals of the task force was to develop recommendations regarding the use of OMs for individuals poststroke. Through the use of a Delphi process, consensus was reached among 7 physical therapists with clinical and research expertise in stroke rehabilitation. The review criteria and recommendation categories reported are consistent with established psychometric standards.44,45 The recommendation criteria include clinically relevant issues such as administration time, ease of scoring, equipment required, and copyright issues. Additionally, the use of a CAT while reviewing the evidence on OMs further strengthens the recommendations.
The EDGE template developed by the APTA Section on Research36 was adapted to assess psychometric properties and clinical utility of the OMs reviewed. The revised StrokEDGE template addresses many of the previously described barriers to systematic OM use including time, equipment, and cost. 6,15,17–19 Explicitly evaluating these issues and structuring recommendations to support OMs that can be administered efficiently and with equipment typically available in most clinics may facilitate clinicians to more readily incorporate OM use. Additional barriers to OM use, such as therapist knowledge of OMs and lack of information regarding their utility based upon evidence, have been reported in the literature.6,15,17–19
Feedback received from nearly 400 therapists who have attended the Toolbox course suggested that availability of information on OM is an additional barrier to systematic OM use. The APTA Neurology Section addressed these issues via dissemination of the final StrokeEDGE documents, score sheets, recommendations, and administration information in a Web-based format.46 Furthermore, dissemination will occur via a collaborative agreement with Rehabilitation Measures Database (RMD), a Web-based repository of information on OMs. Beginning in 2013, RMD will include a category of “Professional Association Recommendations” to each OM listed.32 In addition, the collaboration with RMD may help address the concern about updating OM information, as the site conducts regular reviews to ensure content is current. Dissemination of the recommendations also is planned to occur via the “Tests & Measures” section of PTNow, a Web-based information portal developed and sponsored by APTA.47
Following the StrokEDGE task force work, the NS BOD has launched several additional task forces focused on those diagnosis groups commonly treated in neurological practice. These task forces utilized the process developed by the StrokEDGE task force with modifications specific to their target population. Task forces focused on multiple sclerosis, spinal cord injury, and traumatic brain injury made their recommendations in 2012–2013, and groups focused on vestibular disorders and Parkinson disease began work in early 2013. Various groups outside the APTA's Neurology Section also have mounted similar efforts.
Recently, the Centers for Medicare and Medicaid Services (CMS) implemented a claims-based data collection requirement for outpatient physical therapy services by requiring reporting of functional “G-codes” on claims.48 Physical therapists will be required to provide information about a client's status and goals in several areas including walking and moving, changing body position, carrying objects, and self-care. Severity modifiers indicating the percent impairment/limitation/restriction will be required. The CMS encourages the use of an appropriate assessment tool to justify the assigned level of severity. Although clinicians may use clinical judgment, their documentation must indicate how they determined the level of severity. Easy access to and use of recommended OMs may facilitate physical therapists' compliance with the requirements and ultimately enhance the provision of care for Medicare and Medicaid beneficiaries.
The task force recommendations were organized using ICF domains. This framework has been advocated previously to enhance comprehensive clinical examination7,8 and as a useful reference to identify and quantify the concepts of interest in clinical trials.22 Although the authors used the best available evidence and a consensus process among experts to classify measures across the 3 domains of the ICF, not all measures are “homogeneous” with regard to the domains. Some OMs may arguably be categorized in more than one domain (eg, balance), and other measures may contain sample items pertaining to more than one ICF construct. The identification of OMs that evaluate participation-level constructs addresses concerns about the paucity of participation OMs used in clinical practice and research.10,49 The fact that there were fewer OMs in this area (14 participation OMs versus 21 and 28 in body structure/function and activity, respectively) and only 1 participation domain, OM received a rating of 4, suggesting that this is a potential area for additional OM development.
The recommendations developed address what has been advocated previously, that consistent use of agreed-upon, standardized OMs will facilitate clinical decision making,8 guide educators in curricular decisions,10 and enhance the methodological quality and generalizability of clinical trials.4,12–14 The explicit review of criteria in the StrokEDGE template and the definitions of recommendation categories will allow individual physical therapists or facilities to examine existing or newly developed OMs to determine appropriateness. The ability to decide, as a department or service, which OMs to use has been cited as a key factor in successful clinical implementation of OMs.19 Optimally, these OM recommendations may be incorporated into proposed strategies to enhance more widespread OM use.18,29,30,50 The description of the process used along with the detailed recommendation criteria utilized may provide a blueprint for groups interested in developing OM recommendations for other patient diagnostic groups.
We acknowledge several potential limitations of the recommendations developed, which include the challenge of maintaining up-to-date recommendations as the field of OM research evolves and the individual biases of task force members. Although all task force members have clinical practice experience, most are not currently in full-time clinical practice. However, the development of explicit definitions of review categories and use of the Delphi consensus process were intended to mitigate individual biases. Although the task force did not use specific criteria to guide their decision making for the educational content recommendations, the published E-L NC curriculum guidelines were used as a frame of reference when reviewing the available evidence and posing education recommendations. Additionally, the task force has recommended the development of an on-going process to examine newly developed OMs and current information on existing OMs to ensure up-to-date recommendations.
We suggest that the use of the recommended OMs in physical therapist clinical practice, education, and research can provide a common set of tools and a consistent language to capture and describe body function/structure, activity, and participation limitations poststroke. The use of a clearly defined and comprehensive assessment template as used here may facilitate the pooling of data on OMs and contribute the necessary evidence for the determination of best practice guidelines. The explicit description of the process used for developing an evaluation template and discussion of the actual processes involved in evaluating OMs and reaching consensus on recommendations may prove useful for other groups interested in developing recommendations. Although we acknowledge a formal systematic review was not utilized, the approach used ensured that the reviews were detailed and scholarly and that there was expert consensus regarding the recommendations. Therefore, the use of these recommended OMs can assist physical therapists in developing patient-centered care plans that are based upon well-informed, sound decisions.
Appendix.
StrokEDGE Outcome Measure Review Forma
Footnotes
Dr Sullivan, Dr Kluding, Dr Rose, Dr Yoshida, and Dr Pinto Zipp provided concept/idea/project design. Dr Sullivan, Dr Crowner, Dr Kluding, Dr Rose, and Dr Pinto Zipp provided writing. Dr Sullivan, Dr Kluding, Ms Nichols, Dr Rose, and Dr Pinto Zipp provided data collection. Dr Sullivan, Dr Kluding, Dr Rose, and Dr Pinto Zipp provided data analysis. Dr Sullivan and Dr Pinto Zipp provided project management. Dr Sullivan provided the patient and clerical support. Ms Nichols provided consultation (including review of manuscript before submission).
This manuscript derives from work developed for the Neurology Section regional continuing education course “Neurologic Practice Essentials: A Measurement Toolbox.”
- Received December 7, 2012.
- Accepted May 20, 2013.
- © 2013 American Physical Therapy Association
References
Issue highlights
