Various studies have described the benefits of using outcome measures (OMs) in physical therapist practice1–3; however, multiple barriers interfere with their use (eTab. 1).1,3,4 Most notably, a limited understanding of how to select and apply the best OM has been reported to be a barrier.3,4 Wedge et al3 reported that physical therapists do not always critically evaluate the psychometric properties of OMs, are unfamiliar with OM-related resources, and have difficulties determining patient suitability for particular OMs. Physical therapists working with people with multiple sclerosis (MS) have additional challenges when selecting OMs. Multiple sclerosis is a complex, heterogeneous, and progressive disorder causing a wide variety of symptoms among patients.5,6 Symptom variability in individual patients throughout the course of the disease also must be considered. Additionally, people with MS are treated in a variety of settings, which may affect OM selection and use (eg, due to limited space and equipment in a home environment). Thus, selecting 1 or 2 OMs for use with all people with MS is challenging, requiring the clinician to determine the most appropriate OMs to measure relevant constructs. Although some resources exist to assist the physical therapist with identifying and selecting OMs for people with MS,7–13 literature is lacking that describes a consensus-based method to determine which OMs are appropriate for people with MS at various disability levels and in different practice settings.
In 2009, the American Physical Therapy Association (APTA) Neurology Section (“the Section”) began a process to develop recommendations for the use of OMs for patients with neurological conditions, beginning with stroke (information pertaining to the Section's OM recommendations is available elsewhere14). The following year, the Section expanded its efforts by creating a task force to recommend OMs for people with MS. The Multiple Sclerosis Task Force (MSTF) was charged to: (1) develop evidence-based recommendations for OMs for clinical practice, education, and research; (2) develop instruction sheets outlining administration and scoring procedures for each OM; and (3) identify needs for future research on specific OMs for people with MS.
The purposes of this article are to: (1) describe the process used by the MSTF to evaluate the psychometric data and clinical utility of OMs for use in people with MS, (2) describe the consensus process used to recommend OMs for this patient population, and (3) provide evidence-based recommendations to assist clinicians in selecting appropriate OMs for people with MS.
Method
In the fall of 2010, the Section recruited task force members. Criteria for recruitment consisted of physical therapists with geographical diversity and various clinical, research, and educational experiences pertaining to MS (Tab. 1). Task force members also needed to have a sound knowledge of psychometrics and the ability to read and interpret the literature. One member (K.P.) was designated chair of the committee.
Background Information on the Multiple Sclerosis Task Force Membersa
Stage 1: Determination of the Process and Selection of OMs
The process began with an initial review of the literature conducted by the MSTF Chair to identify OMs that have been used in people with MS. The search identified both generic and disease-specific OMs, using the following terms in various combinations: “multiple sclerosis,” “neurological,” “outcome measures,” and “functional outcomes.” The goal was to identify a broad list of OMs with potential relevance to the MS patient population. Several resources were identified and used to generate an initial list of measures for consideration.7–13 Other resources included the Neurology Section's entry-level curricular guidelines document15 that outlines OMs recommended for inclusion in physical therapist education programs, OMs covered in the Section's regional continuing education course “Neurologic Practice Essentials: A Measurement Toolbox,” and the StrokEDGE recommendations.16 The Chair assembled a comprehensive list of all the OMs from the above resources; task force members were asked to review the list and identify additional OMs for consideration.
Task force members initially met for a day-long meeting at the 2011 Combined Sections Meeting of APTA. The MSTF decided unanimously to follow the processes used by the StrokEDGE task force. These processes included: (1) the use of a modified Delphi process to build consensus; (2) use of the International Classification of Functioning, Disability and Health (ICF)17 to classify the selected OMs; and (3) a comprehensive and critical analysis of the literature on each OM. The results of the critical review for each OM and recommendations were entered into an Evaluation Database to Guide Effectiveness (EDGE) form.
A Delphi process is a mechanism for an expert panel to come to consensus.18,19 A modified Delphi process has been proposed as a way to evaluate the literature on OMs for patients with total hip arthroplasty18 and was used by the StrokEDGE task force.20 The MSTF used a modified Delphi process consisting of 3 stages (Figure). The threshold for consensus was set at 70% (at least 5 of the 7 task force members), consistent with the threshold of 70% used by other authors.18,21
Illustration of modified Delphi process and stages. OM=outcome measure; ICF=International Classification of Functioning, Disability and Health; EDGE=Evaluation Database to Guide Effectiveness.
In addition, the MSTF adopted the ICF as a way of categorizing OMs.17 The ICF considers the interactions of the patient's health condition with body function and structure (ie, anatomical and physiological status, respectively), activity (ie, task execution), and participation (ie, involvement in life situations), in addition to environmental and personal factors. Thus, the ICF encourages a holistic view of a patient's health, and it has been suggested that physical therapists should consider using the ICF to ensure a comprehensive examination.10 The MSTF unanimously agreed to include OMs across the ICF levels of body function and structure, activity, and participation. Outcome measures focusing on the categories of environmental and personal factors were not specifically sought.
The MSTF conducted a comprehensive and critical review of the scientific literature pertaining to each OM. This information was summarized on an EDGE form. The EDGE form was developed by the APTA Section on Research EDGE task force to assist in the identification of core sets of tests and measures for clinical practice.22 The EDGE form used by the MSTF (Appendix) was modified from the original EDGE form and the one used by the StrokEDGE task force to include issues relevant to people with MS. It was comprised of 4 sections: OM properties (eg, ICF level and constructs measured, type of measure, instrument description), psychometrics (eg, reliability and validity, including responsiveness), clinical utility, and recommendations.
For each OM, 11 recommendations were made, 9 relating to the use of the OM in clinical practice and 2 related to the use of OMs in physical therapist education and research. Of the 9 recommendations relating to clinical practice, 4 pertained to the use of the OM in patients across the spectrum of disability levels (Appendix). The Expanded Disability Status Scale23 (EDSS) is commonly used in clinical trials to classify the level of disability in patients with MS, from normal neurological function (0) to death from MS (10). For the purposes of this work, participants were categorized into 4 EDSS disability groupings (0.0–3.5, 4.0–5.5, 6.0–7.5, and 8.0–9.5) agreed upon by the MSTF.24 A 4-point rating scale was used to rate the appropriateness of the OM for patients in each EDSS disability group. Scores ranged from 1 (“do not recommend”) to 4 (“highly recommend”) based on the combined strength of the psychometric data in individuals with MS and the OM's clinical utility. The strength of the psychometric data was evaluated based on guidelines described by Portney and Watkins.25
The determination of the OM's clinical utility was based on the time needed to administer and score the measure, the need for specialized equipment or training, and the ease of obtaining a copy of the OM (eg, cost and copyright issues). To achieve a rating of 3 or 4, the measure needed to have good to excellent psychometrics and good to excellent clinical utility (Tab. 2) in people with MS. Responsiveness data were needed for an OM to receive a rating of 4. A rating of 2 (“unable to recommend at this time”) was issued if there was a lack of psychometric data specific to individuals with MS. Lastly, a rating of 1 (“do not recommend”) indicated the OM had poor psychometrics or clinical utility, or both, in people with MS. The MSTF also recommended the use of each OM in various practice settings (acute care, inpatient rehabilitation, outpatient care, skilled nursing facility, and home health), based on the clinical utility of the OM for each setting, using the 4-point rating scale.
Outcome Measure Rating Scalea
Before reviewing potential OMs, the MSTF generated a list of health-related constructs (ie, concerns, problems, or issues) important to individuals with MS. More than 50 constructs pertaining to all ICF levels and multiple areas of MS-related health were identified. The MSTF then narrowed the list to focus on the constructs (eg, strength, balance, role function) that were most closely related to physical therapist practice and typically examined with OMs. The MSTF also sought to include a variety of types of OMs (ie, generic and disease-specific, self-report and performance-based). Constructs such as cognition and emotional state were removed from the original list, as professionals other than physical therapists typically measure these constructs more specifically. Cranial nerve function was excluded from the list because it is not examined with an OM. A total of 28 constructs were unanimously selected by the MSTF (eTab. 2).
The initial list of OMs to be considered consisted of approximately 120 measures. Many factors limited the feasibility of reviewing all 120 OMs, such as time, funding, and the size of the task force. In an attempt to reduce the number of OMs from 120 while retaining a diverse group of OMs addressing all 28 constructs, each OM was categorized according to ICF levels,17 constructs assessed by the OM, and the type of measure (ie, generic or disease-specific, performance-based or self-report). This preliminary determination was made based on the MSTF's collective expertise and familiarity with each measure. A vote was taken as to whether to include each measure in the review, and consensus was reached on all measures. Sixty-three OMs were selected for review by the MSTF.
Stage 2: Initial Peer Review Process
The next stage of the process (Figure) involved the review of each of the 63 OMs by 2 of the MSTF members between March and July 2011. Each OM was assigned a primary reviewer, who conducted a literature review on the OM, designated the constructs measured by the OM, gathered evidence of the measure's psychometric properties and clinical utility, and assessed its clinical relevance. The reviewer considered the purpose and type of the OM, as reported in the literature, and analyzed the OM itself (eg, administration procedures and the OM's items). A summary was entered onto the modified EDGE form.26 Once the initial review was complete, a secondary (unblinded) reviewer evaluated the EDGE form to ensure it was accurate, thorough, and clearly written. The secondary reviewer also indicated his or her agreement or issues with all recommendations and determined whether the recommendations were supported by both the psychometric evidence and the measure's clinical utility. The goal was for the primary and secondary reviewers to develop consensus on each recommendation. If the 2 reviewers did not agree, they would discuss the evidence and recommendation and revise the summary until consensus was achieved. For all recommendations, the primary and secondary reviewers reached consensus.
Stage 3: Group Review Process
The final stage of the process (Figure) involved a group review of the EDGE forms and recommendations and continued deliberations leading to final consensus. This stage occurred from July to October 2011. All EDGE forms were compiled and disseminated to all MSTF members. Each member reviewed each EDGE form, analyzing the strength of the psychometric data and clinical utility for the OM's use in each patient group and practice setting. A confidential online survey was used for each member to indicate agreement (“yes” or “no”) with each recommendation. When in disagreement with a given recommendation, the member was asked to provide a rationale. Two MSTF members (K.P. and E.T.C.) analyzed the results of the survey to identify recommendations lacking consensus, determine whether the recommendation was appropriate given the psychometric evidence and clinical utility, and ensure that the rating scale was applied consistently. Based on this analysis, the 2 MSTF members revised recommendations if appropriate and composed a written rationale. This information was shared with all MSTF members who appraised the compiled survey results, analysis, and revised recommendations. The MSTF continued discussion via e-mail until consensus was reached. At this time, a final vote was taken on the recommendations, and all MSTF members voted unanimously to accept all recommendations.
Results
The MSTF reviewed 63 OMs across all levels of the ICF. Of these, 36 measured aspects of body function/structure, 40 pertained to activity, and 17 pertained to participation.23,27–89 Twenty-two OMs included items pertaining to more than one ICF level. Seventeen OMs were MS specific, whereas 46 were generic or pertained to other diagnostic groups. Twenty-three of the OMs reviewed used a self-report method of administration, 38 used a performance-based method, and 2 used a combination of both methods.
The MSTF ratings for each measure across practice area and level of disease severity are listed in Table 3. The detailed reviews (ie, completed EDGE forms) of all OMs are available elsewhere.90 Of the measures reviewed, nearly half received ratings of 3 or 4 (“recommended” or “highly recommended,” respectively) for use in inpatient rehabilitation (n=28) and outpatient (n=29) settings (Tab. 4). The greatest number of OMs receiving ratings of 3 or 4 was found for outpatient settings (n=29), with the lowest number for acute care (n=22). Nearly half of the OMs were recommended or highly recommended for use in patients across 3 of the 4 EDSS groupings (Tab. 5). The exception was for the EDSS grouping 8.0 to 9.5, where 18 OMs were recommended or highly recommended. Across all settings and all EDSS groupings, a greater number of OMs were rated as 3 (“recommended”) compared with OMs rated as 4 (“highly recommended”). For each measure, consensus was achieved, with 100% of the MSTF members agreeing with each of the 11 recommendations.
Reviewed Outcome Measures by International Classification of Functioning, Disability and Health (ICF) Category, Task Force Recommendations for Outcome Measure Use by Practice Setting and Disease Severity, Outcome Measures Recommended for Professional (Entry-Level) Physical Therapist Education, and Outcome Measures Recommended for Research Usea
Numbers of Outcome Measures Receiving Ratings of 3 (“Recommend”) and 4 (“Highly Recommend”) Across Practice Settingsa
Numbers of Outcome Measures Receiving Ratings of 3 (“Recommend”) and 4 (“Highly Recommend”) Across Expanded Disability Status Scale Groupings
Discussion
The MSTF reviewed 63 OMs across the MS disability spectrum and for use in a variety of health care settings. Appropriate measures exist for people with MS across the disability spectrum, as nearly half of the OMs reviewed received ratings of 3 or 4 (indicating good to excellent psychometrics and clinical utility) for all but one EDSS group. Notably, fewer OMs were recommended for use in patients with MS with severe disability (ie, EDSS score ≥8.0). Although this finding may reflect the types of OMs selected for review by the MSTF, additional OM development for use in people with more advanced disease may be indicated. Also, fewer OMs received ratings of 4 across all EDSS groups compared with ratings of 3, largely due to the limited responsiveness data currently available in the literature for many measures. A large number of OMs received ratings of 2, indicating insufficient evidence in this population to support a higher recommendation at the time of review.
Similarly, there are numerous OMs that are appropriate for people with MS receiving care in a variety of health care settings, particularly inpatient rehabilitation and outpatient practices. Fewer OMs were recommended for use in acute care than in any other practice setting. When considering use of OMs in specific practice settings, the MSTF considered the clinical utility of the OM. The amount of time needed to administer, score, and interpret many measures makes them somewhat less feasible for the acute care environment.
There are many benefits to using the MSTF recommendations in clinical practice. The recommendations have broad applicability for clinicians working with people with MS across the disability spectrum, in any health care setting. The recommendations allow efficient identification of OMs pertinent to constructs important to people with MS, and the summaries of psychometric data on the EDGE documents facilitate sound OM selection. For example, reported responsiveness and normative data provide a basis for measuring change and making comparisons for people with MS. Additionally, the usefulness for particular settings and ability levels for people with MS has been determined, enabling clinicians to efficiently identify those OMs that might have particular relevance.
The ability to find an OM that is recommended for use across multiple disability levels is especially important when working with patients with MS because it facilitates the measurement of outcomes longitudinally as the disease progresses. Because Table 3 identifies the ratings for each OM across each setting and EDSS level, therapists can identify with relative ease which OMs are best for measuring patient status over time or across settings. For example, the 12-Item Multiple Sclerosis Walking Scale, Berg Balance Scale, Multiple Sclerosis Impact Scale, and visual analog scale for fatigue all received ratings of 3 or 4 for the great majority of settings and EDSS levels, indicating their appropriateness for tracking patient status over a long-term period or as a patient transitions across settings. Additionally, one-page summary sheets are available on the APTA Neurology Section website identifying measures that were recommended or highly recommended for use in acute care and inpatient/outpatient rehabilitation to facilitate tracking of patient outcomes across settings. References for evidence on the OMs are reported along with instructions for administration and information on other available resources (eg, web links). The recommendations also can facilitate assessment of clinician and programmatic effectiveness. This information could be used for marketing and public relations efforts. Lastly, the US Department of Health and Human Services Centers for Medicare and Medicaid Services recently implemented a claims-based collection strategy requiring the reporting of functional outcome data (G-codes) at various times during an episode of care for patients with Medicare Part B.91,92 The MS EDGE recommendations can assist therapists with identifying appropriate measures to meet these requirements. To do so, the MSTF suggests that clinicians refer to the EDGE documents to identify what constructs are assessed by each OM, attending to the constructs balance and falls, bed mobility, transfers, gait, reach and grasp, and self-care, as these constructs relate closely to the various G-codes. Doing so should facilitate appropriate selection of OMs to meet specific G-code requirements.
There are several limitations to the recommendations. First, they were developed through a consensus process, not a systematic or meta-analysis review of the literature. However, the MSTF did perform an in-depth critical appraisal of the psychometric data and clinical utility of each OM. Various constraints (eg, time) limited the capability to review all OMs relevant to people with MS. This limitation may have contributed to the relatively limited numbers of recommended OMs for use in acute care and for patients with greater disability. The MSTF aimed to review a broad group of OMs, representing all important sensorimotor and activity constructs. As such, some OMs relevant to physical therapist practice in people with MS (eg, the Fatigue Severity Scale) were not included in the recommendations. Limited availability of evidence was problematic, and many OMs are not validated across the MS disability spectrum, leading to a rating of 2. Others had good reliability and validity data but lacked responsiveness values, leading to a rating of 3. Yet, many of these OMs had acceptable clinical utility. Completion of the EDGE forms relied on the literature available at the time of the reviews, generally the summer of 2011. Additionally, this work was done prior to the G-code requirement discussed previously. Hence, the task force did not attempt to match each OM to related G-code requirements. Lastly, it is unknown whether use of the recommendations will assist in standardizing the selection of OMs and if they will assist clinicians with making better clinical decisions when developing a plan of care.
The recommendations of the MSTF provide significant information to assist with clinical decision making. However, more work is needed. Due to time constraints, some relevant OMs were not reviewed, and others lacked psychometric data. Research contributing to the body of literature is needed. The MSTF determined that 30 OMs reviewed lack sufficient psychometric data to support their use in people with MS. Studies examining the psychometrics of these OMs are needed. Similarly, it would likely be beneficial to develop OMs for use in the acute care setting and with patients who have greater MS-related disability.
The MSTF also recommends that the EDGE forms and recommendations be revised on a regular basis to include more current and complete information. Currently, the APTA Neurology Section is collaborating with the Rehabilitation Institute of Chicago's Center for Rehabilitation Outcomes Research (RIC-CROR) to integrate the Section's OM recommendations with RIC-CROR's Rehabilitation Measures Database.93 Plans are under way by the RIC-CROR to update the psychometric data in the Rehabilitation Measures Database. However, the Section does not currently have a plan to update the EDGE recommendations. The MSTF recommends that a plan be developed to update and revise the recommendations to ensure currency of information for clinicians, including matching each OM to pertinent G-codes to assist clinicians in selecting appropriate OMs for a given G-code. Lastly, the MSTF did not attempt to identify core sets (ie, specific OMs to be used for all patients receiving services in a given practice setting or at a given MS-related disability level). The MSTF contends that the development of core sets should be a next step, to improve consistency and quality of care for people with MS. The EDGE documents and recommendations can be used to inform the development of core sets.
In summary, the recommendations provided by the MSTF provide information that will be helpful when selecting and interpreting OMs for patients with MS. It should be noted that the MSTF does not endorse any particular OM as serving all needs of patients or clinicians. The MSTF does propose that clinicians use a thoughtful process when seeking the appropriate OMs to enhance patient care.
Appendix.
Evaluation Database to Guide Effectiveness (EDGE) Form Used by the Multiple Sclerosis Task Forcea
a Available at: http://www.ptresearch.org/article/84/resources/researchers/edge-task-force-evaluation-database-to-guide-effectiveness/edge-rating-forms. ICF=International Classification of Functioning, Disability and Health; MDC=minimal detectable change; MCID=minimal clinically important difference; EDSS=Expanded Disability Status Scale.
Footnotes
Dr Potter, Dr Cohen, Dr Allen, Ms Brandfass, Dr Widener, and Dr Yorke provided concept/idea/project design. Dr Potter, Dr Cohen, Dr Allen, and Dr Widener provided writing. All authors provided data collection and analysis. Dr Potter and Dr Cohen provided project management. Dr Cohen provided clerical support. Dr Allen, Dr Bennett, and Ms Brandfass provided consultation (including review of manuscript before submission).
- Received April 22, 2013.
- Accepted December 12, 2013.
- © 2014 American Physical Therapy Association
References
Issue highlights
