Abstract
Background Standardized assessment of patients' activity limitations in acute care settings can provide valuable information. Existing measures have not been widely implemented.
Objectives The aim of this study was to provide evidence for validity of scores on Activity Measure for Post-Acute Care (AM-PAC) “6-Clicks” measures of basic mobility and daily activity in acute care.
Design A retrospective measurement study was conducted.
Methods The study used a database from one health system containing “6-Clicks” scores from first and last physical therapist and occupational therapist visits for 84,446 patients. Validity was analyzed by examining differences in “6-Clicks” scores across categories of patient characteristics; the ability of “6-Clicks” scores to predict patients' having more than one therapy visit; correlation of “6-Clicks” scores with Functional Independence Measure (FIM) scores; and internal responsiveness over the episode of care. Internal consistency reliability also was determined.
Results The “6-Clicks” scores differed across patients' age, preadmission living situation, and number of therapy visits. The areas under receiver operating characteristic curves derived using “6-Clicks” scores at the first visit to predict patients receiving more than one visit were 0.703 and 0.652 using basic mobility and daily activity scores, respectively. The “6-Clicks” scores at the final visit were correlated with scores on subscales of the FIM completed on admission to inpatient rehabilitation facilities (r=.65 and .69). Standardized response means were 1.06 and 0.95 and minimal detectable changes with 90% confidence level (MDC90) were 4.72 and 5.49 for basic mobility and daily activity scores, respectively. Internal consistency reliability of basic mobility and daily activity scores was .96 and .91, respectively.
Limitations Using clinical databases for research purposes has limitations, including missing data, misclassifications, and selection bias. Rater reliability is not known.
Conclusions This study provides evidence for the validity of “6-Clicks” scores for assessing patients' activity limitations in acute care settings.
In the acute care setting, the case mix index of hospitals provides a means to measure the medical complexity and burden of illness for the patients they serve and the need for resources. No similar measure exists to measure the burden presented by activity limitations in the acute care setting. Impairments in mobility and activities of daily living (ADL) are important drivers of health care resource utilization and costs.1 Therefore, the implementation of a tool that provides a “footprint” for activity limitations in hospitalized patients could be an important element for improving the value of hospital care. A relatively high proportion of patients with neurological, musculoskeletal, and cardiopulmonary problems in acute care hospitals have limitations in walking, self-care, and changing and maintaining body positions,2 and these types of limitations are the focus of care provided by physical therapists and occupational therapists in this setting.3,4 In an era of constrained resources, there is a pressing need to determine which patients in acute care require skilled rehabilitation services so that personnel can be deployed in an efficient and effective manner.
Studies have shown that the percentage of patients scheduled for physical therapy, yet not receiving these services, may be quite high5 and that the time spent by therapists when patients cannot be treated may be substantial.4 Although some visits that result in nontreatment are due to patients being medically unstable, some may be the result of patients being referred who are sufficiently mobile and have sufficient capacity for functional activity as to not require services.6 This problem has led to exploration of potential screening instruments to identify appropriate referrals and triage patients in order to improve efficiency of rehabilitation services.6,7 Without standardized measurement, clinical recommendations and resource allocation may be subject to excessive variability, error, and bias.
Although several instruments have been developed for measuring patients' activity limitations in the acute care setting,8–16 it appears they have not been widely implemented.17 Recently, Cleveland Clinic Health System hospitals have piloted the use of new standardized instruments that allow physical therapists and occupational therapists to measure basic mobility and daily activity functions of patients in the acute care setting. These tools, called “6-Clicks,” are short forms created from the Activity Measure for Post-Acute Care (AM-PAC) instrument, developed by researchers at Boston University.18 The AM-PAC measures 3 functional domains: basic mobility, daily activities, and applied cognition. It may be used for assessment in adults with a wide range of diagnoses and levels of performance in the 3 domains. The “6-Clicks” instruments may have advantages over the instruments previously developed for the acute care setting in that they are simple and quick to complete, provide a transparent measure of patients' capabilities in functional areas important for prioritization of therapy resources, and use item response theory to derive a common metric that can be linked with other short forms derived from the AM-PAC instrument.19 Thus far, implementation of the “6-Clicks” instruments in acute care at the Cleveland Clinic Health System has allowed clinicians and managers to determine the feasibility of its implementation in routine patient care; however, its psychometric properties have not been tested. The overall purpose of this study, therefore, was to determine clinical validity of the “6-Clicks” basic mobility and daily activity short forms.
Method
Data Source
A clinical database derived and collated from the MediLinks electronic medical record system (EMR) used by Cleveland Clinic Health System physical therapy and occupational therapy departments was the source of data for analyses. It contained more than 200,000 visit entries, with data on 92,899 individual patients seen in acute care by physical therapists or occupational therapists, or both, from April 2011 to December 2012. We identified 84,466 patients for whom we had data for first or last physical therapy or occupational therapy visits (Fig. 1). The data included in the clinical dataset were “6-Clicks” basic mobility and daily activity scores; number of visits by physical therapists and occupational therapists; patient demographic information, including age, primary medical or surgical condition, and preadmission living situation; length of hospital stay; and discharge disposition if patients were discharged home with services or to another facility. If patients were admitted to an inpatient rehabilitation facility (IRF) within the Cleveland Clinic Health System, the admission score on the Functional Independence Measure (FIM) was included.
Data selection flowchart. PT=physical therapist, OT=occupational therapist.
Setting and Procedure
The Cleveland Clinic Health System is a nonprofit system that includes 3,700 beds in the main campus hospital and 8 regional hospitals. The Rehabilitation and Sports Therapy enterprise includes more than 700 therapy professionals, with 90 physical therapists and 45 occupational therapists assigned primarily to acute hospital care. These therapists manage more than 1,900 patient evaluations per week. The “6-Clicks” items and implementation process were introduced to staff at a 1-hour in-service meeting that provided them with the background, rationale, and instructions for completion. The implementation was part of a broad institutional push for providing uniform, high-quality services. Physical therapists were asked to complete the basic mobility short form, and occupational therapists were asked to complete the daily activity short form. The therapists were instructed to determine scores for each item either by observing patients' performance or using their clinical judgment. The therapists were familiar with the EMR and entered their patients' “6-Clicks” scores into the EMR as part of their visit documentation.
Patients were discharged to many settings within the community, some to IRFs within the Cleveland Clinic Health System. In the Cleveland Clinic Health System IRFs, the FIM was administered by trained personnel using the standard protocol.20 The dataset used for this study included FIM data from those settings. Our dataset did not indicate the date of this administration; however, the regulations governing IRFs stipulate the assessment must be done within the first 3 calendar days from admission.20
The institutional review boards of the Cleveland Clinic Health System and the University of Vermont classified this project as “nonhuman subjects” research. The dataset included only de-identified data.
Instrumentation
The “6-Clicks” short forms include items selected from the calibrated AM-PAC item banks. The AM-PAC is a validated measure based on the activity limitation domain of the World Health Organization's International Classification of Functioning, Disability and Health (ICF)21 and is designed to be used for patients receiving postacute care rehabilitation regardless of type of condition or setting.18 One “6-Clicks” form assesses basic mobility function, and another assesses daily activities function. The items for each form were selected for their relevance to patients' capacity for functional activities in an acute care setting and represent types of activities that physical therapists and occupational therapists commonly assess. They also represent the functional activities of most interest to postacute rehabilitation providers screening patients for admission to their facilities. Each item is scored on 1 of 4 levels based on the amount of difficulty a patient has or how much help is needed from another person in completing the task. Because the items in the “6-Clicks” forms were drawn from the calibrated AM-PAC item bank, standardized scores on the t-score scale (where the mean is 50 and the standard deviation is 10) can be generated and used to assess the level of patients' activity limitations.19 Lower scores equate to a greater degree of limitation. The “6-Clicks” short forms and scoring are shown in Appendixes 1 and 2. Although intended for use in postacute care settings, one study of patients with complex medical and postsurgical conditions within 3 days of discharge from acute care showed that the AM-PAC might be useful for assessment of the range of functional activities in patients in acute care.22
The FIM is a standardized measure with 18 items that assess patients' level of independence and dependence in ADL, mobility, bowel and bladder function, and cognitive function. The items are scored on a scale from 1 to 7, with 7 indicating complete independence, based on observations of patients' actual performance.20
Data Analysis
All analyses were conducted using IBM-SPSS Statistics version 20 (IBM Corp, Armonk, New York). Descriptive statistics were derived to describe the patients for whom data were collected at the first or last physical therapist or occupational therapist visit. All remaining analyses were conducted separately for basic mobility and daily activities scores.
To determine possible ceiling and floor effects, we examined the distribution of scores derived from the first and final visits. Internal consistency of initial visit scores for each form was determined with the Cronbach alpha.
To assess the validity of the “6-Clicks” scores, we developed receiver operating characteristic (ROC) curves to determine whether initial scores could be used to determine patients needing only one visit versus more than one visit. The ROC analyses allowed us to determine the sensitivity and specificity for receiving more than one visit for each possible score cutpoint. Using these values for sensitivity and specificity, we calculated the positive and negative predictive values assuming 50% of the sample received more than one visit. Calculations were made based on a standardized 50% prevalence to reduce the effect on predictive values of the study-specific prevalence of receiving more than one visit. We also examined the proportion of patients with the highest “6-Clicks” scores receiving more than one visit. Although we realized that physicians and nurses might request consultations for patients with relatively high levels of functional activity and, therefore, that these patients would be visited by physical therapists and occupational therapists, we hypothesized that in an efficient system, those patients would not receive more than one visit.
To determine the construct validity of “6-Clicks” scores, we used analyses of variance to examine differences in mean first visit scores across 4 age groups (18–40, 41–64, 65–85, and 86+ years) and 5 types of preadmission living situations (home alone, home with others, assisted or independent senior living, IRF or skilled nursing facility, and extended care). We also performed trend analyses. We used t tests to examine the first visit score difference for those with one therapy visit compared with those with more than one visit. We hypothesized that younger patients would have higher scores, that patients living at home would have higher scores than those living in more restrictive settings, and that patients with lower scores would have more than one therapy visit.
Construct validity also was examined using Pearson correlation coefficients to determine the relationship of “6-Clicks” basic mobility and daily activity scores at the last visit in the acute care setting to the mobility (bed/chair transfer, toilet transfer, shower/tub transfer, locomotion, stairs) and ADL (eating, grooming, bathing, dressing upper body, dressing lower body, toileting) component FIM scores, respectively, at the time of admission to an IRF setting.
Internal responsiveness23 of the scores on each form was determined by first estimating the minimal detectable change with 90% confidence level (MDC90) in scores using the standard deviation of all initial scores and Cronbach alpha as the reliability coefficient. We then determined the proportion of patients who had improved, worsened, or stayed the same based on the MDC90. We also calculated the standardized response mean (SRM) and its 95% confidence interval using scores for patients for whom we had both first and last visit data. Because patients' activity limitations in the acute care setting may worsen as well as improve due to medical status, we calculated the SRM using the absolute value of change scores. We interpreted the magnitudes of effect statistics using an ordinal scale proposed by Hopkins.24
Results
Table 1 includes data describing the patients in the sample. The mean age was 69.0 years (SD=15.6), and the majority were women. Approximately 88% of the patients had been admitted from their homes, and they had a wide variety of primary medical and surgical conditions. The number of physical therapist and occupational therapist visits for patients was highly skewed, with medians of 1 (interquartile range=1–2).
Patient Characteristicsa
The distributions of scores for the “6-Clicks” forms completed at the first visit demonstrated 2.7% and 2.3% of responses at the lowest score for basic mobility and daily activity forms, respectively. The highest score was given for 15.5% of patients completing the basic mobility form, and of all possible scores, this was the most frequent. For the daily activities form, the highest score was given for 10.1% of patients. The internal consistency reliability of the basic mobility and daily activity scores was .957 (95% confidence interval [95% CI]=.956, .958) and .911 (95% CI=.909, .912), respectively.
The areas under the ROC curves derived using “6-Clicks” scores at the first visit to predict whether patients required more than one visit compared with one visit only were 0.703 (95% CI=0.699, 0.707) using basic mobility scores and 0.652 (95% CI=0.648, 0.657) using daily activity scores (Fig. 2). Interestingly, 3.0% of the patients with first visit scores indicating no difficulty in performing, or assistance needed, in basic mobility functions, and 4.9% of the patients with no difficulty in daily activities had more than 1 visit from a physical therapist or occupational therapist.
Receiver operating characteristic (ROC) curves for “6-Clicks” scores in determining more than one physical therapist or occupational therapist visit.
Further evidence for validity of the “6-Clicks” forms is delineated in Table 2. Patients with more than one visit had lower first visit scores on both forms compared with those with only one visit (P<.001). There were also significant differences (P<.001) in first visit scores for both forms based on patients' age, with a significant trend of older patients demonstrating lower scores (P<.001). Scores at the first visit also significantly differed by living situation prior to admission (P<.001); the trend was significant (P<.001). Those living at home alone or with others had similar scores; however, scores differed for all other living situations, with those in extended care settings having the lowest scores (P<.001). Validity was further supported by the correlation of “6-Clicks” scores with FIM scores. We had FIM scores for 272 patients who went to IRFs within the Cleveland Clinic hospital system. This sample comprised 7.8% of patients discharged to IRFs. Of these patients, we had final visit basic mobility scores for 170 patients and final visit daily activity scores for 192 patients, 1.2% and 3.1% of all patients for whom we had final visit basic mobility and daily activity scores, respectively. The correlation of the “6-Clicks” basic mobility score at the last visit with the FIM motor subscore was large24 (r=.69 [95% CI=.62, .75]), as was the correlation of the “6-Clicks” daily activities score with the FIM ADL subscore (r=.65 [95% CI=.57, .72]) (Fig. 3).
Evidence for Construct Validitya
Correlation of final acute care visit “6-Clicks” scores with Functional Independence Measure (FIM) scores on admission to the inpatient rehabilitation facility. ADL=activities of daily living.
Mean absolute differences between the scores for first visit and last visit were 5.71 (95% CI=5.61, 5.80) for basic mobility and 4.61 (95% CI=4.49, 4.73) for daily activity. The SRM was 1.06 (95% CI=1.04, 1.08) for basic mobility scores and 0.95 (95% CI=0.95, 0.97) for daily activity scores. Thirty-five percent of the patients improved in basic mobility by at least the MDC90 (4.72), and 10% worsened by at least the MDC90. Approximately 27% of the patients improved in daily activities by at least 1 MDC90 (5.49), and 4% worsened.
Discussion
Our findings support the validity of the “6-Clicks” basic mobility and daily activities scores in assessing the activity limitations of patients with a wide variety of medical and surgical conditions in an acute care setting. The “6-Clicks” forms were implemented by physical therapists and occupational therapists with only brief instruction, and, using EMR, they accomplished data entry in less than a minute as part of routine documentation of the patient encounter. Using EMR, in which scores can be automatically and quickly calculated for the practitioner to view immediately, enhances the feasibility of standardized assessment of activity limitations and is more user-friendly than paper forms that require practitioners to carry out score calculations. Because therapists' judgments can be used to assess patients' functional capabilities using “6-Clicks” forms, patients are not required to perform a specific task. This approach limits missing values and reduces the ambiguity of total scores when item scores for “not tested” are included, as in some other instruments.12,14
Using calibrated items drawn from the AM-PAC item pool, “6-Clicks” forms yield standardized scores in a metric common to all versions of the AM-PAC. The common scoring allows for direct comparison with AM-PAC assessments done over time and across different care settings. Inclusion of only 6 items from the overall AM-PAC, each with a simple, 4-level scoring system, also facilitates application of “6-Clicks.” Existing tools may have many items10 or levels of scoring for each item,9,11,14,15,25 making it challenging for clinicians to make such fine distinctions in activity limitations in the acute care setting. Additionally, application may be slow and interpretation difficult in tools that include different scoring structures for different items.9,16
Internal consistency reliability for both basic mobility and daily activities forms was very high. Validity of both forms was supported by evidence for ceiling and floor effects below 20% at first and last visits. Validity also was supported by differences in scores across groups of patients who would be expected to differ in levels of ability based on age, living situation, and number of therapy visits provided to them. Large correlations24 of scores on both forms at the last acute care visit with admission FIM scores for patients discharged to IRFs within the Cleveland Clinic hospital system further support validity. Given the fact that the mobility subscore of the FIM does not include lower-level activities such as rolling in bed or coming to a sitting position, as does the “6-Clicks” basic mobility form, a greater degree of correlation might not be expected. On the other hand, the items comprising the ADL subscore of the FIM represent identical activities to the “6-Clicks” daily activity form; therefore, a greater degree of correlation might have been expected.
Internal responsiveness of the “6-Clicks” scores also was supported by very large SRMs.24,26 The SRMs were higher than those found using the full AM-PAC measure with computerized adaptive testing for patients with orthopedic problems over a course of outpatient physical therapy27 and similar to the SRMs over a period of 12 months for patients with a variety of conditions discharged from IRFs.28 The MDCs in our study were slightly higher than those reported for patients with orthopedic problems seen in outpatient settings due to higher standard deviation of basic mobility scores at baseline and lower internal consistency of daily activity scores in our study than in the previous study.27
Given the need for maximizing efficiency and productivity of rehabilitation professionals in the fast-paced setting of acute care, other studies have proposed models for triaging patients.6,7 Although these models suggest that decisions about need for services rely on more than patients' functional abilities, our findings demonstrate that initial visit “6-Clicks” scores have the potential to be useful in predicting patients' requirement for more than one visit by a physical therapist or occupational therapist.
The ROC curves and the data included in Figure 2 illustrate the utility of “6-Clicks” scores for making clinical decisions about patients' need for more than one physical therapist or occupational therapist visit in the acute care setting. The greater the area under the curve and the greater the distance from the reference line, the better the test is at classifying patients. Using Table 3, which includes data derived from the ROC curves, one can determine that 63% of patients with a score below a basic mobility cutoff score of 43.7 would be correctly identified as needing more than one visit and 73% of patients with a score above 43.7 would be correctly classified as not needing more than one visit. Generally, the goal in determining cutoff scores is to maximize correct decisions. In making decisions, however, one has to determine the consequences of inaccuracy and select cutoff scores accordingly. Using the above example, therapists would need to determine whether the consequences (eg, higher cost and inefficiency) of providing more than one visit to those who might not need them would have worse consequences than providing only one visit to those who might require more (eg, lost opportunity to address patients' activity limitations). At the same time, the consequences might be weighed differently by therapists depending on what a patient's discharge disposition might be. For example, for patients going to a rehabilitation setting where limitations in mobility and daily activity will be addressed, a lower cutoff score might be acceptable.
Sensitivity, Specificity, and Predictive Values for Various Basic Mobility and Daily Activity Scores
Our findings also suggest that “6-Clicks” scores might be used to audit service efficiency. For example, finding that some patients have more than one visit despite demonstrating no limitations based on “6-Clicks” scores could lead to further examination of whether additional visits were, indeed, necessary or whether those patients required further skilled intervention. It may be that issues other than patients' activity limitations, such as patients' or families' needs for education concerning safety, self-care, or home modifications, drive additional visits. The “6-Clicks” forms do not assess cognitive or communication functions, and these functions could affect physical therapists' and occupational therapists' decisions about the number of visits required. It is also possible that strength, range of motion, or endurance may not be adequate despite high “6-Clicks” scores, leading to additional visits to address those impairments. A recent study of patients receiving physical therapy in the intensive care unit showed improvements in activity limitations but not in strength for those who received services compared with those who did not.11 Physical therapists and occupational therapists also could provide additional visits with the goal of patients' achieving higher levels of mobility and daily activities than determined by “6-Clicks” (eg, walking longer distances than that required in the hospital room). Another possible reason for additional treatment sessions in patients with high levels of capability could be nurses' perceived barriers in allowing patients independent mobility, leading to their requests for additional therapy visits.29,30
We also found that a fairly high proportion of patients seen by physical therapists and occupational therapists had no limitations in basic mobility or daily activity based on “6-Clicks” scores. This finding suggests that some patients may have been referred who did not require skilled services. This phenomenon would not be inconsistent with the experience of therapists in other acute care settings.31 Managers could use information from “6-Clicks” scores to gain understanding of the referral practices in particular clinical units or the practice of particular therapists to determine best approaches to providing high-quality, efficient care. In combination with efficiency audits using “6-Clicks” scores, educational efforts could be directed toward nursing or physician house staff about criteria for referral. Additionally, trials could be done using different “6-Clicks” cutoff scores for deciding whether patients receive more than one visit and then examining their outcomes.
It is not clear why measures of functional activity previously developed for use in the acute care setting have not been adopted; however, in order to be useful to clinicians, measurement tools must be relevant to their patients' conditions, provide information that will help them with their decision making, and be quick to use and easy to interpret.17 Reasons cited for developing new instruments for use in the acute care setting have included perceptions that existing instruments did not include relevant items9,12,13,15; had complicated instructions, scoring, or administration8,10,14; were not able to measure patients' improvement9,10,14; and were too lengthy.10 Although reports of the application of various tools have suggested that they might address some of these concerns, their usefulness in doing so has not been consistently documented. Our findings suggest that “6-Clicks” forms meet many of these criteria, particularly relevance and ease of use.
Limitations
The limitations of this study include the fact that data were collected by a large number of clinicians and we did not assess rater reliability. Another limitation is use of a clinical database to provide data for analyses. The limitations in using clinical databases for research have been well documented.32 The database we used was developed to enhance operations and allow quality assessment, not for research purposes. Clinical databases also are designed to be helpful to clinicians in documenting patients' status. Therefore, they may not include the type of data necessary to address research questions, nor the form of data useful for analysis. Clinical databases that include narrative information, such as the database used in this study, can provide rich descriptions that may be helpful for patient care. Such information, however, may be difficult to code accurately when using it for research purposes and may lead to misclassifications and missing data points.
We had FIM data for a small proportion of patients due to the fact that many patients were discharged to IRFs that were not part of the Cleveland Clinic hospital system; therefore, the data were not in our database. This sample could be biased in ways that we cannot determine. Other biases also might exist that are difficult to identify. For example, we do not know whether implementation of “6-Clicks” forms influenced therapists' decisions about number of visits. Another limitation is that this study took place within only one health care system, and it is unclear how the “6-Clicks” instruments would perform in other settings. On the other hand, the health care system in which this study was done comprises several facilities within which data were collected by a large number of clinicians.
Further Study
Rater reliability of “6-Clicks” forms has not been examined. Assessing intratester reliability may be difficult given the rapid changes in functional abilities that patients in acute care may undergo in a short period of time; however, interrater reliability should be assessed. Determining rater reliability may be particularly important given that “6-Clicks” scores may be based solely on observing patients' performance of each item. Clinicians' judgments of patients' functional capabilities may be more variable than their ability to score actual performance. Studies also could be undertaken to examine further the usefulness of “6-Clicks” scores in determining discharge setting. Finally, further investigation to define cutoff scores for making decisions about need for intervention and the appropriate extent of intervention should be undertaken. We suggest studies in which the efficiency and effectiveness of rehabilitation services are assessed when nurses use “6-Clicks” forms to determine whether to seek referral to rehabilitation services. In a recent study by Drolet et al,33 an algorithm was developed that allowed nurses to assess patients' mobility and make decisions about appropriate physical therapist and occupational therapist referrals. Such an approach may improve the efficiency of rehabilitation services and decrease the costs, and we believe nurses' education and experience would allow them to assess patients' performance on “6-Clicks” items.
Conclusion
The “6-Clicks” basic mobility and daily activities forms are new tools that are based on the AM-PAC and designed for measuring patients' activity limitations in the acute care setting. This study provides evidence for the validity of the measures' scores and their potential ability to address some of the concerns about instruments previously designed and proposed for this purpose.
Appendix 1.
“6-Clicks” Inpatient Basic Mobility Short Forma
a The AM-PAC “6-Clicks” forms are copyright protected by The Trustees of Boston University.
Appendix 2.
“6-Clicks” Inpatient Daily Activity Short Form
a The AM-PAC “6-Clicks” forms are copyright protected by The Trustees of Boston University.
Footnotes
All authors provided concept/idea/research design. Dr D. Jette, Mr Ranganathan, Dr Frost, and Dr A. Jette provided writing. Dr Stilphen, Mr Ranganathan, Dr Passek, and Dr Frost provided data collection. Dr Stilphen, Mr Ranganathan, and Dr Passek provided data analysis. Dr Stilphen, Dr Passek, and Dr Frost provided project management. Dr Frost provided fund procurement and clerical support. Dr Stilphen and Dr Frost provided study participants and facilities/equipment. Mr Ranganathan and Dr Frost provided institutional liaisons. Dr Stilphen, Dr Passek, Dr Frost, and Dr A. Jette provided consultation (including review of manuscript before submission).
- Received May 21, 2013.
- Accepted November 12, 2013.
- © 2014 American Physical Therapy Association
References
Issue highlights
