Advertisement

Cross-Cultural Differences in Knee Functional Status Outcomes in a Polyglot Society Represented True Disparities Not Biased by Differential Item Functioning

Daniel Deutscher, Dennis L. Hart, Paul K. Crane, Ruth Dickstein
DOI: 10.2522/ptj.20100107 Published 1 December 2010

Abstract

Background Comparative effectiveness research across cultures requires unbiased measures that accurately detect clinical differences between patient groups.

Objective The purpose of this study was to assess the presence and impact of differential item functioning (DIF) in knee functional status (FS) items administered using computerized adaptive testing (CAT) as a possible cause for observed differences in outcomes between 2 cultural patient groups in a polyglot society.

Design This study was a secondary analysis of prospectively collected data.

Methods We evaluated data from 9,134 patients with knee impairments from outpatient physical therapy clinics in Israel. Items were analyzed for DIF related to sex, age, symptom acuity, surgical history, exercise history, and language used to complete the functional survey (Hebrew versus Russian).

Results Several items exhibited DIF, but unadjusted FS estimates and FS estimates that accounted for DIF were essentially equal (intraclass correlation coefficient [2,1]>.999). No individual patient had a difference between unadjusted and adjusted FS estimates as large as the median standard error of the unadjusted estimates. Differences between groups defined by any of the covariates considered were essentially unchanged when using adjusted instead of unadjusted FS estimates. The greatest group-level impact was <0.3% of 1 standard deviation of the unadjusted FS estimates.

Limitations Complete data where patients answered all items in the scale would have been preferred for DIF analysis, but only CAT data were available.

Conclusions Differences in FS outcomes between groups of patients with knee impairments who answered the knee CAT in Hebrew or Russian in Israel most likely reflected true differences that may reflect societal disparities in this health outcome.

Physical therapy is commonly used in health service delivery systems to manage patients with musculoskeletal impairments with the goal of maintaining or improving physical functioning.1–4 Musculoskeletal impairments and associated functional limitations are costly.5,6 Comparative effectiveness research has been proposed to identify effective and efficient care processes.7

Such research across cultural groups within a country requires unbiased measures that accurately detect clinical differences among patients.8 Cultural differences may influence the way patients answer otherwise reliable, valid, and responsive surveys, possibly affecting scores. Clinicians, researchers, and policy makers need to know whether observed differences in scores across groups represent a measurement problem, a true health disparity, or some combination of both. A measurement problem can result from nonequivalent item translations or differences in cultural perceptions.9 A health disparity occurs when true differences in the patients' underlying trait exist.10,11 Israel has a polyglot population, similar to many countries where multiple languages are used. Given limited societal resources to address disparities, it seems prudent to investigate whether observed differences represent some underlying systematic inequality or whether they are merely due to a measurement problem.

The current study focuses on one aspect of validity: differential item functioning (DIF). Differential item functioning is present when the relationship between item responses and functional status (FS) measured by the test differs across groups after controlling for FS measures.12,13 When DIF is present, validity of FS estimates is eroded.

Modern psychometric techniques such as item response theory (IRT)14,15 facilitate administering FS questionnaires using computerized adaptive tests (CATs).16–18 A CAT is a data collection method in which an algorithm selects each item by matching item difficulty to the patient's estimated level of ability, represented here as estimated FS. After each item response, the estimated FS is updated. The CAT stops when the patient's estimated FS meets certain criteria, such as a required level of measurement precision.19,20 Computerized adaptive tests minimize the number of items administered, reducing respondent burden, with limited effect on measurement precision.18 A CAT that administers a particular number of items from a psychometrically strong item pool provides better measurement precision than fixed format administration of the same number of items.21

Differential item functioning may be particularly menacing in the CAT setting because of the relatively small subsets of items administered.22 In an item bank with several items favoring one group and several others favoring another group, one can imagine a particularly unlucky scenario in which a CAT selects items that all have DIF in the same direction, producing a biased estimate for FS. Because DIF has been identified in FS items answered by patients from different cultural backgrounds, identifying and accounting for DIF may be important to improve the validity of conclusions about health disparities.11,23

In a previous study, Deutscher et al24 found that among 4,394 patients receiving outpatient physical therapy care due to knee impairments, those who answered a knee FS CAT in Russian (25%) had higher FS at discharge compared with other patients who answered in Hebrew (69%), English (4%), Spanish (1%), or Arabic (1%). This difference in FS scores across groups remained after controlling for patient characteristics at intake and treatment processes. These differences in FS scores among cultural groups raised the question of whether they represented true differences in FS—which might suggest existence of a health disparity—or possible measurement bias related to cross-cultural DIF. In a separate study using the same knee CAT, Hart et al9 found negligible DIF when comparing knee CAT items for patients with knee impairments who spoke English and were treated in the United States or who spoke Hebrew and were treated in Israel. However, it was not known whether similar results would be found across patients in Israel who spoke Hebrew or Russian.

The purpose of this study was to examine the presence and impact of DIF in FS items administered via CAT for patients with knee impairments in Israel who spoke either Hebrew or Russian. The results should facilitate comparative effectiveness research studies across groups.

Method

Design

We conducted a secondary analysis of prospectively collected data.

Participants and Clinics

Data from patients 18 years of age or older were analyzed. Patients were treated between 2005 and 2008 in the Physical Therapy Service of Maccabi Healthcare Services (Maccabi), a public health maintenance organization responsible for the care of 1.8 million people, representing 25% of Israel's population. Maccabi's Physical Therapy Service has been collecting functional outcome data since 2005 using a customized version of Patient Inquiry computer software (version 5.0) developed by Focus On Therapeutic Outcomes (FOTO), Inc.*,3

All participating clinics (n=71) are payer-owned and provide physical therapy services to patients with a variety of musculoskeletal impairments.24 The study included patients who on admission to therapy selected the knee as their primary area of musculoskeletal impairment, independently completed the knee CAT, and answered the CAT in Hebrew or Russian.

The statistical significance criterion used by the DIF detection software for nonuniform DIF in each item is a sample size–dependent chi-square test. Therefore, we balanced samples of participants across language groups to minimize the risk of false-positive and false-negative findings due to unequal sample sizes. Most patients answered the surveys in Hebrew, so we selected a random sample of Hebrew surveys to match the 4,567 available Russian surveys, resulting in a numerically balanced sample of 9,134.

Data Collection

Data were retrieved from the Maccabi Physical Therapy Service database. Maccabi performs routine outcomes collection as part of its patient management strategy.3,24 Demographic data were retrieved from the Maccabi electronic health record system.3 Although the CAT was administered throughout rehabilitation, only intake data were analyzed for this study.

The Knee CAT

Development,16 simulation,16 use,17 and interpretation25 of the knee-specific CAT and its FS measures have been described. Briefly, items from the Lower Extremity Functional Scale26 represent functional activities, such as “putting on your shoes or socks” (Appendix). Patients are asked to rate their ability to perform each activity using 5 responses, from “Extreme difficulty or unable to perform activity” to “No difficulty.”16 Unidimensionality and local independence of the 18-item bank and fit to the Andrich27 rating scale 1-parameter item response theory (IRT) model (Andrich RSM) were supported.16 The knee CAT was developed as a body part–specific CAT.16 The CAT development process followed the logic of Thissen and Mislevy28 and has been described.29–31 Efficiency of knee CAT administration in routine clinical practice has been supported.3,17 Functional status, as assessed using knee CAT items, represents the “activity” dimension of the World Health Organization's International Classification of Functioning, Disability and Health.32 Before the knee CAT was implemented in Maccabi, items were translated into Hebrew and Russian following published procedures.9,33

Statistical Analyses

We followed analytic methods similar to those in our previous study assessing DIF in FS items for patients in the United States who spoke English versus patients in Israel who spoke Hebrew.9 The statistical analyses had 3 main purposes: (1) to identify patient characteristics and CAT usage differences between the 2 samples using descriptive analyses; (2) to identify DIF presence; and (3) when DIF was detected, to estimate the magnitude of individual-level and group-level DIF impact.

Descriptive analyses.

Demographic differences between the Hebrew and Russian samples were assessed using chi-square statistics and 2-sample t tests, as appropriate. To assess item usage, we determined the item exposure rate, defined as the number of times each item was administered divided by the total number of CATs administered.

DIF detection.

Validity of using IRT methods to assess cross-cultural DIF depends on the fit of the data to the model.34 Although Andrich's RSM27 was used to develop and calibrate this scale, we used the publicly available difwithpar software (version 1.0)†,35 which uses Samejima's Graded Response Model (GRM) for DIF analyses.36 Prior to assessing DIF, we tested fit of data from this study to both the RSM and GRM. Fit to the RSM was tested using WINSTEPS‡,37 as we did previously.16 Good item fit was defined as infit and outfit statistics between 0.60 and 1.40.38 There is no recognized best way to assess fit of data to the GRM, particularly for samples larger than 1,500.39 We followed several approaches to assess fit of our data to the GRM. We examined item category curves to ensure they progressed from easier to harder along the FS continuum and ensured that each curve had a maximum at a unique scale interval.36 In addition, we assessed item discrimination parameters (slopes). Slopes of 0.70 or higher are preferred.39

We assessed DIF using 4 steps described previously,9 including the following. First, we performed analyses of items in the Hebrew and Russian data sets separately for DIF with respect to sex, surgical history, acuity of symptoms, exercise history, and age. All of these covariates have been shown to affect FS,17 and we wanted to ensure that differences between patients who spoke Hebrew or Russian were not due to DIF related to these covariates before assessing DIF related to language. Second, we combined Hebrew and Russian data and repeated the above DIF assessments in the combined sample. Third, we assessed DIF by language (Hebrew versus Russian) in the combined data set. Fourth, to account for CAT data acquisition, we investigated whether DIF findings were stable across 2 subgroups stratified by FS levels. Patients with high FS scores answered harder items, so we limited these secondary analyses to the 9 hardest items in the test. In similar fashion, we evaluated only the 9 easiest items for patients with low FS scores.

For each of these 4 steps, each item was assessed for DIF using difwithpar software,40–45 which combines IRT item calibration estimated by the GRM36 using PARSCALE for Windows (version 4.1)§,46 with multiple ordinal logistic regression models for each item and covariate using Stata software (version 10.1).∥,47 Details of the difwithpar framework for DIF detection have been previously published.40–45 Briefly, statistical analyses included the following steps.

  1. Estimates of FS were obtained using GRM.

  2. Those FS estimates were used as conditioning variables in a series of nested ordinal logistic regression models to identify items with DIF.

  3. A new data set was prepared treating items found not to have DIF as anchor items and items found with DIF as separate items for the 2 groups. This data set was used to re-estimate FS scores accounting for the items found with DIF.

  4. The revised estimates of FS adjusted for DIF were used as conditioning variables as in step 2.

  5. Items identified in step 2 and in step 4 were compared. If they were the same, statistical analyses stopped. If they were different, a new data set was prepared as in step 3, and the procedure continued until the same items were identified with DIF as on a prior run.

In steps 2 and 4 of the difwithpar analyses, items were examined for the presence of uniform and nonuniform DIF. Uniform DIF occurs when differences across groups in item responses while controlling for FS are the same across the entire range of FS measured by the test. Uniform DIF was assessed by examining the relative difference between β coefficients of 2 ordinal logistic regression models.45 As recommended by Crane et al40 for capturing small amounts of uniform DIF, items were treated as having uniform DIF when that difference exceeded 5%. Nonuniform DIF exists when differences in item responses between demographic groups while controlling for FS vary at different levels of FS; it was assessed by comparing log likelihoods of 2 ordinal logistic regression models.41 For nonuniform DIF, we have used a Bonferroni adjustment previously42,45,48 to reduce the potential for a type I error. However, because we were interested in identifying smaller amounts of DIF and assessing its impact, we decided to use α=.01, which was a compromise between making the P value small enough for a large data set and not so small that our sensitivity to identify DIF would be compromised.

Within each of the 4 steps outlined above, we performed 2 separate analyses. First, we evaluated each covariate alone, beginning in each case with an unadjusted FS estimate. We followed the steps of the difwithpar process until we were confident items with DIF were identified with respect to that covariate. The difwithpar process produced a final FS score that accounted for DIF with respect to that covariate.

Second, we performed a set of analyses to account simultaneously for DIF related to all of the covariates. We performed all analyses sequentially for sex, surgical history, symptom acuity, age group, exercise history, and language. The approach to DIF with respect to several covariates was performed exactly as has been published.9,43,44

We plotted item category response functions of items with DIF to visually confirm the relationship between the ordinal logistic regression approach embodied by difwithpar and the IRT conceptualization of DIF.40 The curves represent probabilities for responding at or higher than each response category (y-axis) plotted against knee FS measures (x-axis) for each covariate. We assessed concordance of item location parameter estimates between languages using the intraclass correlation coefficient (ICC [2,1]), which we also analyzed graphically.49

DIF impact.

In large samples, statistically significant DIF may have little practical importance or impact.9,40,48 This may be a concern in this case, as we used a relatively lenient P value of .01 for nonuniform DIF and had large sample sizes. Therefore, we assessed patient-level and group-level DIF impact.

For patient-level DIF impact, we performed 2 analyses. First, we assessed concordance between unadjusted FS estimates and FS estimates that accounted for DIF with respect to all of the covariates using an ICC (2,1). Second, we determined FS differences (ie, the difference between unadjusted [“naive”] FS estimates and FS estimates that accounted for DIF) as an indicator of DIF impact. We determined FS differences from analyses of DIF with respect to each covariate separately and FS difference from the analysis with all of the covariates together. The rationale for these calculations is that the only factor producing a difference between the naive FS score and the FS score that accounted for DIF with respect to a single covariate is DIF related to that covariate; the difference in scores represents the cumulative effect of DIF on that individual with respect to that covariate across all of the items. In a similar fashion, the difference between the naive FS score and the FS score that accounted for all sources of DIF represents the cumulative impact of all sources of DIF on that individual across all of the items.

Smaller values for these differences indicate less DIF impact, whereas larger values represent more DIF impact. We have in prior publications indexed these differences to the median standard error of measurement (SEM) for the cohort as a whole and called differences larger than this “salient DIF impact.”9,43,44 We plotted these FS differences using a box-and-whisker plot. We determined the number of patients with salient DIF.

For group-level DIF impact, we assessed the magnitude of mean differences on the (0,1) standard normal metric between unadjusted FS estimates and FS estimates that accounted for all sources of DIF in percentage of standard deviation units of unadjusted FS estimates.

Results

Descriptive Data

On average, patients who answered the CAT in Russian were older, had more chronic symptoms, were more likely to be female, received less surgery, and exercised less compared with patients who answered the CAT in Hebrew (Tab. 1). Item exposure rates are displayed in Figure 1, which shows that the Hebrew sample, compared with the Russian sample, was exposed to slightly lower rates of lower-level items and slightly higher rates of higher-level items. The average number of items administered to patients for the 2 languages were similar, with 6.5 (SD=1.5) for Hebrew and 6.4 (SD=1.3) for Russian. Three to 17 items were required to meet CAT stopping rules in both languages.

View this table:
Table 1.

Patient Characteristics at Therapy Intakea

Figure 1.
Figure 1.

Item exposure rate (ie, percent of computerized adaptive tests [CATs] with each item from all CATs). Items are sorted by location parameters of the combined sample, from low (left side of x-axis) to high (right side of x-axis) level of difficulty. See Appendix for item descriptions.

Fit of Data to the IRT Model

The CAT data from the Hebrew and Russian samples fit the RSM well, with all fit statistics ranging between 0.68 and 1.40 except for the squatting item in the Russian sample (infit=1.6, outfit=1.5). This result was similar to fit analysis from the original knee CAT assessment, with only the squatting item exceeding the 1.4 fit statistics, but was less than 2, thus representing adequate fit.16 The CAT data also fit the GRM well, with category response curves progressing as expected from easier to harder along the FS axis. Each response curve had a maximum at a progressively higher unique scale interval. Item slopes for both Hebrew and Russian language samples, as well as for the combined sample, were all greater than 0.89 (Tab. 2). Results were interpreted as suggesting good fit of the Hebrew and Russian data to both RSM and GRM.

View this table:
Table 2.

Item Discrimination and Location Parameters (Graded Response Model)a

DIF Presence

In the Hebrew sample (Tab. 3), only one item (HOBBY) demonstrated uniform DIF with respect to age. A few items demonstrated nonuniform DIF with respect to the other covariates analyzed. In the Russian sample, only 2 items (SQUAT and HOP) showed nonuniform DIF with respect to surgical history; no other items showed uniform or nonuniform DIF with respect to any of the other covariates (Tab. 4). Overall, the amount of DIF detected was interpreted as small for the Hebrew sample and almost nonexistent in the Russian sample.

View this table:
Table 3.

Presence of Differential Item Functioning (DIF) Related to Covariates in the Hebrew Sample (n=4,567)a

View this table:
Table 4.

Presence of Differential Item Functioning (DIF) Related to Covariates in the Russian Sample (n=4,567)a

Small amounts of nonuniform DIF were detected in the combined sample with respect to age (3 items), symptom acuity (1 item), sex (3 items), and surgical history (5 items). Only 1 item (HOBBY) was found to have uniform DIF with respect to age and language (Tab. 5).

View this table:
Table 5.

Presence of Differential Item Functioning (DIF) Related to Covariates in the Combined Samplea

In the combined sample DIF analysis for language, we found strong concordance of item difficulty parameter estimates between languages (ICC [2,1]=.96). An identical ICC was found when using item difficulty parameter estimates from the RSM using WINSTEPS. Only 1 item (HOBBY) was found to have uniform DIF, and 3 items (SQUAT, CAR, and STAND) were found to have nonuniform DIF (Tab. 5). The smallest P value for nonuniform DIF was found for the SQUAT item. The item boundary response function plot for the SQUAT item supported DIF identified by difwithpar (Fig. 2). This boundary response function plot demonstrates nonuniform DIF by language for the SQUAT item. These curves represent the probability (y-axis) of each response category for the 2 languages. Nonuniform DIF is evident when the probability differences between the boundary response function curves of each language, seen as the gap between the solid (Hebrew-speaking) and dashed (Russian-speaking) curves for each boundary, are different at different ability levels. The SQUAT item seems to have DIF in the lower ability levels of the scale and almost no DIF at the higher ability levels. It represents the single item with the greatest amount of DIF; all other curves were even more similar.

Figure 2.
Figure 2.

Boundary response function plot of the item (ie, SQUAT) with the largest amount of nonuniform differential item functioning for the combined sample. See Appendix for item descriptions.

DIF Impact

Group-level results supported minimal FS differences between unadjusted FS estimates and FS estimates that accounted for all sources of DIF, with the largest difference of 0.003, an amount so small as to be of no clinical or practical importance. Differences as percentages of the standard deviation of the unadjusted FS estimates were small (largest difference was 0.29%). Patient-level results supported similar, negligible impact for DIF, with concordance values (ICC [2,1]) between unadjusted FS estimates and FS estimates accounting for sources of DIF greater than 0.999. No patient in the combined sample had FS differences as large as the median SEM (Fig. 3). As described by Crane et al,44 the box-and-whiskers plot displays the impact of DIF for each covariate found to have significant DIF on estimated knee functional status measures in the combined sample. Values in the plot are the differences between IRT measures that accounted for DIF and IRT measures that ignored DIF for each covariate that had significant DIF. The box indicates the 25th and 75th percentiles, and the line in the middle of the box indicates the median. The whiskers indicate 1½ times the interquartile range, and the diamonds indicate more extreme values. No difference would be 0. Vertical lines placed at multiples of 0.32 on a standard normal metric (0,1) indicate the median SEM found in this sample. Any differences greater than the median SEM are said to be associated with salient scale-level differential item functioning. None were found. Results were similar in the Hebrew sample, with only 2 of 4,567 patients showing salient DIF impact. No patients had salient DIF impact in the Russian sample.

Figure 3.
Figure 3.

Individual-level differential item functioning impact: combined Hebrew and Russian sample (N=9,134).

Results were similar when we stratified patients by level of FS and evaluated items selected by their location parameters. A few items with DIF were found, and these items produced negligible impact in the low and high FS subsets. In the low FS subset, there were no patients showing FS differences larger than the median SEM. In the high FS subset, only 11 of 4,567 patients (0.24%) showed salient DIF with respect to surgical history.

Discussion

The knee CAT is efficient in routine clinical use.3,17,50 Computerized adaptive testing FS estimates are reliable, valid, sensitive to change, and responsive16,17 and can be interpreted in clinically useful ways.25 In addition, prior analyses of the knee CAT for patients in the United States who spoke English and patients in Israel who spoke Hebrew found negligible DIF impact.9 In the current study, we investigated the validity of the knee CAT among Hebrew and Russian speakers in Israel. We found a few items with DIF with respect to a variety of covariates, but all DIF detected was associated with negligible individual-level and group-level impact.

The sample of patients who answered the CAT in Russian had demographic characteristics different from those of patients who answered the CAT in Hebrew (Tab. 1). The reasons for these differences have been described previously.51 In theory, differences in patient characteristics between the 2 samples could have led to finding DIF with respect to language. However, very little item-level and no group-level DIF impact was identified or associated with these demographic characteristics, a result that strengthens our confidence in the cross-language DIF analyses. The negligible DIF impact across all analyses supports our confidence in the cross-cultural validity of the knee CAT. This confidence also is buttressed by the robustness of our findings across 2 IRT models.

Analysis of item exposure rates (Fig. 1) showed similar though not identical exposure rates between samples, as patients from the Russian sample had greater exposure to easier items and less exposure to harder items. This item exposure rate difference could result from the Russian sample being older and having more chronic conditions. Indeed, FS estimates at intake were slightly lower in the Russian sample (mean=−0.16, SD=0.96) compared with the Hebrew sample (mean=0.11, SD=0.98). As a preliminary means of addressing variable item administration across language groups, we analyzed subsets of the data set defined by FS scores, and results were minimally affected.

Our results suggest that DIF may be safely ignored for analyses of knee FS within Israel between patients who speak Hebrew or Russian, as differences in FS across language groups are unlikely to be due to DIF and more likely to represent true differences. This result emphasizes the need to identify factors responsible for the difference in knee FS found across language groups in our previous work.24 Once these factors are identified, ways to reduce disparities across cultural groups can be suggested, implemented, and quantified.

We analyzed DIF using CAT data because complete data from patients who answered all items were not available. Use of CAT data to assess DIF is challenging because of the adaptive nature of the test, which results in different subsets of items being administered to different participants (Fig. 1).52 However, the software (PARSCALE, difwithpar, and Stata) used to generate FS estimates with CAT data managed missing data well, as we have seen previously.9 We believe that our relatively large sample size and our methods, which included analyzing the full item bank and analyzing subsets of items stratified by difficulty levels, provided an adequate data set with which to assess DIF.

We found a few items with DIF in the Hebrew and combined samples but almost no items with DIF in the Russian sample. This finding suggests that the source of DIF in the combined sample may be Hebrew speakers, which might have affected our previous results.9 Our Hebrew-speaking sample may be more heterogeneous than the Russian-speaking sample, as it included people who immigrated to Israel or were children of immigrants to Israel from all around the world. In contrast, most of the patients who answered the CAT in Russian immigrated from Russia during the 1980s and early 1990s, thus representing a more culturally homogeneous sample.

The HOBBY item had DIF with respect to age in the Hebrew and combined samples and with respect to language in the combined sample (Tab. 5). The HOBBY item also showed relatively lower fit to the data (Tab. 2). Patients from different age or language groups might have different hobbies that represent different levels of challenge to the knee, which thus are perceived differently. If DIF impact had not been negligible, a revision of this item's content might seem reasonable to improve fit to the data and decrease the identified statistically significant DIF. Even with the HOBBY item included, only 2 patients had salient DIF impact. These findings suggest negligible DIF impact; any culturally pertinent differences in hobbies were of no practical consequence in the assessment of FS.

The FS measures estimated using the knee CAT have been assessed for the smallest difference patients perceive as beneficial, known as the minimal clinically important difference (MCID).53 Although important change was dependent on intake FS score, the cut-point of 9 out of 100 FS units was identified for the entire scale as an amount of change relevant to patients.17 The knee CAT included a stopping rule of SEM<4 out of 100 scale units, which is equivalent to SEM<0.5 logits, similar to other published stopping rules for CATs.54,55 The median SEM we used as an indicator of salient DIF impact was ∼0.3, or ∼25% of the MCID. These results suggest that there was no relevant DIF impact43 related to the covariates assessed in this study.

Crane et al42 have described what we found in our data, namely, use of large samples can detect item-level DIF with negligible impact. Our results suggest that DIF in the knee CAT can be ignored in group comparisons.42 Patients in Israel who answered the knee CAT in Hebrew or Russian with equivalent FS levels respond to the knee items similarly, regardless of differences found in patient characteristics or cultural differences. These results support the validity of the translations of the FS items into Russian. The Russian CAT is practically equivalent to the Hebrew CAT, which was previously found to be equivalent to the original English version of the knee CAT.9

Future cross-cultural comparative effectiveness studies should use standardized, patient-centered measures that have been assessed for construct validity across nations or across subpopulations who speak a variety of languages. It is not recommended to evaluate disparities in health outcomes between patients who answer surveys in different languages if the health outcome is not assessed in an identical fashion for all populations.

Limitations

We would have preferred to analyze complete data where patients answered all items in the scale, but only CAT data were available. There are different views regarding the appropriate IRT model used for DIF assessment. We felt that the methods for DIF detection described by Crane and colleagues,40–45 which combine multiple ordinal logistic regression and 2-parameter IRT FS estimates, are appropriate when the data fit the GRM. However, because our data also fit the 1-parameter RSM, we also tested DIF presence by language using another 1-parameter model, the Partial Credit Model,56 and found similar results, with only the HOBBY item representing important uniform DIF (results available upon request). A full analysis of DIF presence and impact using other DIF techniques is beyond the scope of our article but might be beneficial.57

In conclusion, we examined possible reasons for cultural differences in functional outcomes related to DIF in a knee-specific CAT item bank. Several items were found to have DIF, but we found no practical DIF impact. Differences in FS scores across groups may reflect some underlying health disparity rather than being due to DIF. Results support the validity of the Russian-translated item pool compared with the Hebrew translation. This suggests cross-cultural comparisons of outcomes measures from the knee CAT used in Israel can progress, so ways to improve quality of life of patients discharged with lower outcomes can be identified. Additional studies are needed to assess presence and impact of cross-cultural DIF for other languages used to answer the knee CAT (Spanish and Arabic) or between languages used to answer other CATs before using these measures with confidence for cross-cultural comparative effectiveness research.

The Bottom Line

What do we already know about this topic?

The knee computerized adaptive test (CAT) is efficient and produces measures of functional status (FS) that are valid, sensitive to change, responsive, and can be interpreted in clinically useful ways. The CAT has a negligible differential item functioning (DIF) impact for patients who speak English in the US when compared with patients who speak Hebrew in Israel, which supports the validity of the Hebrew translation.

What new information does this study offer?

The knee CAT found a negligible DIF impact for patients in Israel who spoke Hebrew compared with patients in Israel who spoke Russian, which supports the validity of the Russian translation. Differences in FS between patients who speak Hebrew and those who speak Russian are likely to represent true differences that are not biased by DIF.

If you're a patient, what might these findings mean for you?

This research suggests that the knee CAT is an appropriate measure for use in future studies of treatment effectiveness, regardless of the cultural origin of the participants. Such studies may help identify ways to improve FS outcomes for patients with knee impairments.

Footnotes

  • All authors provided concept/idea/research design and writing. Mr Deutscher provided data collection, project management, fund procurement, participants, and facilities/equipment. Mr Deutscher, Dr Hart, and Dr Crane provided data analysis. Dr Dickstein provided institutional liaisons. Dr Hart, Dr Crane, and Dr Dickstein provided consultation (including review of manuscript before submission).

  • The authors thank Laura Gibbons, PhD, from the Department of Internal Medicine at the University of Washington for her assistance throughout the data analysis process. They also thank Moshe Gutvirtz, MHA, national director of physical therapy services, and Ilana Ariel, PT, BPT, Ditza Gotleib, PT, MScPT, Sandra Maron, PT, BPT, Tal Nitzan, PT, BPT, and Noa Ben-Ami, PT, BPT, the 5 district directors of the physical therapy service, all from Maccabi Healthcare Services, for their continued devotion and successful management of the continuous functional outcomes collection process. A special acknowledgment is dedicated to the hundreds of physical therapists who managed the on-site data collection process. Their dedicated work enabled this study, promoting outcomes-based clinical practice, for the benefit of our patients.

  • The Maccabi Healthcare Services Institutional Review Board for the Protection of Human Subjects approved the study.

  • This study was funded by Maccabi Healthcare Services–HMO, Israel.

  • Dr Hart is an employee of, and investor in, Focus On Therapeutic Outcomes, Inc, a database management company and owner of the outcomes collection software used to collect functional outcome data for the study.

  • ↵* Focus On Therapeutic Outcomes, Inc, PO Box 11444, Knoxville, TN 37919 (Web site: www.fotoinc.com).

  • ↵† Crane P, Gibbons LE, Jolley I, van Belle G, University of Washington, Seattle, Washington, 2005

  • ↵‡ Winsteps, PO Box 811322, Chicago, IL 60681-1322.

  • ↵§ Scientific Software International Inc, 7383 N Lincoln Ave, Suite 100, Lincolnwood, IL 60712-1747.

  • ↵∥ StataCorp LP, 4905 Lakeway Dr, College Station, TX 77845.

  • Received March 24, 2010.
  • Accepted July 16, 2010.

Appendix.

Appendix.
Appendix.

Item Descriptions

References

  1. ↵
    Guide to Physical Therapist Practice. 2nd ed. Phys Ther. 2001;81:9–746.
    OpenUrlPubMedWeb of Science
  2. ↵
    1. Carter SK,
    2. Rizzo JA
    . Use of outpatient physical therapy services by people with musculoskeletal conditions. Phys Ther. 2007;87:497–512.
    OpenUrlAbstract/FREE Full Text
  3. ↵
    1. Deutscher D,
    2. Hart DL,
    3. Dickstein R,
    4. et al
    . Implementing an integrated electronic outcomes and electronic health record process to create a foundation for clinical practice improvement. Phys Ther. 2008;88:270–285.
    OpenUrlAbstract/FREE Full Text
  4. ↵
    1. Swinkels IC,
    2. Hart DL,
    3. Deutscher D,
    4. et al
    . Comparing patient characteristics and treatment processes in patients receiving physical therapy in the United States, Israel and the Netherlands: cross-sectional analyses of data from three clinical databases. BMC Health Serv Res. 2008;8:163.
    OpenUrlCrossRefPubMed
  5. ↵
    1. Cunningham LS,
    2. Kelsey JL
    . Epidemiology of musculoskeletal impairments and associated disability. Am J Public Health. 1984;74:574–579.
    OpenUrlPubMedWeb of Science
  6. ↵
    Medicare Payment Advisory Committee. Toward better value in purchasing outpatient therapy services. In: Report to the Congress: Increasing the Value of Medicare. Washington, DC: Medicare Payment Advisory Committee; 2006:117–141.
  7. ↵
    1. Freburger JK,
    2. Carey TS
    . Comparative effectiveness research: opportunities and challenges for physical therapy. Phys Ther. 2010;90:327–332.
    OpenUrlFREE Full Text
  8. ↵
    1. Hahn EA,
    2. Holzner B,
    3. Kemmler G,
    4. et al
    . Cross-cultural evaluation of health status using item response theory: FACT-B comparisons between Austrian and U.S. patients with breast cancer. Eval Health Prof. 2005;28:233–259.
    OpenUrlAbstract/FREE Full Text
  9. ↵
    1. Hart DL,
    2. Deutscher D,
    3. Crane PK,
    4. Wang YC
    . Differential item functioning was negligible in an adaptive test of functional status for patients with knee impairments who spoke English or Hebrew. Qual Life Res. 2009;18:1067–1083.
    OpenUrlCrossRefPubMedWeb of Science
  10. ↵
    1. Petersen MA,
    2. Groenvold M,
    3. Bjorner JB,
    4. et al
    . Use of differential item functioning analysis to assess the equivalence of translations of a questionnaire. Qual Life Res. 2003;12:373–385.
    OpenUrlCrossRefPubMedWeb of Science
  11. ↵
    1. Tennant A,
    2. Penta M,
    3. Tesio L,
    4. et al
    . Assessing and adjusting for cross-cultural validity of impairment and activity limitation scales through differential item functioning within the framework of the Rasch model: the PRO-ESOR project. Med Care. 2004;42(1 suppl):I37–I48.
    OpenUrlPubMed
  12. ↵
    1. Camilli G,
    2. Shepard LA
    . Methods for Identifying Biased Test Items. Thousand Oaks, CA: Sage Publications; 1994.
  13. ↵
    1. Millsap RE,
    2. Everson HT
    . Methodology review: statistical approaches for assessing measurement bias. Appl Psychol Meas. 1993;17:297–334.
    OpenUrlCrossRefWeb of Science
  14. ↵
    1. Hambleton RK,
    2. Swaminathan H,
    3. Rogers HJ
    . Fundamentals of Item Response Theory. Newbury Park, CA: Sage Publications; 1991.
  15. ↵
    1. Hays RD,
    2. Morales LS,
    3. Reise SP
    . Item response theory and health outcomes measurement in the 21st century. Med Care. 2000;38(9 suppl):II28–II42.
    OpenUrlPubMed
  16. ↵
    1. Hart DL,
    2. Mioduski JE,
    3. Stratford PW
    . Simulated computerized adaptive tests for measuring functional status were efficient with good discriminant validity in patients with hip, knee, or foot/ankle impairments. J Clin Epidemiol. 2005;58:629–638.
    OpenUrlCrossRefPubMedWeb of Science
  17. ↵
    1. Hart DL,
    2. Wang YC,
    3. Stratford PW,
    4. Mioduski JE
    . Computerized adaptive test for patients with knee impairments produced valid and responsive measures of function. J Clin Epidemiol. 2008;61:1113–1124.
    OpenUrlCrossRefPubMedWeb of Science
  18. ↵
    1. Wainer H
    1. Wainer H
    . Introduction and history. In: Wainer H ed. Computerized Adaptive Testing: A Primer. 2nd ed. Mahwah, NJ: Lawrence Erlbaum Associates; 2000:1–21.
  19. ↵
    1. Hart DL,
    2. Wang YC,
    3. Stratford PW,
    4. Mioduski JE
    . Computerized adaptive test for patients with foot or ankle impairments produced valid and responsive measures of function. Qual Life Res. 2008;17:1081–1091.
    OpenUrlCrossRefPubMedWeb of Science
  20. ↵
    1. Jette AM,
    2. Haley SM,
    3. Tao W,
    4. et al
    . Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings [erratum in Phys Ther. 2007;87:617]. Phys Ther. 2007;87:385–398.
    OpenUrlAbstract/FREE Full Text
  21. ↵
    1. Rose M,
    2. Bjorner JB,
    3. Becker J,
    4. et al
    . Evaluation of a preliminary physical function item bank supported the expected advantages of the Patient-Reported Outcomes Measurement Information System (PROMIS). J Clin Epidemiol. 2008;61:17–33.
    OpenUrlCrossRefPubMedWeb of Science
  22. ↵
    1. Wainer H
    1. Steinberg L,
    2. Thissen D,
    3. Wainer H
    . Validity. In: Wainer H ed. Computerized Adaptive Testing: A Primer. 2nd ed. Mahwah, NJ: Lawrence Erlbaum Associates; 2000:185–229.
  23. ↵
    1. Custers JW,
    2. Hoijtink H,
    3. van der Net J,
    4. Helders PJ
    . Cultural differences in functional status measurement: analyses of person fit according to the Rasch model. Qual Life Res. 2000;9:571–578.
    OpenUrlCrossRefPubMedWeb of Science
  24. ↵
    1. Deutscher D,
    2. Horn SD,
    3. Dickstein R,
    4. et al
    . Associations between treatment processes, patient characteristics, and outcomes in outpatient physical therapy practice. Arch Phys Med Rehabil. 2009;90:1349–1363.
    OpenUrlCrossRefPubMedWeb of Science
  25. ↵
    1. Wang YC,
    2. Hart DL,
    3. Stratford PW,
    4. Mioduski JE
    . Clinical interpretation of computerized adaptive test generated outcomes measures in patients with knee impairments. Arch Phys Med Rehabil. 2009;90:1340–1348.
    OpenUrlCrossRefPubMedWeb of Science
  26. ↵
    1. Binkley JM,
    2. Stratford PW,
    3. Lott SA,
    4. Riddle DL
    ; North American Orthopaedic Rehabilitation Research Network. The Lower Extremity Functional Scale (LEFS): scale development, measurement properties, and clinical application. Phys Ther. 1999;79:371–383.
    OpenUrlAbstract/FREE Full Text
  27. ↵
    1. Andrich D
    . A rating formulation for ordered response categories. Psychometrika. 1978;43:561–573.
    OpenUrlCrossRefWeb of Science
  28. ↵
    1. Wainer H
    1. Thissen D,
    2. Mislevy RJ
    . Testing algorithms. In: Wainer H ed. Computerized Adaptive Testing: A Primer. 2nd ed. Mahwah, NJ: Lawrence Erlbaum Associates; 2000:101–134.
  29. ↵
    1. Linacre JM
    . Estimating measures with known polytomous item difficulties. Rasch Measurement Transactions. 1998;12:638.
    OpenUrl
  30. ↵
    1. Lord FM
    ed Applications of Item Response Theory to Practical Testing Problems. Hillsdale, NJ: Lawrence Erlbaum Associates; 1980.
  31. ↵
    1. Sands WA,
    2. Waters BK,
    3. McBride JR
    eds. Computerized Adaptive Testing: From Inquiry to Operation. Washington, DC: American Psychological Association; 1997.
  32. ↵
    International Classification of Functioning, Disability and Health: ICF. Geneva, Switzerland: World Health Organization; 2001.
  33. ↵
    1. Lewin-Epstein N,
    2. Sagiv-Schifter T,
    3. Shabtai EL,
    4. Shmueli A
    . Validation of the 36-Item Short-Form Health Survey (Hebrew version) in the adult population of Israel. Med Care. 1998;36:1361–1370.
    OpenUrlCrossRefPubMedWeb of Science
  34. ↵
    1. Bjorner JB,
    2. Kreiner S,
    3. Ware JE,
    4. et al
    . Differential item functioning in the Danish translation of the SF-36. J Clin Epidemiol. 1998;51:1189–1202.
    OpenUrlCrossRefPubMedWeb of Science
  35. ↵
    DIFwithPar. [computer program]. Version 1.0. Seattle, WA: University of Washington; 2005.
  36. ↵
    1. Samejima F
    . Estimation of latent ability using a response pattern of graded scores. Psychometrika. 1969:monograph 17.
  37. ↵
    1. Linacre JM
    . A User's Guide to WINSTEPS. Chicago, IL: MESA Press; 2008.
  38. ↵
    1. Bond TG,
    2. Fox CM
    . Applying the Rasch Model. Mahwah, NJ: Lawrence Erlbaum Associates; 2001.
  39. ↵
    1. Fliege H,
    2. Becker J,
    3. Walter OB,
    4. et al
    . Development of a computer-adaptive test for depression (D-CAT). Qual Life Res. 2005;14:2277–2291.
    OpenUrlCrossRefPubMedWeb of Science
  40. ↵
    1. Crane PK,
    2. Hart DL,
    3. Gibbons LE,
    4. Cook KF
    . A 37-item shoulder functional status item pool had negligible differential item functioning. J Clin Epidemiol. 2006;59:478–484.
    OpenUrlCrossRefPubMedWeb of Science
  41. ↵
    1. Crane PK,
    2. van Belle G,
    3. Larson EB
    . Test bias in a cognitive test: differential item functioning in the CASI. Stat Med. 2004;23:241–256.
    OpenUrlCrossRefPubMedWeb of Science
  42. ↵
    1. Crane PK,
    2. Gibbons LE,
    3. Ocepek-Welikson K,
    4. et al
    . A comparison of three sets of criteria for determining the presence of differential item functioning using ordinal logistic regression. Qual Life Res. 2007;16(suppl 1):69–84.
    OpenUrlCrossRefPubMedWeb of Science
  43. ↵
    1. Crane PK,
    2. Gibbons LE,
    3. Narasimhalu K,
    4. et al
    . Rapid detection of differential item functioning in assessments of health-related quality of life: the functional assessment of cancer therapy. Qual Life Res. 2007;16:101–114.
    OpenUrlCrossRefPubMedWeb of Science
  44. ↵
    1. Crane PK,
    2. Cetin K,
    3. Cook KF,
    4. et al
    . Differential item functioning impact in a modified version of the Roland-Morris Disability Questionnaire. Qual Life Res. 2007;16:981–990.
    OpenUrlCrossRefPubMedWeb of Science
  45. ↵
    1. Crane PK,
    2. Gibbons LE,
    3. Jolley L,
    4. van Belle G
    . Differential item functioning analysis with ordinal logistic regression techniques: DIFdetect and difwithpar. Med Care. 2006;44(11 suppl 3):S115–S123.
    OpenUrlCrossRefPubMedWeb of Science
  46. ↵
    PARSCALE for Windows [computer program]. Version 4.1 Lincolnwood, IL: Scientific Software International; 2003.
  47. ↵
    Stata Statistical Software [computer program]. Release 9.2. College Station, TX: StataCorp LP; 2007.
  48. ↵
    1. Hart DL,
    2. Werneke MW,
    3. George SZ,
    4. et al
    . Screening for elevated levels of fear-avoidance beliefs regarding work or physical activities in people receiving outpatient therapy. Phys Ther. 2009;89:770–785.
    OpenUrlAbstract/FREE Full Text
  49. ↵
    1. Shrout PE,
    2. Fleiss JL
    . Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86:420–428.
    OpenUrlCrossRefPubMedWeb of Science
  50. ↵
    1. Hart DL,
    2. Connolly J
    . Pay-for-Performance for Physical and Occupational Therapy: Medicare Part B Services. Grant #18-P-93066/9-01. Washington, DC: Centers for Medicare and Medicaid Services, US Dept of Health and Human Services; 2006.
  51. ↵
    1. Schifris G
    . Immigrant Population From the USSR (Former): Selected Data 2000–2001. Jerusalem, Israel: Israeli Central Bureau of Statistics; 2004. Available at: http://www.cbs.gov.il/www/publications/ussr/intussre.pdf. Accessed April 24, 2009.
  52. ↵
    1. Nandakumar R,
    2. Roussos L
    . CATSIB: A Modified SIBTEST Procedure to Detect Differential Item Functioning in Computerized Adaptive Tests (CT-97-11). Newtown, PA: Law School Admission Council Inc; 2001.
  53. ↵
    1. Jaeschke R,
    2. Singer J,
    3. Guyatt GH
    . Measurement of health status: ascertaining the minimal clinically important difference. Control Clin Trials. 1989;10:407–415.
    OpenUrlCrossRefPubMedWeb of Science
  54. ↵
    1. Cook KF,
    2. Choi SW,
    3. Crane PK,
    4. et al
    . Letting the CAT out of the bag: comparing computer adaptive tests and an 11-item short form of the Roland-Morris Disability Questionnaire. Spine. 2008;33:1378–1383.
    OpenUrlCrossRefPubMedWeb of Science
  55. ↵
    1. Elhan AH,
    2. Oztuna D,
    3. Sutlay S,
    4. et al
    . An initial application of computerized adaptive testing (CAT) for measuring disability in patients with low back pain. BMC Musculoskelet Disord. 2008;9:166.
    OpenUrlCrossRefPubMed
  56. ↵
    1. Masters GN
    . A Rasch model for partial credit scoring. Psychometrika. 1982;47:149–174.
    OpenUrlCrossRefWeb of Science
  57. ↵
    1. Hambleton RK
    . Good practices for identifying differential item functioning. Med Care. 2006;44(11 suppl 3):S182–S188.
    OpenUrlCrossRefPubMedWeb of Science
View Abstract
Back to top
Vol 96 Issue 12 Table of Contents
Physical Therapy: 96 (12)

Issue highlights

Email
Print
Cross-Cultural Differences in Knee Functional Status Outcomes in a Polyglot Society Represented True Disparities Not Biased by Differential Item Functioning
Daniel Deutscher, Dennis L. Hart, Paul K. Crane, Ruth Dickstein
Physical Therapy Dec 2010, 90 (12) 1730-1742; DOI: 10.2522/ptj.20100107

Citation Manager Formats

Save to my folders

Cross-Cultural Differences in Knee Functional Status Outcomes in a Polyglot Society Represented True Disparities Not Biased by Differential Item Functioning
Daniel Deutscher, Dennis L. Hart, Paul K. Crane, Ruth Dickstein
Physical Therapy Dec 2010, 90 (12) 1730-1742; DOI: 10.2522/ptj.20100107
del.icio.us logo Digg logo Reddit logo Technorati logo Twitter logo CiteULike logo Connotea logo Facebook logo Google logo Mendeley logo

Subjects