Abstract
Background People with lower limb amputations frequently have impaired balance ability. The Berg Balance Scale (BBS) has excellent psychometric properties for people with neurologic disorders and elderly people dwelling in the community. A Rasch analysis demonstrated the validity of the BBS for people with lower limb amputations of all ability strata, but rater reliability has not been tested.
Objective The study objective was to determine the interrater reliability and intrarater reliability of BBS scores and the differences in scores assigned by testers with various levels of experience when assessing people with lower limb amputations.
Design This reliability study of video-recorded single-session BBS assessments had a cross-sectional design.
Methods From a larger study of people with lower limb amputations, 5 consecutively recruited participants using prostheses were video recorded during an in-person BBS assessment. Sixteen testers independently rated the video-recorded assessments. Testers were 3 physical therapists, 1 occupational therapist, 3 third-year and 4 second-year doctor of physical therapy (DPT) students, and 5 first-year DPT students without clinical training. Rater reliability was calculated using intraclass correlation coefficients (ICC [2,k]). Differences in scores assigned by testers with various levels of experience were determined by use of an analysis of variance with Tukey post hoc tests.
Results The average age of the participants was 53.0 years (SD=15.7). Amputations had occurred at the ankle disarticulation, transtibial, and transfemoral levels because of vascular, trauma, and medical etiologies an average of 8.2 years earlier (SD=7.9). Berg Balance Scale scores spanned all ability strata. Interrater reliability (ICC [2,k]=.99) and intrarater reliability of scores determined in person and through video-recorded assessments by the same testers (ICC [2,k]=.99) were excellent. For participants with the lowest levels of ability, licensed professionals assigned lower scores than did DPT students without clinical training.
Limitations Intrarater reliability calculations were based on 2 testers.
Conclusions Berg Balance Scale scores assigned to people using prostheses by testers with various levels of clinical experience had excellent interrater reliability and intrarater reliability.
Although many factors can affect the gait of people with lower limb amputations, balance was the only physical capacity measure shown to have a strong relationship with gait performance in a systematic review of 48 studies.1 Instrumented measures of postural control assess the physiologic systems of balance at work in people after lower limb loss2 but are not feasible in most clinical settings. Clinical assessments of balance abilities have often been limited to self-report measures, such as the Prosthetic Evaluation Questionnaire, which are open to subjectivity and may not accurately reflect physical ability.3,4 Clinical performance measures of prosthetic and mobility functions, such as the Amputee Mobility Predictor, incorporate but do not specifically assess balance ability.5
The performance-based clinical assessment of balance ability most commonly used by rehabilitation practitioners is the Berg Balance Scale (BBS),6 which consists of 14 items that challenge functional balance ability. Each item is rated on a 5-point scale from a lowest level of 0 to a highest level of 4, and a cumulative score is reported.7 The psychometric properties of the BBS have been examined in people with neurologic conditions, and moderate intrarater reliability and excellent interrater reliability have been reported.6 Interrater reliability and intrarater reliability also have been consistently excellent for older adults dwelling in the community; the latter population is similar to people who have lower limb loss and dwell in the community—a population that includes many older adults.8–12
The BBS has been used to assess balance in a general population of patients undergoing rehabilitation for various disorders, including patients with lower limb amputation.13 The few studies that have used the BBS to examine balance ability specifically in people with lower limb amputation have been limited to small samples of 12 or fewer people in specific groups of athletes,14 young people after traumatic amputations,15 or elderly people with vascular amputations.16 Recently, a larger sample was used in a Rasch analysis that demonstrated the internal validity of the BBS in people who dwelled in the community and had various etiologies and levels of lower limb amputations, prosthetic use, and balance ability.17 This Rasch analysis identified 4 distinct levels of balance ability, from lowest (I) to highest (IV), based on BBS total scores,17 that could be used to determine the risk of falls or poor prosthetic gait performance. For balance ability strata I, II, III, and IV, the BBS total scores ranged from 0 to 33, 34 to 49, 50 to 53, and 54 to 56, respectively.
However, the rater reliability of the BBS as a clinical assessment of balance in people with lower limb amputation has remained untested. Inconsistent patient performance because of common problems such as pain, fatigue, and poor prosthetic fit as well as fluctuating blood pressure, blood glucose levels, and integument conditions18 can complicate test-retest reliability in this medically vulnerable population. The aims of this methodological study were to use video recordings of single-session clinical balance assessments of people who used prostheses to determine the interrater reliability of BBS scores, the intrarater reliability of BBS scores assigned on the basis of in-person and video-recorded assessments, and differences in BBS scores assigned by testers with various levels of experience. It was hypothesized that both the interrater reliability and the intrarater reliability of BBS scores for people who used prostheses would be similar to those found in other populations and that some differences would be apparent between testers with clinical experience and those without.
Method
Five consecutively enrolled participants from a larger cohort of people who had lower limb amputations and who had undergone balance assessment as part of an ongoing longitudinal study provided separate informed consent for optional video recording of their BBS assessments. Inclusion criteria for the longitudinal study included people who dwelled in the community and had lower extremity amputations of any level or etiology, whether bilateral or unilateral. Exclusion criteria included medical issues affecting balance, such as uncontrolled blood pressure, past stroke, vestibular dysfunction, or blindness. Thus, people dwelling in the community, of either sex or any race, and with unilateral or bilateral lower limb amputations of any level or etiology were included. Participants who did not use a prosthetic leg were excluded. The number of participants exceeded the 2 or 3 considered optimal for a design with 10 testers, based on an alpha value of .05 and a power of 80% for expected observed reliability coefficients of greater than .9.19
Procedure
The single in-person balance assessments were video recorded by an assistant using a handheld iPhone 3GS-4 (Apple, Cupertino, California; http://www.apple.com/iphone/compare-iphones/) to allow both the participant and the assessor to remain in the camera frame at all times, even as they moved. The BBS was scored independently during this single session by 2 in-person raters: 1 licensed physical therapist, who scored participants B, D, and E, and a third-year doctor of physical therapy (DPT) student, who scored participants A and C. The BBS was scored by use of the standard 14 tasks.7 All 5 participants performed each task using their prosthesis but without any other assistive device. In accordance with the method validated for the use of the BBS in people with lower limb amputations, tasks that could be performed in ways that emphasized or minimized prosthetic leg use were performed both ways, and the best score was recorded.17 Thus, standing on 1 leg was attempted on both limbs, although it was performed more easily on the sound limb. Standing with 1 foot in front was also attempted both ways; the participants demonstrated a variety of strategies, including leaning back with the sound limb in back or leaning forward with the sound limb in front.17 The video recordings were later downloaded to a research desktop computer for viewing and scoring.
Testers involved in the ongoing longitudinal study were invited to participate in the present study, and all but 1 participated. All 16 testers trained with the author in a 1-hour session that included individual viewing and scoring of a separate video-recorded BBS assessment of another person who had limb loss but was not included in the study analysis, followed by a small-group discussion of the scoring. Of the 16 testers, 11 had clinical experience: 4 licensed allied health professionals (3 physical therapists and 1 occupational therapist), 3 third-year DPT students, and 4 second-year DPT students. The 11 testers had general clinical experience as well as specific hands-on experience assessing balance by use of the BBS for people with limb loss. The licensed professionals had at least 1 year of general clinical practice experience, the third-year DPT students had completed 2 full-time clinical internships (18 weeks), and the second-year DPT students had completed 1 full-time clinical internship (8 weeks). The other 5 testers were first-year DPT students who had not completed a formal clinical training internship or did not have hands-on clinical experience with people who had lower limb amputations; they assessed the video recordings as part of their clinical research training. After a period of 4 to 6 weeks, a time interval deemed sufficient to exclude accurate recall by the 2 in-person testers of their earlier ratings,20 the video recordings of all 5 participants performing the BBS tasks were individually viewed and independently scored20 by the 16 testers with various levels of clinical experience.21
Utilizing the methods of an interrater and intrarater reliability study with video-recorded assessments,20 the scores assigned by all testers were used to determine the interrater reliability and the differences in the scores assigned by the testers with various levels of experience. The scores assigned in the initial in-person assessments and the scores assigned in the video-recorded assessments by the same tester were used to determine the intrarater reliability of the BBS total scores without the participant performance variability that would occur with a clinical retest.
Data Analysis
The BBS scores were totaled for analysis. The scores on each BBS task also were recorded for a separate analysis. In addition, the average total scores were calculated for raters with various levels of experience: the licensed therapists and the DPT students with different levels of clinical training. For determination of the interrater reliability of the 16 testers, the 2-way random intraclass correlation coefficient (ICC [2,k]) and the 95% confidence interval (CI) were calculated for BBS total and item scores.22 The intrarater reliability of the in-person raters scoring the BBS on the basis of in-person and video-recorded assessments also was calculated, with the ICC (2,k) value reported. For determination of differences in BBS total scores assigned by testers with various levels of experience, mean BBS scores and the scores at each stratum were compared by use of an analysis of variance with Tukey post hoc tests. All statistical analyses were performed with PASW Statistics 18.0 for Mac (SPSS-UK Ltd, Surrey, United Kingdom).
Role of the Funding Source
This research was supported, in part, by grant 1 R49 CE002096 from the National Center for Injury Prevention and Control, Centers for Disease Control and Prevention, to the Center for Injury Epidemiology and Prevention at Columbia University. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the Centers for Disease Control and Prevention. The funding source had no impact on the design, conduct, or reporting of this study.
Results
The average age of the 5 participants was 53.0 years (SD=15.7). Amputations had occurred an average of 8.2 years earlier (SD=7.9), and the participants wore their prostheses at least 4 h/d (Tab. 1). Their BBS total scores from the single in-person assessments (4, 42, 53, 56, and 56) fell within the 4 distinct ability strata ranging from lowest (I) to highest (IV).17
Characteristics of Participantsa
The 11 testers with clinical experience fell into 3 categories: experienced licensed professionals, student clinicians with clinical experience, and students without clinical experience. The 4 licensed professionals had an average of 8 years of experience (range=1–19 years) assessing balance in people with lower limb amputations. Their primary clinical work environments were acute care hospital (1 tester), acute rehabilitation (2 testers), and outpatient (1 tester) settings. In addition to hands-on experience assessing balance in people with lower limb amputations, the second- and third-year DPT students had completed clinical internships; the first-year DPT students had no hands-on clinical training (Tab. 2).
Number of Raters and Rater Clinical Experiencea
The interrater reliability of the BBS total scores assigned by all 16 testers was excellent (ICC=.99, 95% CI=.99 to 1.00). The interrater reliability (ICCs) of each of the BBS item scores ranged from .72 to 1.00, except for unsupported sitting (Tab. 3).
Interrater Reliability of Each Berg Balance Scale Item Score Assigned by All Testers to All Participantsa
The intrarater reliability of the BBS total scores assigned by the same 2 testers during the initial in-person and the follow-up video-recorded assessments was excellent (ICC=.99, 95% CI=.96 to 1.00) for single measures analyzed for absolute agreement.
There were no differences between testers at all experience levels in the mean BBS total scores for the 5 participants (P>.05). A comparison (by use of an analysis of variance) of the mean scores from all testers for participants in each ability stratum demonstrated a significant difference between tester experience levels (P<.05); the BBS scores assigned by the licensed professionals were significantly lower than the scores assigned by the first-year DPT students to participants in the 2 lower-ability strata (stratum I: P=.04, SE=2.41, 95% CI=−14.75 to −0.45; stratum II: P=.04, SE=1.73, 95% CI=−10.48 to −0.22) (Figure).
Comparison of mean Berg Balance Scale (BBS) scores assigned to participants with lower levels of ability (strata I and II) by raters with different levels of experience and in-person BBS scores (for reference). The mean BBS scores assigned by licensed physical therapists (PT)/occupational therapists (OT) and first-year doctor of physical therapy (DPT) students were significantly different (P=.04) for participants in strata I and II.
However, analysis of each item scored for the participant in stratum I (lowest level of ability) revealed that only item 1 (sit to stand) and item 5 (transfer to a chair) were scored significantly differently by testers with different levels of experience (P=.01 and P=.03, respectively). No individual items were scored differently by the testers for participants in strata II to IV (P>.05).
Discussion
Because balance impairment is common and is an important determinant of gait performance in people with lower limb amputations,1 a reliable measure of balance ability may help clinicians assess functional gait potential and fall risk. The results of the present study of people who had lower limb amputations and used prostheses suggested that the interrater reliability of BBS total scores was comparable to that reported for other populations with balance impairments (ICCs=.87–.99).6,8–12 For people with lower limb amputations, who are vulnerable to medical problems, functional ability can be affected by pain, fatigue, and prosthesis fitting problems as well as fluctuating blood pressure, blood glucose levels, and integument conditions.18 When fluctuating health conditions cause performance to vary, as in people with lower limb amputations, the interrater reliability of a measure may provide more useful information than the intrarater reliability.21 For participants in the present study, who used prosthetic legs and whose balance ranged from the lowest-ability stratum to the highest-ability stratum, the BBS demonstrated excellent interrater reliability (ICC=.99), similar to that for other populations of patients.6,8–12
Interrater reliability was derived from scores obtained from a single assessment by multiple testers and may have varied with the level of experience of the testers. The interrater reliability of BBS scores for participants in the present study was similar to that in previous studies with experienced, novice, or student BBS raters. One study in which BBS raters with various levels of clinical experience assessed people with Parkinson disease revealed excellent interrater reliability among 2 licensed physical therapists and 1 DPT student (ICC=.93).23 All 3 testers had read the directions, had practiced scoring video-recorded assessments, and had clinical experience with the BBS,23 similar to the testers in the present study. Excellent interrater reliability (ICC=.99) also was demonstrated in a study in which 10 physical therapists who had recently graduated and 8 experienced physical therapists scored single-session BBS assessments of elderly people.21 Likewise, in the present study, the 16 testers, including experienced professionals and DPT students with and without clinical training, scored the video-recorded BBS assessments with excellent interrater reliability.
An important disadvantage of using video-recorded assessments for scoring is that such assessments inherently lack the physicality of in-person assessments. Nevertheless, scores for physical tests such as the Action Reach Arm Test of upper extremity function in people with stroke have been obtained from video-recorded assessments.20 Excellent intrarater reliability was demonstrated when scores from in-person assessments and video-recorded assessments obtained 4 to 6 weeks later were compared (ICC>.98), and interrater reliability values were similar.20 When a methodology similar to that used for the Action Reach Arm Test was used in the present study,20 the intrarater reliability for absolute agreement of in-person and video-recorded BBS scores assigned by the same tester was also excellent (ICC=.99). Similar BBS intrarater reliability coefficients have been found for older adults dwelling in the community (ICCs=.95–.98)8–12 and for people with neurologic conditions (ICC>.85).6
An advantage of video-recorded assessments is that rater reliability can be determined without being confounded by variations in participant performance.20 When test-retest reliability—a common form of intrarater reliability—is calculated from 2 separate sessions, the assessment scores reflect both rater reliability and participant performance variability. Fluctuating health conditions that affect participant performance ability can confound results and lead to an underestimation of rater reliability, as in a study of people with Parkinson disease.23 Inconsistent participant performances in 2 separate assessment sessions yielded test-retest reliability coefficients for student physical therapists (ICC=.79) and licensed physical therapists (ICC=.80) that were lower than the interrater reliability coefficient for all testers in a single assessment (ICC=.93).23 The results of the present study demonstrated that BBS scores not affected by variations in participant performance can be obtained from video-recorded assessments with excellent interrater reliability and intrarater reliability.
Although the rater reliability of the BBS for assessing participants' balance in the present study was excellent, some differences in BBS scores among testers with various levels of experience were expected. However, no difference in the mean BBS total scores assigned to 5 participants by 16 testers with various levels of experience was observed (P>.05). This finding was consistent with that of a previous study of elderly people in which no difference in mean total scores assigned by 18 novice and experienced testers was observed.21 Consideration of mean scores for a group of participants whose abilities vary from the highest level to the lowest level may not reveal potential differences in participant scores in each stratum.
The BBS total scores assigned to participants in each balance ability stratum by testers with clinical experience and testers without clinical experience differed. For participants in the lower-ability strata (I and II), licensed professionals assigned lower scores than did testers without clinical training. Lower scores often require clinical judgment of the assistance level provided, a task that may be difficult without clinical training. The use of video-recorded assessments to evaluate physical ability may make such judgments more difficult. Visual assessment of mean BBS total scores for each tester experience level further suggested this possibility; although scores for participants in the 2 higher-ability strata appeared to be similar to those assigned through in-person assessments regardless of tester experience, scores assigned to participants in the 2 lower-ability strata by testers with less experience were increasingly higher than scores assigned through in-person assessments (Figure).
Differences in scores assigned by licensed professionals and students without clinical experience were apparent for the participant in the lowest-ability stratum (I) on 2 particular tasks: sit to stand and transfer to a chair. Conversely, participants in higher-ability strata were more likely to perform the tasks without assistance and therefore may have been more consistently graded on the basis of video recordings. No difference between the groups of testers was apparent for any BBS tasks performed by participants in strata II to IV (P>.05).
Although reliable measurements can be obtained, scores assigned by students without clinical experience on the basis of video recordings should be viewed with caution. The nonphysical experience of video-recorded assessments may make judging the degree of physical ability difficult. This difficulty may be especially true for the most vulnerable people, with the lowest level of ability, or when assessors have no experience with hands-on balance assessment of people who have lower limb amputations and use prostheses. In the present study, the least reliable BBS item to score was item 3 (sit unsupported) (Tab. 3). Although the task was easy for all participants, body shifting, incidental arm contact with the chair, or imprecise timing within the 2 minutes allowed led to unexpected score variability.
Reliable scoring of video-recorded assessments may create new opportunities. Video-recorded assessments can be scored by an unlimited number of testers not constrained by a specific time or place. It is conceivable that video recordings of BBS tasks being performed could be posted on the Internet and viewed and scored asynchronously by trained rehabilitation therapists for people who do not have physical access to a therapist with experience assessing balance in people with lower limb amputations. The confounding effects of untrained people following directions and assisting patients in the performance of BBS tasks and the impact on test accuracy would have to be explored.
Limitations
Although the sample of participants in the present study was more than sufficient for an overall analysis of the interrater reliability of BBS total scores, only the highest-ability stratum (IV) was represented by the recommended minimum of 2 participants. Therefore, further research is needed to determine reliability for each ability stratum. The present research was limited to people using prostheses; additional study of the reliability of the BBS for people not using prostheses is needed for application to all people with lower limb loss. In addition, the intrarater reliability was determined from the in-person and video-recorded assessments by only the 2 in-person testers. Study limitations also included the use of a handheld video recording device. Because the present study did not include a detailed motion analysis, it was deemed sufficient for the video-recorded segments to include both the participant's body and the tester's body for the analysis of balance ability. However, handheld video recording may not have resulted in optimal video quality. Neither the length of time allowed for review and scoring of video-recorded BBS assessments nor environmental distractions that could have varied among testers and video reviewing sessions were controlled.
Conclusions
The present study demonstrated excellent interrater reliability and intrarater reliability among testers who had various levels of clinical experience and used video recordings to assess the balance of people with lower limb amputations with the BBS. The results were comparable to those found for other populations, and the study hypotheses were accepted. There was no difference in mean BBS total scores among testers with various levels of experience. However, the participant in the lowest-ability stratum received a lower score from licensed professionals than from DPT students with no clinical experience. Future study of the usefulness of the BBS as an indicator of fall risk or as a measure of rehabilitation outcomes for people with lower limb amputations is now recommended.
Footnotes
This study was approved by the Institutional Review Board of the Columbia University Medical Center.
This research was supported, in part, by grant 1 R49 CE002096 from the National Center for Injury Prevention and Control, Centers for Disease Control and Prevention, to the Center for Injury Epidemiology and Prevention at Columbia University. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the Centers for Disease Control and Prevention.
- Received May 6, 2013.
- Accepted October 1, 2013.
- © 2014 American Physical Therapy Association
References
Issue highlights
