Reliability of the ECHOWS Tool for Assessment of Patient Interviewing Skills

Jill S. Boissonnault; Kerrie Evans; Neil Tuttle; Scott J. Hetzel; William G. Boissonnault

doi:10.2522/ptj.20150172

Abstract

Background History taking is an important component of patient/client management. Assessment of student history-taking competency can be achieved via a standardized tool. The ECHOWS tool has been shown to be valid with modest intrarater reliability in a previous study but did not demonstrate sufficient power to definitively prove its stability.

Objective The purposes of this study were: (1) to assess the reliability of the ECHOWS tool for student assessment of patient interviewing skills and (2) to determine whether the tool discerns between novice and experienced skill levels.

Design A reliability and construct validity assessment was conducted.

Methods Three faculty members from the United States and Australia scored videotaped histories from standardized patients taken by students and experienced clinicians from each of these countries. The tapes were scored twice, 3 to 6 weeks apart. Reliability was assessed using interclass correlation coefficients (ICCs) and repeated measures. Analysis of variance models assessed the ability of the tool to discern between novice and experienced skill levels.

Results The ECHOWS tool showed excellent intrarater reliability (ICC [3,1]=.74–.89) and good interrater reliability (ICC [2,1]=.55) as a whole. The summary of performance (S) section showed poor interrater reliability (ICC [2,1]=.27). There was no statistical difference in performance on the tool between novice and experienced clinicians.

Limitations A possible ceiling effect may occur when standardized patients are not coached to provide complex and obtuse responses to interviewer questions. Variation in familiarity with the ECHOWS tool and in use of the online training may have influenced scoring of the S section.

Conclusion The ECHOWS tool demonstrates excellent intrarater reliability and moderate interrater reliability. Sufficient training with the tool prior to student assessment is recommended. The S section must evolve in order to provide a more discerning measure of interviewing skills.

Footnotes

All authors provided concept/idea/research design and contributed to writing. Dr J. Boissonnault, Dr W. Boissonnault, Dr Evans, and Dr Tuttle provided data collection. Dr Tuttle, Mr Hetzel, Dr J. Boissonnault, and Dr W. Boissonnault provided data analysis. Dr J. Boissonnault provided project management and fund procurement. Dr J. Boissonnault and Dr Tuttle provided participants. Dr J. Boissonnault, Dr W. Boissonnault, Dr Evans and Dr Tuttle provided facilities/equipment. All authors provided institutional liaisons and consultation (including review of manuscript before submission).
The authors acknowledge the assistance of Nicholas Conte, DPT, LAT, for his help in development of the ECHOWS training module, and Sarah Stream, MS, DPT, for her assistance in data input. Additionally, they recognize and thank the University of Wisconsin–Madison and Griffith University students, and the US and Australian faculty and clinicians who assisted the project as participants and assessors.
The study received UW–Madison Social Science Institutional Review Board approval and Griffith University Institutional Review Board approval.
Research funding was provided by a grant from the Department of Orthopedics and Rehabilitation of the University of Wisconsin–Madison School of Medicine and Public Health.