Abstract
Background Implementation fidelity is poorly addressed within physical therapy interventions, which may be due to limited research on how to develop and implement an implementation fidelity protocol.
Objective The purpose of this study was to develop a feasible implementation fidelity protocol within a pilot study of a physical therapy–led intervention to promote self-management for people with chronic low back pain or osteoarthritis.
Design A 2-phase mixed-methods design was used.
Methods Phase 1 involved the development of an initial implementation fidelity protocol using qualitative interviews with potential stakeholders to explore the acceptability of proposed strategies to enhance and assess implementation fidelity. Phase 2 involved testing and refining the initial implementation fidelity protocol to develop a finalized implementation fidelity protocol. Specifically, the feasibility of 3 different strategies (physical therapist self-report checklists, independently rated direct observations, and audio-recorded observations) for assessing implementation fidelity of intervention delivery was tested, followed by additional stakeholder interviews that explored the overall feasibility of the implementation fidelity protocol.
Results Phase 1 interviews determined the proposed implementation fidelity strategies to be acceptable to stakeholders. Phase 2 showed that independently rated audio recordings (n=6) and provider self-report checklists (n=12) were easier to implement than independently rated direct observations (n=12) for assessing implementation fidelity of intervention delivery. Good agreement (79.8%–92.8%) was found among all methods. Qualitative stakeholder interviews confirmed the acceptability, practicality, and implementation of the implementation fidelity protocol.
Limitations The reliability and validity of assessment checklists used in this study have yet to be fully tested, and blinding of independent raters was not possible.
Conclusions A feasible implementation fidelity protocol was developed based on a 2-phase development process involving intervention stakeholders. This study provides valuable information on the feasibility of rigorously addressing implementation fidelity within physical therapy interventions and provides recommendations for researchers wanting to address implementation fidelity in similar areas.
Implementation fidelity has been defined as “the degree to which an intervention, treatment, or program is implemented as intended”1(p79) and helps to increase scientific confidence that changes in study outcomes are due to the influence of the intervention being investigated and not due to differences or variability in the implementation of the intervention.2 The importance of addressing fidelity within research is well established,3 and the recently published Template for Intervention Description and Replication (TIDieR) reporting guidelines (produced to improve completeness of intervention reporting for replication and implementation purposes) have emphasized the importance of addressing fidelity within clinical research.4 Despite its importance, implementation fidelity is still poorly addressed within physical therapy interventions,5 and our recent review showed that 18 of 22 included studies of physical therapy-led self-management interventions had “low” fidelity scores (<50%).6 The reasons for this finding are unclear but may be attributed to a lack of knowledge of fidelity and the practicalities and logistics of addressing it in a comprehensive and rigorous manner.7,8 Currently, there is limited guidance in the literature on the processes involved in developing and implementing a feasible and rigorous implementation fidelity protocol within complex interventions.9
In 2005, the National Institutes of Health Behavior Change Consortium (NIHBCC) developed fidelity guidance, which was updated in 2011.2,10 The 40-component framework details a combination of strategies or methods that aim to enhance, establish, and ensure fidelity (eg, intervention manuals) and to assess or monitor it (eg, direct observations) (Appendix 1). The framework categorizes implementation fidelity into 5 specific domains: study design, training of providers, treatment delivery, treatment receipt, and treatment enactment (Tab. 1).11 Developed specifically to address the fidelity of behavior change interventions in pragmatic clinical settings, the framework has been shown to have good interrater reliability10,12 and good construct validity.12 Despite its rigorous development and relevance for addressing implementation fidelity, it is still not used widely.13 Of the studies that have used it, most have not used the updated 2011 version, nor have they addressed all of the 5 domains.14–16 Furthermore, few studies have fully explained the rationale for choosing certain fidelity strategies over others where numerous options exist, limiting the translation of implementation fidelity approaches to other settings.13,17 For example, in vivo observations and self-report records are 2 potential strategies for assessing treatment delivery,3 but studies that have directly compared and contrasted these methods to inform selection are lacking.
Definitions of National Institutes of Health (NIH) Behavior Change Consortium (NIHBCC) Fidelity Framework Domainsa
The aim of the Self-management of Osteoarthritis and Low back pain through Activity and Skills (SOLAS) pilot study was to evaluate the feasibility of a group-based complex behavior change intervention to promote self-management for people with osteoarthritis (OA) of the hip or knee or chronic low back pain (CLBP) compared with usual individual physical therapy.18 Osteoarthritis was defined according to the 2014 National Institute for Health and Care Excellence criteria (ie, 45 years of age or older, activity-related joint pain, and either no morning joint stiffness or no stiffness lasting no longer than 30 minutes),19 and CLBP was defined as nonspecific low back pain of mechanical origin with or without radiation to the lower limb for 3 months or longer.20 The intervention consisted of 6 weekly 90-minute sessions delivered in person by primary care physical therapists to groups of 6 to 8 participants. Each session targeted a specific self-management behavior or skill and was structured to include an educational discussion, exercise, and facilitated goal setting. Additional materials, such as pedometers, were provided to supplement and enhance participant understanding and uptake of skills (Appendix 2).
The aim of this study was to describe the development of a comprehensive, feasible implementation fidelity protocol based on existing evidence-based fidelity guidelines. The study aimed to provide a working example of how this guidance was applied and tested within the context of the SOLAS pilot study and, in doing so, to offer further information for researchers wanting to address implementation fidelity in similar areas. The study focused on exploring the overall feasibility of applying this protocol within a research setting, with specific emphasis on the processes involved in the development of assessment strategies within the domain of treatment delivery. The development of the enhancement and assessment strategies in the remaining domains is beyond the scope of this article and will be published separately. To our knowledge, this is the first study to provide in-depth description of the practical development of a feasible implementation fidelity protocol, engaging and involving all stakeholders, in the context of a complex physical therapy intervention.
Method
The implementation fidelity protocol was developed in 2 phases (Figure), consistent with Medical Research Council guidance for complex interventions that promote the inductive, iterative processes of (1) development and (2) feasibility and pilot testing.21 First, an exploratory phase (phase 1) informed the development of an initial implementation fidelity protocol using the NIHBCC framework. Next, a testing/refinement phase (phase 2) took place that involved testing this initial fidelity protocol during the SOLAS pilot study and subsequently refining it to develop a feasible finalized implementation fidelity protocol. For clarity, physical therapist participants will be referred to as “physical therapists,” and people with CLBP or OA will be referred to as “participants” throughout this article.
Diagrammatic representation of the development of the finalized implementation fidelity protocol. IF=implementation fidelity, IFP=implementation fidelity protocol, TD=treatment delivery, NIHBCC=National Institutes of Health Behavior Change Consortium, SOLAS=Self-management of Osteoarthritis and Low back pain through Activity and Skills.
Exploratory Phase (Phase 1)
The exploratory phase aimed to explore the potential barriers to and enablers of using strategies for enhancing and assessing implementation fidelity from the perspective of both the physical therapist and the participant.
Phases 1a and 1b: qualitative studies with physical therapists and participants.
Prior to the commencement of the SOLAS pilot study, 2 focus groups (n=14 × 2) were conducted with primary care physical therapists from all geographic areas that would potentially be involved in the pilot study (phase 1a). Purposive sampling was used to recruit physical therapists who were currently providing group physical therapy classes, had previously provided group classes, or were likely to be providing group classes as part of the SOLAS pilot study, as identified by their managers. During the focus groups, strategies used to assess and enhance implementation fidelity in similar studies6 were presented to the physical therapists, followed by a semistructured discussion to explore potential barriers to and enablers of these strategies. Following the focus groups, 6 semistructured individual interviews were conducted with a convenience sample of adults with CLBP or spinal OA (as defined earlier) who had recently participated in a similar group-based primary care physical therapy intervention to promote self-management (phase 1b). Invitation letters were sent to all therapists who had completed the intervention in the preceding 6 months (n=22), identified through records by the physical therapist who had delivered the intervention because of ethical requirements. Data for both focus groups and interviews were audio recorded, transcribed verbatim, and analyzed separately using content analysis.22 This analytic approach was deemed the most appropriate for these data sets given the structured nature of the research questions and the limited depth of the responses.23
Testing/Refinement Phase (Phase 2)
Based on the findings from the exploratory phase, an initial implementation fidelity protocol was designed. The testing/refinement phase then sought to test the feasibility of this initial protocol in order to develop a refined and feasible finalized protocol. First, the treatment delivery assessment strategies were tested within the SOLAS pilot study (phase 2a), where 2 intervention sites (n=3 physical therapists, n=8 participants) participated. Subsequently, stakeholder interviews (phases 2b and 2c) were conducted to explore the overall feasibility of the full fidelity protocol in practice, including feedback on any specific enhancement or assessment strategies from any NIHBCC domain. Informal feedback from the researchers involved in this phase also was sought regarding the feasibility of the fidelity protocol from a research viewpoint (eg, conducting observations). Feasibility was addressed across 3 key areas (ie, practicality, acceptability, and implementation), as defined previously by Bowen et al.24 Successful feasibility was determined if issues pertaining to any of these areas emerged from the integrated results of the testing/refinement phase and could be easily addressed in a finalized protocol (ie, continue with modifications).25,26 Integration of mixed methods occurred at an interpretative level during narration of results27 between the qualitative physical therapist interviews (phase 2b) and the quantitative treatment delivery assessment strategies findings (phase 2a) in relation to feasibility of these strategies.
Phase 2a: treatment delivery assessment strategies.
This phase aimed to test the feasibility of 3 methods of assessing implementation fidelity within treatment delivery (ie, direct observations, audio recordings, and self-report checklists) and to ascertain the agreement of audio recordings and self-report methods with the “gold standard” approach of direct observations.11 Direct observations were conducted during all intervention sessions (n=12, 6 sessions per site) during the SOLAS pilot study. The direct observations were conducted using a checklist developed by the research team to assess the fidelity of the delivery of sessions and the treatment dose (eAppendix 1). Items on the checklists were scored as present (“yes”), absent (“no”) or “attempted/unsure.” Audio recordings were selected as a potentially more feasible alternative and were chosen over video recordings, as previous evidence suggests they are less intrusive and more feasible to implement.3 Audio recordings of half of the intervention sessions (n=6, 3 sessions per site) were completed during the pilot study and were used to evaluate fidelity retrospectively, using the same checklist as for the direct observations. Two raters (E.T. and A.K.) independently rated the audio-recorded data to give an estimate of the interrater reliability of rating implementation fidelity using the observation checklist and audio-recorded data. Self-report treatment record checklists developed by the research team were used by physical therapists at the intervention sites (n=12, 6 sessions per site) to assess self-reported fidelity in both groups (eAppendix 2). The levels of agreement between the findings of the audio recordings and the self-report with direct observations for the intervention group were analyzed using concordance (percentage level of agreement). Due to the predominance of “yes” replies within all checklists, Cohen kappa was found to be invalid and, therefore, was not applied.28,29
Phases 2b and 2c: qualitative interviews with SOLAS intervention physical therapists and participants.
Approximately 1 week after the last SOLAS intervention session, individual semistructured telephone interviews were conducted with the 3 physical therapists who had delivered the intervention (phase 2b). Within 2 weeks of the last SOLAS session, individual semistructured telephone interviews were conducted with a convenience sample of 5 people with CLBP or OA from the intervention sites of the SOLAS pilot study who were willing to participate in interviews (phase 2c). Participants were recruited at the end of the pilot study by research physical therapists who had been observing classes at intervention sites. Data from all interviews were audio recorded and transcribed verbatim. Deductive thematic analysis30 was used to analyze the interviews to enable the findings to refine the implementation fidelity protocol by coding for constructs relevant to the specific domains of the NIHBCC fidelity framework.2 Relevant units of text were summarized and coded within each interview and then grouped across interviews. Initial codes were reviewed and continually refined into more concise final themes. A reflective diary of the analytical process was kept by the primary author (E.T.), and the method for analysis was discussed and planned with another member of the research team (J.M.) a priori. Data from both sets of interviews (ie, physical therapists and participants) were initially analyzed separately, with the findings then integrated using triangulation31 to give overall feedback on the feasibility of the full implementation fidelity protocol in practice. Any findings specific to the feasibility of the treatment delivery assessment strategies were extracted from the physical therapist interview results and triangulated with quantitative findings from phase 2a following analysis.31
Role of the Funding Source
This study was funded as part of Health Research Award HRA_HSR/2012/24 by the Health Research Board of Ireland.
Results
Exploratory Phase (Phase 1)
Phases 1a and 1b: integrated findings of qualitative studies with physical therapists and participants.
The participant demographics and characteristics of the exploratory phase qualitative studies are provided in Table 2. Table 3 details the integrated findings of the studies and how they influenced the development of the initial implementation fidelity protocol. The assessment and enhancement strategies of the initial fidelity protocol that was developed as a result of the exploratory phase are detailed in Table 4.
Characteristics of Qualitative Data Collection Methods for Phase 1 (Exploratory Phase) and Phase 2 (Testing/Refinement Phase)a
Exploratory Phase Findings From Qualitative Stakeholder Data Collectiona
Initial Implementation Fidelity Protocol: Strategies to Enhance and Assess Implementation Fidelity in the Pilot Studya
Testing/Refinement Phase (Phase 2)
Phases 2a and 2b: Integrated findings of treatment delivery assessment strategies and SOLAS physical therapist interviews—feasibility of assessment strategies.
Overall levels of agreement ranged from 79.8% (between direct observations and rater 1 audio recordings) to 92.8% agreement (between direct observations and self-report), suggesting good to excellent agreement (Tab. 5).32 Interrater agreement for the audio recordings was 82.3%. Of the sections of the intervention, “Introduction/Recap and Review” and “Review and Planning” (eg, goal setting) had the lowest agreement among all 3 methods.
Testing/Refinement Phase 2a: Agreement Among Treatment Delivery Assessment Strategiesa
A minor issue emerged regarding the implementation of assessment strategies in relation to the checklists' scoring system, as the “unsure/attempted” option lacked clarity due to the difference in meaning between “attempted” and “unsure.” Direct observation was the most comprehensive method for assessing fidelity of treatment delivery, consistent with its gold standard status, as the audio recordings were unable to detect the performance of certain items on the checklist, such as “room setup for exercise,” and one audio recording was unusable due to technical issues. However, in terms of implementation, direct observations were time-consuming and resource-intensive, and the self-report forms and audio recordings were found to be more practical for researchers. In the testing/refinement phase interviews (phase 2b), the physical therapists felt that the 3 assessment strategies were acceptable and raised no concerns regarding their implementation. One physical therapist, however, suggested that direct observations may be more intrusive for participants, stating, “More for the patients than anything else. I think that they felt there was an awful lot of people in the room watching” (physical therapist 3, site B).
For assessing treatment delivery in the finalized implementation fidelity protocol, therefore, obtaining self-report checklists and audio recordings of all intervention sessions was proposed because of their good agreement with the gold standard and to directly observe 24 randomly selected sessions from across all intervention sites. A sample of 24 sessions was chosen, as it has been previously shown to be the minimum number needed for initial instrument development within pilot studies.33 Based on feedback from the testing/refinement phase, we decided to refine the scoring system of the checklists for the finalized fidelity protocol, changing the option for “unsure/attempted” on all checklists to “attempted.” As agreement within the scoring of “Introduction/Recap and Review” and “Review and Planning” sections had been low, we decided to ensure that the structure, aims, and strategies relevant to these sections (eg, adequate goal setting) would be clarified further with physical therapists during physical therapist training and with the raters of the audio-recorded data prior to completion of rating in the finalized implementation fidelity protocol.
Phases 2b and 2c: integrated findings of qualitative interviews with SOLAS physical therapist and participants—overall feedback on implementation fidelity protocol.
The participant demographics and characteristics of the testing/refinement phase qualitative interviews are detailed in Table 2. Overall, the stakeholder feedback obtained regarding the specific SOLAS enhancement and assessment strategies within the 5 domains showed the implementation fidelity protocol to be feasible from both physical therapist and participant perspectives. In terms of acceptability, participants found the intervention materials and resources that comprised strategies to enhance treatment receipt/enactment very useful, particularly the pedometer, but they were unsure of whether they would continue using activity diaries as self-monitoring tools in the long term. However, as a potential strategy of assessing treatment receipt, participants felt that the collection of activity diaries by researchers would be acceptable: “No, I wouldn't have minded that [collection of activity diaries] at all. As I say, I would have only filled in, maybe filled in 1 or 2, because I wouldn't have been there for the some of them. But no, I wouldn't have minded” (participant 1, site B). The participant interviews themselves also were found to be a useful and acceptable means of further assessing treatment receipt and enactment, as participants spoke about an enhanced knowledge of their condition and pain management skills and of increasing physical activity levels and use of pain management strategies since completing the program.
Regarding the practicality of the implementation fidelity protocol, physical therapists reported minor technical issues surrounding relaxation CDs and access to the projectors used to deliver the education component (treatment delivery enhancement strategies). One physical therapist remarked, “It's just the hassle, if you like, of setting up the PowerPoint [Microsoft Corp, Redmond, Washington], and I had to get used to that” (physical therapist 1, site A). In terms of implementation of the fidelity protocol strategies, physical therapists felt that they delivered the SOLAS intervention with good levels of fidelity but that the goal-setting section was challenging. One physical therapist also felt that the fidelity of delivery had been adversely affected by the amount of time between the therapist's training and the intervention start: “The training was completed a little bit earlier, and there was a bit of a gap then…. Things were a lot fresher in my head after the initial training…. It was a good bit earlier than the start of the program” (physical therapist 2, site B).
Subsequent to this feedback, it was deemed necessary to ensure that, in the future, any intervention materials and equipment should be carefully tested in each site in advance to avoid any technical issues, and the time between training and delivery would need to be considered. Compounding the earlier findings regarding treatment delivery assessment strategies, it was felt that appropriate and adequate goal setting would be of paramount importance in the finalized implementation fidelity protocol. For assessing treatment receipt and treatment enactment in the finalized fidelity protocol, we decided to assess participants' activity diary use at the end of the 6-week intervention and to conduct further participant interviews.
Testing/refinement phase output—finalized implementation fidelity protocol.
Based on the findings of the testing/refinement phase, a feasible finalized implementation fidelity protocol was developed that addresses each component of the NIHBCC framework. The finalized implementation fidelity protocol, structured according to domain, is shown in the eTable.
Discussion
This article describes the development of a feasible implementation fidelity protocol using the SOLAS pilot study as a vehicle for its development. The article provides a working example of each component of the updated NIHBCC fidelity framework addressed in a complex behavior change intervention and is one of the first articles to explore pragmatic issues of implementation fidelity from all stakeholder perspectives.
A key strength of this article is that it evaluates the feasibility and appropriateness of multiple strategies for assessing implementation fidelity within the domain of treatment delivery. In a special series report on implementation research within physical therapy research, Huijg et al5 highlighted the need for multiple methods of data collection in order to comprehensively address implementation fidelity. Ideally, interventions should aim for gold standard methods (eg, direct observations).11,16 However, implementation fidelity assessment strategies ought to be tailored to the intervention in question34; therefore, the appropriateness and feasibility of these gold standard measures may need to be explored, and alternative methods (eg, provider self-report, audio recordings) should be concurrently evaluated for suitability in context and feasibility, as shown in this study.
The use of qualitative data collection involving all intervention stakeholders is an important aspect of implementation fidelity35 and is crucial in developing an acceptable implementation fidelity protocol. By collecting qualitative data in the exploratory phase, we identified and explored potential barriers to implementation fidelity strategies consistent with factors previously identified to influence fidelity, such as participant characteristics,36 time constraints,7,37,38 and availability of resources.39 This approach allowed us to develop a more feasible initial fidelity protocol from the outset, thus reducing potential waste of research resources. The interviews completed in the testing/refinement phase provided valuable feedback regarding the overall feasibility of implementing the fidelity protocol and the enhancement and assessment strategies that had been used within each domain of the implementation fidelity protocol. To our knowledge, no other studies have both prospectively and retrospectively explored the acceptability of implementation fidelity strategies to stakeholders to develop an implementation fidelity protocol.
This study, to our knowledge, is also the first to develop an implementation fidelity protocol through a 2-phase process of testing and refinement. In a recent editorial, Vernooij-Dassen and Moniz-Cook35 highlighted the need for more interventions that use feasible implementation fidelity protocols to plan for and address implementation fidelity from the outset. Previously, Poltawski et al39 described their experience of addressing fidelity in the development of a clinical stroke rehabilitation trial, using the NIHBCC framework to structure their work. Although their mixed-methods study explored implementation fidelity in more depth than previous work, the study focused more on how the protocol was applied, rather than how the protocol was developed. The authors acknowledged the lack of further testing of the protocol as a limitation of their study, for which future research was indicated. We believe the 2-phase process described in the current study further strengthens the comprehensiveness of the finalized fidelity protocol.
There were some limitations to the current study. First, the assessment checklists have been developed to be specific to the intervention in question, and their reliability and validity have yet to be thoroughly tested, which may limit the internal validity of the study. However, we have attempted to address criterion (concurrent) validity of the audio recordings and self-report checklists by assessing their agreement with the gold standard of a direct observation checklist, which was developed to address all intervention components. Second, the use of blinded raters would enhance the assessment of treatment delivery; however, this was not possible in the current study due to lack of resource availability. Third, the physical therapist and participant samples used in this study were specific to the SOLAS pilot study inclusion and exclusion criteria, and the intervention took place in a primary care setting, which may limit the generalizability of findings. Nonetheless, this study contains pragmatic information about implementation fidelity that is applicable across a variety of research contexts and will facilitate how fidelity is addressed in future complex interventions.
Key “Take-Home” Points
The following points summarize how and when researchers may best use the findings of this study. First, the finalized implementation fidelity protocol (eTable) can be used as a pragmatic and feasible example of how to address each component of the NIHBCC framework within complex interventions—thus ensuring a comprehensive approach to addressing implementation fidelity in future research. The finalized implementation fidelity protocol can be used during the planning and development stages of a complex intervention to plan for addressing implementation fidelity in a research study. Second, the assessment checklists (eAppendixes 1 and 2) can be used by researchers as templates for developing similar treatment delivery fidelity assessment checklists. Finally, the finalized implementation fidelity protocol can be used as an aid in evaluating the implementation fidelity of similar complex interventions with the knowledge that this is an example that has been tested and found to be feasible for use in a research setting.
In conclusion, this article describes the development of a comprehensive implementation fidelity protocol within the context of a complex physical therapy intervention. Future work will apply the finalized protocol to a randomized controlled feasibility trial and explore the factors influencing implementation fidelity results.18 This article contributes much-needed guidance on the feasibility of addressing fidelity in complex intervention, and findings can be used to enhance how implementation fidelity is addressed in physical therapy research in addition to other research fields.
Appendix 1.
National Institutes of Health Behavior Change Consortium (NIHBCC) Fidelity Frameworka
a Adapted with permission from: Bellg AJ, Borrelli B, Resnick B, et al. Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH Behavior Change Consortium. Health Psychol. 2004;23:443–451.
Appendix 2.
Structure of Self-Management of Osteoarthritis and Low Back Pain Through Activity and Skills (SOLAS) Intervention
Footnotes
All authors provided concept/idea/research design. Ms Toomey and Dr Hurley provided writing and project management. Ms Toomey and Dr Guerin provided data collection. Ms Toomey provided data analysis. Dr Hurley provided facilities/equipment. Dr Matthews and Dr Hurley provided consultation (including review of manuscript before submission). The authors thank Alison Keogh for her help in independently rating of audio-recorded data.
Ethical approval for the research was granted by the University College Dublin Human Research Ethics Committee.
This study was funded as part of Health Research Award HRA_HSR/2012/24 by the Health Research Board of Ireland.
- Received August 14, 2015.
- Accepted February 24, 2016.
- © 2016 American Physical Therapy Association
References
Issue highlights
