Abstract
The knowledge-to-practice gap in rehabilitation has spurred knowledge translation (KT) initiatives aimed at promoting clinician behavior change and improving patient care. Online KT resources for physical therapists and other rehabilitation clinicians are appealing because of their potential to reach large numbers of individuals through self-paced, self-directed learning. This article proposes best practice recommendations for developing online KT resources that are designed to translate evidence into practice. Four recommendations are proposed with specific steps in the development, implementation, and evaluation process: (1) develop evidence-based, user-centered content; (2) tailor content to online format; (3) evaluate impact; and (4) share results and disseminate knowledge. Based on KT evidence and instructional design principles, concrete examples are provided along with insights gained from experiences in creating and evaluating online KT resources for physical therapists. In proposing these recommendations, the next steps for research are suggested, and others are invited to contribute to the discussion.
The knowledge-to-practice gap in rehabilitation has spurred the development of knowledge translation (KT) initiatives aimed at changing clinician behaviors and improving patient care.1,2 Knowledge translation is defined as “a dynamic and iterative process that includes synthesis, dissemination, exchange and ethically-sound application of knowledge to improve the health of Canadians, provide more effective health services and products and strengthen the health care system.”3 Strong interest exists in determining the most effective methods to transfer knowledge to clinical practice.4–6 For physical therapists, active strategies involving multiple components, rather than more passive approaches, have been most effective at improving evidence-based behaviors.7
The Internet can be an important platform for KT initiatives. E-learning refers to technology or Web-based education or training.8 The appeal of e-learning lies in its potential to guide large numbers of individuals through self-paced, self-directed learning of up-to-date multimedia content.9,10 A primary application of e-learning in physical therapy has been within professional (entry-level) and postgraduate continuing education.11–14 In contrast, we define online KT resources as e-learning products that translate evidence-based knowledge to disseminate information that increases awareness, informs clinical practice, and stimulates practice change.
Online KT resources can take many forms of varying complexities, including (but not limited to): websites, downloadable PDFs, blogs, wikis, chat forums, groups, educational modules, really simple syndication (RSS) feeds, podcasts, tags, social bookmarks, and social networking applications.9,11,15,16 Mairs et al17 undertook a systematic review of online KT strategies to promote interaction and knowledge exchange in health care and found that online KT can be a cost-effective way to link different stakeholders for timely and relevant information exchange, especially across geographic boundaries, but that challenges exist with access and users' knowledge of technology. Although online KT resources have been found to have a positive impact on self-reported knowledge and skills, few studies have documented their impact on therapist behavior change or patient outcomes.10 Stroke Engine,18 which is recognized for its rigor as a source for evidence-based content related to stroke rehabilitation, is an example of an online KT resource that has been documented as effecting changes in practice.19,20
Researchers and clinicians looking to develop and evaluate online KT resources must consider the relative complexities involved in creating different types of tools and may benefit from key information in several areas. First, how can theory inform these processes? A KT theoretical framework is recommended to inform decisions relating to different stages of the KT process.21,22 The knowledge-to-action (KTA) cycle22 is one such framework that guides researchers in the process of creating and implementing KT initiatives, including online strategies,20 that target user needs, incorporate evaluation, and promote the sustainability of knowledge use (Fig. 1).
The knowledge-to-action framework. Reprinted with permission of John Wiley & Sons from: Straus SE, Tetroe J, Graham ID, eds. Knowledge Translation in Health Care: Moving From Evidence to Practice. 2nd ed. Chichester, United Kingdom: John Wiley & Sons Ltd; 2013.
Second, how can evidence-based KT strategies be incorporated into online KT development? “Integrated” KT, which engages stakeholders or end users from the beginning of the process, is one strategy that can increase KT effectiveness and sustainability.3 Usability testing of the online resources is a critical component of this partnership.19 Additionally, an awareness of the barriers to and facilitators of practice change can assist in targeting actions to maintain change over time.20 Multicomponent active KT strategies also are encouraged,1,23 but the multiple components must be described.4,23,24 Each of these considerations can contribute to the selection of an appropriate online format for KT.
Finally, how can principles of instructional design inform online KT resource development? Instructional design principles provide concrete systematic guidance to inform the development of well-planned, effective, high-quality instruction that is replicable.25,26 Examples of these principles include tailoring content to the learner, creating learning objectives to guide content development and the selection of engaging teaching strategies, and establishing a method of evaluating learning.25,27,28 The ADDIE (Analysis, Design, Development, Implementation, Evaluation) model25,29 (Fig. 2) is an instructional design process that has been used to guide the development of KT resources involving simulated learning environments to target practice skills in novice nurses26 and for improving patient safety through knowledge transfer about teamwork and communication.30 This popular model also has been adapted to inform the structure and format of e-learning by taking into account factors related to the online context.27,31–35 However, outside of curriculum development, demonstrations of the application of the ADDIE model to KT and, more pertinently, to online KT within the field of rehabilitation have been lacking.
Five phases of the ADDIE (Analysis, Design, Development, Implementation, Evaluation) model.
Integrating KT theory with instructional design principles while incorporating effective KT strategies for online formats can be complex. The purpose of this article is to address these challenges by proposing best practice recommendations for the development, implementation, and evaluation of online KT resources designed to translate evidence into practice in rehabilitation. The guidance provided applies to a variety of online resource formats but is especially pertinent to the development of more complex, customized designs (eg, websites, e-learning modules). These recommendations draw upon research evidence from the KT and instructional design literature as well as on our experiences in developing, implementing, and evaluating online KT resources. Subsequent steps are then proposed that will be needed to guide further research.
Recommendation Development Process
Each of the authors has recently created, implemented, and evaluated online KT resources for physical therapists. One project was designed to translate research evidence about, and to train clinicians in, the use of virtual reality (VR) systems within neurorehabilitation.36 Another project involved the development of a module to translate research evidence about best practice management principles in physical therapy for children with developmental coordination disorder (DCD).37,38 Two additional projects supported the implementation of evidence-based practice (EBP) processes by clinicians: an e-learning course to train knowledge brokers (KBs) in facilitating EBP within their interprofessional teams and Web-based resources guiding clinicians in Sackett's 5 steps of EBP.39 The Appendix provides a description of each of these projects. All projects involved consultation with and feedback from practicing clinicians to direct development of the content, design, and evaluation, as described in subsequent sections.
The authors met as a group to identify common challenges inherent in the process of developing online KT resources, to clarify lessons learned, and to outline a plan for developing the proposed recommendations. A literature review and our experiences with online KT development informed the generation of an initial list of recommendations. An iterative consensus-building process was used to link the recommendations to the KTA framework and ADDIE model components and to subsequently group them into 4 recommendations. Each recommendation contains specific steps for online KT resource development, implementation, and evaluation. Finally, concrete examples from our own experiences were incorporated to illustrate each of the recommendations (Table).
Summary of Recommendations, Steps, and Examplesa
Recommendations
We propose 4 recommendations for the development, implementation, evaluation, and reporting of online KT resources: (1) develop evidence-based, user-centered content; (2) tailor the content to an e-learning format; (3) evaluate impact; and (4) share results and disseminate knowledge. Figure 3 illustrates these recommendations, and the Table summarizes their associated steps and provides relevant examples.
Framework for the development, implementation, evaluation, and reporting of knowledge translation (KT) resources.
Recommendation 1: Develop Evidence-Based, User-Centered Content
Step 1: assess audience needs.
Once a knowledge-to-practice gap has been identified, the KTA framework suggests use of a “knowledge creation funnel” to develop evidence-based content and products that are targeted to user needs and local context.22 As such, involving end users (eg, representatives from the target audience) early on is encouraged.3 It may be valuable to complete a formal or informal assessment of end users' needs and to determine their current level of knowledge about the topic, as outlined in the analysis phase of the ADDIE process.25,29
Step 2: summarize evidence to address audience needs.
Tailoring content to the learner is a foundational principle of instructional design25,40 and a key concept of the knowledge creation phase of the KTA framework. The methods used to present the content will vary depending on the goal of the resource. For example, if the intent is to inform best practice in the clinical implementation of a tool, videos of experts using the tool may be required, whereas if the goal is to translate evidence about a disorder or treatment method, a systematic literature review might be necessary. Synthesizing knowledge is crucial, as too much information can disengage or overwhelm the learner, limiting retention.41 Providing key messages (“need to know”) while directing individuals to additional, in-depth resources (“nice to know”) is one way of addressing this dilemma. Alternatively, providing different streams or paths through the resources allows novices the opportunity to access all of the information in one location while more knowledgeable or experienced clinicians can bypass information with which they are already familiar.
It may be beneficial to sketch out a rough map based on the length of the online resource you would like to create. Length can be defined in terms of number of hours that users should spend working through the resource or the number of pages or units of content.
Step 3: use a theory, framework, or model.
Although the KT literature emphasizes the importance of using a theory, framework, or model to inform resource development, this is not often done. Colquhoun and colleagues42 found limited use of the Theory of Diffusion of Innovations, the Theory of Planned Behavior (TPB), or the Theory of Promoting Action on Health Behavior in KT rehabilitation research. This and other reviews provide recommendations about how to increase theory use in the design and implementation of KT strategies.5,6,42–44 A perusal of the available KT, behavioral change, or instructional design literature may help in selecting a framework or model (eg, KTA, TPB, ADDIE) that is suitable to guide the selection of content or the format of your material.
Step 4: select an appropriate KT format.
A needs assessment can help to select, tailor, and target a specific KT format to meet the needs of the end user.17 To maximize learning throughout a Web-based resource, incorporating content in formats that target different learning styles is beneficial (eg, including written, visual, auditory, and hands-on features).45 A discussion forum, blog, wiki, or webinar in which users interact with each other and with resource developers might be another way to enhance participation in learning; indeed, it is the social aspect of exchanging ideas and building connections that is key to the success of these KT formats.17
A systematic review by Mairs et al17 provides an account of the various online KT interventions applied in health care, including discussion forums (eg, for access to current knowledge, promoting reflection, and creating social connections), online communities of practice (eg, for sharing, collaboration, and exposure to new experts and resources to facilitate best practices), knowledge management strategies (eg, for sharing practice information to foster collaboration and enhance care), and conferencing (including webinars for real-time communication to a broad audience in multiple formats involving social interaction). Mairs et al also cited studies involving wikis, blogs, and social media tools; however, no concrete recommendations are provided for identifying the most optimal online strategies for a specific audience or outcome, apart from undertaking an analysis of the users and their needs. Knowledge of the users, their local context, the evidence being translated, and the intended outcome of the KT intervention provides the foundation for choosing appropriate intervention mechanisms or strategies.1,46,47
Step 5: develop learning objectives.
Linking activities and assessments of knowledge or skill gains with specific learning objectives is a central principle of effective instructional design.34 Establishing learning objectives occurs early in the ADDIE process (design phase).25,29 Determining the key knowledge you hope to translate to your audience will allow you to develop content, craft learning experiences, and integrate evaluations (where appropriate) that support these objectives and that are meaningful to the user.25 A needs assessment can be helpful in generating relevant learning objectives from the perspective of the target audience. This step can be formalized within the resource by including assessment methods that determine whether learning objectives have been achieved (see recommendation 3 below) and that give learners clear expectations about the resource content.
Step 6: include multimedia content.
Including multimedia content in an online KT resource makes it user-friendly and visually appealing, differentiating it from more didactic media. Use of relevant media can enhance the effectiveness of learning opportunities if content is targeted toward the learning objectives.31,34 Videos provide learners with concrete examples of how knowledge can be applied in practice. However, the time, cost, and skill required to produce these resources also must be considered, especially if pop-up animation or voice-overs are included. Technical and practical issues exist, such as the amount of Internet bandwidth required, the platform on which the resource will be posted (eg, access to YouTube), and the software required. When posting videos of patients online, privacy issues related to confidentiality are paramount.
Recommendation 2: Tailor Content to Online Format
Step 1: partner with a Web developer.
The design phase of the ADDIE process as adapted for e-learning informs the planning of the resource format and structure.32 Good communication with the Web developer who will design the format and organization of your resource is imperative when customized Web platforms or applications are to be used. As with any working relationship, a discussion of expectations and the individual's background and experience will inform decisions. Deadlines, methods of sharing content with the developer, and clarifying opportunities to make changes once content is “live” should all be discussed.
Step 2: “mock up” content and navigational structure.
An online KT resource can take many forms (eg, learning module or course format, interactive website, YouTube video). It may be difficult to visualize the final appearance of a website or online module. For customized designs, Web developers encourage the “mapping out” of each page of the site, including the navigational structure involved when users click on embedded links or move between Web pages. This planning can be time-consuming but will allow both you and the developer to visualize the end product. It may be helpful to consult with the Web designer for advice about how to make navigation through the resource interactive and nonlinear, but easy to follow (eg, using a navigation bar or a tutorial explaining how to navigate the site).
Step 3: consider Web sustainability.
Online resources are unique because they have the potential to be “living” documents. Consider methods that will make it easy to update the resource when new evidence becomes available, or when new tools or content are created. Practical tips include keeping track of the date the domain name was purchased and its expiration date (if your resource is a website), minimizing use of any external Web links that need to be maintained, and anticipating the impact and cost of future additions on navigation. Interactive resources also will require a moderator to monitor content contributions by users. Ensuring that the developers of the resource have access to make changes, in addition to the Web designer, is crucial.
Step 4: pilot test with intended audience.
To help refine the resource, users from the intended audience should work through it to give formal or informal feedback about aspects, such as content, visual appeal, interactivity, technical functionality, navigation, and learner support.48 This step is part of the implementation phase of the ADDIE model.25,29,44 Consider engaging both expert and novice practitioners to field test your resource prior to its launch to ensure that the level of content is appropriate and comprehensive for all levels of learning and that external links (eg, to other websites, videos, or PDFs) work. A number of usability testing methods are available to direct this process,49–51 some specific to52,53 or applied to19,54 e-learning. This testing may include methods, such as surveys, video or screen capture analysis, think-aloud protocols (eg, verbalizing thought processes during resource use), or heuristic evaluation (eg, evaluating against principles of good design).52,53 Methods should be selected based on the intended goals, the nature of the resource being evaluated, and the availability of resources.53,54
Recommendation 3: Evaluate Impact
Step 1: embed evaluation.
The KTA cycle encourages outcome evaluation,22 as does the final phase of the ADDIE process.25,29 Regardless of whether the online resource was created as part of a research study, it is important to evaluate how effective it was, regardless of whether the goal was to increase awareness, transfer knowledge, or elicit behavior.
Comprehension and retention can be tested, critical thinking can be evaluated using case scenarios where users make choices along a decision path, required task completion and analysis of responses can provide insights into competencies, and self-evaluation can assess learner confidence. Feedback also should be obtained on user perceptions about resource content and format to inform improvements.
A resource may be designed for single or repeated viewing. Design parameters of the Web platform may not allow learners to bypass required task completion on subsequent visits. A challenge lies in balancing the need for data collection with the ability of individuals to refresh their knowledge and access resources on which they have already been assessed. Making creative interactive activities or evaluations that can be feasibly accomplished without a heavy time burden on learners can be difficult.
Step 2: collect usage data.
An online KT resource provides an optimal forum for automatic data collection as users peruse its content. Tracking the number of times people have signed in, accessed content or interacted with the online route taken to access and navigate the resource, time spent per visit, and the number of total users can be useful for budgeting, planning more widespread dissemination, and gaining insights into usability and utility. For instance, the sequence of webpage browsing may differ from that intended, or certain resources may be overlooked or highly accessed.
Step 3: build in evaluation of learning, retention, and behavior change.
The inclusion of interactive methods to assess learners' understanding and retention of information while they are progressing through an online resource is essential. This evaluation could include multiple-choice questions, knowledge checks, open-ended questions, and activities in which users complete a task or skill. To make this element effective, developers can ensure that content development includes the provision of feedback to learners for each method of evaluation (eg, making the correct answer available after the user has selected his or her response). As noted in recommendation 1, linking evaluation to learning objectives is an essential component to facilitate engagement and the evaluation of learning over time. Consideration also can be given to the accreditation or continuing education requirements of the target population (for instance, by offering certificates indicating completion of a module or course).
If the goal of the resource is practice change, its impact on user behavior change should be assessed. The KTA framework emphasizes evaluating the sustainability of transferred knowledge.22 If implementing a series of KT resources, the retesting of learning introduced in earlier resources is one method of evaluating retention of knowledge or skills over time. Asking participants to complete questionnaires examining self-reported change in skills, knowledge, or behavior is one way to assess sustainability.
Recommendation 4: Share Results and Disseminate the Knowledge
Step 1: write scientific papers.
Sharing results with the scientific and clinical communities will advance the field of KT. It is important to report the process used to develop and evaluate the resource and to use recognized data collection techniques (including qualitative or mixed methods or evaluation tools that have established psychometric properties). Doing so will help others to interpret study results and replicate KT strategies in other contexts. Because KT interventions geared toward changing clinicians' behavior can be complex, the field is working toward identifying their “functional components” through better description of content, outcomes, and consensus on terminology for different intervention strategies and through the development of recommendations for describing important elements of the intervention.24
Step 2: share information with target audience.
Your target audience may not have access to the scientific literature, making multiple methods of knowledge dissemination essential. Presentations or short articles posted on relevant websites are 2 such KT strategies for knowledge exchange of study results. Also, consider sharing your resource more widely. Distribution strategies include the use of social media and listserves, e-mails to relevant organizations or associations, the hosting of workshops, and promotions at conferences, during outreach visits, and at stakeholder meetings.
Step 3: maintain knowledge “currency.”
The team should have an individual whose role is to maintain and update the resource, including content, navigation, software and server requirements, and external Web links, as applicable. Ensure that funds are built into a grant proposal or that organizational partnerships are established to support this work.
Discussion
We have drawn from the literature and our experiences to propose best practice recommendations for the development, implementation, evaluation, and reporting of online KT resources in rehabilitation. Four specific recommendations were presented, along with accompanying steps. Although these are preliminary suggestions, they advance KT science by suggesting recommendations in this area that integrate evidence and theory from pertinent fields. The recommendations proposed may help to promote the development of high-quality online KT resources and, with an emphasis on evaluation, should encourage the development of resources that have greater potential for changing practice.
We invite discussion, debate, and further development on these proposed recommendations to explore their utility across contexts, as well as their validity for online formats of varying complexities. Anticipated next steps include having researchers and clinicians implement these recommendations in the development of other KT resources and provide feedback on their utility. More research is needed to explore how online KT resources can incorporate strategies that lead to lasting behavior change and how the sustainability of knowledge transfer can be evaluated following an online KT intervention. A scoping review being conducted on this topic by Tricco and colleagues55 may prove useful in developing a framework for assessing the sustainability of KT interventions. Developing a standardized method of evaluating the content of online KT resources and comparing different methods of disseminating similar types of content also will be essential. Indeed, although checklists for quality assurance and evaluation are available for e-learning course developers,33,56 none are specific to online KT resources.
Conclusion
The recommendations provided here may be useful to guide researchers and others through the development, evaluation, and reporting of online KT resources. We invite discussion about, and further development of, these recommendations to support their integration into practice. Knowledge translation science is advanced when knowledge from different domains is combined with relevant experience. We do so here to promote and advance the undertaking and reporting of online KT initiatives that encourage behavior change that advances practice and improves the quality of patient care.
Appendix.
Projects Supporting the Implementation of EBP Processes by Cliniciansa
a EBP=evidence-based practice; VR=virtual reality; IREX=GestureTek's Interactive Rehabilitation Exercise; GX=Gesture Xtreme; SMART=specific, measurable, achievable, realistic, time-related; DCD=developmental coordination disorder; KB=knowledge broker.
Footnotes
All authors provided concept/idea/study design and writing. Ms Glegg, Dr Camden, and Ms Rivard provided data collection and analysis. Dr Levac provided project management. Ms Glegg, Dr Camden, Ms Rivard, and Dr Missiuna provided consultation (including review of the manuscript before submission).
- Received October 31, 2013.
- Accepted October 6, 2014.
- © 2015 American Physical Therapy Association