Best Practice Recommendations for the Development, Implementation, and Evaluation of Online Knowledge Translation Resources in Rehabilitation

Step 1: assess audience needs.

Once a knowledge-to-practice gap has been identified, the KTA framework suggests use of a “knowledge creation funnel” to develop evidence-based content and products that are targeted to user needs and local context.²² As such, involving end users (eg, representatives from the target audience) early on is encouraged.³ It may be valuable to complete a formal or informal assessment of end users' needs and to determine their current level of knowledge about the topic, as outlined in the analysis phase of the ADDIE process.^25,29

Step 2: summarize evidence to address audience needs.

Tailoring content to the learner is a foundational principle of instructional design^25,40 and a key concept of the knowledge creation phase of the KTA framework. The methods used to present the content will vary depending on the goal of the resource. For example, if the intent is to inform best practice in the clinical implementation of a tool, videos of experts using the tool may be required, whereas if the goal is to translate evidence about a disorder or treatment method, a systematic literature review might be necessary. Synthesizing knowledge is crucial, as too much information can disengage or overwhelm the learner, limiting retention.⁴¹ Providing key messages (“need to know”) while directing individuals to additional, in-depth resources (“nice to know”) is one way of addressing this dilemma. Alternatively, providing different streams or paths through the resources allows novices the opportunity to access all of the information in one location while more knowledgeable or experienced clinicians can bypass information with which they are already familiar.

It may be beneficial to sketch out a rough map based on the length of the online resource you would like to create. Length can be defined in terms of number of hours that users should spend working through the resource or the number of pages or units of content.

Step 3: use a theory, framework, or model.

Although the KT literature emphasizes the importance of using a theory, framework, or model to inform resource development, this is not often done. Colquhoun and colleagues⁴² found limited use of the Theory of Diffusion of Innovations, the Theory of Planned Behavior (TPB), or the Theory of Promoting Action on Health Behavior in KT rehabilitation research. This and other reviews provide recommendations about how to increase theory use in the design and implementation of KT strategies.^5,6,42–44 A perusal of the available KT, behavioral change, or instructional design literature may help in selecting a framework or model (eg, KTA, TPB, ADDIE) that is suitable to guide the selection of content or the format of your material.

Step 4: select an appropriate KT format.

A needs assessment can help to select, tailor, and target a specific KT format to meet the needs of the end user.¹⁷ To maximize learning throughout a Web-based resource, incorporating content in formats that target different learning styles is beneficial (eg, including written, visual, auditory, and hands-on features).⁴⁵ A discussion forum, blog, wiki, or webinar in which users interact with each other and with resource developers might be another way to enhance participation in learning; indeed, it is the social aspect of exchanging ideas and building connections that is key to the success of these KT formats.¹⁷

A systematic review by Mairs et al¹⁷ provides an account of the various online KT interventions applied in health care, including discussion forums (eg, for access to current knowledge, promoting reflection, and creating social connections), online communities of practice (eg, for sharing, collaboration, and exposure to new experts and resources to facilitate best practices), knowledge management strategies (eg, for sharing practice information to foster collaboration and enhance care), and conferencing (including webinars for real-time communication to a broad audience in multiple formats involving social interaction). Mairs et al also cited studies involving wikis, blogs, and social media tools; however, no concrete recommendations are provided for identifying the most optimal online strategies for a specific audience or outcome, apart from undertaking an analysis of the users and their needs. Knowledge of the users, their local context, the evidence being translated, and the intended outcome of the KT intervention provides the foundation for choosing appropriate intervention mechanisms or strategies.^1,46,47

Step 5: develop learning objectives.

Linking activities and assessments of knowledge or skill gains with specific learning objectives is a central principle of effective instructional design.³⁴ Establishing learning objectives occurs early in the ADDIE process (design phase).^25,29 Determining the key knowledge you hope to translate to your audience will allow you to develop content, craft learning experiences, and integrate evaluations (where appropriate) that support these objectives and that are meaningful to the user.²⁵ A needs assessment can be helpful in generating relevant learning objectives from the perspective of the target audience. This step can be formalized within the resource by including assessment methods that determine whether learning objectives have been achieved (see recommendation 3 below) and that give learners clear expectations about the resource content.

Step 6: include multimedia content.

Including multimedia content in an online KT resource makes it user-friendly and visually appealing, differentiating it from more didactic media. Use of relevant media can enhance the effectiveness of learning opportunities if content is targeted toward the learning objectives.^31,34 Videos provide learners with concrete examples of how knowledge can be applied in practice. However, the time, cost, and skill required to produce these resources also must be considered, especially if pop-up animation or voice-overs are included. Technical and practical issues exist, such as the amount of Internet bandwidth required, the platform on which the resource will be posted (eg, access to YouTube), and the software required. When posting videos of patients online, privacy issues related to confidentiality are paramount.

Step 1: partner with a Web developer.

The design phase of the ADDIE process as adapted for e-learning informs the planning of the resource format and structure.³² Good communication with the Web developer who will design the format and organization of your resource is imperative when customized Web platforms or applications are to be used. As with any working relationship, a discussion of expectations and the individual's background and experience will inform decisions. Deadlines, methods of sharing content with the developer, and clarifying opportunities to make changes once content is “live” should all be discussed.

Step 2: “mock up” content and navigational structure.

An online KT resource can take many forms (eg, learning module or course format, interactive website, YouTube video). It may be difficult to visualize the final appearance of a website or online module. For customized designs, Web developers encourage the “mapping out” of each page of the site, including the navigational structure involved when users click on embedded links or move between Web pages. This planning can be time-consuming but will allow both you and the developer to visualize the end product. It may be helpful to consult with the Web designer for advice about how to make navigation through the resource interactive and nonlinear, but easy to follow (eg, using a navigation bar or a tutorial explaining how to navigate the site).

Step 3: consider Web sustainability.

Online resources are unique because they have the potential to be “living” documents. Consider methods that will make it easy to update the resource when new evidence becomes available, or when new tools or content are created. Practical tips include keeping track of the date the domain name was purchased and its expiration date (if your resource is a website), minimizing use of any external Web links that need to be maintained, and anticipating the impact and cost of future additions on navigation. Interactive resources also will require a moderator to monitor content contributions by users. Ensuring that the developers of the resource have access to make changes, in addition to the Web designer, is crucial.

Step 4: pilot test with intended audience.

To help refine the resource, users from the intended audience should work through it to give formal or informal feedback about aspects, such as content, visual appeal, interactivity, technical functionality, navigation, and learner support.⁴⁸ This step is part of the implementation phase of the ADDIE model.^25,29,44 Consider engaging both expert and novice practitioners to field test your resource prior to its launch to ensure that the level of content is appropriate and comprehensive for all levels of learning and that external links (eg, to other websites, videos, or PDFs) work. A number of usability testing methods are available to direct this process,^49–51 some specific to^52,53 or applied to^19,54 e-learning. This testing may include methods, such as surveys, video or screen capture analysis, think-aloud protocols (eg, verbalizing thought processes during resource use), or heuristic evaluation (eg, evaluating against principles of good design).^52,53 Methods should be selected based on the intended goals, the nature of the resource being evaluated, and the availability of resources.^53,54

Step 1: embed evaluation.

The KTA cycle encourages outcome evaluation,²² as does the final phase of the ADDIE process.^25,29 Regardless of whether the online resource was created as part of a research study, it is important to evaluate how effective it was, regardless of whether the goal was to increase awareness, transfer knowledge, or elicit behavior.

Comprehension and retention can be tested, critical thinking can be evaluated using case scenarios where users make choices along a decision path, required task completion and analysis of responses can provide insights into competencies, and self-evaluation can assess learner confidence. Feedback also should be obtained on user perceptions about resource content and format to inform improvements.

A resource may be designed for single or repeated viewing. Design parameters of the Web platform may not allow learners to bypass required task completion on subsequent visits. A challenge lies in balancing the need for data collection with the ability of individuals to refresh their knowledge and access resources on which they have already been assessed. Making creative interactive activities or evaluations that can be feasibly accomplished without a heavy time burden on learners can be difficult.

Step 2: collect usage data.

An online KT resource provides an optimal forum for automatic data collection as users peruse its content. Tracking the number of times people have signed in, accessed content or interacted with the online route taken to access and navigate the resource, time spent per visit, and the number of total users can be useful for budgeting, planning more widespread dissemination, and gaining insights into usability and utility. For instance, the sequence of webpage browsing may differ from that intended, or certain resources may be overlooked or highly accessed.

Step 3: build in evaluation of learning, retention, and behavior change.

The inclusion of interactive methods to assess learners' understanding and retention of information while they are progressing through an online resource is essential. This evaluation could include multiple-choice questions, knowledge checks, open-ended questions, and activities in which users complete a task or skill. To make this element effective, developers can ensure that content development includes the provision of feedback to learners for each method of evaluation (eg, making the correct answer available after the user has selected his or her response). As noted in recommendation 1, linking evaluation to learning objectives is an essential component to facilitate engagement and the evaluation of learning over time. Consideration also can be given to the accreditation or continuing education requirements of the target population (for instance, by offering certificates indicating completion of a module or course).

If the goal of the resource is practice change, its impact on user behavior change should be assessed. The KTA framework emphasizes evaluating the sustainability of transferred knowledge.²² If implementing a series of KT resources, the retesting of learning introduced in earlier resources is one method of evaluating retention of knowledge or skills over time. Asking participants to complete questionnaires examining self-reported change in skills, knowledge, or behavior is one way to assess sustainability.

Step 1: write scientific papers.

Sharing results with the scientific and clinical communities will advance the field of KT. It is important to report the process used to develop and evaluate the resource and to use recognized data collection techniques (including qualitative or mixed methods or evaluation tools that have established psychometric properties). Doing so will help others to interpret study results and replicate KT strategies in other contexts. Because KT interventions geared toward changing clinicians' behavior can be complex, the field is working toward identifying their “functional components” through better description of content, outcomes, and consensus on terminology for different intervention strategies and through the development of recommendations for describing important elements of the intervention.²⁴

Step 2: share information with target audience.

Your target audience may not have access to the scientific literature, making multiple methods of knowledge dissemination essential. Presentations or short articles posted on relevant websites are 2 such KT strategies for knowledge exchange of study results. Also, consider sharing your resource more widely. Distribution strategies include the use of social media and listserves, e-mails to relevant organizations or associations, the hosting of workshops, and promotions at conferences, during outreach visits, and at stakeholder meetings.

Step 3: maintain knowledge “currency.”

The team should have an individual whose role is to maintain and update the resource, including content, navigation, software and server requirements, and external Web links, as applicable. Ensure that funds are built into a grant proposal or that organizational partnerships are established to support this work.