Printer Friendly

An exploratory study on the teaching of evidence-based decision making.

Occupational therapists are urged to make clinical decisions in consultation with their clients and based on their professional expertise as well as scientific evidence. This approach to decision making is referred to as evidence-based practice (EBP). The goal of EBP is to optimize care and improve patient outcomes. The EBP process is said to include five steps:

1. Ask a clinical question.

2. Access research evidence to answer the question.

3. Appraise the validity, importance, and applicability of the evidence.

4. Apply the appraised evidence into decision making in practice.

5. Audit the effectiveness of the EBP process. (Young, Rohwer, Volmink, & Clarke, 2014)

Every step in the EBP process requires different skills and knowledge. For example, identifying a gap in one's knowledge and proceeding to formulate an answerable question requires reflection (Step 1). Information literacy skills are necessary to search effectively for relevant literature (Step 2). Critically appraising the evidence (Step 3) requires knowledge about research methodology. And evidence-based decision making (EBDM) requires critical thinking (Step 4). Critical thinking is a concept that encompasses two main components: cognitive skills and thinking dispositions (Facione, 2007; Stanovich, 2011; West, Toplak, & Stanovich, 2008). Occupational therapists use generic cognitive skills and dispositions to support EBDM. They rely on different sources of knowledge, consider multiple options for treatment, and generate supporting or refuting arguments for these options based on the judicious selection of reliable information. Moreover, under ideal circumstances, EBDM occurs in collaboration with the client. It follows that EBDM also necessitates teaching and consultation skills (Thomas, Saroyan, & Dauphinee, 2011). Therefore, Step 4 (apply) requires action, such as using evidence in decision making and engaging in collaborative decision making. A certain behavior should be observable in this step. It is widely acknowledged that to show a certain behavior, the self-confidence in the ability of such behavior is important. Indeed, EBP self-efficacy has been found to be a strong predictor of research use in practice (Salbach, Guilcher, Jaglal, & Davis, 2010; Thomas & Law, 2013). The final step of the EBP process, audit, requires critical thinking about the EBP process and the outcomes of one's clinical decision.

Challenges Relating to EBDM in Research and Education

Previous studies have shown that students in health education experience specific difficulties with EBDM. Challenges include making sense and assessing the relevance of the evidence for practice, dealing with conflicting evidence, and having the opportunity to explore the outcome of evidence-based decisions in authentic practice situations (Gillam & Gillam, 2008; Lam, Fielding, Johnston, Tin, & Leung, 2004).

Teaching Steps 1, 2, and 3 of EBP (ask, access, and appraise) has been shown to be effective among undergraduate students in health education (Young et al., 2014). Authors agree that the teaching should be multifaceted, i.e., using multiple educational strategies and integrated into authentic situations (Khan & Coomarasamy, 2006; Thomas et al., 2011; Young et al., 2014). However, there is less evidence on how to promote EBP behavior or critical thinking changes effectively in undergraduates, which as explained earlier, lie at the heart of EBDM. In fact, multifaceted interventions have only been shown to be effective in knowledge, skills, and attitudes in undergraduates. Moreover, studies do not report in detail which intervention caused which effect on which particular skill development. Therefore, it is unclear whether critical thinking increases as a result of multifaceted interventions (Young et al., 2014).

The limited research examining EBDM may be explained by the challenges associated with measuring and teaching this step of the EBP process. First, EBP skills and behaviors are not well defined, and EBDM is seldom operationalized (Harris et al., 2011). Second, there is a lack of valid and reliable assessments of Step 4 (Tilson et al., 2011). Lastly, opportunities for authentic EBDM learning experiences in the academic setting are limited. Although the application of EBDM should ideally take place during fieldwork, documentation of underused research findings in clinical practise (Kloda & Bartlett, 2009) poses a challenge. Underutilization in clinical practice is explained by compelling evidence supporting individual barriers (e.g., lack of time, lack of research skills) and organizational barriers (institutional support, limited access to resources) to EBP (Diermayr, Schachner, Eidenberger, Lohkamp, & Salbach, 2015; Dijkers, Murphy, & Krellman, 2012; Thomas & Law, 2013). As such, the fieldwork environment may not be an ideal learning setting either. For example, studies show that education on EBP in fieldwork is suboptimal, largely due to a lack of preceptor guidance (Coster & Schwartz, 2004; Lam et al., 2004), and that EBP skills actually decrease after fieldwork (Crabtree, Justiss, & Swinehart, 2012).

On the one hand, this literature suggests that EBDM requires self-efficacy and complex skills, such as critical thinking, consultation, and teaching. EBDM is a step of EBP that requires multifaceted education in authentic situations. On the other hand, authentic situations, such as fieldwork, represent challenges in teaching EBP. Moreover, it is not yet clear whether teaching leads to effective and significant improvements in mastering Step 4 (apply) or EBDM, because this step is not easy to operationalize or to assess as a learning outcome (Harris et al., 2011; Khan & Coomarasamy, 2006; Thomas et al., 2011; Tilson et al., 2011; Young et al., 2014).

The overall purpose of the study was to evaluate the impact of innovative, theoretically grounded teaching aimed particularly at promoting self-efficacy and the cognitive skills used in EBDM among undergraduate occupational therapy students. The specific teaching comprised five teaching sessions, all of which included using the theoretical principles of situated learning and cognitive apprenticeship and teaching critical thinking. The sessions consisted of critical thinking exercises in which evidence is used to formulate opinions about authentic case-based occupational therapy issues. The research questions that guided the study were:

* What are undergraduate occupational therapy students' and their teachers' views on the elements selected and the benefits of teaching aimed at promoting self-efficacy and the cognitive skills used in EBDM?

* What is the impact of teaching aimed at promoting self-efficacy and the cognitive skills used in EBDM on undergraduate occupational therapy students' self-efficacy and the cognitive skills used in EBDM?

Methods

Educational Research in Natural Settings

The research was conducted in the context of ongoing education as part of a four-year undergraduate occupational therapy bachelor's program at Zuyd University of Applied Sciences in Heerlen, the Netherlands. The underlying philosophical worldview is pragmatism (Creswell, 2014) and design-based educational research. Pragmatism and design-based educational research focus on what works in a naturalistic context. Design-based educational research aims to develop educational, theory-driven interventions in the real world. The focus is on understanding and improving educational interventions in naturalistic settings (Van den Akker, Gravemeijer, McKenney, & Nieveen, 2006). Educational interventions are complex; they demand complex behavior from teachers and students alike and involve many interacting factors that are difficult to control for in a naturalistic setting. Therefore, this research is also aligned with the Medical Research Council's model for developing complex interventions (MRC; see Figure 1) (Craig et al., 2008). It is strongly recommended that research on complex interventions include the development and piloting of potential interventions. This ensures that researchers use appropriate theoretical frameworks or rationale for an intervention, and that they can argue for the feasibility of complex interventions (Craig et al., 2008). These approaches match up with the aim of this research: We wanted to develop and evaluate potentially interesting teaching methods promoting self-efficacy and cognitive skills in EBDM in order to produce knowledge about practical, potentially useful elements in education on this topic.

Research performed in naturalistic settings has advantages in terms of practical relevance and applicability in other educational practices, but it also presents a challenge in that many factors cannot be controlled in naturalistic settings. For instance, the policy rules of our university did not allow us to rigorously change the curriculum and apply new teaching methods, so we had to apply the five teaching sessions as an add-on to existing education. This research, therefore, is a pilot. The research involved a first cycle of the development of, experimentation with, and evaluation of a new teaching intervention (Van den Akker et al., 2006) on self-efficacy and cognitive skills, specifically in EBDM in a naturalistic setting. This provides descriptive knowledge about potentially useful elements to teaching EBDM to inform educational practice and further research.

Research Design

In line with the above-mentioned approaches to research, we conducted a convergent parallel mixed-method design (Creswell, 2014). Parallel qualitative and quantitative data gathering methods were used to evaluate the teaching sessions. This is an appropriate research design, considering that our aim was to understand the applied new teaching intervention in depth and to explore the learning processes that occurred. Using different methods to gather data, we could check whether a convergent picture of the impact of the teaching emerged. A teaching intervention consisting of five teaching sessions was developed and piloted in a group of 17 students between September 2013 and December 2013. The qualitative arm consisted of an evaluative descriptive design (Sandelowski, 2000), which is a suitable qualitative design to describe the participants' experiences of the intervention. Therefore, semi structured interviews were conducted with the participants after the teaching sessions in January 2014. The quantitative arm consisted of a pre-post design. Students' self-efficacy, generic cognitive skills, and critical thinking were measured in September 2013 (before the teaching sessions) and in January 2014 (after the teaching sessions).

Participants

Students were recruited on a voluntary basis at the end of a preceding academic year. They received informative emails and short presentations about the research during on-going education at the end of their second academic year. Fourteen of the second-year students were going to be away from the university for fieldwork education in September of their third academic year, so these students could not participate. Of the remaining 59 students that were going to be attending school from September 2013, 17 (28.8%) were willing to participate. The School of Occupational Therapy integrates EBP knowledge and skills throughout the curriculum, so students had strong prior knowledge of EBP. Prior EBP education included training in incorporating scientific literature into their assignments. This was supported by information literacy workshops in collaboration with librarians. In addition, before following the five teaching sessions provided in the present study, during the second year of the program, students also attended a mandatory course on the principles of EBP, including critical appraisal of the research literature.

The students gave written informed consent, and general ethical procedures were followed. Data was processed anonymously, efforts were made to ensure that the teaching sessions would not interfere with the lessons the students were obliged to follow, no information was shared that could have a negative impact on the students, and they were informed that they could stop participating. In addition, this research was approved by an authorized body linked to the School of Occupational Therapy. This authorized body reviewed the research plan, including ethical procedures, and approved the use of this course consisting of five teaching sessions as a pilot.

Five of the students were excluded from the analysis, as they had attended three sessions or fewer. The students missed these sessions due to pragmatic, mainly logistical issues, such as travel problems, sudden changes in the scheduling of other obligatory lessons, and medical appointments. Of the remaining 12 students, four attended four sessions and eight attended all sessions. The students' ages ranged from 19 to 29 years; nine of the participants were women and three were men. The School of Occupational Therapy uses a European Credit Transfer and Accumulation System (ECTS), which is in standard use throughout the European Union. The ECTS aims to make programs and the performance of students in higher education more transparent and comparable Europe wide and is intended to replace or complement the different local (national) standards in Europe. When the students complete a course, seminar, or module, they are awarded ECTS credit points. Every ECTS credit point represents a certain workload. Typically, 1 year corresponds to 60 ECTS credits. The total number of ECTS credits the participants in the present study had accrued in September 2013 ranged between 37 and 63, whereby 60 ECTS credits is the nominal value. The mark they had attained for the mandatory course on the principles of EBP that they had attended previously was 6.5 on average (SD 0.6, on a scale of 1-10, whereby 1 indicates that the student failed badly, a 6 indicates a pass, and a 10 is excellent), and four students had to retake a test, having failed it the first time. This suggests that at the start of the five teaching sessions provided in the present study, the students were not a selected group, in terms of either their general academic performance or their EBP performance in prior education.

Educational Intervention Relating to Evidence-Based Decision Making

As stated in the introduction, the idea was to design innovative, theoretically grounded teaching aimed at promoting the self-efficacy and cognitive skills used in EBDM, or Step 4 (apply) of EBP. Therefore, the objectives or learning goals of the teaching intervention were

* to increase students' skills in (a) selecting and sorting information included in research evidence and client information, (b) analyzing this information and making pro and contra arguments, and (c) coming up with a logical conclusion following this analysis in light of a clinical decision to be made;

* to increase students' self-efficacy in EBDM; and

* to increase students' critical thinking skills.

Theoretical Underpinnings

As advocated by Thomas et al. (2011), Cognitive Apprenticeship (CA) and Situated Learning (SL) can support the design of EBP teaching interventions. CA emphasizes the use of modeling, scaffolding, and reflection in authentic contexts, all of which were applied in our intervention, as specified in the description of the teaching intervention below. SL focuses on the social aspects of learning and stresses the importance of real-life work experiences. The students were required to make clinical decisions in situations most likely to be experienced in occupational therapy practice in a simulated authentic situation. In addition to CA and SL, the intervention was informed by the literature on promoting critical thinking. Purposeful training, including the use of argument maps, is considered important in mastering critical thinking (Facione, 2007; Ter Berg, van Gelder, Patterson, & Teppema, 2009). An argument map organizes reasoning into a hierarchical visual representation, with arguments and propositions arranged in boxes and connected by arrows that highlight relations.

Development of the teaching intervention. The educational intervention needed to be developed and constructed. Therefore, an exploration and analysis took place of what should be taught, in what way, and with which endpoints in mind, all in relation to EBDM. Throughout the exploration and analysis, information was gathered which led to general design requirements for the educational intervention. These design requirements are called design conjectures, i.e., core ideas that underpin--and are used as input for--design. Design conjectures were made based on the abovementioned theoretical underpinnings and research literature addressing EBDM teaching. Local opinions and experiences relating to EBDM instruction were collected from five teachers involved in teaching EBP, five occupational therapists involved in fieldwork, and six occupational therapy students with fieldwork experience, in order to further refine the design conjectures. Details on methods and analysis are described elsewhere (Baarends, 2015). Design conjectures relating to the endpoints of the educational intervention, the educational material, and the way in which the education should be supported were used for the actual construction of the five teaching sessions. The first author constructed the five teaching sessions based on these design conjectures, and two fourth-year occupational therapy students provided feedback during the process. The researcher and the teacher prepared every teaching session and engaged in reflection together after each session. While preparing the teaching sessions, the design conjectures were checked to make sure they were understood and addressed in the teaching sessions. Furthermore, the reflection on the teaching sessions was guided by informal notes made by a fourth-year student who observed the lessons.

Description of the Teaching Intervention

The teaching intervention consisted of five 2 hr sessions performed in a simulated authentic situation at our university. To further provide a simulated authentic situation, we developed written patient scenarios based on real clients' stories. The scenarios required students to make authentic treatment decisions based on a predefined question, such as, "Would it be effective to apply mirror therapy for this patient who has suffered a stroke?" In Table 1, a condensed description of a session in which this example question was central is given. Moreover, as Table 1 shows, six clients were present during four of the five sessions, which allowed the students to have conversations and debates with clients. The clients were instructed to ask critical questions to ensure that the rationale behind the students' decisions was understandable. Also, the clients were instructed to be spokespersons for the client's view of the decision. The scientific evidence for the patient scenario was provided, since in these teaching sessions the students were informed that Steps 1, 2, and 3 of the EBP process had already been taken for them and that the focus should be on EBDM.

Modeling was offered through worked-out examples of argumentation maps. Scaffolding was applied by suggesting a certain structure for reasoning to be used in the assignments. The teacher was explicitly instructed to think out loud and to ask critical questions to stimulate the consideration of alternative perspectives and the thorough argumentation of clinical decisions. In addition, the teacher reflected with the students on his or her argumentation maps, providing the students with feedback.

To ensure specific instruction on critical thinking, the students received homework assignments based on authentic problems, structured in such a way that they practiced the cognitive skills and subskills involved in critical thinking (Facione, 2007). Similar assignments were done during the training sessions. The students received a software program in which they could draw argument maps (Rationale[TM]).

Data Collection

Methods used for data collection to address Research Question 1 (qualitative arm). In January 2014, following the five teaching sessions, the student and teacher experiences were collected using 45-min semi-structured interviews. The main interview questions were: (a) How did you experience development in EBDM during and after the five teaching sessions? and (b) In which way was development in EBDM supported or limited by the education received? The interviews were recorded and transcribed verbatim.

Methods used for data collection to address Research Question 2 (quantitative arm): Self-efficacy in EBP. In September and December 2013, the Evidence-based Practice Confidence Scale (EPIC) was used to assess self-efficacy in EBP (Salbach, Jaglal, & Williams, 2013). This instrument was translated into Dutch, translated back into English, and checked by an independent coresearcher to ensure high quality translation. Ultimately, the translated Dutch version of the EPIC was used. The EPIC contains 11 items (n = 11 items) that target the 5 steps of the EBP process. Participants rate their level of confidence on an 11-point scale (0% confidence to 100% confidence). Item level scores can be averaged to obtain a total score ranging from 0 to 100 percentage points. The scale has excellent internal consistency (0.89; 95% confidence interval 0.86 to 0.91) and test-retest reliability (0.89; 95% confidence interval 0.85 to 0.91) (Salbach et al., 2013). To be able to compare studies or students in relation to EPIC outcomes, we calculated mean percentage self-efficacy for the whole EPIC. In addition, since our focus is on EBDM, we also calculated mean percentage self-efficacy for the decision part of the EPIC. There are three statements in the EPIC questionnaire that specifically refer to EBDM. Critical Thinking Test

In September and December 2013, an existing Dutch online version of the validated critical thinking test, the Health Sciences Reasoning Test (HSRT), was used to assess general critical thinking skills (Insight Assessment, 2014). This test consists of 33 scenarios, each with one multiple-choice question. The test stops after 50 min, regardless of whether or not the student has finished. Construct validity derives from the definition of critical thinking (Facione, 2007) and has been positively shown in research (Huhn, Black, Jensen, & Deutsch, 2011). The Dutch version had a good internal consistency, evidenced by a Cronbach's alpha of 0.72 (Paans, Sermeus, Nieweg, Krijnen, & van der Schans, 2012).

Generic Cognitive Skills in EBDM

Before the teaching sessions in September 2013 but also following the teaching sessions in December 2013, the students completed a written performance-based assignment to monitor possible improvement in the cognitive skills the students had been trained in during the teaching sessions. The assignment was to deliver an argumentation behind an opinion about a statement for a certain clinical case and occupational treatment decision. The students had to write an argument in which they had to demonstrate the skills described as objectives of the teaching sessions. Thus, to set up the argument they had to:

* Select relevant information in light of the treatment decision to be made (select);

* formulate arguments for or against this decision (analyze); and

* formulate a congruent conclusion following the arguments and the statement about the treatment decision given (conclude).

The students were instructed to use the following given information: (a) a rich case description, (b) an evidence-based occupational therapy treatment guideline (which they studied during a previous course in their bachelor program), and (c) a research article that students received a week before the assignment. After the five teaching sessions, the students received a similar assignment but with another statement relating to a treatment decision, case, and evidence. This was done because during the teaching sessions the students had received feedback on their first argument. Comparability of the two assignments was peer reviewed by two coresearchers, concluding that positive scores in cognitive skills could not be explained by a decreased difficulty in the final assignment.

Data Analysis

Transcripts were member checked and analyzed using principles of inductive content analysis with Nvivo 10 (Elo & Kyngas, 2008). Content analysis is a suitable method used in qualitative descriptive design (Sandelowski, 2000) to reduce interview data into a general description of the research topic in several categories, to understand and describe the topic in depth, and to present the data content in a representative manner. Data was analyzed in an inductive way, since the teaching intervention studied had not been studied previously and inductive coding is recommended in cases where there is no former knowledge about a phenomenon. Fragments were inductively coded, keeping the research topic in mind and adhering closely to the content of the fragments. The open coding of the first two interviews were peer-reviewed by a co-researcher; discussions about coding helped the first author to select relevant fragments and find the right words to cover the content. After coding eight interviews, using a constant comparative method, the codes were placed into subcategories. These subcategories were compared, analyzed, and put together into higher order categories and finally into main categories. This was performed by the first author. A concept description of the seven main categories was made, based on the content of fragments in the subcategories, and this was verified by a group of seven researchers. These researchers received multiple fragments and placed them in the main categories. The description of the main categories and the understanding of the subcategories and corresponding codes were deepened through discussion in this peer-review with seven researchers, and the analysis of eight interviews was checked accordingly. The remaining interviews were analyzed also, revealing that the seven main categories had been maintained.

The quantitative data were analyzed using descriptive statistics and a nonparametric Wilcoxon signed rank test, using SPSS. Ap value < 0.05 was considered statistically significant. Effect sizes (ES) were obtained by calculating Cohen's d, using the mean outcome before and after the intervention and the pooled standard deviation into the equation. The ES was classified as negligible ([greater than or equal to] -0.15 and < 0.15), small ([greater than or equal to] 0.15 and < 0.40), medium ([greater than or equal to] 0.40 and < 0.75), large ([greater than or equal to] 0.75 and < 1.10), very large ([greater than or equal to] 1.10 and < 1.45), and huge ([greater than or equal to] 1.45) (Thalheimer & Cook, 2002). We used mean imputation to deal with two missing values, which were due to logistic problems (loss of one HRST and EPIC measurement before the teaching sessions). One HRST score was excluded from the analysis, because this student's results (before and after the intervention) were considered unreliable by the software (too few scenarios answered).

To measure whether the cognitive skills demonstrated in the argument had changed after the teaching sessions, the arguments were analyzed using the SOLO (the Structure of Observed Learning Outcomes) (Biggs & Tang, 2011). The SOLO describes learning outcomes from lower to higher cognitive levels, ranging from prestructural (which means missing the point), through uni structural, multi structural, and relational (different aspects have become integrated into a coherent whole) to the abstract level (understanding of the integrated knowledge, which can be generalized or transferred to new contexts). The assignment was considered as at a relational level. A scoring rubric was created by the first author for the three categories mentioned earlier: (a) selection, (b) analysis, and (c) conclusion. The possible scores included 0 (prestructural), 1 (uni-structural), 2 (low multistructural), 3 (high multi-structural), and 4 (relational level). According to the SOLO, for each skill category and each level of cognitive performance, a specific and concrete description was developed, so that the performance of the students' arguments could be compared to the description. For instance, if a student misinterpreted information or selected irrelevant information, the selection category was scored low. No earlier research has been carried out on the validity and reliability of this rubric. However, we applied several strategies to increase the reliability of scoring. The scoring rubric was verified by two coresearchers using sample essays. The first author performed the final scoring using a blind method, i.e., the arguments were labeled with numbers so that the first author could not link them to specific students while scoring. The overall SOLO score was computed as the mean SOLO score of the three categories.

Results

Research Question 1: Perceptions of the Benefits of the Educational Intervention

Seven main categories emerged from the semi-structured interviews that were held after the five teaching sessions aimed at promoting self-efficacy and the cognitive critical thinking skills used in EBDM. The description of the categories provides a comprehensive summary of the participants' experiences, followed by the content analysis method described earlier. Quotations are used to illustrate each description and are an example of the fragments that lead to the description of the categories. The first four categories related to learning processes that were perceived to have resulted from the teaching.

Deeper understanding of integration of evidence and client information. (Eighty-one fragments belonging to this category were coded; these fragments were taken from all 13 interviews). Although students found that it was more difficult to integrate evidence and the clients' values and wishes than they had anticipated, they gained a greater understanding of the necessity and value of doing so. They expressed a deeper understanding of how to balance evidence and the clients' input and mentioned the importance of communication in this process. The students added that they still wondered whether they are able to find the right balance and make the best decisions. One student reported, "What I have learned is that it is important to take into account both the client and the evidence, and now I have an understanding of how to go about it."

Enhanced ability to sort and select information. (Seventy fragments belonging to this category were coded; these fragments were taken from 11 interviews). The students expressed difficulty in organizing information, as exemplified by one student who said, "Before, everything in my head was a bit chaotic." The students felt that they were better able to distinguish between relevant and irrelevant information and to organize data coming from research evidence as well as from clients. They felt better able to identify arguments for and against a course of action. Specifically searching for these arguments in the data helped them gain a better understanding of the subject at hand. One student said,
   I noticed when writing my second
   assignment that my approach was far more
   structured. I stated clearly "this is taken
   from the guidelines, here is an argument
   for, here is an argument against, this is
   taken from the research paper, I would like
   to draw a distinction here between this and
   that...." Unconsciously, you apply more
   structure, and that makes it easier for you
   to gain a clear overview.


Questioning information and opinions and being more cautious when formulating arguments or drawing conclusions. (Ninety-five fragments belonging to this category were coded; these fragments were taken from 12 interviews). The participants found that they had learned to think more deeply before forming opinions, and they expressed a better understanding of how to formulate arguments to support their opinions. Their comments suggest that they had developed a broader perspective on reasoning: exploring more options, coming up with more arguments for and against, and questioning arguments, as shown by this student:
   Yes, don't jump to conclusions; always
   maintain a broader perspective. For
   instance, if research results suggest that the
   chance of a positive outcome is small,
   don't immediately think 'Oh well, that
   won't work then'. You can also take the
   opposite approach. If, for example, a
   client says, 'I want to be able to do such
   and such,' and you are almost sure it's not
   possible, keep searching for another
   solution. Don't just accept no for an
   answer.... Keep your options open.


More interaction with clients, paying closer attention to their experiences. (Seventy-five fragments belonging to this category were coded; these fragments were taken from 11 interviews). The students stated that they had learned to interact better with clients and have more genuine conversations. They discovered that they needed to probe deeper when asking clients about their experiences and opinions, be more open to different views, and treat clients as partners in the decision-making process. They also discovered the importance of using layman's language. They experienced that the contact became more natural over time, and that they became better listeners. One student said,
   Yes, see the client more as an equal
   because, yes, you are the professional, but
   the client is also an expert, and you will
   achieve the best result if you manage to
   work together as a team. Maybe it has to
   do with daring to be vulnerable toward the
   client. You don't have to 'know it all';
   you can seek the way forward together.
   Perhaps daring to be open like that is
   actually a sign of self-confidence.


Furthermore, there were three additional categories relating to perceptions of positive parts of the teaching, as well as possible aspects of the teaching that could be improved.

Clients have a significant positive impact on motivation and initiated learning. (One hundred and twenty-two fragments belonging to this category were coded; these fragments were taken from all 13 interviews). The most consistent statements were about the involvement of clients in the teaching sessions. The clients' views surprised students and impacted them emotionally. They were genuinely touched by the clients' stories. It also seemed significant that the clients were not 'their' clients, but were there as spokespersons for the documented cases, which stimulated even more open communication. Client involvement motivated students to better prepare for the lessons. They appreciated clients' positive comments about what they did or said. One student said,
   For me, the most valuable aspect was
   having the clients there. They gave me a
   whole new perspective. We usually tend
   to focus on our books and teaching
   materials, but they showed us a completely
   new angle. We'll meet clients when we do
   fieldwork, of course. But in this setting
   they were able to express their views
   freely. To my mind, that added value.


Active, guided instruction motivates students and intensifies the learning experience. (Three hundred and forty-nine fragments belonging to this category were coded; these fragments were taken from all 13 interviews). The students were positively challenged when working with clients. They stated that they had fun. They valued the positive, supportive, and safe atmosphere in the lessons, including the teacher's coaching and support. They also appreciated working in small groups. In addition, some students commented that the argument maps helped them gain a good overview of arguments. The model answers were considered helpful in learning how to reason, specifically because answers were used in the lessons as a basis for discussion and debate with the clients. The teacher commented,
   I want to start. There was lots of initiative-taking
   in the group. We never had to mess
   about waiting for someone to set the ball
   rolling, no, [name of the student] was
   already sitting on the edge of her chair,
   keen to get started. And then very bright
   faces, no bored stares, no Facebook
   checking, nothing but hard work
   throughout the session. We had to open
   the windows. All that exertion in the
   group generated lots of heat. You could
   see the steam rising from their heads.


Learning to think critically while considering evidence and that client input requires more time, practice, and in-depth study. (One hundred and forty-two fragments belonging to this category were coded; these fragments were taken from all 13 interviews). The participants felt that too little time was devoted to assignments. They would have appreciated more in-depth discussion and detailed explanation. They suggested similar exercises during the entire EBP process. Other suggestions were made in relation to details of the educational intervention that would increase more effective skills training. The students failed to see a link between the content of the course and the critical thinking test. As one student noted,

Perhaps more emphasis on skills instead of knowledge. In those four or five lessons,
   it was made clear that the opinions of
   clients are important, but more meetings
   would help, because I might want to know
   how exactly to go about it and how to find
   the right balance.


Research Question 2: The Impact of the Intervention on Self-Efficacy and Cognitive Skills

Table 2 contains the results of the impact of the intervention on self-efficacy in relation to EBP and EBDM, the generic cognitive skills used in EBDM, and critical thinking skills. Self-efficacy improved significantly with an average increase of 15.2% (SD = 8.1 %, p = .002). Self-efficacy in relation to the EBDM part of the EPIC increased by 15.1% (SD = 9.2%, p = .002) on average. There was greater variability in the essay SOLO scores, but scores still improved significantly: The average increase was 0.8 (SD = 1.0, p = .03), 0.9 (SD = 0.9,p = .015), and 0.9 (SD = 1.0,p = .016) for selection, analysis, and conclusion, respectively. The increase in the overall SOLO score was 0.9 (SD = 0.8,p = .012) on average. The results of the critical thinking test were highly variable; there was no progression at a group level.

Discussion

The purpose of this mixed-method descriptive study was to describe the impact of an intervention promoting self-efficacy and the generic cognitive skills used in EBDM. Empirical evidence regarding necessary elements of EBP teaching can inform the development of theoretically sound teaching interventions in practice. To our knowledge, this is the first study exploring a theory-driven intervention directed toward the fourth step of the EBP process, EBDM, using quantitative outcome measures that include self-efficacy in relation to EBDM and the generic

cognitive skills used in EBDM.

The intervention was well received. The students expressed better interactions with clients and a deeper understanding of the importance of individually tailoring evidence-based decisions. They also expressed improvements in their selection and organization of information, including research evidence. An increased understanding of the cognitive strategies used in reasoning and caution in reasoning was experienced. These positive perceived learning outcomes were in line with objective gains in self-efficacy in relation to EBDM and the generic cognitive skills used in EBDM. Working with real clients, as well as performing active, guided assignments, were perceived as powerful strategies for learning EBDM. However, the students were unanimous in their opinion that the intervention was too short and that more time was needed to pursue the issue in greater depth. This may explain why there were no improvements on the critical thinking test. Therefore, in the present study, quantitative and qualitative data provided a convergent picture, i.e., that while the applied teaching intervention included potentially useful elements in terms of providing important prerequisites in EBDM, such teaching requires more time and depth.

There are several findings that merit further discussion. First, we measured objective outcomes, such as the quality of the generic cognitive skills involved in EBDM. This has yet to be done in research on EBP education. Systematic reviews of EBP education report that the focus on outcome measures in EBP is on critical appraisal skills, and that other outcomes "were often only described narratively as improved or not, with vote counting used" (Young et al., 2014, p. 10). Though some studies have shown increased use of evidence (Harris et al., 2011; Maggio, Tannery, Chen, ten Cate, & O'Brien, 2013; Van Lew & Singh, 2010), these studies do not provide in-depth information on how this evidence is applied. Thorough, evidence-based decisions applied in shared decision making require more than a superficial application of research evidence, implying a sort of cut and paste action. Critical thinking should be applied thoroughly in decision making. This includes consulting the client about the options and the decision to be made. Considering that research evidence is often difficult for students to appraise, apply, and explain to clients, it is probably especially important to monitor how evidence is used in reasoning and interacting with clients.

Secondly, the study highlights the importance of implementing critical thinking exercises in the context of EBDM and making their purpose explicit. A meta-analysis has shown that active, purposeful training in critical thinking is needed to increase critical thinking skills. Just mentioning critical thinking in the course objectives, without explicit instruction in these courses, has only limited effects (Abrami et al., 2014)

Thirdly, several features of the teaching intervention were informed by previous research and valued by the participants. For example, the participants valued real clients as spokespeople for the written cases, which is in accordance with research that stresses the importance of authentic situations in EBP education (Khan & Coomarasamy, 2006; Thomas et al., 2011). The presence of clients during the teaching sessions was perceived as highly motivating, which is consistent with research on the importance of engagement in effective education (van der Vleuten & Driessen, 2014). Hence, the perceived positive elements of the teaching are in line with the theoretical rationale supporting the educational intervention.

Limitations

There are limitations in this study. Our goal was to use a strong theoretical basis to pilot a teaching intervention and learning outcome measures in preparation for a larger scale multi-site study. In a larger-scale study, a control group can be applied in the research design to check the specific effect of the teaching intervention as a supplement to other education. The naturalistic context of this study involved challenges, such as the fact that students missed sessions. However, since we checked reasons for missed sessions and since these reasons were not related to the teaching intervention in any way, it is not likely that this influenced our results. Design-based educational research involves studying teaching interventions in multiple cycles, where education is developed, executed, evaluated, adapted, executed, and evaluated again. The present study represents a first cycle in this line of research and provides knowledge for further cycles. With regard to these limitations, it should also be said that we ensured quality by employing trustworthiness methods, such as member checks and an expert panel review of the data.

Another limitation was that the intervention was too short and not sufficiently in-depth. Indeed, experiential learning and critical thinking requires time and considerable rehearsal in different contexts for these complex skills to be thoroughly incorporated (van Merrienboer & Kirschner, 2012). This requires additional teacher contact time, which can place a heavy demand on resources. This study should be viewed as a first step in the research. It was informative in providing a first idea of potentially useful elements in a teaching intervention and for promoting self-efficacy and the cognitive skills used in EBDM. A secondary product of this research is a set of potential design conjectures, or recommendations, for requirements to be applied when designing EBDM education (see Appendix). These design conjectures guided the design of our educational intervention, were discussed with the teacher during the intervention, and were slightly revised after the final teaching session. These design conjectures inspire educational practice and can be refined in further research. The next steps in research will require larger numbers of students, application of a control group, and validated assessments to evaluate the effectiveness of such teaching.

Conclusions

Our findings suggest that the use of theory-driven teaching of EBDM leads to progression in client-centered decision making and improved generic cognitive skills and dispositions in decision making. Moreover, student experiences were supported by objectively measured significant increments of vital EBDM requisites, namely self-efficacy and the cognitive skills used in argumentation. It would appear to be important to provide such teaching in a simulated authentic context with clients, using critical thinking exercises in an active, safe, and supporting learning environment. Competence in these aspects of EBDM requires additional time and sustained practice.

Future research should assess the effectiveness of prolonged, targeted, theory-driven teaching on consultation and critical thinking skills for the whole EBP process in undergraduate as well as postgraduate students from different academic levels. Moreover, future studies on EBDM should use outcome measures that assess quality in applying critical thinking dispositions and generic cognitive skills in decision making, as well as patient consultation.

Erica Baarends, PhD, OT: Zuvd Hogeschool of Applied Sciences, research Centre for Educational Innovation and Continues Professional development and School of Occupational Therapy, Heerlen, The Netherlands.

Marcel Van der Klink, PhD: Zuvd Hogeschool of Applied Sciences, Chair at the Research Centre for Educational Innovation and Continuous Professional Development, Heerlen ; Maastricht University, Department of Educational Research and Development of the School of Business and Economics, Maastricht, The Netherlands.

Aliki Thomas, PhD, OT: McGill University, Assistant Professor at McGill University, School of Physical and Occupational Therapy; Center for Medical Education, Montreal, Canada.

Cover Page Footnote

We especially would like to thank all of the students and clients that were involved in this research. In addition, we want to thank the involved colleagues of Zuyd University of Applied Sciences who performed expert review on the analysis of the data.

References

Abrami, P. C., Bernard, R. M., Borokhovski, E., Waddington, D. I., Wade, C. A., & Pcrsson. T. (2014). Strategies for teaching students to think critically: A meta-analysis. Review of Educational Research, 85(2), 275-314. http://doi.org/10.3102/0034654314551063

Baarends, E. M. (2015). Evidence-based onderwijs voor Evidence-Based Practice. Een ontwerpgericht onderzoek over hoe ergotherapiestudenten te leren evidence-based te handelen. [Evidence-based education for evidence-based practice. A design-based research about how occupational therapy students learn to apply principles of evidence-based practice.] Heerlen, Zuyd University of Applied Sciences.

Biggs, J., & Tang, C. (2011). Teaching for quality learning at university: What the student does (4th ed.). New York, NY: McGraw-Hill.

Coster, W., & Schwartz, L. (2004). Facilitating transfer of evidence-based practice into practice. Education Special Interest Section Quarterly, 14(2), 1-3.

Crabtree, J. L., Justiss, M., & Swinehart, S. (2012). Occupational therapy master-level students' evidence-based practice knowledge and skills before and after fieldwork. Occupational Therapy In Health Care, 26(2-3), 138-149. https://doi.org/10.3109/07380577.2012.694584

Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008). Developing and evaluating complex interventions: The new Medical Research Council guidance. British Medical Journal, 337, a 1655. Retrieved from http://wwwbmi.com/content/bmi/337/bmi.al655.full.pdf

Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approachs. Thousand Oaks: SAGE publications.

Diermayr, G., Schachner, H., Eidenberger, M., Lohkamp, M., & Salbach, N. M. (2015). Evidence-based practice in physical therapy in Austria: current state and factors associated with EBP engagement. J Eval Clin Pract, 21(6), 1219-1234 1217p.

Dijkers, M. P., Murphy, S. L., & Krellman, J. (2012). Evidence-based practice for rehabilitation professionals: concepts and controversies. Arch Phys Med Rehabil, 93(8 Suppl), S164-176.

Elo, S., & Kyngas, H. (2008). The qualitative content analysis process. Journal of Advanced Nursing, 62(1), 107-115. https://doi.org/10.1111/j.1365-2648.2007.04569.X

Facione, P. A. (2007). Critical thinking: What it is and why it counts. Retrieved from http://wwwinsightassessment.com/Resources/ Critical-Thinking-What-It-Is-and-Why-It-Counts/ Critical-Thinking-What-It-Is-and-Why-It-Counts-PDF

Gillam S. L., & Gillam, R. B. (2008). Teaching graduate students to make evidence-based intervention decisions: Application of a seven-step process within an authentic learning context. Topics in Language Disorders, 28(3), 212-228. https://doi.org/10.1097/01.TLD.0000333597.45715

Harris, J., Kearley, K., Heneghan, C., Meats, E., Roberts, N., Perera, R, & Kearley-Shiers, K. (2011). Are journal clubs effective in supporting evidence-based decision making? A systematic review. BEME Guide No. 16. Medical Teacher, 33(1), 9-23. https://doi.org/10.3109/0142159X.2011.530321

Hulin, K., Black, L., Jensen, G. M., & Deutsch, J. E. (2011). Construct validity of the Health Science Reasoning Test. Journal of Allied Health, 40(4), 181-186.

Insight Assessment. (2014). The health sciences reasoning test HSRT [Test manual]. Retrieved from https://wwwinsightassessment.com/Products/Products-Summary/ Critical-Thinking-SkillsTests/Health-Sciences-Reasoning-Test-HSRT

Khan. K. S., & Coomarasamy, A. (2006). A hierarchy of effective teaching and learning to acquire competence in evidence-based medicine. BMC Medical Education, 6(59). https://doi.org/10.1186/1472-6920-6-59

Kloda, L. A., & Bartlett, J. C. (2009). Clinical information behavior of rehabilitation therapists: A review of the research on occupational therapists, physical therapists, and speech-language pathologists. Journal of the Medical Library Association, 97(3), 194-202. https://doi.org/10.3163/15365050.97.3.008

Lam, W. W. T., Fielding, R., Johnston. J. M., Tin, K. Y. K., & Leung, G. M. (2004). Identifying barriers to the adoption of evidence-based medicine practice in clinical clerks: A longitudinal focus group study. Medical Education. 38(9). 987-997. https://doi.org/10.1111/j.1365-2929.2004.01909.X

Maggio, L. A., Tannery, N. H., Chen, H. C., ten Cate, O., & O'Brien, B. (2013). Evidence-based medicine training in undergraduate medical education: A review and critique of the literature published 2006-2011. Academic Medicine. 88(1). 1022-1028. https://doi.org/10.1097/ACM.0b013e3182951959

Paans, W., Senneus, W., Nieweg, R. M., Krijnen, W. P., & van der Schans, C. P. (2012). Do knowledge, knowledge sources and reasoning skills affect the accuracy of nursing diagnoses? A randomised study. BMC Nursing. 11(11). https://doi.org/10.1186/1472-6955-11-11

Salbach, N. M., Guilcher, S., Jaglal, S. B., & Davis, D. A. (2010). Determinants of research use in clinical decision making among physical therapists providing services post-stroke: A cross-sectional study. Implementation Science. 5(11). 1-9. https://doi.org/10.1186/1748-5908-5-77

Salbach, N. M., Jaglal, S. B., & Williams, J. I. (2013). Reliability and validity of the evidence-based practice confidence (EPIC) scale. Journal of Continuing Education in the Health Professions. 33(1), 33-40. https://doi.org/10.1002/chp.21164

Sandelowski, M. (2000). Whatever happened to qualitative description? Research in Nnursing and Health. 23(4), 334-340. https://doi.org/10.1002/1098-240X(200008)23:4<334::AID-NUR9>3.0.CO;2-G

Thalheimer, W., & Cook, S. (2002). How to calculate effect sizes from published research: A simplified methodology. Work-Learning Research. 1-9. Retrieved from http://wwwbwgriffin.com/gsu/courses/edur9131/ content/Effect_Sizes_pdf5.pdf

Thomas, A., & Law, M. (2013). Research utilization and evidence-based practice in occupational therapy: A scoping study. American Journal of Occupational Therapy. <57(4), e55-e65. https://doi.org/10.5014/ajot.2013.006395

Thomas, A., Sa royan. A., & Dauphinee, W. D. (2011). Evidence-based practice: A review of theoretical assumptions and effectiveness of teaching and assessment interventions in health professions. Advances in Health Sciences Education. 16(2). 253-276. https://doi.org/10.1007/sl0459-010-9251-6

Tilson, J. K., Kaplan, S. L., Harris, J. L., Hutchinson. A., Ilic, D., Nicdcrman. R., ... Zwolsman, S. E. (2011). Sicily statement on classification and development of evidence-based practice learning assessment tools. BMC Medical Education. 11(78). https://doi.org/10.1186/1472-6920-11-78

Van den Akker, J., Gravemeijer, K., McKenney, S., & Nieveen, N. (2006). Educational design research. London. New York: Routledge.

van der Vleuten, C. P. M., & Dricsscn. E. W. (2014). What would happen to education if we take education evidence seriously? Perspectives on Medical Education. 3(3), 222-232. https://doi.org/10.1007/s40037-014-Q129-9

Van Lew, S., & Singh, N. (2010). Integrating the centennial vision into an evidence-based fieldwork-learning experience. Occupational Therapy in Health Care. 24(1). 68-73. https://doi.org/10.3109/07380570903410852

van Merrienboer, J. J. G., & Kirschner, P. A. (2nd ed.). (2012). Ten steps to complex learning: A systematic approach to four-component instructional design. London: Routledge.

West, R. F., Toplak, M. E., & Stanovich, K. E. (2008). Heuristics and biases as measures of critical thinking: Associations with cognitive ability and thinking dispositions. Journal of Educational Psychology, 100(4), 930.

Stanovich, K. (2011). Rationality and the reflective mind. Oxford: Oxford University Press

Young, T., Rohwer, A., Volmink, J., & Clarke, M. (2014). What are the effects of teaching evidence-based health care (EBHC)? Overview of systematic reviews. PLOS ONE. 9(1). e86706. https://doi.org/10.1371/journal.pone.0086706

Appendix

Proposed Design Conjectures for the Construction of an Educational Intervention Focusing on Evidence-Based Decision Making

The table below gives a short description of the design conjectures arising from the research. The concept conjectures used to develop the five teaching sessions were refined in the process of the present study in consultation with the teacher, but also following the results of the present study, by the first author. Further research and development of educational interventions aimed at improving EBDM in health professional students could refine these conjectures.
Subject       Design conjectures Proposed requirements when designing
              education aimed at EBDM

Goals and     More self-efficacy in EBDM.
learning
process       More skills in empathic consultation with clients.

              Increased critical thinking skills including reflection
              skills to improve reasoning relating to both research
              evidence and client information.

                * The student explores a premise by asking questions
                  about arguments for and against.
                * The student explores the information basis of the
                  arguments.
                * The student asks questions concerning the
                  trustworthiness of this information.
                * The student explores other possible options
                  relating to the premise.

Assignments   Assignments involve EBDM in authentic situations or
              tasks where a decision needs to be made; first steps of
              EBP are given.

and context   Involve real clients as spokespersons of written cases;
              clients can ask critical questions and support
              engagement in the assignments. These clients need to be
              able to project themselves into a case or problem that
              is not entirely the same as their own.

              Make sure that relevant research evidence is available
              for the decision at hand.

              Multiple decisions or options are realistic, so that
              discussion is prompted.

              Make sure that in the lesson active didactic forms are
              used that allow discussion, consultation, and
              communication with peers, the lecturer, and the
              clients.

              Assignments and lessons should be directed at training
              the cognitive skills involved in critical thinking.

              Nurture and safeguard a safe, open, non-judgmental
              learning environment whenever possible.

Support       Consciously apply methods of Cognitive Apprenticeship
and means     and Situational Learning.

              Apply educational principles relating to critical
              thinking, such as the use of argument maps to visualize
              reasoning.

              Ensure that assignments develop from simple to more
              complex so that students experience successes.


Erica Baarends

Zuyd Hogeschool of Applied Sciences, erica.baarends@zuyd.nl

Marcel Van der Klink

Zuyd Hogeschool of Applied Sciences, marcel.vanderklink@zuyd.nl

Credentials Display

Erica Baarends, PhD, OT; Marcel Van der Klink, PhD; Aliki, Thomas, PhD, OT

DOI: 10.15453/2168-6408.1292

Caption: Figure 1. Model of the Medical Research Council of developing and testing complex interventions. Reprinted with permission from Craig et al. (2008).
Table 1 Condensed Description of a Teaching Session

Teaching session 3: Insight into the reasoning to be applied to
occupational therapy issues.

Goal:

* Knowledge about and elementary application of complex reasoning.

* Knowledge about and elementary application of pro and contra
arguments in evidence-based decision making.

Preparation of the students prior to the session:

* Students read information about reasoning. Subjects were, for
instance, the difference between facts, premises, arguments, and
rebuttals, or differences between a simple reasoning structure and
complex reasoning structures.

* Students performed assignments to foster reasoning skills. For
instance, based on given evidence, students made an overview of
available questionnaires on assessing fatigue. They drew up an
argument map, displaying pro and contra arguments for these
questionnaires.

Content of the lesson:

* Every session started by repeating the ground rules in the
sessions, which were set up in the first session. These ground
rules ensured a safe learning environment and were meant to
increase questioning, critiquing, being open to other options,
opinions, etc.

* Homework assignments were discussed and feedback was received.

Students compared their own argument maps with a worked out example
argument map (a model example).

* The six clients who had been invited to assist in the educational
intervention entered the session.

* Students interviewed clients in subgroups, using two fatigue
questionnaires. This was followed by a discussion with the clients
about pro and contra arguments for using these questionnaires. The
students used their previously prepared argument maps, but also
added new pro and contra arguments that were generated by the
discussion with clients. Important issues in reasoning behind the
selection of a certain questionnaire and balancing pro and contra
arguments were discussed plenary.

* Students tried out two occupational intervention methods with the
clients in subgroups. The first intervention method was a virtual
reality game that has been used to support upper extremity
rehabilitation after suffering from stroke (Wii game). The second
intervention method was mirror therapy, also applied in upper
extremity rehabilitation, after suffering from stroke. They did
some simple exercises, together with the clients, to have a basic
experience with these intervention methods. This was followed by a
discussion with the clients (in subgroups) about pro and contra
arguments for and against these intervention methods. The students
and the clients applied these arguments, keeping in mind a written
case scenario patient who had a stroke (Mrs Stevens). During the
discussion, the students made concept argument maps when
considering these intervention methods in stroke rehabilitation.
The clients asked questions, articulated intuitive remarks and
opinions about these intervention methods, and ensured that
Mrs Stevens' voice was heard. First ideas about reasoning behind
therapy choices were discussed plenary.

The homework assignment for the next session was to expand the
argument maps further using the experiences from the session and
given evidence to prepare a debate about these intervention
methods.

* Finally, the goals of the session were evaluated and the session
was closed.

Table 2
Outcome Measures in Self-efficacy, Generic Cognitive Skills,
and Critical Thinking

Measurement         Mean    SD     Min--Max    Cohen's d

EPIC before         59      7      45-72
* decision part     * 68    8      (60-87)
EPIC after          74      8      62-90       1.8 (1.7)
* decision part     * 83    8      (70-97)     (very large)
SOLO sel before     1.7     0.8    1-3
SOLO sel after      2.4     0.5    2-3         0.7 (large)
SOLO analy before   1.7     0.8    1-3
SOLO analy after    2.6     0.7    1-3         0.9 (large)
SOLO conci before   1.3     0.9    0-2
SOLO conci after    2.3     0.9    0-3         0.85 (large)
HSRT before         20      2      17-24
HSRT after          21      4      18-25       0.4 (small)

Note. * decision part: There are three statements in the EPIC
questionnaire that specifically refer to EBDM. The mean score
of these three statements was computed before and after
the intervention.
COPYRIGHT 2017 Western Michigan University, Department of Occupational Therapy
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2017 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Baarends, Erica; Van der Klink, Marcel; Thomas, Aliki
Publication:Open Journal of Occupational Therapy
Article Type:Report
Date:Jun 22, 2017
Words:9249
Previous Article:Determining the feasibility, content validity, and internal consistency of a newly developed care coordination scale for people with brain injury.
Next Article:Effectiveness of a multidisciplinary rehabilitation program following shoulder injury.
Topics:

Terms of use | Privacy policy | Copyright © 2020 Farlex, Inc. | Feedback | For webmasters