Scaffolded reflection as a tool for surfacing complex learning in undergraduate research projects.

Design of Undergraduate Research Projects that are Assessed for Degree Credit

Providing opportunities for undergraduate students to engage in research is widely seen as desirable for a variety of reasons, including improved connections between a university's research and teaching activities, improved student motivation and retention, and valuing students' learning about research (Howitt et al. 2010; Hunter et al. 2006; Kardash 2000; Linn et al. 2015; Lopatto 2007; Russell et al. 2007; Seymour et al. 2004). Students may benefit by deepening both their knowledge of their discipline (the questions currently preoccupying it, the methods available to address those questions, the kinds of evidence seen as valid) and generic skills (problem-solving, critical and analytical thinking, independence, and creativity). Thus there are very good reasons to promote a variety of research experiences, spanning the spectrum from individual supervised projects to structured, course-based activities, as normal components of the undergraduate curriculum. However, once such experiences become graded activities that contribute to a student's GPA, new questions arise regarding their design, supervision, and assessment, as well as what they contribute to students' learning outcomes.

In the following, we focus mainly on apprentice-style undergraduate science research experiences (UREs). Supervisors and other faculty may hope that UREs will facilitate rich learning about the complex nature of research and lead to development of professional research expertise (Wilson et al. 2015b), including the ability to generate and frame research questions, as well as understanding the necessity of adopting a critical stance regarding one's own methods and data, of being simultaneously rigorous and flexible, and of making judgments and decisions at all stages of the research process. However, although many students who undertake UREs report "learning what research is like" in general terms, only a very few appear to attain high levels of understanding of the nature of research (Howitt et al. 2010; Hunter et al. 2006; Kardash 2000; Linn et al. 2015).

This suggests there is a need to reconsider the structure and assessment of research experiences to focus on developing professional expertise in addition to disciplinary knowledge and skills. Developing from novice to expert researcher requires a sense of the interconnectedness, incompleteness, contingency, and ambiguity of current knowledge, in addition to the disciplinary context. It thus requires the development of the kinds of "wicked competences" (Knight and Yorke 2008) that are needed to recognize, understand, and address "wicked problems." We argue that if UREs are to be made more effective in encouraging steps along this novice-to-expert transition, we need to pay attention to two key issues: re-aligning assessment and assuring high-quality supervision.

Assessment is widely acknowledged to drive learning, and the effective constructive alignment of assessment activities has been demonstrated to result in improved learning outcomes (Biggs and Tang 2011). UREs, however, are often assessed primarily through a final report based on a scientific paper. As we have argued elsewhere (Howitt and Wilson 2014), this is typical of much science education in which there is a focus on the outputs rather than on the process and practice of science. Textbooks and scientific papers give what appears to be a complete story, but conceal the process of discovery, leaving the misleading impression that knowledge is certain. Project reports modelled on such formal academic communication may therefore result in students' attention being focused on experimental outcomes (Wilson et al. 2015b), rather than on seeing the practice of science, including the inevitable troubleshooting and cycles of experimental design and re-design, as part of their learning. Problems may be seen as delays in getting results rather than as opportunities for learning. Some form of activity that allows students and their supervisors to recognize and value learning about the process and practice of science could be a valuable addition to the assessment that leads to grading of a URE.

The supervisor of a research project is key to the success of the experience for the student (Howitt et al. 2010; Lopatto 2003; Russell et al. 2007). Research projects that are done for degree credit occupy contested ground for the supervisor, somewhere between a teaching experience with a focus on student learning and a contribution to the supervisor's program of research with a focus on the results obtained. Where the supervisor sits along this continuum will have a big impact on his or her approach to supervision. Although we might expect that the focus would be on student learning outcomes, our experience with a research-intensive undergraduate degree shows that this may not be the case (Howitt et al. 2010; Wilson et al. 2012). Interviews with supervisors revealed that while most valued development of "wicked" competences and an understanding of the nature of research as learning outcomes for their students, they agreed that these outcomes were difficult to assess (Wilson et al. 2015b).

Instead, supervisors often rely on under-articulated "markers for growth" (Laursen et al. 2010, 170) to make judgments of student progress and achievement. However, these do not provide the kind of evidence on which effective teaching interventions or grading decisions can reliably be made. Thus, supervisors experience a lack of clarity both in the aims of the research project and the value of the learning experience to the student. Quality of supervision might be improved if supervisors had a greater awareness of these issues, potentially leading to greater equity in the way projects are framed, supported, and assessed.

Introducing Structured Reflection: the TREASURE project

One way in which students' attention can be focused on process and their thoughts and progress made more visible to supervisors is through the use of structured reflection. Reflection is seen as an essential component of a model for the learning cycle in inquiry-based projects (Justice et al. 2007), increasing engagement and building students' capacity for self-evaluation. Reflection can assist the students to deconstruct an experience by considering what has happened and why and thus help them to adjust to the unfamiliar demands of research or inquiry. However, students need some guidance and structure in order to reflect effectively. In addition, there is evidence that without incentive to do so, significant proportions of students do not regularly reflect on their learning (Spronken-Smith et al. 2011), and thus may benefit less from inquiry experiences than they might.

Reflective journals are commonly used during work placements and internships in professional degrees because they allow students the space to come to terms with the complexity of the workplace, by directing students' attention to the processes, interactions, and decisions that structure the workplace; they also are tools for self-assessment and ongoing development. The realization that the wicked problems of the workplace are different from the structured problems usually experienced in undergraduate courses can be difficult for students. Undergraduate research experiences share this feature with work placements and thus reflective journals are also likely to add value to the experience.

A project we named TREASURE (Teaching Research-Evaluation and Assessment Strategies for Undergraduate Research Experiences) allowed us to introduce structured reflective logbooks in UREs. Our aim was to reveal and enhance "invisible learning" by providing a scaffold for both students and supervisors to recognize and value increased understanding of the nature and practice of research and the learning of generic skills. By asking students to respond to questions regularly throughout the URE, we wanted to document their changing ideas and perceptions. Previous publications provide details of the project and our findings ((Howitt et al. 2014; Wilson et al. 2015a; Wilson et al. 2015b; Wilson et al. 2015c). Our intention here is to briefly summarize the approach and consider some implications.

The reflective journals were structured by the inclusion of questions that asked students about their learning and their experiences of research. The Prompt Question Framework (see Table 1) was developed for our project. Prompt questions need to be carefully tailored to course contexts and desired learning outcomes; while they are effective in directing students' attention toward aspects of their experience, they can also direct attention away from other aspects. The questions sow the seeds of reflection, and since students only have limited time, they are likely to reflect on what is being highlighted for them by the questions in front of them. Our questions were initially designed for science UREs and were developed after workshops and interviews with supervisors about their learning outcomes for students' projects. At the workshops, supervisors were able to suggest questions and comment on the value of existing questions. Although some core questions were always included, others were modified in response to supervisors' feedback or to accommodate other disciplines, including arts and social sciences and lecture courses that included embedded research experiences.

The questions used as prompts should reflect the learning outcomes of the course or other research experience, and can also draw attention to particular aspects of the research experience. For example, we added a question on collaboration in courses where the research project was done in groups. We found the approach to be widely applicable. During the two years of the TREASURE project, 330 students from three institutions participated.

The reflective journals were delivered to students in the form of a private blog, known as their Learning Logbook, that was accessible to the student, his or her supervisor, and the TREASURE project team. Students could choose which questions from the Prompt Question Framework to answer; they were required to do this about five times per semester. Where possible, the questions were incorporated into assessment as a low-stakes item, with marks allocated for answering the questions rather than for the quality of the answers. Our aim was to give students some incentive to take the Learning Logbook seriously, but to avoid issues that may arise when reflective writing is formally assessed on the perceived quality of the reflections themselves (Kember et al. 1996; O'Connell and Dyment 2011).

Around 80 percent of students complied with the requirements, suggesting that this was not too onerous a burden. Most logbooks provided useful insights into the progress of the research project and the students' thinking and feelings about their achievements and problems. About one-third of the logbooks demonstrated high-quality reflection. Our aim was not to train students in high-level reflection, but rather to give them a space to consider what they were learning and doing. Different assessment structures and increased support for students to develop reflective capacity could lead to different outcomes or meet different goals.

We found that the Learning Logbooks illustrated the development of wicked competences in a way that traditional assessment does not (Wilson et al. 2015a). Questions asking about unanticipated problems were particularly successful in prompting students to write about the process of science as they experienced it. At their best, the logbooks showed that students understood the complexity of research and were able to generalize from their own experience to the nature of science. For example, some students reported becoming more critical of the literature now that they understood how data were generated. Others expressed surprise at the role of analysis and creativity in experimental design, recognizing that science was more than simply good experimental technique. We have previously provided evidence from the logbooks showing students thinking critically, gaining independence, and developing a sense of themselves as scientists (Wilson et al. 2015a; Wilson et al. 2015c).

Issues and Benefits of Learning Logbooks for Stakeholders

Students. By prompting students to think about their projects regularly, logbooks may help students see the bigger picture. In a semester-long project, with most of the assessment at the end, there is a tendency for students to focus on the experimental procedures at the expense of thinking about the context and aims of their projects. Many students found that simply having to make regular posts forced them to think about what they had achieved, helping them keep on track. This suggests that the logbooks are successful in assisting students to engage with their projects in ways that promote deeper learning, as shown by the following two quotes from student-feedback surveys:

"I got a lot more out of my actual project because it [the Learning Logbook] encouraged me to engage with the process and think about my actions throughout it."

"I think also that the process of writing helped to shape and focus a 'world view' of scientific research that was formerly fragmented (at least in my head)."

There is also some evidence that keeping logbooks or any reflective journal may help students develop metacognitive skills; there appears to be a link between writing and learning, as the process of having to articulate thoughts in writing can be beneficial (Bean 2011; Rivard 1994). Some students found the logbook posts a burden, however, and others wrote in a purely descriptive fashion, more appropriate for a laboratory notebook.

Reflective writing may be more effective when regular feedback is given. How much guidance and feedback is given is an issue to consider. We had hoped that supervisors would provide feedback because they were given access to their student's logbook. However, most supervisors did not regularly check their students' posts, in some cases because they were unfamiliar with the blog system used and in others because they felt no need of an extra communication mechanism. This did not seem to inhibit use of the logbooks by students, but incorporating feedback may increase the journals' effectiveness. More interactive approaches in other projects have involved emailing responses to supervisors so that they respond directly or, in a classroom context, regular checking of logbooks by the instructor (Audet et al. 1996; Campbell and Lom 2006).

URE Supervisors. Learning Logbooks can provide a window into students' thinking and thereby help supervisors better understand their students' conceptions and misconceptions about research. This could lead to timely interventions for a particular student but also could feed into better project design in the future. Logbooks can therefore act as a professional development mechanism for supervisors by helping them reflect on their strengths and weaknesses in supervising undergraduate researchers. This has the potential for significant improvements to UREs because, as noted above, the supervisor is one of the major factors that determine the quality of the student experience (Howitt et al. 2010; Lopatto 2003; Russell et al. 2007). Although most supervisors in the TREASURE project did not read the student researcher's logbook during semester, we sent the responses to each supervisor at the end of the semester. Later interviews with supervisors demonstrated that the logbooks often provided a different perspective on the students' experience, as the following comments illustrate:

"I felt like maybe that's what I didn't do such a good job of this semester. I guess reading [the Learning Logbook] it was good to see him go through that process, but maybe if I had spoken to him or if we had had that direct contact about those sort of issues that he had, we could've ... maybe it would've been a bit more of a successful project."

"She seems in some ways a little bit less confident in the blog than she projects when talking to us, so that was interesting. So she felt . I think in the blog she said she sometimes finds it difficult to ask for help."

Some supervisors became aware that they were taking too much about the research environment for granted. It appears common to overestimate the students' knowledge both about particular disciplines and about their understanding of the nature of research. Logbooks can assist supervisors in developing more realistic expectations and project aims. One striking feature of the logbooks is the emotional engagement of students as they describe their responses to the highs and lows of their projects. Recognition of the degree to which students identify with and take ownership of their projects may also assist supervisors to deal more effectively with their students.

Course convenors/URE program directors. Logbooks can allow convenors and program directors to monitor the success of a URE program by maintaining oversight of students' progress, problems, and learning. For example, the logbooks could be used to identify common problems or misconceptions, provide feedback on whether the course is meeting its aims, or identify areas in which students need more help. Common issues could be dealt with through improved student induction procedures or supervisors' professional development. This is particularly important where the URE is done for degree credit, when a more structured program may be desirable to ensure equity for students. When the research experience is embedded within a lecture course, the logbooks provide an effective monitoring mechanism that allows the course convenor to identify and respond to common issues in a timely fashion. This could be done at a whole-class level, without necessarily identifying individual students.

Occasionally, individual students wrote about potentially serious problems in their logbooks--for example, a communication breakdown between a student and his or her supervisor or pressure from a supervisor to spend too much time on the project. This may raise ethical issues concerning the need for intervention. Expectations and responsibilities need to be made clear to all students and staff involved in the course. In particular, a decision needs to be made about the confidentiality of logbooks, and students need to know who has access to them and how their responses will be used.

Institutions. Logbooks can provide a mechanism by which development of generic skills and graduate attributes (such as wicked competences) can be monitored. The results could be used in a variety of contexts, for example, to demonstrate meeting externally imposed accreditation criteria or to justify the existence and structure of a particular course. For research-intensive institutions, logbooks can provide evidence that students benefit from exposure to the research environment.

Implementation of logbooks has costs that need to be considered. Implementation across a program may be better done centrally, allowing control of access to the logbooks and facilitating their oversight. However, maintenance of the blog system we used required some time, adding to staff costs. Less formal systems based on email or learning-management systems may be appropriate in other contexts. Training for students in reflective writing and for supervisors in providing feedback or assessing reflective writing could potentially add other costs. We found that science faculty often lacked confidence in using reflective writing as an assessment item, although they recognized it might be valuable. In the TREASURE project, support was provided by the project team through discussions with faculty prior to and during the course in which research was conducted, as well as the provision of guidelines for students (Howitt et al. 2014) and the Prompt Question Framework. Without such support, implementation may be less successful. It is important that issues of implementation be considered in light of the aims of the undergraduate research program so that the program is sustainable and fit for purpose.

Conclusion

When research experiences are introduced into degree programs and are assessed for credit, the issues of students' learning outcomes and equity in supervision and assessment should take priority over simply giving students a taste of research. The typical focus of URE assessment on students' presentation of findings can result in students' undervaluing learning about the process of research, even though such learning is highly valued by their supervisors. The TREASURE project's prompt questions were effective in redirecting students' attention to learning about the process and practice of science, showing that structured reflection can be used to provide evidence of particularly complex learning outcomes, such as critical, independent, and creative thinking, and improved understanding of the nature of a discipline. In addition to promoting student learning, this small change to assessment may also contribute to supervisors' professional development by assisting them to see projects through their students' eyes. We conclude that, although somewhat complex to introduce, the process of structured reflection adds significant value to UREs.

Acknowledgment

We are grateful to the Australian Government Office for Learning and Teaching for funding for the TREASURE project.

References

Audet, Richard H, Paul Hickman and Galina Dobrynina. 1996. "Learning Logs: A Classroom Practice for Enhancing Scientific Sense Making." Journal of Research in Science Teaching 33 (2): 205-222. doi: 10.1002/(sici)10982736(199602)33:2<205::aid-tea5>3.3.co;2-b.

Bean, John C. 2011. Engaging Ideas. The Professor's Guide to Integrating Writing, Critical Thinking and Active Learning in the Classroom. San Francisco: JosseyBass.

Biggs, John and Catherine Tang. 2011. Teaching for Quality Learning at University: What the Student Does. Maidenhead : Open University Press.

Campbell, A. Malcolm and Barbara Lom B. 2006. "A Simple E-mail Mechanism to Enhance Reflection, Independence, and Communication in Young Researchers." CBE-Life Sciences Education 5 (4): 318-322. doi: 10.1187/ cbe.06-06-0170.

Howitt, Susan and Anna Wilson A. 2014. "Revisiting 'Is the scientific paper a fraud?'." EMBO Reports 15 (5): 481-484. doi: 10.1002/embr.201338302.

Howitt, Susan, Anna Wilson and Denise Higgins D. 2014. "TREASURE (Teaching Research - Evaluation and Assessment Strategies for Undergraduate Research Experiences)." Sydney: Australian Government Office for Learning and Teaching.

Howitt, Susan, Anna Wilson, Kate Wilson and Pam Roberts. 2010. "'Please Remember We are Not All Brilliant': Undergraduates' Experiences of an Elite, Research intensive Degree at a Research intensive University."

Higher Education Research & Development 29 (4): 405-420. doi: 10.1080/07294361003601883.

Hunter, Anne-Barrie, Sandra L. Laursen and Elaine Seymour. 2006. "Becoming a Scientist: The Role of Undergraduate Research in Students' Cognitive, Personal, and Professional Development." Science Education 91 (1): 36-74. doi: 10.1002/sce.20173.

Justice, Christopher, James Rice, Wayne Warry, Sue Inglis, Stefania Miller, S. and Sheila Sammon .2007. "Inquiry in Higher Education: Reflections and Directions on Course Design and Teaching Methods." Innovative Higher Education 31 (4): 201-214. doi: 10.1007/s10755-006-9021-9.

Kardash, CarolAnne M. 2000. "Evaluation of an Undergraduate Research Experience: Perceptions of Undergraduate Interns and Their Faculty Mentors." Journal of Educational Psychology 92 (1): 191-201. doi: 10.1037//0022-0663.92.1.191.

Kember, David, Alice Jones, Alice Loke, Jan McKay, Kit Sinclair Harrison Tse, Celia Webb, Frances Wong, Marian Wong, Po Wa Yan and Ella Yeung. 1996. "Developing Curricula to Encourage Students to Write Reflective Journals." Educational Action Research 4 (3): 329-348. doi: 10.1080/0965079960040304.

Knight, Peter and Mantz Yorke M. 2008. "Assessment Close Up: The Limits of Exquisite Descriptions of Achievement." International Journal of Educational Research 47 (3): 175-183. doi: 10.1016/j.ijer.2008.01.005.

Laursen, Sandra L., Anne-Barrie Hunter, Elaine Seymour, Heather Thiry and Ginger Melton. 2010. Undergraduate Research in the Sciences: Engaging Students in Real Science. San Francisco: Jossey-Bass.

Linn, Marcia C., Erin Palmer, Anne Baranger, Elizabeth Gerard and Elisa Stone. 2015. "Undergraduate Research Experiences: Impacts and Opportunities." Science 347 (6222): 1261757-1261757. doi: 10.1126/science.1261757.

Lopatto, David. 2003. "The Essential Features of Undergraduate Research." CUR Quarterly 23 (3): 139-142.

Lopatto, David. 2007. "Undergraduate Research Experiences Support Science Career Decisions and Active Learning." CBE-Life Sciences Education 6 (4): 297306. doi: 10.1187/cbe.07-06-0039.

O'Connell, Timothy S. and Janet E. Dyment. 2011. "The Case of Reflective Journals: Is the Jury Still Out?" Reflective Practice 12 (1): 47-59. doi: 10.1080/14623943.2011.541093.

Rivard, Leonard P. 1994. "A Review of Writing to Learn in Science: Implications for Practice and Research." Journal of Research in Science Teaching 31 (9): 969-983. doi: 10.1002/tea.3660310910.

Russell, Susan H, Mary P. Hancock, James McCullough. 2007. "THE PIPELINE: Benefits of Undergraduate Research Experiences." Science 316 (5824): 548-549. doi: 10.1126/science.1140384.

Seymour, Elaine, Anne-Barrie Hunter, Sandra L. Laursen and Tracee DeAntoni. 2004. "Establishing the Benefits of Research Experiences for Undergraduates in the Sciences: First Findings from a Three-Year Study." Science Education 88 (4):493-534. doi: 10.1002/sce.10131.

Spronken-Smith, Rachel A., Rebecca Walker, Julie Batchelor, Billy O'Steen, and Tom Angelo. 2011. "Enablers and Constraints to the Use of Inquiry-based Learning in Undergraduate Education." Teaching in Higher Education 16 (1): 5-28. doi: 10.1080/13562517.2010.507300.

Wilson, Anna, Susan Howitt, and Denise Higgins. 2015a. "Assessing the Unassessable: Making Learning Visible in Undergraduates' Experiences of Scientific Research." Assessment & Evaluation in Higher Education. In press. doi: 10.1080/02602938.2015.1050582.

Wilson, Anna, Susan Howitt and Denise Higgins. 2015b. "A Fundamental Misalignment: Intended Learning and Assessment Practices in Undergraduate Science Research Projects." Assessment & Evaluation in Higher Education. In press. doi: 10.1080/02602938.2015.1048505.

Wilson, Anna, Susan Howitt, Denise Higgins and Pam Roberts. 2015c. "Making Critical Thinking Visible in Undergraduates' Experiences of Scientific Research." In The Palgrave Handbook of Critical Thinking in Higher Education, edited by Martin Davies and Ronald Barnett, 491-508. New York: Palgrave MacMillan. doi: 10.1057/9781137378057.0037.

Wilson Anna, Susan Howitt, Kate Wilson and Pam Roberts. 2012. "Academics' Perceptions of the Purpose of Undergraduate Research Experiences in a Research-intensive Degree." Studies in Higher Education 37 (5): 513-526. doi: 0.1080/03075079.2010.527933.

Susan Howitt, Australian National University, Anna Wilson, University of Stirling

Susan Howitt

Australian National University, susan.howitt@anu.edu.au

Susan Howitt heads the Division of Biomedical Science and Biochemistry and also is deputy head of biology teaching and learning in the Research School of Biology at the Australian National University. Originally a biologist, she completed a master's degree in higher education in 2009, and her main research interest now is student perceptions of science and scientific research and how those views impact their learning.

Anna Wilson is a former nuclear physicist who has worked at universities in the UK, United States, and Australia. She has received two teaching awards from the Australian Learning and Teaching Council. While at the Australian National University, she and Susan Howitt worked together to develop the project described in this article. With Howitt, she has co-taught an interdisciplinary freshman course aimed at increasing students' understanding and critical thinking regarding the nature and practice of scientific research. Wilson is currently undertaking a second PhD, in education, with the sponsorship of the UK's Higher Education Academy, at the University of Stirling.

doi: 10.18833/curq/36/4/8

Table 1. Prompt Question Framework for Undergraduate
Science Research Projects

First Learning Logbook questions:

Why have you chosen to do a research project and
what are you expecting to get out of it?

Have you undertaken a research project previously?
Describe it.

What are you expecting to be different in this
research project experience from your normal
coursework?

What skills do you think you need to be a good
researcher?

Regular questions (students choose three to answer
in each logbook post):

How have your recent activities helped you address
your research question?

Have you made progress in the last fortnight?
  * If so, what allowed you to make progress?
  * What kind of activities did you engage in
    that helped you make progress?

Problems and obstacles are a normal part of
research. Did you encounter any?
  * If so, what made them problems?
  * How did you go about solving them?
  * What would have helped you overcome them?

What might you have done differently if you had
known two weeks ago what you know now?

Has your research question changed? If so, why,
and what has it changed to?

Have you found/learned anything unexpected?
Explain.

Has anything you've learned shifted the focus or
aims of your project? How?

How have you chosen the approach or methods that
you are using for your project?

What are the connections between your research
activities and your other studies?

Can you see ways in which you could apply what you
have learned to other activities, in or out of
university? How?

What have you learned about your project topic,
science or research more generally?

What have you learned about yourself from doing
this project?

Has your view of what research is changed from
your project experience? Explain how.

Has this fortnight's activities raised any
questions you would like to discuss with your
supervisor? If so, list them.

Final logbook questions:

Has your research project met your expectations?
Why/why not?

What have you learned from undertaking this
research project?

Would you do another research project if you had
the opportunity? Why/why not?

What skills do you think you have developed or
strengthened through your research project?

Article Details

Printer friendly Cite/link Email Feedback
Title Annotation:	From the International Desk
Author:	Howitt, Susan; Wilson, Anna
Publication:	Council on Undergraduate Research Quarterly
Article Type:	Report
Geographic Code:	1USA
Date:	Jun 22, 2016
Words:	4628
Previous Article:	Undergraduate research on short-term, faculty-led study abroad.
Next Article:	Development of a client-based undergraduate molecular biology laboratory.
Topics:	College students Social aspects Education, Higher Research Educational research Higher education Research