Assessing higher levels of learning in post-secondary education. (Online Instruction).

Abstract

This article describes the results of a study that investigated how to assess higher levels of online learning in post-secondary education. The results of a six item open-ended questionnaire indicated that traditional forms of assessment typically used (i.e., paper and pencil exams) has limited application in online learning environments, indicating a need for alternative assessment strategies. Alternative assessment strategies suggested in the results of this study include the use of negotiated contracting, embedded assessment, learning portfolios, presentations, and repertory grids.

**********

Background: Why Alternative Assessment?

To receive course credit in formal post-secondary learning environments, we must provide our learners with opportunities to demonstrate that they have acquired an understanding of the content presented. Yet most of us know from our own practices that assessing learning is a difficult process, especially higher levels of learning. How best to assess learning has been the topic of many papers and studies, with diverse outcomes and recommendations. For example, Reeves (2000) maintains that "traditional assessment (commonly called testing) is being challenged in many academic circles by those who favor alternative assessment approaches" (p. 103). Taylor, Marienau and Fiddler (2000) assert further that "many conventional assessment methods, including essays, unseen exams, and laboratory reports, allow students to take a surface approach or even implicitly encourage and reward such an approach" (p. 309). Based on this rationale, Renner (1997) maintains that assessment activities should be integrated into every learning activity, irrespective of intended learning outcomes.

The assessment process is typically even more difficult in online learning environments, especially when we want to move beyond reward systems that encourage surface approaches. While more authentic assessment activities tend to be effective - most specifically at determining whether learners can apply their knowledge and skill to a real (or authentic) task - it is often unclear what kinds of assessment strategies effectively achieve this aim, and whether or not they can be facilitated in online learning. The purpose of this study was to extend our understanding about the use of alternative assessment approaches for assessing higher levels of online learning in postsecondary environments. This study was guided by the following question: What assessment activities can effectively assess higher levels of online learning?

This study focused on student assessment in online learning environments within postsecondary institutions. Online learning is referred to as the use of asynchronous Internet integrated distributed learning environments (e.g., WebCT, FristClass, Virtual U, Top Class). Within the scope of student assessment, this study was further focused on formative assessment for higher levels of learning and was concerned with investigating higher levels of learning. This kind of learning has been referred to as higher order learning by Fabro and Garrison (1998) and Resnick (1987). Irrespective of whether it is referred to as higher ordered learning or higher levels of learning, the essence is on the construction of new knowledge.

Method

This study was built upon the results of a prior study (Kanuka, 2001), which identified six elements as evidence that higher levels of learning was occurring in online environments. The six elements included negotiable learning, instructional, performance-based, new and/or multiple perspectives, assumption identification, and an ability to use a variety of learning strategies. A survey was developed based on these elements and sent to a group of selected experts and scholars in the area of online learning from Canada and the United States. Experts and scholars were defined as those who had a PhD, scholarly publications, and experience using the Web to facilitate teaching and learning in post-secondary institutions. Selection was based on prior publications in academic journals and conference proceedings within North America. Survey participants were asked to provide an example (or examples) of an online assessment activity for each of these elements. The survey was designed where each of the six items were presented as a statement. After each item, the following open-ended question was asked: If you agree with this statement, please provide an example (or examples) of an assessment activity. If you do not agree, please explain why.

Results

The survey was sent to 78 individuals; twenty-six responded for a response rate of 33%. All respondents had experience facilitating learning on the Web (average = 4.5 years), all taught courses in post-secondary institutions where the Web was used in some way to facilitate learning activities, and 77% taught courses where the Web was the only communication tool used to interact with students. All respondents had published in the area of Web-based teaching and learning, with 40% having published in books, 83% in refereed journals, 67% in non-refereed journals, and 75% in other areas (e.g., conference proceedings, technology reports, newsletters, CDs, electronic journals). The survey participants indicated that there are many complexities to the implementation of these assessment elements. These complexities are discussed in combination with the examples provided for implementation.

Item 1

Higher levels of learning typically include an assessment process that is personally meaningful to each learner and can be facilitated in online learning by negotiated assessment.

The survey participants indicated that, when appropriate - which is dependent upon the discipline and objectives - negotiated assessment (e.g., negotiable contracting, peer assessment, contracting, and/or self assessmen) could be effectively facilitated using the Web. Research has supported this view. For example, research by Marantz and England (1992) on conducting negotiated learning in online environments revealed that when using computer mediated conferencing it is possible to "achieve at least as much, and often more, than what face-to-face [provides] by way of close collaboration and meaningful mutual learning." However, many of the survey respondents also indicated that there are some serious drawbacks to negotiated assessment. In agreement with these data, O'Donnell and Caffarella (1991) outline the disadvantages as:

   1. discomfort with the unknown,

   2. quality of the learning,

   3. time pressures, and

   4. not suitable for all situations. (p. 46)

A suggestion to overcome some of the disadvantages was to use the SOLO taxonomy (Biggs, 1996) as a framework for negotiable contracting. The SOLO taxonomy is an orderly way of describing a hierarchy of complexity, which learners show mastery of academic work. The main strength of the SOLO taxonomy is its generality, which is not content dependent, making it useable across a number of subject areas. It has five levels of sophistication:

   1. Prestructural - the task is not attacked appropriately;

   2. Unistructural - one or a few aspects of the task are picked up and used
   (understanding as nominal);

   3. Multi-structural - several aspects of the task are learned but are
   treated separately (understanding as knowing about);

   4. Relational - the components are integrated into a coherent whole, with
   each part contributing to the overall meaning (understanding as
   appreciating relationships);

   5. Extended abstract - the integrated whole at the relational level is
   reconceptualized at a higher level of abstraction, which enables
   generalization to a new topic or area, or is turned reflexively on oneself
   (understanding as far as transfer and as involving metacognition).

Using this taxonomy can overcome at least two of the limitations of negotiated learning: discomfort with the unknown and quality of learning. Levels four and five, for example, can be seen to be qualitatively different from the lower levels in that they require the learner to integrate new knowledge and skills into a coherent structure, resulting in the construction of new knowledge.

Item 2

Higher levels of learning typically include an assessment process that is meaningful to each learner and can be facilitated in online learning by developing assessment activities in a way where they will also be used as part of the instructional process.

The survey participants indicated that instructional assessment could be facilitated in a variety of ways in online learning. For example, quoting the learners' contributions from the online transcripts for personal reflection can be effective as it encourages learners to use their own work as part of the instructional process. The most frequent example provided to make assessment part of the instructional process was through the use of embedded assessment. The assumption underpinning embedded assessment is that there are better ways than testing learners to determine what has been learned. When assessment is embedded in the learning process, it means that the authentic task is part of the ongoing instruction. Thus, learners who are engaged in an embedded assessment activity would appear to be doing an instructional activity - but the instructor is, in fact, using the process (or products of the process) to evaluate the learners' understanding and planning for further instruction. According to Shank (1992) (see also Simmons, 1994), alternative assessment - such as embedded assessment - provides a more holistic picture of the learners' understandings. Related to this point, a suggestion made by one of the survey respondents was that the presentation of archived messages could incrementally build on the learners' meaning-making process by providing ongoing formative feedback.

Item 3

Higher levels of learning typically include an assessment process that is meaningful to each learners, and can be facilitated in online learning by selecting assessment activities that are performance based.

Many of the survey participants indicated that the Web could be an appropriate medium for facilitating performance based assessment and using the Web to exhibit student work brings a wider audience for evaluation (e.g., peers, instructors, and experts in the field). These views are in keeping with related literature (see Collett, et al., 1999). Survey participants also indicated considerable success with presentations on the Web, as well as the use of Web-based learning portfolios. Learning portfolios are an instructional activity where learners are required to keep a record of their learning processes, typically based on a learning contract where the learners and instructors negotiate on pre-determined competencies. Reeves (2000) describes portfolio assessment as "any method by which a student's work is stored over time so that it can be reviewed in relationship to both process and product" (p. 108). One survey participant suggested that portfolios could provide learners with an opportunity to be challenged to accomplish a task that is as close to a real world performance, or authentic activity, as possible. Again, this is in keeping with related literature (Wielenga, Ritzen & Kosters, 2000). Benefits cited by many survey participants, and are supported by Wielenga, et al. include:

   1. Through hyperlinks, learners can easily show the relationship between
   different parts of the portfolio and so demonstrate the coherence between
   different elements in the course.

   2. Many learners find it' motivating to present themselves on their own Web
site.

   3. It contributes to the necessary development of future technology skills.

   4. Increases communication between and among learners and instructors.

   5. Learners can continue to develop their portfolio after finishing their
   studies and can present themselves to future employers on their own
   homepage.

The most important aspect of learning portfolios, however, is its use as "an instrument belonging to the student; students use the portfolio to direct their learning process and to reflect on their development and growth" (Wielenga, Ritzen & Kosters, 2000).

Item 4

Higher levels of learning typically require learners to provide evidence of new understandings and ways of thinking, and can be facilitated in online learning by providing opportunities for learners to demonstrate they have acquired new and/or multiple perspectives of the phenomena presented.

The survey data for this item provided a number of examples. One respondent indicated, for example, that learners should be provided with the opportunity to write about their experiences (where students individually reflect on what the material meant to them) in a summative way so instructors can check for understanding. The survey data also revealed that there are a number of ways that this process can be facilitated using the Web, with the most common example being online presentations and reflective papers. According to Collett, et al. (1999), presentations are well suited to the Web.

Item 5

Higher levels of learning typically require learners to provide evidence of new understandings and ways of thinking, and can be facilitated in online learning by providing opportunities for learners to become explicit about their assumptions.

The survey data indicate that this process can be difficult in conventional settings, but even more difficult in online learning. While acknowledging that this process is difficult, there are ways that can facilitate this process. One example from the survey data was through providing a contentious or a controversial view on the issues or problems presented and then challenging the learners' current assumptions. Another example provided was through the use of a repertory grid. As a learning activity, the purpose of repertory grid development is to raise the level of awareness of a learner's assumptions, as well as a platform for "conscious analysis of those assumptions toward greater self-understanding" (Taylor, et al., 2000, p. 50). Essentially, a repertory grid is an attempt to selectively sample a learner's frame of reference. In its simplest form, it consists of presenting the learner with three items from an array of qualitatively similar elements (e.g., situations from which learners have learned, books they have read, teaching methods experienced, etc.). The instructor then asks the learners to describe in what ways the elements are similar and different. This process is repeated a number of times with different combinations of elements (referred to as triads), producing a series of bipolar descriptors that can be used by the learners to differentiate the items in the array - not just the three used to generate the original grid. The result is a two-way matrix with constructs down one side and the elements on the alternate side. At this point the learners give a rating to each construct. This process, according to Candy (1991), is a particularly powerful way to enable learners to surface and examine hidden assumptions. The survey respondent who suggested the use of repertory grids noted that this activity can be effectively facilitated through Web-based conferencing systems.

Item 6

Higher levels of learning typically require learners to provide evidence of new understandings and ways of thinking and can be facilitated in online learning by providing opportunities for learners to demonstrate their ability to use a variety of strategies to produce meaningful understandings.

The data from the survey indicate that some participants hold the opinion that online learning does not give the same opportunity for students to demonstrate a wide range of learning strategies, as does a face-to-face class. For those who agreed that this construct could be facilitated in online learning, there was similarity within the examples provided. The basic gist of most of the comments for tose who agreed with this item was that in order for students to demonstrate their ability to use a variety of learning strategies, instructors must provide a variety of online instructional activities. To assess a learner's ability to use a variety of learning strategies, it was suggested that learners be given the opportunity to write about their experience and feelings in a summative way and submit the summary at the end of each standard unit within the course structure. This process is easily facilitated using Web pages that have text boxes with submit forms.

Discussion and Conclusion

The aim of this study was to extend our understanding of alternative assessment strategies for higher levels of online learning. Examples of ways to facilitate higher levels of learning included negotiated assessment, embedded assessment, learning portfolios, reflective papers, and repertory grids. However, it also needs to be acknowledged (as many of the survey participants indicated) that in credentialled postsecondary institutions many of the policies and regulations currently governing assessment may act as barriers to the execution of the examples presented. Moreover, literature on alternative assessment (i.e., self assessment or negotiated contracting) indicates that there are three major ethical issues in assessment of this nature (Crowe, 2000): learner readiness, evaluation credibility, and power issues. To resolve these issues, Crowe suggests a middle ground that combines traditional assessment techniques with alternative assessment techniques - such as triangulated assessment. Nevertheless, while it is important to acknowledge that there may be barriers to the implementation of alternative assessment strategies in online learning, the results of this study provides an important step toward understanding what activities are effective at assessing higher levels of learning in online post-secondary settings.

References

Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32, 347-364.

Candy, P. C. (1991). Self-direction for lifelong learning. San Francisco: Jossey-Bass.

Collett, D., Kanuka, H., Blanchette, J., & Goodale, C. (1999). Learning technologies in adult distance education. Edmonton, AB: University of Alberta Press.

Crowe, J. L. (2000). Evaluation of adult learners: Ethical issues. New Horizons in Adult Education, 14(3), 4-10.

Fabro, K. R. & Garrison, D. R. (1998). Computer conferencing and higher-order learning. Indian Journal of Open Learning, 7(1), 41-53.

Kanuka, H. (2001). A principled approach to facilitating Web-based distance education in post-secondary institutions. Unpublished doctoral thesis, University of Alberta, Alberta.

Marantz, B. & England, R. (1992). Closing the distance: a CMC learning contract tutorial. DEOSNEWS, 2(4). <http:/lwww.ed.psu.edu/acsde/deosldeosnewsldeosnews2_4.asp>

O'Donnell, J. M. & Caffarella, R. S. (1991). Learning contracts. In M. W. Galbraith (Ed.), Adult Learning Methods (pp. 133-160). Malabar, FL: Krieger Publishing Company.

Reeves T. C. (2000). Alternative assessment approaches for online learning environments in higher education. Journal of Educational Computing Research, 23(1), 101-111.

Renner, P. (1997). The art of teaching adults: How to become an exceptional instructor and facilitator. Vancouver, BC: The Training Associates.

Resnick, L. B. (1987). Educational and learning to think. Washington, DC: National Academy Press.

Shank, G, (1992). Educational semiotic: threat or menace? Educational Psychology Review, 4(2), 195-221.

Simmons, R. (1994). The horse before the can: assessing for understanding. Educational Leadership, February, 1994, 22-23.

Taylor, K., Marienau, C. & Fiddler, M. (2000). Strategies for teachers and trainers. Developing Adult Learners. San Francisco: Jossey-Bass.

Wielenga, D., Ritzen, M., & Kosters, J. (2000). EFA's Digital Portfolio System. <http://www.efa.nl/publicaties/publiexplo/english/portfo12.html>.

Heather Kanuka is a post-doctoral fellow. Her research interests include investigating the impact of open and flexible learning in higher education.