How useful are portfolios to assess online learning.

Abstract

Portfolios are becoming increasingly popular both in the assessment of writing as well as a mode of assessment to measure teacher learning. In this paper I will describe a portfolio used for measuring student learning outcome in four partially web-based courses in a university in Hong Kong. Students had completed web tasks set by the teachers, put in journal entries to record what they had learnt, and regularly participated in web-based bulletin board discussions of key issues. To create the portfolio students were asked to choose 3 or 4 items from the range of work done on the web to show what they had learnt from the course. The data consisting of an analysis of the portfolios created and interviews with 22 students indicate that learning portfolios have the potential for bringing in a reflective dimension in assessing student learning in online classrooms.

**********

Learning Portfolios to Assess Online Participation

The question of alternative modes of assessment applies to all teaching situations irrespective of the mode of delivery. Within an alternative approach, assessment of student learning is increasingly seen in terms of a process and not a singular objective/standardized test. Such assessment should require students to demonstrate deep understanding of learning, not just mastery of a set of discrete facts or skills (Zessoules & Gardner, 1991). Wiggins (1991) argues that direct assessment of performance on desired tasks is the only means of assessment to understand whether there has been "a measurable effect on teaching and learning" (1991). One way to measure effect on learning is portfolio assessment. Portfolios can be seen as a "purposeful collection of student work and records of progress and achievement assembled over time" (Valencia and Calfee, 1991). Portfolios reflect a student's own perceptions of the learning process, thus bridge the gap between assessment and instruction (Paulston et al, 1991).

Integration of computers into the curriculum brings changes to every aspect of the curriculum (Patricks, 1997) including assessment of learning. Alternative modes of assessment have become increasingly important because as Jonassen et al (1999) argue, "If you agree that learning is or should be an active, constructive, intentional, authentic and cooperative process, and if you agree that technologies should be used as learning tools for students to learn with, then you must probably also challenge your beliefs about how to assess and evaluate learning by students".

Today web-teaching environments incorporating conferencing and online pedagogic tools are becoming increasingly popular as a means to dialogic and flexible learning. Participation in interactive online learning needs to be recognized when we measure learning outcomes. Portfolios may well provide a viable option for measuring the contribution of such participation in students' overall learning. In this paper, I describe why and how portfolio assessment was chosen for four partially web-based courses, the kinds of learning outcome student portfolios demonstrated and the opinions 22 students expressed regarding learning portfolios in follow-up interviews.

The Context

To understand why portfolios were used to measure student learning, we need a brief description of the four classrooms where they were used. The four classes were:

   1. Image and Text was offered in year three to eight students of a
   three-year BA in Contemporary English (BACEL). The subject largely examined
   concepts in semiotics and text analysis.

   2. English for specific purposes: Image and Text was offered in year one to
   12 students of BACEL. The focus was on the reading, writing and analyses of
   multimodal texts.

   3. General and Academic English focused on language proficiency. There were
   19 students - all in the first year of a bachelor's degree in Language and
   Business which aimed at developing fluent tri-lingual graduates.

   4. English and the media was offered to 13 second and third year students
   as an optional subject for BACEL. The focus was on the use of English in
   the media.

All four were typically 3-credit courses that lasted for a semester of 14 weeks with 3 hours of teaching time allocated for each week. Two hours were dedicated to face-to-face meetings while one hour was spent online (see Appendix 1 for extracts from outlines). All homework was also done online. The courses were firmly situated within a social constructionist philosophy. Each web classrooms incorporated the following:

   * A bulletin board: for discussion of central issues/concepts.
   Participation in these discussions was voluntary, although the portfolio
   required a selection from this component.

   * Participant pages: Students had individual pages to write their learning
   logs, evaluate the learning experience, critique assigned readings and
   respond to classmates' logs. The log entries were compulsory. The teacher
   also had a page and kept a learning log.

   * Task pages: The one-hour on the web was spent on completing tasks and
   responding to peers' tasks. The tasks arose from the classroom input. For
   example, some tasks required students to conduct web search to find
   information on an issue e.g. bias in the media, and write a description of
   the site for the class.

The web-based work received regular feedback but was formally assessed through a portfolio accounting for 25% of the course grade. All students were asked to hand in a portfolio consisting of a selection of three or four contributions from the range of online work (e.g. any kind of log entry or a response to a classmate or a bulletin board posting or a web-task) to demonstrate what was learnt in the course (see Appendix 2). The teacher also created a mini portfolio and put it on the web as a working model.

Why Portfolio?

There was a need for an assessment instrument for the web-based work that would be fair by ensuring that students have the chance to evaluate and be evaluated on the range of work they had produced online over the semester. It is suggested that writing portfolios need to be made fair by ensuring that they are purposeful (not just a folder of collected work), involve students in self-reflection, include guidelines for work selection, and ensure student participation in this selection process (Arter and Spandel, 1992). So fairness was incorporated by allowing selection of a range of evidence and explanation of how each selection reflected the specific aims of the course.

There was also a need for an assessment instrument for the web-based work that would be cost effective in terms of investment of time by teachers and students. There was a huge quantity of textual work archived on the online classroom that needed to be encapsulated within a readable summary. The choice of portfolios were informed by the general literature on alternative assessment in language classrooms (see Hamp-Lyons and Condon, 1996), since issues of assessment of student performance on the online mode in language classrooms had just begun (see Debski et al, 1997).

Portfolios also served the need for an assessment that matched the social constructionist, student-centered, reflective philosophy of the course and made it possible to connect the full spectrum of the work i.e. both face-to-face and the web components; concepts and their application; learning and teaching (giving feedback to peers and teacher).

What kind of portfolio?

Portfolios can take the form of showcases (featuring a student's best, self-selected work) or documentation (showing evidence of student work and progress across time, but with no selective or evaluative process). While documentation portfolios store relevant authentic language learning processes over time and entail the idea of negotiated curriculum, showcase portfolios simply include the selection of a sample and a reflective explanation of the choice. A showcase portfolio was used following the CRADLE approach (Gottlieb, 1995) that entails:

(1) Collections of student work (always dated) containing expressions of contextual learning processes as well as evidence of learning.

(2) Reflections by students on their perceptions and interpretations of the learning experience.

(3) Assessments made possible by systematic collection of data.

(4) Documents of achievement that include not only student work, but also narratives that substantiate the choices.

(5) Linkage systems between student and teacher in the learning process (see process below)

(6) Evaluations or summary data for further development

Though this was not a high stakes context, with the portfolios only accounting for 25% of the grades in a 3-credit course, the issue of validity was not ignored. The portfolios described here followed a task-centered construction by allowing the use of multiple, varied and naturally occurring tasks for determining desired performances, and making the scoring criteria and performance rubrics a part and parcel of the performance itself as suggested by Messick (1994). Reliability was taken into account by rating portfolios according to a set of criteria that were agreed on and trialed by the stakeholders involved (see process below). The process below demonstrates that portfolios had a formative dimension, though they were done summatively i.e. at the end of the course.

The process of development

   1. Course outline (week 1): Stated that a portfolio would be used

   2. Bulletin board (week 8-10): The initial description of portfolio put in
   and views sought. Twenty bulletin board postings and 12 e-mails (typically
   with questions) were received between weeks 10-16.

   3. Teacher posted preliminary rubrics based on views received - seeking
   comments (week 10).

   4. Teacher gave printed copy of rubrics, criteria, deadline based on
   discussion (week 11). For example, the teacher added the idea of a
   follow-up to the selection because students felt that some of their earlier
   work they select should be revised on the basis of the later input and
   discussions.

   5. Teacher posted own portfolio in her page (week 13) and summarized issues
   raised online on the bulletin board. Students were asked to evaluate the
   teacher's portfolio on the basis of the criteria to refine the criteria.

   6. Individual conferences between teacher and student of preliminary drafts
   were held (week 14)

   7. Students handed in portfolios one week after end of course

   8. Written feedback and grades made available in 3 days.

What did the portfolios contain?

To examine whether the portfolios had been an effective measure of learning outcome, all 52 portfolios were re-read by the teacher and a research assistant to find the common elements across the four classrooms. We found the following:

1. Every portfolio contained self praise often situated within the following aspects of learning outcomes (numbers of portfolios containing each are in brackets):

   * The ability to apply concepts to real life contexts (all 52)

   * Personal satisfaction with good work (42)

   * Confidence arising from successful task completion and teacher praise
(38)

   * Gradual learning from initial confusion (19) as the following extract
demonstrates.

I have chosen two bulletin board contributions for my second selection because the one posted on 27.2 shows that I am still a bit confused about the idea of connotation and denotation. But in the same thread, I have again posted a revision on 9.3 which uses an example from an advertisement I see everyday when I cross the tunnel on the bus. It shows that I have thought about the idea and developed the kind of awareness that match the course aims..... (I&T)

2. There was self criticism too. Often this related to being lazy (27) but twenty-three contained choices of "bad" work and an evaluation of why it was considered bad by the student and then a revised work that would be considered "better". The following extract from a portfolio demonstrates this:

I choose this because this is the first "big" task I need to do with a partner. And I have made a big mistake in doing that synopsis. The most important thing of writing a synopsis is objectively but what I did in the synopsis is just give out my own opinions. Besides I just copy pan of the article from other web sources. These are two big mistakes. So if I go back, I will write..... (sic from GAE)

3. Inevitably there was course evaluation as well in 49 portfolios. Overall the praise and criticism balanced each other out. Criticism typically was tied to appropriating blame for the confusion reported (17) and 42 portfolios concluded with a piece of advice to teacher such as the following. Overall I learnt a lot about the media, although it would be better if we spent more time on television and less on the print media. (EMM)

4. Though all portfolios tried to refer to course objectives/aims, quite often (25) there were more explicit references to personal goals/interest/understanding. Students justified choices with reference to their final, year dissertation (11) and personal interest in the kinds of analyses and discussions in the course (e.g. I have always found advertisements very interesting ... (ESP).

5. Finally there was an attempt to contextualise the portfolio into a coherent narrative by showing how the online and the face-to-face elements worked together in their thinking, learning and (24). Fifteen explained the difficulties in the selection process of a single item as the work online reflected ongoing learning. Indeed all indicted a degree of reflection on the work they had done, but for many first year students, the reflection stopped short of being critical as the last lines in the sample in appendix 2 indicates. The student is able to see that there are alternative ways of interpreting a text but does not critically consider how that perception impacts on his/her learning of a language.

What did the students say about the portfolios?

In order to ascertain whether portfolios were seen as an effective measure of performance, 22 random selection of students were interviewed by a research assistant. All said the portfolio was a "good" idea because:

   * it made them think (22). I liked the portfolio because it was one of the
   few assignments that made me think. You know all those long essays and
   analysis - it is mechanical almost - but suddenly..

   * it gave them a chance to show their "clever" work (17)

   * it was quite good for revision purposes (13)

   * it helped them to take stock (11) as the following interview extract
   shows: I can easily say what I learn for I&T - the portfolio made me go
   back - I never go back because I feel ashamed of my work but I saw that I
   was not bad - and I understood some of those ideas better ...

But all thought it was extremely time consuming. The rubrics were also criticized as sixteen students said that to show learning is to show "good" work that is praised by the teacher but the criteria implied that this was likely to be "thoughtless". Finally, the reference to course aims was seen as limiting. Fourteen students said that directly connecting the selection with the course aims was artificial, as it was hard to find a direct match.

An overall evaluation: How did the portfolio work?

The data seem to indicate that portfolios can be an informed and negotiated selection to illustrate learning from all the work done online in a course. It can integrate aims, process, product and reflection. This "informed" selection required a set of strategic competence of taking stock of what is needed to demonstrate learning, deciding on appropriate selections, and being able to use the selections to the best advantage.

However, the "informed" selection needs a valid and effective reference point. In the portfolios described here, a syllabus-based construct was built in by referring to the course objectives and making a direct connection with the curriculum. But this seemed to prove problematic as the students suggested. In fact a more open reference point is being piloted now.

This paper would be incomplete without discussing exactly what the portfolios measured. Students seemed to be measuring teacher performance/curriculum as much as student performance. Indeed one could argue that this conflation is a strength as it bridges the gap between assessment and instruction (Paulston et al, 1991). In reality it had a measurable effect on teaching (Wiggins, 1991) as the courses were revised on the basis of some of the points made by students. However, we need to find a robust set of criteria for measuring "reflective self evaluation".

Finally, we must ask if this is indeed a "fair" measure? For example, could anyone else apart from the teachers grade the portfolios fairly? Recent work shows even teachers have different expectations. There is obviously a need for more work, but portfolios may be well worth considering for measuring student learning outcomes in online instruction.

References

Arter, J. A., & Spandel, V. (1992). Using portfolios of student work in instruction and assessment. Educational Measurement: Issues and Practice, 11(1), 36-44.

Debski, R., Gassin, J. and Smith M. (Eds.) 1997. Language learning through social computing Australia: Applied Linguistics Association.

Gottlieb, M. (1995). Nurturing student learning through portfolios. TESOL Journal, 5(1), 12- 14.

Hamp-Lyons, L and Condon, W. (1996) Questioning assumptions about portfolio-based assessment. College composition and communication. 44 (2) 176-190.

Jonassen, D., Peck, K. & Wilson, B. (1999). Learning with technology: a constructivist perspective. Englewood, NJ: Prentice Hall.

Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13-23.

Patricks, P.C. (1997). The evolution of computer technology in foreign language teaching and writing. In Debski et al (pp. 159-177).

Paulson, F. L., Paulson, P. R., & Meyer, C. A. (1991). What makes a portfolio a portfolio? Educational Leadership, 48(5), 60-63.

Valencia, S. W., & Calfee, R. (1991). The development and use of literacy portfolios for students, classes, and teachers. Applied Measurement in Education, 4(4), 333-345.

Wiggins, G. (1991). A response to Cizek. Phi Delta Kappan, 72, 700-703.

Zessoules, R., & Gardner, H. (1991). Authentic assessment: Beyond the buzzword and into the classroom. In Perrone, V. (Ed.). Expanding student assessment. Alexandria, VA: Association for Supervision and Curriculum Development.

Sima Sengupta teaches English language at the Department of English. One of her research interests is to investigate the role of the online mode in the literacy development. This research was funded by a grant (no. G-S988) from the Department of English, Hong Kong Polytechnic University.