Assessing the wizards.

Abstract

Today it is commonplace for the K-12 curricula to include multimedia projects, which encompass presentation programs, graphic arts, digital photography and web design classes. On the surface teachers applaud these innovations as education for the new millennium but assessment for this new interactive style presents a challenge. A clear and credible rationale for assessment of such classes has not yet been developed. While assessment is the challenge, criteria are the dilemma.

Introduction

The multimedia curricula teach the use of computers and computer software, digital and video cameras, the web, and telecommunications as tools to be used in the context of a required task. In addition to using web resources in the classroom, students are taught to design and create their own websites. A major concern of educators is the appropriate assessment of student learner outcomes. All students, at all levels of ability, are entitled to have performance based assessment. These are referred to as alternative assessments, because they do not fall under the traditional form of evaluation, which is generally a pen and paper test or exam or a standardized test.

In the last two decades, the educational community has embraced Gardner's theory of multiple intelligences (MI), negating the former concept of intelligence as a single capacity which can be expressed for all individuals in a single manner (Gardner, 1995). Howard Gardner, the acclaimed Harvard cognitive psychologist, emphasizes that the MI theory recognizes the significance of human differences and elevates the dignity and giftedness of every individual (Diaz-Lefebvre, 2004). Thus, alternative assessments based on individual learning styles should be used to ascertain student learning. More recently, federal legislation, such as the Individuals with Disabilities Education Act (IDEA) of 1997, Pub. L. No. 105-17, has mandated that all students need to be educated in the least restrictive environment (2003) and, even more recently, the No Child Left Behind (NCLB) Act of 2001, the revised Elementary and Secondary Education Act, has mandated accountability for every student's education (2003-2004).

Because of the mandates of IDEA and NCLB, all students must take part in accountability testing, even those with severe cognitive disabilities (SCD). The development of these alternative assessments is both time-consuming and expensive. This creates special challenges for many schools in rural areas, where adequate funding is not available to develop alternative assessments. Performance criteria must reflect mastery of skills. Researchers Hager and Slocum (2005) define mastery as that which gets the job done. The difficulty schools face is designing alternative assessments to promote monitoring of ongoing progress and support effective classroom practices, within the limitations of school budgets and in alignment with state standards. This article is based on a review of the existing literature, in conjunction with and supporting an action research project undertaken by this educator.

Impact of Legislation

The implementation of IDEA and NCLB has impacted each and every classroom in the country. When web-based technology is used in the classroom, skills in manipulating telecommunications and multimedia are taught, to be eventually used in other school subjects and in the outside world (Becker, 1993). However, the actual incorporation of these skills in other school subjects has been spotty at best (Jancich, 2005.) Students are taught these skills in their technology classes, but many teachers of other subjects do not feel qualified to plan lessons requiring the usage of these techniques in their particular classes. Many educators do not feel comfortable using technology themselves, and are reluctant to require it of their students. NCLB Title IID continues to build the pressure since a primary goal is to

   encourage the effective integration of technology resources and
   systems with teacher training and curriculum development to
   establish research based instructional methods that can be widely
   implemented as best practices by state and local educational
   agencies (Jancich, 2005).

The interpretation of the NCLB mandate goes beyond the notion of providing website resources with the subject matter text. It requires that 25% of the millions of dollars available for technology be spent on the professional development of the teachers. This need is approaching a crisis stage. Teachers are not being adequately prepared for teaching and utilizing technology in their classes.

Researchers Ross, Hogaboam-Grey and Hannay (2001) found a significant relationship between teacher efficacy and student achievement. In-service training is not enough to produce teachers who feel confident about their ability to teach technology to students. In-service programs for teachers will be more effective when the programs are adapted to meet the individual needs of the teachers, and when ongoing teacher support is provided for instructional, rather than software, issues. This reinforcement is vital in promoting teacher efficacy. Student achievement and teacher self-efficacy increase when students are taught by teachers "with greater confidence in their ability to accomplish goals requiring computer skills, or in their ability to teach students how to use computers" (Ross, Hogaboam-Grey & Hannay, 2001).

In addition, there is a shortage of technology education teachers in the United States, particularly on the middle school level (Ritz & Manning, 2004). During adolescence youngsters develop their cognitive abilities, and are able to learn about technology and its varied applications. Many middle school students enjoy instruction and projects dealing with multimedia. Unfortunately, most of the individuals who elect to teach technology are interested in teaching only on the high school level. The next most desired level for teaching technology to students is the elementary group. The least popular segment is the middle school level (Ritz & Manning, 2004).

Assessment of Multimedia Projects

Teaching in the post-Sesame Street world which assumes that all learning will be fun has allowed the technology of the new millennium to morph into the K-12 curriculum that includes presentation programs, graphic arts, digital photography and web design. Evidence of this can be seen when observing classrooms on various grade levels (Ross, Hogaboam-Grey & Hannay, 2001). On the surface, teachers applaud the application of these enjoyable innovations as future necessities having great potential for interactive learning; however, these new age programs open a Pandora's Box of assessment problems. The interactive style falls well outside the traditional pen and paper exam, generally composed of multiple-choice and constructed-response items. Although understanding the theory of any subject matter is important, a multiple choice exam on web design is clearly a less valid measurement of skill than the ability to create a working website. One complication in the creation of a practical assessment is the design of the larger school structure that expects the pen and paper standard. The advantage of assessing with these traditional standardized tests is that they provide manageable, measurable information for schools. Conversely, performance assessments yield more complex and unfamiliar information that must be carefully interpreted (Rabinowitz, 2001).

Information from multiple-choice and constructed-response tests is used to drive curriculum and provide information for the placement, or tracking, of students. Such tests may be the most efficient way of dealing with specific policy concerns (Rabinowitz, 2001). On the other hand, performance assessment is expensive to design, score, and report. There is a problem created by the length of time between the administration of the test and the reporting of results. These results must be available when needed for administrative decisions regarding a student's education plan (Rabinowitz, 2001). School administrators sometimes question how much better a performance based approach is over the traditional pen and paper counterpart, in regard to needs, time and money (Rabinowitz, 2001).

A clear and credible rationale for assessment for multimedia projects has not yet been established. While assessment is the challenge, criteria are the dilemma. The multimedia based curricula teach the use of the computer as a tool. While this emphasis presents the correct definition for computer usage it creates a teaching vacuum. This tool must be learned in the context of a necessary task. Simply following a number of outlined steps to the completion of a particular task does not necessarily increase the problem solving ability of the users or their knowledge of the computer as a tool.

A major concern of educators is the effective assessment of student learner outcomes. The intersection of the teacher's knowledge of computer applications and subject matter of core curriculum with the final product impact the choice of criteria. The master schedule of most schools does not have the flexibility to have the students' PowerPoint presentations, for example, assessed by both the English teacher for content and the business or technology teacher for manipulation of the computer software; therefore, every teacher in every subject area needs to assume dual roles. Ideally, in schools encouraging interdisciplinary projects across several content areas, two or more teachers would be able to assess one project. Two major areas of assessment warrant consideration: Content and Presentation.

While content has always been a requirement, technology offers alternatives that certainly are not the content originally available in the one room schoolhouse era. A PowerPoint presentation requires a specialized strategy for sequencing information. Teachers and students alike believe that such a presentation is more free-flowing and creative than an essay, while in reality, a PowerPoint presentation utilizing web resources needs to be even more concise and structured. The typical essay explores a particular idea. This same idea cloaked as a presentation, by the nature of the tool involved (PowerPoint), needs to show a result. Honing this skill is wonderful preparation for the world outside of school but complicates the criteria challenge. The idea and the computer tool must mesh, and both are assessed across different criteria.

This assessment challenge has caused confusion and as a result, many teachers will accept the presentation only after the completion of the pen and paper assignment. Students love effects and many teachers have been left feeling that they have been machined gunned by the rapid-fire sound effects of flying text. The end result is that the potentially useful business tool is used as if it were a toy. A few schools are successfully addressing this challenge. The Beacon School, a rigorous college-preparatory public high school in New York City, is one such school. The school's mission infuses technology and the arts into the curriculum. Beacon students are required to prepare course-related portfolios for performance based assessment (Beacon School, 2005).

To avoid the situation of overwhelming special effects sometimes found in students' PowerPoint presentations, some teachers may require storyboards. A storyboard defines the parameters of the project, not only in content but also with the artistic elements of graphics and audio (Valenza, 2005). An important teachable moment is the constraint imposed by the delivery of the project. The students must learn to use the tools and time at their disposal to create the desired result.

Brad Astin, a mentor teacher with Options for Youth in Sacramento, an alternative high school, (personal communication, August 11, 2005) provides an example of such a project:

   While working in southern California 1 had a tremendously talented
   student who often struggled mightily with written assignments. The
   reason I say this student was tremendously talented was because of
   his art work. He competed in graffiti competitions as far as
   Paris, France and was an unbelievable artist. I gave him a basic
   drawing class and instead of receiving rote drawings of shapes and
   lines I received imaginative murals of all sorts. He was thrilled
   to be in an art class and to be rewarded for his talents.
   Recognizing this I began changing many of the assessments in
   classes such as history and science where many assessments were
   based on writing but where I felt a student's ability to write was
   not necessary to reflect his grasp of the curriculum. Instead of
   having this particular student write essays that were completely
   meaningless to him I asked him to draw pictures and use his
   creativity to reflect his knowledge of the subject matter. The one
   example that comes to mind was the alternative assessment of "The
   Battle of the Bulge." I had the student draw a story board
   reflecting his understanding and interpretation of the "Battle of
   the Bulge" and orally describe what was occurring in each frame.
   It was an eye opening experience. The student showed a deep
   understanding of the subject matter and drew interesting
   conclusions. It became clear that appealing to a student's
   interest and creative strengths promotes a greater understanding
   of the subject matter and a more positive learning experience for
   all involved.

Use of Checklists

In preparing for alternative or project-based assessment, checklists are useful to focus the student on the elements of the assignment: spelling and grammar, design, color, number of slides, and number of animations and other factors, depending on which multimedia project is being created. When this list is available to the students as they complete the assignment they can spotlight their own progress. Students may also use a checklist as a time management tool and gauge their strengths and weaknesses. During the action research of this writer students are encouraged to complete the assignment in the most basic form and then double back to animate and beautify it. Stressing completion aids the notion of meeting deadlines and it keeps the class working on the same elements in greater depth.

Use of Rubrics

Rubrics are also useful assessment tools. A checklist requires a yes/no response but the rubric questions quality. In the ideal classroom students would be given the rubric at the project's inception and evaluate their work as it progressed. From the teacher's point of view the rubric requires thought and practice. Rubric templates and generators abound on the web but from this researcher's experience, a rubric needs individual fine tuning. Students need to develop a self-critical eye for their own work. The utopian rubric would be developed collaboratively by the teacher and the students in the class.

In a study done on assessments on the college level, Lewis, Berghoff and Pheeney (1999) found that some professors stress certain instructional areas more than others. They desire creativity on the part of the students, and hope that learning will go beyond the moment of assessment. In order to convey the goals of the university courses, the researchers found three means of communicating what the instructor expects to what the students know and can do. One is the negotiated rubric, prepared jointly by students and teacher. Another is the rubric provided by the teacher and prepared by the teacher, indicating scales and levels of performance. A third is test specifications, where the teacher provides the basic concepts to be included on a test, and notes important course content. Although this study focused on university students, the same concepts apply in the context of K-12 education.

When teachers and students jointly create a rubric, students are learning both the content and the process. The rubric becomes a focusing tool. Students will have a better understanding of the task, and a tendency to aim for a high level of achievement. The rubric stimulates self-evaluation on the part of the students (Lewis, Berghoff & Pheeney, 1999). Wiggins and McTighe stress the importance of encouraging students to use a rubric and to reflect and self-assess while working on a project (Brown, 2006).

The researchers Lewis, Berghoff and Pheeney refer to Wiggins' (1993) Postulate 3 which states that an alternative assessment process must be based on clear, known, non-arbitrary standards and criteria (Lewis, Berghoff & Pheeney, 1999). An assessment must serve learning, not only monitor learning. The greatest value of a rubric is when it is offered early in the course. It can affect the teaching, studying, and thinking involved in an assignment, and lead to the finishing of that task. Wiggins and McTighe, in their Understanding by Design Professional Development Workbook (2004) recommend criterion-based rubrics, to be presented at the beginning of a task by the teacher, in addition to models and samples of desired outcomes with the teacher applying evaluation criteria to them (Brown, 2006).

Action Research Project

This researcher utilized a pre-existing rubric designed to assess multimedia projects and adapted it for the assessment of a specific project. The assignment was designed to give the students an opportunity to work with the basic function of DreamWeaver, a computer software program which manipulates multimedia, and is taught in the high school introductory computer technology class. This multimedia software can also be used in core classes to enhance student presentation of course materials, such as history, English, social studies, and other subject matter. As a class, a table was developed to be used on all four pages of the student project website. All students had the opportunity to choose a creature for their individual website and show variations of that creature on the subsequent three pages. For the purpose of this assignment the students were told that their text could be copied and pasted from a group of selected sites as long as no page contained more than 100 words and made sense. They were required to add pictures and links. The introduction of this assignment to the students included the admonition that this teacher was most interested in the on-time delivery of any and all assignments. The rubric for this assignment was available to the students at the inception of the project. The students were also advised to gather all materials and edit their content before they attempted to incorporate data into DreamWeaver, the web authoring tool.

Several of the 40 students in this action research study had problems on their sites that were not reflected in the rubric. One student had made excellent choices as far as color and style but had not completed the assignment. The site was so far off task that it was redundant to grade this project on the rubric from the _teacher's perspective. Although the student's good decisions were rewarded through the rubric it is questionable at this writing whether that student will engage in the self evaluation necessary to make performance judgments in future assignments. In the ideal school students choose classes based on their interests and abilities. In most high schools in the USA, ironically, students may take classes based on a choice to not take another class. Unfortunately, many students taking this researcher's web design class are not particularly interested in the subject matter, but are less interested in taking Microbiology or AP History.

Conclusion

To make the rubric experience a worthwhile learning tool it must be used consistently. Perhaps with constant use the students will become more engaged in the process and will attempt to adjust the rubric to gain an advantage. As the websites become more complicated students may realize that they have certain strengths and enjoy some aspects of the site building more than others. When this occurs the teacher needs to encourage the class to seek this advantage. Requiring the students to self evaluate using the rubric should help them to become more aware of their skills (Brown, 2006). This process might be even more effective when the assignment is based on group work since the evaluation of the site then could be done individually.

The DreamWeaver action research project is an ongoing study. This educator will continue to modify the rubric and monitor the results. Authentic assessment requires creative grading, which necessitates a diagnostic rubric. A reflective educator plus student evaluations provide insight into the teaching and learning process.

Diaz-Lefebvre (2004), in an ongoing nine-year study at the Glendale Community College, finds rubrics "an effective, interdisciplinary approach to learning, teaching and creative assessment" (p. 49). Comments made by faculty in their reflections indicate that students must take an active part in their learning, be exposed to different ways of learning, and must be accountable for their own learning. Wiggins and McTighe report similar findings in their research (Brown, 2006). This empowers students, and they develop enthusiasm about their work. With the evolution of change, and the transformation of education, it is vital to have a forum for the exchange of ideas and pedagogies. There is a great need for further research in the area of rubrics for fairly and objectively coping with authentic assessment development utilizing the realm of technology and multimedia, and the implementation and evaluation of these rubrics and assessments.

References

Beacon School [Mission Statement]. (2005). Retrieved August 26, 2005 from http://www.beaconschool.org/

Becker, G. (1993). Human capital: A theoretical and empirical analysis with special Reference to education (3rd edition). Chicago, the University of Chicago Press.

Brown, J. (2006) Assessing for Understanding. ArtsEdge. Retrieved June 26, 2006 from http://artsedge.kennedy-center.org/content/3648/

Diaz-Lefebvre, R. (2004). Multiple intelligences, learning for understanding and creative assessment: Some pieces to the puzzle of learning. Teachers College Record, 106(1), 49-57.

Gardner, H. (1995). "Multiple intelligences" as a catalyst. English Journal (High School Edition), 84(8), 16-19.

Hager, K. & Slocum, T. (2005). Using alternate assessment to improve educational outcomes. Rural Special Education Quarterly, 24(1). 54-60.

Individuals with Disabilities Act (IDEA) of 1997, Pub. L. No. 105-17. (2003). Retrieved June 26, 2006 from http://www.ed.gov/offices/OSERS/Policy/IDEA/the_law.html

Jancich, H. (Project Director). (2005). Are your teachers teaching effectively with technology? [PowerPoint presentation]. MyTarget2. Retrieved October 3, 2005 from http://mytarget.iassessment.org/

Lewis, R., Berghoff, P. & Pheency, P. (1999). Focusing students: Three approaches for learning through evaluation. Innovative Higher Education [H.W.Wilson--EDUC], 23(3), 181. Retrieved June 13, 2005 from ProQuest database.

No Child Left Behind (NCLB) Act of 2001, the revised Elementary and Secondary Education Act. (2003-2004). Retrieved June 26, 2006 from http://nclb2.ecs.org/projects_centers/index.aspx? issueid=gen&IssueName=General

Rabinowitz, S. (2001). Four things to consider about performance assessments. The School Administrator Web Edition. Retrieved September 6, 2005 from http://www.aasa.org/publications/sa/2001_12/rabinowitz_sidebar.htm

Ritz, J. & Manning, M. (2004). Alternative strategies for preparing middle school technology education teachea's. Childhood Education, 80(3), 142-147.

Ross, J., Hogaboam-Grey, A. & Hannay, L. (2001). Effects of teacher efficacy on computer skills and computer cognitions of Canadian students in grades K-3. Elementary School Journal, 102(2), 141-157. Retrieved January 20, 2005 from ProQuest database.

Valenza, J. (2005). PowerPoint effective, but often misused. The Never Ending Search. Retrieved August 26, 2005 from http://joycevalenza.com

Wiggins, G. (1993). Assessing student performance: Exploring the purpose and limits of testing. San Francisco: Jossey-Bass.

Carol M. Shepherd, National University, CA

Ann Mary Mullane, Nutley High School, NJ

Shepherd, Ed. D. is Professor of Teacher Education, and Mullane, M.A., is high school teacher of computer skills.