Problems with assessment validity and reliability in web-based distance learning environments and solutions.

Assessment of learning is critical to the learners, teachers, and designers of learning environments. Current assessment techniques in web-based distance learning apply age-old techniques to a new medium and are not adequate for web-based distance learning environments (WBDLE). The goals of this article are to extend existing critiques of assessment validity to web-based distance learning environments, define the problems with current WBDLE assessment methods using a learner, task, inference framework, and present three unique solutions to the problems. We describe our solutions, present validity and reliability issues for each method, and report instructor and student experiences with these solutions.

**********

Web-based distance learning environments (WBDLE) are growing exponentially and appear to be driven by a technology focus and not a learning focus. WBDLEs are being used with students from K-12 to post-graduate levels and the best thing people say about them is that they are no worse than traditional classrooms! No significant difference is the WBDLE buzzword. There are many challenges in WBDLEs. They include the validity and reliability of our measures, quality of assessments, and the myth that web-based course management systems provide an all inclusive learning package. This article focuses on validity and reliability, evaluation of current assessment practices, challenges that must be addressed (including web course management systems), and solutions to the problems.

"Educational assessment seeks to determine how well students are learning and is an integral part of the quest for improved education. It provides feedback to students, educators, parents, policy makers, and the public about the effectiveness of educational services" (National Research Council, 2001, p. 1). Measurement theories on reliability and validity are critical components in any assessments that lend credibility to the inferences drawn. WBDLEs present unique challenges to assessment of learning and are in desperate need of information that can give credibility to its assessment practices. Brookhart (2003) suggested that classroom assessment measurement theory is significantly different from its parent large scale assessment theory because of differences in the context, integration of assessment and instruction, and formative-summative assessment purposes. Since web-based distance learning environments are significantly different from traditional classrooms (Suen & Parkes, 1996) Brookhart's line of thinking must be extended further to rethink the measurement theories for WBDLEs. The first goal of this paper is to extend the measurement theories on reliability and validity to include WBDLEs.

Suen and Parkes, (1996) and Reeves, (2000) have called for improved practices in assessing web-based learning environments. For example Reeves suggested that online learning environments have to concentrate on improving assessment practices by concentrating on alternative assessments such as cognitive, performance, and portfolio assessments. He also stated that "colleges and universities are increasingly expected to help their student develop positive habits of mind such as commitment, motivation, and ethics as well as higher order outcomes such as problem solving, intellectual curiosity, and critical analysis. These new outcomes are much harder to assess than discrete knowledge and skills" (p. 104). Suen and Parkes (1996) suggested that distance learning "has a myriad of implications for the learning and assessment processes. Among these implications are (a) the isolation of the learner from resources, support, and peers; (b) the lack of face-to-face interaction with instructors; and (c) delayed feedback. These factors have, in part, necessitated a larger quantity and diversity of media and technology." The second goal of this article is to use the extended reliability and validity issues to evaluate current assessments in WBDLEs and see whether current assessments use new and diverse media or have just re-invented the wheel (and rather poorly) by converting methods used in traditional classroom settings to web-based learning environments.

Assessment in WBDLEs are being influenced significantly by the growing numbers of web course management systems and their perceived efficiency and functionality. No evaluation of WBDLE assessments will be useful and complete unless we evaluate these methods in the context of web-based course management systems (WBCMS). The third goal of this article is to show how existing functionality of WBCMSs can be used to improve assessments and some instances where the WBCMS do not afford the capabilities necessary to create effective assessments.

The final goal of this article is to describe three innovative practical solutions to the challenges of assessment in WBDLEs. Each solution is briefly described, validity issues are discussed, and stakeholder experiences with the solutions are shared.

EXTENDING MEASUREMENT THEORY TO WEB-BASED DISTANCE LEARNING ENVIRONMENTS

Students, families, teachers, principals, and anyone interested in learning get their information about what, how, and how much is being learned from assessment. Measurement theories on reliability and validity help us understand the measures, interpret the results, and use it to improve learning. In traditional classrooms the reliability and validity factored in formative and summative purposes of assessments, the contextual factors such as the presence of classmates and teacher(s), and immediate feedback. Inferences about student learning and actions were made based on the assessment information and were useful to the student, teachers, and stakeholders. The validity goals were to compare the student performance against "ideal" work. Assessments were part of the learning and good assessment is an "episode of genuine learning" (Brookhart, 2003). The teacher's understanding of the whole learning environment also affects the outcomes. Reliability in this environment is to make the assessment information as stable as possible.

On the other hand, web-based distance learning environments are quite different. Students are isolated from their peers, do not get immediate feedback, and the context in which the performance happens can vary significantly. The validity and reliability issues now have new meaning. Table 1 compares classroom assessment with WBDLEs in terms of the context, purpose of assessment, and feedback.

In summary, the WBDLEs are significantly different from traditional classrooms and we need to address the following issues carefully so that we do not lead students and stakeholders down the wrong path. Since assessment contributes to the learning experience, we must provide the necessary infrastructure so that students get the self-regulatory skills to learn from these environments. Based on the information shown in Table 1 and other research reviewed (Bachman, 2002; Moss, 2003) we can state that WBDLEs need to:

1. Focus on what constructs are measured and how we measure them.

2. Support multiple samples of learner performance to improve the reliability and validity.

3. Train students on self-regulatory skills for working in a distance learning environment.

4. Structure tasks to provide formative feedback to students.

5. Evaluate student performance and provide immediate feedback to the learner.

6. Convert the evaluation information to a useable format for students so that they may actually have a "genuine learning experience."

7. Summarize the assessment information in more meaningful ways for stakeholders.

8. Provide learners with tasks that can showcase their skills and still be useful with scaffolds and feedback.

9. Bring closure to the learning experience by creating a feedback loop to the student.

10. Take advantage of technologies to assess students.

11. Train students on the self-regulation skills necessary to observe their performances in isolation, reflect, seek help, and learn from the experiences.

Next we evaluate assessment methods currently used in WBDLE using the measurement theory adapted for WBDLE.

EVALUATION OF ASSESSMENT METHODS

"Every assessment, regardless of its purpose, rests on three pillars: a model of how students represent knowledge and develop competence in the subject domain, tasks or situations that allow one to observe students' performance, and an interpretation method for drawing inferences from the performance evidence thus obtained" (National Research Council 2001, p. 2). Using these three dimensions, validity, and reliability issues we explore and critique two frequently used assessments in WBDLE. A review of over 500 WBDLE courses conducted by the authors is the source for the information presented here. The courses reviewed included 300 undergraduate and 200 graduate courses in the sciences (Physics, Chemistry, Mathematics, Computer Science, Earth and Mineral Sciences), social sciences (English, Economics, Sociology, Psychology), health sciences (Nutrition Education, Physical Education, Hotel and Restaurant Management), business (Business Management, Management Information Systems, Accounting, and Marketing), and others (Safety Sciences). The courses were offered at eight educational institutions (universities and community colleges) and used many different design, development, and delivery approaches ranging from completely online courses with course management systems such as WebCT, Blackboard, and ANGEL to mixed delivery techniques (periodic face-to-face meetings and proctored tests). The courses included notes, lectures (in print, audio, and video), Powerpoint notes, case-studies, problem-based learning, chat rooms, discussion boards, timed quizzes, tests, and portfolios. Each assessment method described next includes background information, validity issues, role of course management systems, and possible solutions.

Multiple Choice

Multiple choice tests are used frequently in WBDLEs. They are easy to create, administer, and grade in online settings. Measures of reliability can be generated for multiple choice tests. There are four problems with multiple choice tests.

First, creating good multiple choice questions that can actually allow students to represent their knowledge well is difficult. A review of the literature on assessment research shows that short answer or multiple choice (even when carefully constructed) is a poor match to assessing higher order learning skills like problem solving (Suen & Parkes, 1996).

Second, using multiple choice tests as sole method of assessment in WBDLEs is a serious problem. Not only does it not measure how much the students know, how deep their knowledge is, and whether they can actually apply the knowledge, it is open to cheating by students who are taking the class online.

Third, creating online multiple choice tests may appear easy because of the web-based course management systems and easy to grade since the technology allows that, it really cannot be relied on to make inferences about student learning, and/or skills.

Fourth, multiple choice does not usually give informative feedback that students can use to self-assess their learning. This method does not give the teachers too much information on what the student know as well.

Finally, course management system's multiple choice tests are usually isolated from the learning modules and the feedback does not help the student in getting self-regulated about learning.

Possible solutions include, embedding multiple choice tests inside the notes/readings section or if the test feedback linked back to the relevant portions of the readings for the student.

Portfolios and Projects

Portfolios and projects are suggested as alternatives to multiple choice assessments in WBDLEs because they capture more working knowledge and knowledge in use. While it is difficult to create information on reliability, validity is said to be better with this method.

When portfolios are done in traditional classrooms, the student usually organizes their thoughts and presents samples of their skills. They also defend their choices or make a presentation to their colleagues and teachers. This allows us to actually draw inferences about the student's knowledge and transfer of skills. This experience is one of those "episodes of genuine learning" (Brookhart, 2003).

Portfolios in the traditional classroom can also contain multiple rich mediums for presentation and include paper, electronic, and other materials. In the web-based medium the presentation may appear rich because of the multi-media capabilities but it remains a very narrow view of the product.

On the other hand, WBDLE portfolios are gaining popularity but missing the critical component of the student presenting their portfolio and defending their choices. It is an exercise in creating a portfolio in isolation and receiving delayed feedback that may or may not be useful to the student and teachers. Portfolios in the course management systems may be limited to the multi-media presentation mode and be actually limited in its richness.

Possible solution to the problem includes student "presentations" of their portfolio in a chat room or Macromedia Breeze[TM] style of interactive presentation system.

Summary of Problems

Multiple choice and short answer tests are acceptable as long as we can actually draw inferences from the data. Portfolios and performance assessments are also acceptable as long as we can determine that the person can actually do more than just create a document and whether they can argue in favor of their positions.

In the examples presented here designers and course management system developers have taken existing assessment techniques used in traditional classrooms and converted them into online learning environments without exploiting all that the new technologies offer. In fact there are instances like portfolios that have completely lost their "defense" component that were actually a great learning and assessment experience for the student, peers, and teacher.

Given this description it is hard to make the case that assessment has changed with the invention of web-based technologies or that "new" assessment techniques have been introduced for web-based distance learning environments.

There are some opportunities afforded by the new technologies that can improve how we assess learning and present that information to promote learning. We present some innovative ideas for assessing web-based learning environments that have been implemented and successful. Each assessment technique is described briefly, validity issues are identified, and the techniques are justified next.

SOLUTIONS TO PROBLEMS IN WBDLES

Chat-Room Discussions as Assessment

Chat rooms are part of most web-based course management systems like ANGEL, Blackboard, E-College, and WebCT. They are used for synchronous discussions among students or between students and the teacher. A review of many WBDLEs show that this tool is frequently used as a social gathering place for students or as an avenue to answer student questions and discuss issues usually relating to course projects or problems.

A closer look at the functionality provided by chat-rooms shows that it is a good tool for allowing students the opportunity to "think on their feet" and communicate in a time constrained environment. This tool allows us to create WBDLEs that have a counterpart for class presentations, project presentations and discussions. It may also serve the need for an interactive style for teachers who like to ask students questions in class and get responses to gauge how well they understand the course content.

However, current chat tools are limited by their text-based interface, the need for good typing skills, poor organizing structure (anybody can type in anything at any time and discussions can appear disjointed), and some limits to the number of persons who can interact in a session. We created multiple courses that make effective use of the chat tool in assessment and minimize the concerns.

Applying chat room assessments. The first example shows a chat room activity that was structured. Students were e-mailed an agenda, discussion questions, and procedures for communication two days prior to the scheduled chat session. Students were instructed to prepare a Word document with their answers to the questions to minimize the typing necessary during the chat. The chat sessions were limited to 5-6 persons and students were polled on availability to ensure that all of them had an opportunity to participate. The teacher moderated the discussion and used a round-robin questioning technique to direct questions to the students in the chat room. After the student had the opportunity to answer the question, others were asked for comments and examples. The teacher then followed up with any feedback and moved on to the next question. Using this method the teacher was able to interact with students "live" and provide the students an opportunity to think in time constrained situations. A grading scheme was designed to identify the main learning tasks and the instructor assessed the individual learner's responses using the chat logs.

A second example shows how the chat was incorporated into project presentations in web-based distance learning courses. Students were required to submit their project documents to the teacher and their peers prior to the chat. The student led the discussion on their product by answering questions from their peers and teacher. Again, this served as an assessment forum that was useful to the student, their peers, and the teacher. Macromedia Breeze[TM] provides yet another advanced interactive chat that allows sound and video with capabilities to include Powerpoint[TM], Whiteboard, and other presentation tools.

Validity and reliability of chat room assessment. These assessment techniques make use of the available technologies and improve the validity of the assessment techniques currently practiced in web-based distance learning courses. Face validity and construct validity are higher in this approach than when students learn in isolation at their own pace and never have to work in time-constrained situations. The instructor and student can interact about projects and discuss or debate issues as well. This allows us to actually make inferences about the depth of the student's understanding and ability to apply their knowledge.

The students are also able to gauge their understanding from the assessment chats. The questions posed by the instructor and peers, the reactions from the student, how they responded to the questions are all part of the learning cycle. The logs of these types of chat assessments were always posted by the instructor for the whole group to read later. This allowed reflection on the part of the students and helps their self-regulation of learning.

Course management systems and chat rooms. WBCMS--Chat room technology can be improved with the addition of advanced moderating capabilities, organization, and searchable texts. Most importantly the tools must include moderation capabilities that can be easily linked back to learning modules. It will also be useful to allow the teacher to review the logs and highlight critical parts of the discussion to help students focus on important portions of the discussion.

Student and faculty experiences. Experiences reported by students in these organized chat rooms were that they learned a great deal from them. Students commented that they had never used chat rooms in such an organized manner and that the format reminded them of formal meetings in their companies. Some students working on teams also said they learned this new technique that allowed them to use chat rooms very effectively.

Three professors who used this chat format were thrilled with the outcome. Two of the three had taught online classes where the chat room was a social gathering place and they both commented on how much they learned from this experience. They commented that the lack of face to face interactions in online courses had left them feeling that they had no synchronous conversations with the students. One professor had tried talking to each student at least once a semester using the telephone but felt that was not enough and not practical. This modified chat facilitated synchronous conversations that allowed the students to "think on their feet". These professors started relying on the chat logs to see what the students were understanding and also wanted to include the logs as part of the final grade.

Problem-Solving Logs as Assessment (using procedural analysis & simulations)

Learning environments that require students to learn how to perform a procedural task, generate a multi-step proof, or design a product are difficult to assess in WBDLEs. In face-to-face classrooms the instructor can observe the student performing the task or have the student write the proof and provide immediate feedback on the correct approach and solutions. In some cases correcting the errors immediately is critical in making sure that the student is on the correct path to the solution and does not spend too much time following dead-end paths. In WBDLE courses this may be pronounced with the student working in isolation and not having immediate feedback or scaffolding on their problem solving process. To address these concerns we created two special environments.

Applying Problem-Solving Logs as Assessments

The first example we created has the student interacting with a web-based interface built using PERL programming language scripts to ask questions, get responses from the learner, log the responses, and provide feedback to the learner. This method was adapted from the computerized tutoring methods used by many research teams (Anderson, Lorbett, Koedinger, & Pelletier, 1995; Graesser, Woemer-Hastings, Wiemer-Hastings, Kreuz, & The Tutoring Research Group, 2000; Wijekumar & Meyer, in press) but simplified to work on a long-term basis. The ideas borrowed from these cognitive tutors include the tracking of learners' choices and adapting the responses of the system to give immediate feedback to the learners. The technique was used in most of the learning modules in courses that required students to perform many steps to reach their objectives.

The logged interactions were parsed using another script and formatted into an Excel spreadsheet to show the students' pattern of actions on the tasks and graded by the professor based on accuracy of responses in each step and final product. This additional step was useful to the instructor to gauge student understanding.

The second example used logs created in student interactions within simulations as an assessment technique. Students worked within an environment where they could build different computer network configurations and test them. The configurations created by the students and actions performed were logged. The log was parsed using a predesigned activity matrix and sequenced. The teacher was able to grade the effectiveness of the final outcome as well as the appropriateness of the steps leading to the product.

Similar techniques have been reported by Reeves (2002) where students interact with simulations or diagnostic systems and their activities and outcomes are part of the assessment of learning. Our method goes beyond the use of the final product to delve deeper into the problem solving approach by using the activity logs and comparing them against "ideal" solving paths created by using expert approaches to problem solving as well as possible exceptions.

Validity and reliability. Even though face-validity in this approach is limited the construct validity is high when aligned with the higher-order learning goals.

Course management systems and problem-solving logs. Most computers and related environments have a wealth of data on user interactions with the system that is rarely used. Extracting this information in a meaningful form and combining the data with the learning objectives and instructional techniques creates an assessment tool that is rich in information for purposes of grading higher-order activities.

Student and faculty experiences. Students reported that the immediate feedback in our interactive problem solving tool allowed them to stay focused on the problem solving tasks. The most interesting part of this example was how excited the professors were in seeing the numbers of attempts the students had on each step and finding out the bottlenecks in the problem solving tasks. Each professor who taught with this type of simulation/feedback problem solving mechanism received a daily report on each student's performance, attempts, responses, and so forth. The professors reacted to these reports immediately by sometimes emailing hints to the students and in many instances posting comments to a discussion board on common problems encountered in the problem solving process.

Discussion Board Content Analysis as Assessment

Solving complex real-life problems (sometimes referred to as ill-structured problems) is an important goal of our learning environments today (National Research Council, 1996). Creating WBDLEs that allows learners the experiences of solving such problems and assessing their skills is very important.

Discussion boards are frequently used in web-based distance learning courses as supplements to activities. The justification for its use is the research on collaborative learning and constructivist learning philosophies. However, a closer look at both areas suggests that collaborative learning shows improved learning in those students who give explanations and does not improve learning for the students who asked the questions (Webb, Troper, & Fall, 1995). Additionally, a review of 20 web-based distance learning course discussions in progress by the authors shows that many learners paraphrase each other's postings and contribute little significant new information in unstructured discussions. Finally, most discussions are skewed by the personalities in the class. Some highly motivated, engaged students can monopolize the discussion and/or contribute most of the good ideas whereas the passive students may or may not even be reading the postings.

Using a recent research study on argumentation (Wiley & Voss, 1999) as the basis for improving learning in collaborative learning environments we use debates as the learning and assessment technique in many of our course discussions. Argumentation is also an integral part of solving ill-structured problems (Jonassen, 1997).

Applying discussion board content analysis as assessments. An example of this is a structured discussion on the use of herbal supplements like Echinacea for the common cold. Students are assigned to two teams (Team A--For the use of the supplements and Team B--Against the use of the supplements). Each team is given the charge of creating a case brief that is reviewed by the teacher stating the main points they will cover in making a convincing argument for their position. After the paper is reviewed by the teacher the debate is opened on the discussion board with a posted time frame for submissions and rebuttals. The whole activity is limited to two weeks to maintain interest and allow the students to engage in a lively discussion. Each team is assessed on how convincing their arguments were.

Validity and reliability. This method has higher face and construct validity when compared to the previous technique of using computer interaction logs alone. This method allows evaluation of more than just individual learning but collaborative learning processes. Each student contributes their experiences and contributes to the discussion and that improves face and construct validity. However, we must be clear about how much the previous postings influenced each individual student's posting. For example, if a student paraphrases previous postings then is that proof of their learning?

Course management systems and discussion boards. Course managements tout discussion boards as one of their great tools for supporting collaboration and learning. What they rarely tell you is that just making students talk on a discussion board does not imply they are learning. Promoting reflection on the discussion board postings is one critical element that currently requires the professors to carefully coordinate the discussion. Even with professors navigating the course not all students participate actively and the discussions can be hard to synthesize.

There are solutions to these problems with new tools such as KIE that ask students to reflect on their posting and classify them (Scardamalia & Bereiter, 1999). Additionally, there are tools like Rotisserie (created by Harvard University) that request responses from passive students, remind students to contribute and time the discussions.

Student and faculty experiences. Students participating in the debates described earlier (example, Echinacea debate) were always nervous about what was expected. They frequently quizzed the professor in the chat rooms and asked questions like, "what would a good answer be?", "How many points do we have to make to earn a good grade?" Once the debate started students enjoyed the conversations. They commented that they learned more from being forced to think about the positive and negative effects of the supplements. They also looked critically at each others postings and the sources for the conversations.

Professors who use the discussion board debates were all surprised about the number of postings and how engaged the students were. The professor who conducted the nutrition related discussions also conducted debates on healthy living and found students freely spoke of their habits online. The same professor had taught the same course in a traditional classroom and students there had been much more reticent about their lifestyles. She concluded that the relative anonymity in the online environment may have resulted in the postings.

CONCLUSION

Web-based learning environments are significantly different from their in-class counterparts and require rethinking the measurement theories, assessment methods, and inferences drawn from them. It is very important that we not lose sight of the context and constructs measured in web-based assessments and allow students to participate in their assessment learning cycle and allow teachers the ability to gauge learning and adapt the learning environment to what the assessment is telling us. We have shown examples of three techniques used in web-based distance learning courses that attempt to do this. These examples have addressed some concerns about assessment in web-based distance learning and adapted existing technologies to serve our needs. In some cases the course management system tools had to be modified to improve the assessment quality and in others we created separate simulations and interactive environments for assessment. We hope that students, teachers, designers, and stakeholders will push much harder to make real improvements to assessment in web-based distance learning environments.

References

Anderson, J. R., Corbett, A. T., Koedinger, K., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4(2), 167-207.

Bachman, L.F. (2002). Alternative interpretations of alternative assessments: Some validity issues in educational performance assessments. Educational Measurement: Issues and Practice, 21(3), 5-18.

Brookhart, S.M. (2003). Developing measurement theory for classroom assessment purposes and uses. Educational Measurement: Issues and Practice, 22(4), 5-12.

Graesser, A., Wiemer-Hastings, K., Wiemer-Hastings, P., Kreuz, R., & The Tutoring Research Group (2000). AutoTutor: A simulation of a human tutor. Journal of Cognitive Systems Research, 1, 35-51.

Jonassen, D.H. (1997). Instructional design for well-structured and ill-structured problems. Educational Technology Research and Development, 45(1), 656-694.

Moss, P.A. (2003). Reconceptualizing validity for classroom assessment. Educational Measurement: Issues and Practice, 22(4), 13-21.

National Research Council (1996). National science educational standards. Washington, DC: National Academy Press.

Reeves, T.C. (2000). Alternative assessment approaches for online learning environments in higher education. Journal of Educational Computing Research, 23(1), 101-111.

Reeves, T.C. (2002). Keys to successful e-learning: Outcomes, assessment and evaluation. Educational Technology, 42(6), 23-29.

Scardamalia, M., & Bereiter, C. (1999). Schools as knowledge building organizations. Retrieved January 21, 2006, from http://ikit.org/fulltext/1999schoolsaskb.pdf

Suen, H.K., & Parkes, J. (1996). Challenges and opportunities for student assessment in distance education. The Distance Education Online Symposium, 6(7) [Online serial]. Retrieved January 21, 2006, from http://www.music.ecu.edu/DistEd/EVALUATION.html

Webb, N.M., Troper, J.D., & Fall, R. (1995). Constructive activity and learning in collaborative small groups. Journal of Educational Psychology, 87(3), 406-423.

Wijekumar, K., & Meyer, B.J.F. (in press). Design and pilot of a web-based intelligent tutoring system for the structure strategy. Computers in the Schools.

Wiley, J., & Voss, J. (1999). Constructing arguments from multiple sources: Tasks that promote understanding and not just memory for text. Journal of Educational Psychology, 91(2), 301-333.

KAY WIJEKUMAR

The Pennsylvania State University 

USA

[e-mail] 

LON FERGUSON AND DIANE WAGONER

Indiana University of Pennsylvania 

USA

[e-mail]

[e-mail]

             Traditional Classroom
             Environments (Source:    Web-Based Distance Learning
Validity     Brookhart, 2003)         Environments

Purpose      Inferences made and      The goals are the same as the
             actions taken are        traditional classroom where the
             internal to the          assessment is supposed to help the
             measurement process      stakeholders as well as the
                                      students improve learning. The
                                      goal is not to make comparisons
                                      between groups of students as is
                                      done in large-scale assessments.
Context      Students are observers   Students are isolated from the
             jointly with teachers    teachers and peers. They have to
                                      be self-regulated to gather useful
                                      information from assessments.
                                      Unless students are given explicit
                                      training on how they can manage
                                      their learning, multiple choice
                                      and other tests in WBDLEs may not
                                      provide useful information for the
                                      student.
Context and  Students' awareness of   Again, the assessment process can
Feedback     and benefit from         become so isolated that the
             assessment information   "information" is not readily
             are part of the          available to the student and they
             "information" itself     may not know how to actually use
                                      the information.
Purpose      Validity goal is to      Student performance in WBDLEs may
             compare students' work   be compared to what is considered
             against "ideal" work     ideal but the available measures
                                      and tools are limited. We do not
                                      get the rich observation data and
                                      face-to-face interactions in
                                      WBDLEs.
Context      The measurement context  It is difficult to identify the
             is construct-relevant    measurement context and use it for
                                      assessment.
Feedback     Assessment is part of    In WBDLEs assessment is usually
             Instruction and becomes  separated from the instruction.
             part of the learning     Tests and activities are usually
             itself                   compartmentalized into a separate
                                      link and rarely connected back to
                                      the instruction. (For example,
                                      they do not link back to the
                                      instruction and tell the student
                                      that they need to brush up on some
                                      topic or that they should re-read
                                      certain information).
Context      Content specifications   The domain objectives may remain
             reflect both the domain  the same as classroom instruction
             (learning objectives)    (in fact most designers and
             and instruction (modes,  instructors insist they are).
             and activities)          However, the modes and activities
                                      in WBDLEs appear to be restricted
                                      to reading notes, performing tasks
                                      on their own time, viewing video
                                      tapes, and in some instances
                                      interactive simulation types of
                                      environments.
Context      Teacher beliefs,         The teacher beliefs and
             teacher instructional    instructional practices can
             practices, and teacher   actually take a back seat to the
             understanding of both    technology driven creation of
             the subject matter and   WBDLEs. If a teacher usually uses
             students are relevant    an interactive teaching style
             validity concerns        where they constantly ask
                                      questions of students and then
                                      adapt their class to the student
                                      responses, WBDLE technologies
                                      rarely allow this mode of
                                      interaction. So the validity
                                      issues may actually be less
                                      influenced in WBDLEs. However, the
                                      student's understanding of the
                                      environment can be affected by the
                                      distance of space, time, and
                                      feedback.
Purpose      Reliability is           The number of available measures
             sufficiency of           for assessing student performance
             information              has been limited by the
                                      technologies promoted by web-based
                                      course management systems. Special

                                      programs that can capture student
                                      performance in simulation types of
                                      environments can actually give
                                      more credibility to the
                                      reliability of the measures.
Purpose      Reliability goal is      The reliability goal is still
and Context  stability of measured    stability but the contextual
             information on the gap   factors influencing the measures
             between the ideal and    can vary greatly and cannot be
             actual performance       controlled.