Assessing Distance Teaching and Learning.

Abstract

Recently, the number of college and university programs delivered via forms of distance education has increased. In the wake of this growth, questions are raised about the quality of distance teaching and learning, and about how to assess such teaching and learning. There is a lack of evaluation tools and the majority of distance education publications are opinion pieces and how-to-articles rather than original research. So how can the quality of distance teaching and learning be assessed? We present a theoretical framework for three researched assessment approaches: "Faculty/Student Assessment of Distance Education Practices," course-long student journal, and student focus group.

Introduction

In recent years the number of college and university programs being delivered via forms of distance education, particularly via the Interact, has increased. The U.S. Department of Education found that their number increased by 72%, to 1,190 and "certificate programs grew from 170 to 330" from 1995 to 1998. In 1998 alone, 54,000 online education courses were taken by 1.6 million students. In 1995, 33% of higher education institutions offered distance education courses, and by 1998 the percentage grew to 44%. The Internet was the primary medium for delivering these courses and programs: 66% in 1998, 22% in 1995. (National Center for Education Statistics, Distance Education at Postsecondary Education Institutions: 1997-98. December, 1999. <http://nces.ed.gov>).

Quality Questions

There are, in the wake of this growth, questions being raised about the quality of distance teaching and learning. Neo-Luddites, a congressional commission, conscientious professors, and even the U.S. Department of Education have raised this important issue.

Neo-Luddites

David F. Noble, a professor of history at York University, Toronto, and long time critic of the role of technology in culture, speaks of distance education as "fools' gold" that "is tempting some administrators to put the core values of their institutions at risk." He refers to the low-tech, old-fashioned classroom as "sacred space" and says, "In person, you get a sense of me you can't get online. I'm convinced of that we have five senses. Why artificially narrow the bandwidth?" (Young, The Chronicle of Higher Education, March 31, 2000, A47-A49).

Congress

The Congressional Commission on Web-Based Education, chaired by Senator Bob Kerrey (Democrat, Nebraska), announced that it would encourage distance education providers to offer high-quality programs (Carnevale, The Chronicle of Higher Education, February 18, 2000, A56).

Faculty

Faculty have expressed concerns about the quality of distance education. For example, a January 2000 report, Teaching at an Internet Distance, is from a faculty study committee from the University of Illinois' three campuses. The committee members' initial perspectives for distance education were balanced between skepticism and enthusiasm. However, the report of the faculty raised concerns about the quality of teaching and learning. "Computer mediated instruction may indeed introduce new and highly effective teaching paradigms, but high-quality teaching is not always assured. Administrative decisions made without due consideration to pedagogy, or worse, with policies or technology which hampers quality, may cause much wasted time, money, and effort of both faculty and students." See (Young, The Chronicle of Higher Education, January 14, 2000, A48).

Elsewhere

The Faculty Senate at San Diego State University, on April 6, 2000, adopted a five page distance education policy focused on balancing the rights of professors with quality control of courses delivered online. Professorial oversight of distance education courses in their field; student interaction with faculty and other students in the course that is "substantial, personal, and timely"; student access to appropriate resources and services; and full-time professors were among the quality concerns addressed. (See <http://www.rohan.sdsu.edu/dept/senate/sendoc/distanceed.apr2000.html>. (Carnevale, The Chronicle of Higher Education, May 12, 2000, A49).

Department of Education

Marshall Smith, acting deputy secretary of education has said that the spread of online courses "begins to raise the question about quality. This is going to heighten the importance of gathering performance data" (Carnevale, The Chronicle of Higher Education, January 7, 2000, A57).

Assessment Questions

In addition to questions being raised about the quality of distance teaching and learning, there are questions surfacing about how to assess such teaching and learning. There is a lack of evaluation tools. Randy Bass, executive director of the Georgetown University Center for New Designs in Learning and Scholarship, suggests that higher education has not yet developed methods to adequately evaluate online courses. He suggests that a "culture of reflective" practice should be developed to discuss how online instruction influences the learning of students (Kiernan, The Chronicle of Higher Education, April 28, 2000, A45-46).

Our Question

Distance education has become more prevalent in higher education and is one of the most powerful new forces influencing the direction of higher education (Connick, 1997; Gandolfo, 1998). However, Merisotis and Phipps' review of distance education literature from 1990-99 (1999) found that the majority of distance education publications are opinion pieces and how-to-articles rather than original faculty- or student-based research. Given the wave of distance education provision by colleges and universities, the level of concern about the quality of distance education teaching and learning, and the importance of assessing distance teaching and learning, how can distance teaching and learning be assessed?

Methodology

Earlier stages of the authors' work related to assessing distance teaching and learning have been published, and readers will be referred to two of those sources in particular (Lockhart, Borland, & Howard, 2000; Lockhart & Borland, 2000). The focus of this article is to provide a composite and review of the conceptual framework for assessing distance teaching and learning, and a review of the processes and the instrument the authors developed for assessing distance teaching and learning. Further, the lessons learned about using these approaches to assessing distance teaching and learning are highlighted. The three approaches are 1) the "Faculty/Student Assessment of Distance Education Practices" (a planning inventory and assessment instrument), 2) the course-long student journal, and 3) the student focus group.

Theoretical Framework for Assessing Distance Education

Thompson (1990) identifies interaction as a significant component in teachers' promoting learning in distance education. However, interaction between the learner and the instructor, and between learners is frequently reported as the missing ingredient in distance education (Wagner 1997). Moore (1989) proposed three types of interaction in distance education (1-3 below) that could be viewed as principles of good practice and ought to be considered in planning and assessment. A fourth type of interaction (4 below), also to be considered as part of planning and assessment, was offered by Hillman, Willis, and Gunawardena (1994). These interactions are (1) learner-content, (2) learner-instructor, (3) learner-learner, and (4) learner-interface (technology).

Chickering and Gamson (1991) compiled findings from decades of research on the undergraduate experience and published Seven Principles for Good Practices in Undergraduate Education. Based on effective education (teaching and learning) as active, cooperative, and demanding, they state that good teaching practices:

1. encourage contacts between students and faculty;

2. encourage cooperation among students;

3. encourage active learning;

4. give prompt feedback;

5. emphasize time on task;

6. communicate high expectations; and,

7. respect diverse talents and ways of learning (Chickering and Gamson 1991).

They believe that the seven principles can benefit all types of students: young, old, male, female and all races, as well as various types of programs. Using the principles as a framework, Chickering and Gamson developed a faculty inventory to help faculty members identify methods and activities they might include in their traditional face-to-face classrooms.

The four types of interactions of distance education in combination with these seven principles provide the conceptual framework (theoretical and conceptual models and principles) for the planning and assessment of distance teaching and learning.

Assessing Distance Teaching and Learning

The authors' work has focused on assessing whether distance education faculty and students found Seven Principles' teaching strategies (Chickering & Gamson, 1991) to be the most effective in promoting learning. Three approaches were utilized. 1) A researcher-designed "Faculty/Student Assessment of Distance Education Practices," 2) the course-long student journal, and 3) the student focus group.

The authors provide a composite and review of the three processes and the instrument the authors developed for assessing distance teaching and learning. Further, the lessons learned about using these approaches to assessing distance teaching and learning are highlighted.

Faculty/Student Assessment of Distance Education Practices

Development. For a detailed description of how the "Faculty/Student Assessment of Distance Education Practices" instrument was developed in its initial form (as a planning and assessment instrument) see Lockhart, Borland, & Howard "Distance Teaching and Learning with Technology: A Cross-Disciplinary Planning and Assessment Instrument" in Journal for the Art of Teaching, volume VII, Number 1, Spring 2000. In short, the instrument emerged from a grant-funded study at Montana State University where all 45 faculty members who were teaching distance education courses were asked:

?? how they communicated with students other than through the main distance delivery method

?? how students communicated and cooperated with each other, what methods were used to promote active learning,

?? how feedback was provided to students,

?? how faculty provided time-on-task structure to students,

?? how they communicated high expectations, and

?? how they respected diverse talents and ways of learning.

Additionally, faculty were asked what distance teaching and learning strategies were most effective.

The results were analyzed and compared to the conceptual framework and to student responses. For details see Lockhart, Borland, Howard, 2000 (noted above) and Lockhart & Borland, "Effective Teaching and Learning with Technology," National Conference on the Adult Learner in Higher Education Conference Proceedings, Atlanta, 5/2000. What emerged was a planning and assessment instrument for distance teaching and learning that is complementary to the Chickering and Gamson Seven Principles. It has been further developed as an assessment instrument to gather both faculty and student perspectives on the same course. Both the Faculty and the Student versions of the Assessment of Distance Education Practices instrument are printed at the end of this article.

The instrument has been used to compare the perspectives of the faculty member against those of his/her students in given distance education courses. Not only does it measure the faculty members' and their students' perceived frequency of the methods that correspond to each of the Seven Principles, but it also establishes the perceived effectiveness of each utilized method regarding helping students to learn the course material. Both teaching and learning can be assessed using the Faculty/Student Assessment of Distance Education Practices.

Validity/Reliability

Prior to its distribution, the instrument was reviewed for content and face validity, readability, and clarity by six faculty members in the Department of Education at Montana State University. Three of these members had taught distance courses in the past; three had never taught a distance course. Since its initial uses, the Faculty/Student Assessment of Distance Education Practices instrument has been tested for validity by numerous distance education faculty members at Montana State University and is being studied by a graduate student, Jan Thiessen, under the direction of Dr. Mohamed Alley, at Canada's all distance education institution, Athabasca University. Thiessen hopes to further establish the instrument's validity and to determine institution-specific factors to be added to the instrument. The instrument is also being considered by faculty development experts at the British Open University.

Lessons

There are three lessons that have been learned regarding the use of the Faculty/Student Assessment of Distance Education Practices. First, the instrument does provide useful data relative to teaching and learning in distance education. One can assess the differences between faculty and student perspectives, and the choice of pedagogical techniques relative to effective learning; and adjustments to the pedagogical plan can be made.

The second lesson relative to the Faculty/Student Assessment of Distance Education Practices has to do with the process of administering the instrument to distant students. In a spring 2000 attempt to distribute the instrument to about 100 students from three courses, a procedural challenge emerged. The instrument, in several word processor formats, was loaded on a shared online platform as attached files to an e-mailed invitation to complete the instrument and submit it to the assessor/researcher. Much to the surprise of the assessor/researcher and the three courses' faculty members, the vast majority of the students were interested in completing the instrument, but unable to open the instrument in any of the word processor formats. The lesson: be sure that online platforms, word processor formats, and students' abilities and hardware and/or software are compatible.

Finally, the instrument, based upon Chickering and Gamson's Seven Principles, does help to plan and assess quality distance education in terms of teaching and learning.

Course-Long Student Journal

A second Seven Principles-based approach to assessing distance education quality used by the researchers was the course-long student journal. This approach was used during the conduct of an on-line graduate course. The teaching and learning strategies used in the course were

?? e-mail discussions with a study partner,

?? e-mail correspondence with the instructor,

?? telephone conversation with instructor,

?? interviews conducted by students with a professional working in the field,

?? independent research conducted by students,

?? opportunities to relate material to past experiences,

?? opportunity to relate material to future use,

?? reading text/supplemental material,

?? studying for exam,

?? homework and assignments returned promptly, and

?? ability to work on material at a time and place convenient to the student.

Design

Each student in the course was required (as a course-embedded assignment) to journal regarding his/her experiences and perspectives regarding the online teaching-learning process. For course-level assessment, all of the students' course-long journals were analyzed using the constant comparative method associated with qualitative research. Themes surfaced related to the Seven Principles. For student assessments, the journals provided detailed perspectives from individual class members.

Lessons

There are two lessons that arise from this assessment technique that the assessor of distance education could apply. First, the course-long student journal is a good form of on-going assessment. Second, it provides a basis for formative evaluation and adjustments. The instructor can, on a periodic basis, note the progression of students relative to the teaching-learning process and Chickering and Gamson's Seven Principles. In this study, a progression from apprehension and uncertainty about online instruction and the effectiveness of such instructional techniques toward greater student confidence about their distance learning and distance education as "a great way to learn" was noted. Via the on-going feedback (consistent with the Seven Principles) the instructor had the opportunity to encourage and mediate the progression via individual and group student-instructor and student-student interactions and support (also consistent with the Seven Principles). The instructor noted that course-long student journals are a source of depth and richness unlike other assessment approaches.

The Student Focus Group

A third approach to assessing teaching and learning for distance courses, according to Chickering and Gamson's Seven Principles, is the student focus group.

Design

A focus group was held at the end of a course. Each student in the course was invited to a central location to meet, face-to-face, with the professor and his/her classmates. The instructor conducted a summative focus group around the Seven Principles and permitted other aspects of the distance education experience to surface as students interacted with each other about the distance teaching-learning process.

Lessons

First, instructors who also utilized the course-long student journal technique noted that the focus groups did not add to what they already knew via the journals: One said she had "heard it all." This suggests the power of the formative journal process. Second, the focus group provides a chance for face-to-face discussion that supplements the written journal data with body language and emotion concerning the teaching-learning process. Third, students in focus groups have a chance to share their course experiences, and one student's comments may encourage other students to reflect on their experiences differently. The focus group dynamic provides a means of learning from peers. Fourth, students enjoy being with each other and their faculty members. One student compared the journal with the focus group (as well as the distance teaching-learning process with a face-to-face process). They paralleled it with banking via a teller and with an automated teller machine: The focus group is a very human assessment process and most people like being with people, even in assessing their teaching-learning processes. Finally, the focus group, unless it is held during the course, is summative rather than formative.

Conclusion

Distance education has become more prevalent in higher education and is one of the most powerful new forces influencing the direction of higher education; however, the majority of distance education publications are opinion pieces and how-to-articles rather than original faculty- or student-based research. Given the wave of distance education provision by colleges and universities, the level of concern about the quality of distance education teaching and learning, and the importance of assessing distance teaching and learning, how can distance teaching and learning be assessed?

The authors, drawing upon the work of Chickering and Gamson (the Seven Principles for Good Practice in Undergraduate Education) presented three approaches to assessing distance teaching and learning: the "Faculty/Student Assessment of Distance Education Practices," the course-long student journal, and the student focus group. The lessons learned from the use of each assessment approach suggest that these approaches can be effectual in terms of informing faculty about the quality of distance education instruction and learning. However, given the limitations and the advantages of each of the approaches, it is best to consider and utilize a combination of these approaches. One other lesson from the authors' experiences in assessing distance education should not be lost via focusing on the approaches. The Seven Principles of Good Practice in Undergraduate Education presented by Chickering and Gamson are also a fitting basis for assessing distance education.

References

Carnevale, D. (2000, January 7). "Survey finds 72% rise in number of distance-education programs." The Chronicle of Higher Education, A57.

Carnevale, D. (2000, February 8). "A key senator dismisses need for legislation to assure the quality of distance education." The Chronicle of Higher Education, A56.

Carnevale, D. (2000, May 12). "A college's detailed policy on distance education: At San Diego State, a panel of professors and administrators seeks to balance faculty rights and quality control in online courses." The Chronicle of Higher Education, A49.

Chickering, A. & Gamson, Z. (1997). Seven principles for good practice in undergraduate education. In New Directions for Teaching and Learning 47, 63-69.

Connick, G. (1997). Issues and trends to take us into the twenty-first century. In New Directions for Teaching and Learning: Teaching and Learning at a Distance: What it Takes to Effectively Design, Deliver, and Evaluate Programs 71, 7-12.

Cyrs, T. (1997). Competence in teaching at a distance. In New Directions for Teaching and Learning: Teaching and Learning at a Distance: What it Takes to Effectively Design, Deliver, and Evaluate Programs 71, 15-18.

Dasher-Alston, R. & Patton, G. (1998). Evaluation criteria for distance learning. Planning for Higher Education 2 7, 11-17.

Faculty Senate. April 6, 2000 San Diego State University. Available http://www.rohan.sdsu.edu/dept/senate/sendoc/distanceed.apr2000.html

Gandolfo, A. (1998). Brave new world? The challenge of technology to time-honored pedagogies and traditional structures. In New Directions for Teaching and Learning 76, 23-38.

Hillman, D., Willis, D. & Gunawardena, C. (1994). Learner-interface interaction in distance education: An extension of contemporary models and strategies. The American Journal of Distance Education 8(2), 30-42.

James, W. & Gardner, D. (1997). Learning styles: Implications for distance learning. In New Directions for Adult and Continuing Education: Facilitating Distance Education 67, 19-32.

Kiernan, V. (2000, April 28). Rewards remain dim for professors who pursue digital scholarship: A case study at Indiana University shows that ways to evaluate online work are still lacking. The Chronicle of Higher Education, A45-46.

Lockhart, M. & Borland, K. (2000, May). Effective teaching and learning with technology. National Conference on the Adult Learner in Higher Education Conference Proceedings. Atlanta, 5/2000.

Lockhart, M., Borland, K., & Howard, R. (2000, Spring). Distance teaching and learning with technology: A cross-disciplinary planning and assessment instrument. Journal for the Art of Teaching, 7(1).

Merisotis, J., and Phipps, R. (1999). What's the difference? Outcomes of distance vs. traditional classroom-based learning. Change 13-17.

Miller, G. (1997). The undergraduate curriculum and distance education. In New Directions for Adult and Continuing Education: Facilitating Distance Education 67:31-50.

Moore, M. (1989). Three types of interaction. The American Journal of Distance Education 3(2): 1-6.

National Center for Education Statistics. (1999, December). Distance education at postsecondary education institutions: 1997-98. Available http://nces.ed.gov

Schrum, L. (1998). On-line education: A study of emerging pedagogy. In New Directions for Adult and Continuing Education: Adult Learning and the Internet 78, 53-62.

Simonson, M. (1997). Evaluating teaching and learning at a distance. In New Directions for Teaching and Learning: Teaching and Learning at a Distance: What it Takes to Effectively Design, Deliver, and Evaluate Programs 71, 87-94.

Thompson, G. (1990). How can correspondence-based distance education be improved? Journal of Distance Education 5(1), 53-65.

Ragan, L. (1999). Good teaching is good teaching: An emerging set of guiding principles and practices for the design and development of distance education. CA USE/EFFECT 22 1-7.

University of Illinois. (2000, January). Teaching at an Internet distance. Report from a faculty study committee. Available http://www.vpaa.uillinois.edu/tid/report/

Wagner, E. (1995). Distance education: Success factors. Adult Learning 7, 18-27.

Wagner, E. (1997). Interactivity: From agents to outcomes. In New Directions for Teaching and Learning: Teaching and Learning at a Distance: What it Takes to Effectively Design, Deliver, and Evaluate Programs 71, 19-26.

Young, J. (2000, January 14). Faculty report at University of Illinois Casts Skeptical Eye on Distance Education. The Chronicle of Higher Education, A48.

Young, J. (2000, March 31). David Noble's battle to `Defend the Sacred Space' of the classroom: Jeremiads against online education attract followers; critics say he's an ill-informed Luddite. The Chronicle of Higher Education, A47-A49.

Note

The author, Anuradhaa Shastri, wishes to thank Jeanne Ellis Ormrod, Professor Emerita of Educational Psychology, University of Northern Colorado, and Adjunct Professor of Education, University of New Hampshire for her valuable suggestions to this article "Examining Content Knowledge Gains in Academic Service-Learning: A Study in an Educational Psychology Course."

Academic Exchange Quarterly Spring 2000, pp 47-53.

Dr. Kenneth W. Borland is Assistant Professor of Higher Education. E-mail: <[e-mail]>. Dr. Marilyn S. Lockhart is Assistant Professor of Adult and Higher Education. Dr. Richard D. Howard is Associate Professor of Higher Education.