Evaluating the use of instructional technology in higher education.

Abstract

Faculty are implementing instructional technology into their higher education classrooms at an exponential rate. With this increase comes the need for models of evaluation aimed at measuring quality implementation. The authors consider the issue of identifying and articulating quality implementation of instructional technology within the following framework: all students' opportunities for increased access to both their instructors and their course materials; the impact that using the technology has on student motivation, ability to learn, and satisfaction with the course; the administrative support made available; and the cost effectiveness of using technologies in higher education classrooms. A brief review of the literature, and associated evaluation theory is also provided.

Introduction

For decades colleges and universities have experimented with various technological innovations to deliver instruction. However, during the period of 1990 to present, higher education has experienced an exponential growth in the type and nature of instructional technology (IT) used in the traditional classroom to increase the nature and type of instructional opportunities for their students. These courses are cutting across curricula, including courses in engineering as well as subjects such as foreign languages, women's studies and virtually every discipline (Burnaska, 1998). A consequence of this growth of IT is the need to increase evaluation efforts that consider its impact on the both the student and the curriculum.

Our Perspective

While quality is typically considered a theoretical evaluation issue, we present a more a discussion of evaluating the quality of IT implementation and use in higher education. We take the blended view that determining quality in a program is highly contextually based yet specific quality indicators are necessary to help guide program administrators. Educational programs that positively impact their particular stakeholders, yet lack predetermined indicators of success, may still be considered as high quality by an evaluator. On the other hand, pre-determined indicators of success help newly forming programs make critical decisions such as allocating resources and personnel. With this second issue in mind, we present four possible quality indicators (Access, Student Learning, Administrative Support, and Cost Effectiveness) of IT use at the program level in higher education. Our indicators can be used by evaluators of IT in higher education programs as a basis for quality representation.

The Need for Criteria of Worth at the Program Level

Indicators at the program level will help provide some structure and framework to the evaluation of the implementation of IT as it spreads more and more to various curricula across the academy. Further, focusing at the program level allows evaluators to look more at the integration of IT and the effect it has on the whole education of the student. There is also a scale of economy involved as evaluations move from the individual classroom use of IT to the program level.

Once again, it is important to say that all evaluation should be contextualized. As we move to the program level complexity increases and the need for recognizing context becomes even more critical. Simply put, documenting that these four indicators of quality are present is not enough to stamp the word "quality" on a program that uses technology to enhance its curricula in higher education. However, it does provide a basis for discussion among evaluators and program implementers as well as provide them with a starting point when designing an evaluation or planning a program.

Take for example the message of Peggy Lent, keynote speaker at the 1999 Faculty Summer Institute on Learning Technologies held at the University of Illinois in Urbana-Champaign. Dr. Lant talked about the need for the academy to more fully understand the impact that IT programs have on the higher education classroom. She felt that many programs use IT naively and that evaluations yield information about indicators that seem to be consistent across curricula were necessary to inform effective implementation (Bullock & Basi, 1999).

Four Proposed Quality Indicators

We propose four indicators of success as a place to begin future discussions of identifying quality when conducting IT evaluation in higher education programs. These four criteria, important across discipline, are access, student learning, administrative support, and cost effectiveness. These indicators have been pulled from conversations with over 100 instructors, surveys of over 7,000 students, and quasi-experiments we have conducted, our personal teaching experiences, and discussions with colleagues and experts in the field (Bullock & DeStefano, 1998; Ory, Bullock, Burnaska, 1997). While our emphasis is on the program level, some evaluation issues are still discussed at the classroom level. Consider that samples of classrooms may be used, or that individual classrooms can be evaluated to present information pertinent to program administrators. Because of this, we do include examples that involve specific classrooms.

Access

Access, as defined in this article, is separated into the three distinct categories of: access to hardware/software; access to course members and materials; and access for students with disabilities. A good place to begin considering the goodness, the quality, of IT in a higher education program is the quantity and quality of computers available for the students. For example, students can simply be asked whether or not they feel they have adequate access to both computers and the necessary software. Another example is that evaluator could collect information on whether there are enough computers in enough computer labs on the campus for all of the students in the IT program to have adequate access. Finally, information about the ability of students to dial in and connect to the campus infrastructure might also be collected.

Data could be collected on the number of labs that exist, their hours of operation and numbers of computers, and whether or not all of the latest versions of the software used in the IT enhanced classrooms are installed on the computers. Other data might include determining how often the software is accessed at each available site and if there are particular times or days of the week when access increases. The evaluator may find that coordination between the courses may be necessary to minimize overcrowding of computer labs. These results may suggest that assignment schedules be shared to allow instructors to coordinate times when their students will most likely use the computer labs. Another common decision based on this type of evaluation information is when instructors could make their computer-based assignments due. Say, for example, that the evaluation determined that several of the computers were being accessed very late at night in order to meet a midnight deadline. This resulted in large numbers of 18 year olds walking across campus to labs well after dark. Based on this type of information, instructors may choose to make their submission deadline late afternoon rather than encouraging procrastinating students to walk across campus at 10:00 p.m. to find a lab to complete homework.

One trend that is becoming more common is for campuses respond to access issues by mandating entering students to own personal computers. Here a measure of quality becomes whether or not the dormitories have the necessary infrastructure to allow these students access necessary software via their personal computers.

Another dimension to hardware/software access is for evaluators to investigate the type of support personnel are available for students with both hardware and software problems. One of the authors surveyed over 100 computer lab personnel and found that only about 50% felt they were competent in the most commonly used instructional technologies on campus (Ory, Bullock, Burnaska, 1996). They frequently reported not being able to answer students' questions because they had not specifically been trained in the common types of IT, such as computer conferencing packages.

And the nature of computing and networking technologies itself are changing quickly. Evaluations of these programs could consider the long-term commitment made by the administration to provide personnel capable of operating and maintaining these systems. This would include allocation of resources to provide on-going training that would allow personnel to remain up to date and operate efficiently (Downey, 2000).

Other types of training provided could also be considered within the evaluation. How are the students and teaching assistants trained to implement IT in the classrooms, for example? Is this training at the beginning of the course, on going, needs based? Is it mandated? These are some questions that might help guide the evaluator in collecting this type of information (Ory, Bullock, Burnaska, 1998). Instructional technologies can efficiently and effectively increase the communication within courses and across curricula (Hiltz, 1997). The degree to which they are successful at this, in a variety of courses, is an indicator of quality at the program level. Types of information that reflects communication could include monitoring interactions that occur within course conferences to look at frequency of use and proportion of students who use the technology, for example. Course members may communicate via chat spaces in Web pages, within conferencing packages such as PacerForum or FirstClass, or through electronic packages established through their University. Even when users report high levels of satisfaction with access to their classmates, further analysis could be conducted to determine the substantive quality of these interactions. Some indicators of quality would include conferencing which is: (1) facilitated by the instructor rather than left unmonitored; (2) constructed in a manner to encourage team work and active learning; and (3) used to decrease class time necessary to deal with routine class management such as announcements of test times and homework changes (Aryan, et al 1998). Frequencies of participation by variables such as gender and ethnicity might be considered to determine if the electronic arena is facilitating the participation of all students. The evaluator may even consider collecting statistics such as the numbers of interactions that occur and the proportion of students who read each generally posted response.

Other statistics might include the timeliness of response to instructional postings within conferencing by both other students and faculty. Counts can be made to determine the average time span from the point where a student posts her question to the point where a response is made, either by the instructor or by other students. Providing this type of data collected at program level helps inform the policies and course structures encouraged by administration.

Critical to an evaluation of learning technology use and implementation in higher education, is a consideration of the type of information that is provided to students. Evaluators should consider whether programs emphasize that instructors should understand the different nature of on-line courses. This means that instructors are encouraged to tailor the material that they present to the new medium. Instructional technology provides a way for information previously unattainable to be presented. Students, for example, in a bioveterinary program can now have access to virtual cow's hearts. The question of quality is whether or not the program encourages instructors to use this new medium to effectively present course material otherwise not possible.

Another indication that implementation and use will be successful is the consideration given to download times. If the typical course materials are too tedious to download, students may not use them. Evaluators could use electronic tools to browse through selected course Web pages and download images, making note of the time it took from home computers. Simple counts such as how many images took over one minute to download via modern could be made. Again, the emphasis made at the program level will help guide the individual courses use of IT.

Consideration should also be given to the quality of the materials contained on the Web pages. Specific to technology, this includes the dynamic or static nature of course postings by the instructor. Evaluators could determine if, and to what extent, technology has been used creatively to allow course material to remain fluid rather than stagnate. An example of this could be posting questions asked during class allowing students access to more relevant course material.

Issues of Access for Student with Disabilities

Learning technologies hold the promise of providing quality instruction with increased accessibility to students typically underserved (Russell, 1999). Evaluations should collect information about the attention given to access issues for students with disabilities in technology-enriched courses. For example, while Web pages cannot be read in Braille and deaf students cannot access audio streaming, reasonable accommodations and sensitivities are possible. Variables to consider in the evaluation might be whether full text versions have been included for course Web pages and if image tags have adequate descriptions rather than vague terms used whenever possible? Other evaluation considerations might be as simple as font size and color contrasts designed with learning disabilities or visual impairments in mind. Clearly, input from persons with disabilities would help inform this aspect of the evaluation.

Considering the Impact on Student Learning

Learning technologies bring a world of knowledge to students and for some (students transitioning from the high school environment, for example) an overload that might actually serve to deter rather than increase their learning (Burnaska, 1998). Collecting data about the various strategies used to monitor students' learning experience in the technology-enriched classrooms can provide useful evaluation information.

Student Motivation

Student motivation can broadly be categorized into intrinsic versus extrinsic. Students intrinsically motivated more easily learn on their own because their stick and/or carrot are internal. They more often study because they want to succeed in and of itself (Wager, 1992). Consequentially, they succeed not because it is expected of them, but because they expect it of themselves. Extrinsically motivated students' stick and/or carrot is more typically carried and wielded by an outside force. They study because they want to avoid failure, and often need external motivating factors to make it clear that it is time to study (Petri, 1986). Collecting data on the types of instructional technologies and compatibility of learning styles could yield valuable data. For example, a common type of learning technology used in higher education is on-line homework where the student receives immediate feedback after submitting a response. One feature of this type of use is that the instructor can pre-determine how many tries each student will have to submit the homework. Some faculty give their student's unlimited attempts with no punitive action to their grades, some limit the times, and others allow only one submission. An evaluation might investigate the philosophy of the various instructors and the impact that these variations have on their student's summative grade.

Another well accepted premise in higher education impacting motivation is that involving students more actively in their learning increases their instructional benefit (Stage, et. al. 1998) Evaluations of IT might collect data on the passive or active nature of the various course delivery methods within the program. For example, a common practice may be to have PowerPoint lectures distributed to students prior to lectures. Data could be collected to determine whether this encourages students to stay with the instructor, or rather gives them passive permission to tune him out. Small-scale experiences, such as leaving certain slides blank on the handout might be conducted to see if this has any impact on student's attention during these lectures. Other examples include assessing the use of techniques such as virtual demonstrations that actively engage the students. These might be virtual labs or interactive homework assignments. The critical evaluation issues involve determining the level of student engagement in the activity and the potential of the technology to further engage students in their learning.

Certain quasi-experimental techniques lend themselves to collection of the types of data mentioned in this section. Quasi-experiments that look at achievement score differences between: (1) Technology-enriched and non-technology sections of the same course; (2) semesters taught with technology and semesters taught without ALN of the same course (historical comparisons); and (3) like courses where one of the professors used technology and the other taught without technology have been found to provide valuable information useful at the program decision making level (Ory, Bullock, Hanson, 1998). Along with comparisons of parallel items on tests in these courses, students can also be asked comparative questions that self-report their learning.

Administrative Support

The shift from administrative support at the instructor level to the institutional level may be indeed be a necessary component of successful implementation of learning technologies at the program level (Ernst, Katz, and Sack, 1994). Departments may not have the expertise or the money to support their own faculty in developing and implementing technology. A centralized support, which provides both technical and pedagogical expertise to faculty across curricula, may produce the most promise for campus wide implementation. However, there are departmental level quality indicators of support for faculty implementing learning technologies in their classrooms. Possible scenarios for consideration would be evaluating the priority that administration gives in the use of technology-enhanced courses. For example, how does the extra time necessary for quality implementation impact the instructors' promotion and tenure review and what types of financial allocations are given for technical support and training?

In the case where a campus wide support structure has been established, key issues of quality would include the attention given to the pedagogy of implementation along with the actual technologies promoted and supported. An optimal scenario would include instructional development personnel, highly cognizant of learning technologies, in place to provide advice, training, and support to faculty across curriculum should either be employed or regularly consulted by a central unit of this nature. These work in tandem with personnel trained specifically in technical issues.

Quality Indicators Associated with Cost Effectiveness

Using learning technologies can reduce specific costs of classroom instruction. This seems to be consistent across curricula (Arvan, et al., 1998) Departments can, for example, eliminate the costs of either graduate or undergraduate graders by putting assessments on-line. Enrollment can be increased in some courses, and it could be argued that homework discussion sections are no longer necessary because feedback is given on-line rather than face-to-face. One indicator of quality is to determine the nonmonetary cost of eliminating students' ability to discuss their homework answers with the grader with cost savings.

Indicators specific to the monetary savings would include the degree with which one faculty can share or transfer their knowledge to another; and the degree to which technology allows instructors to focus on areas that were previously cost prohibitive.

Issues of Transferability

There is considerable start up time associated with implementing learning technologies in a quality manner into any higher education classroom (Campbell, Lenear, & LaFleur, 1999). This can involve developing Web pages, creating on-line homework, Java scripting for white boards, and establishing on-line conferences. Even if there is competent technical help readily available, each of these steps require faculty to integrate them carefully into the curriculum. It is not uncommon that faculty must take responsibility for both the technical and pedagogical aspects of technology-enriched courses. Consistently, faculty report that implementing technology into their course comes at a considerable cost in terms of time and effort. But, a cost savings can occur if more and more courses are put on line using the seasoned faculty model (Arvan, et al., 1998). In this model, Adjunct Faculty or Lecturers could be used in lieu of tenured faculty. A seasoned tenure track faculty who has developed a model for learning technology use could mentor these instructors. The cost savings are determined by amortizing the costs of teaching these courses over time. Consideration is given that the course enrollments can typically be increased and the instructor's salary is usually less than tenure track faculty. The indicator of quality is whether or not this model, while probably saving costs, maintains the quality of instruction. Inherent in collecting this type of information would be to determine the administration responsibility to these non-tenured faculty and whether or not the mentoring model is in place and functioning well.

Indicators for transferability could include comparisons of student ratings and instructor self-reported satisfaction, along with measures of student achievement, between established courses and those with the mentored instructors.

One quality indicator inherent within cost savings is whether students can now have instructional experiences that were previously cost prohibitive. To illustrate this concept, we present the story of one veterinary medicine faculty member we encountered in one of our evaluations. In the past it was prohibitively expensive for students to operate on cow's hearts. Students could not have the unique experience of conducting the surgery themselves, because the school could not (animal rights aside) afford a cow's heart for each student. However, by implementing a virtual lab, each student in her class now does have the individual experience of operating (albeit virtually) on a cow's heart. The indicator of quality here is two-fold. First, did this provide students with an experience that they would not have had otherwise. Second, that the student's experience closely modeled that of the non-technology-enriched course, while maintaining this significant cost savings.

Summary

Technology has found a mainstay in higher education programs. Instructional technologies are implemented and used across curricula in many universities across the country. With this comes increased use and a clear need for evaluations of learning technologies in the academic programs where they our housed. Further, the presentation of quality indicators can be a useful guide for program administrators. We have proposed four indicators (certainly not meant to stand-alone) that may be useful in guiding future evaluations of IT programs in higher education.

Bibliography

Arvan, L., Ory, J., Bullock, C., Burnaska, K., and Hanson, M (1998). The SCALE efficiency projects.. The Journal of Asynchronous Learning Networks 2, 2.

Bullock, C., & Basi, M. (1999). An Evaluation of the 1999 Interinstitutional Faculty Summer Institute. Unpublished Evaluation Report, University of Illinois at Urbana-Champaign.

Brownlee, Jean (1996). Voices from networked classrooms. Educational Leadership. 53, 3, p34-36.

Bullock, C., & DeStefano, L. (1998). A Study of the Utility of Results from the 1992 Trial State Assessment (TSA) in Reading for State-Level Administrators of Assessment. Educational Evaluation & Policy Analysis, 20, 1, p47-51.

Bullock, C., & DeStefano, L. (1998). Considerations inherent in teaching with technology. Paper presented at the 1998 American Evaluation Association Meeting, San Diego, CA.

Burnaska, K. K. (1998) . The effects of using instructional technology on student motivation, achievement and resource management skills in higher education (Self-Paced c). Unpublished Dissertation, University of Illinois at Urbana-Champaign.

Campbell, K., Lenear, P., & LaFleur, J. (1999). An evaluation of the HRS on-line program. Unpublished Evaluation Report. University of Illinois at Urbana- Champaign.

Clark, D. (1992) Getting results with distance education. The Journal of Distance Education, 12,1, 38-51.

Clarke, R.E., (1992) Media use in education. In M.C. Alkin (Ed.) Encyclopedia of Educational Research in Education. NY: Macmillan.

Cooley, W.W. and Lohnes, P.R. (1976) Evaluation research in education. NY: Irvington.

Cousins, J.B. and Earl, L. (1992) The case for participatory evaluation. Educational Evaluation and Policy Analysis, 4, 397-418.

Crump, R.E. (1928) Correspondence and class extension work in Oklahoma. Doctoral dissertation, Teachers College, Columbia University.

Draper, S. Brown, M. Henderson, F., and McAteer, E. (1996). Integrative evaluation: an emerging role for classroom studies of CAL. Computers and Education, 26, 1-3, pp. 17.

Ernst, D., Katz, L., & Sack, D (1994). Organizational and Technological Strategies for Higher Education in the Information Age. CAUSE Professional Paper Series, #13, 32.

Ester, D.P. (1995) CAI, lecture, and learning style: The differential effects of instructional method. Journal of Research on Computing in Education, 27, 4, 129-139.

Gold, L. and Maitland, C. (1999) What's the difference? Paper prepared by the Institute for Higher Education Policy.

Goldberg, M.W. (1997) CALOS: First results from an experiment in computer-aided learning. Proceedings of the ALM's 28th SIGCSE Technical Symposium on Computer Science Education.

Greene, J. (in press) Participatory evaluation. In L. Mabry (Ed.) Evaluation and the Post Modern Dilemma, Greenwich, Connecticut: JAI.

Hiltz, R. S. (1997) Impacts of college-level courses via asynchronous learning networks: Some Preliminary results. Journal of Asynchronous Learning Networks, 1, 2.

Hiltz, S., (1998, November) . Collaborative Learning in Asynchronous Learning Networks: Building Learning Communities. Paper presented at the WebNet 98 World Conference of the WWW, Internet, and Intranet, Orlando, FL.

Jones, A., Scanlon, E., Tosunoglu, C., Ross, S., Butcher, P., Murphy, P., and Greenberg, J. (1996) Evaluating CAL at the Open University: 15 years on. Computers in Education, 26, 19, 5-15.

Kearsley, G. (1990) Designing education software for international use. Journal of Research on Computing in Education, 23, 2, 242-250.

Khalili, A. and Shashaani, L. (1994) The effectiveness of computer applications: A meta-analysis. Journal of Research on Computing in Higher Education, 27, 1, 48-61.

Kulik, C.C. and Kulik, J.A. (1991) Effectiveness of computer-based instruction: An updated analysis. Computers and Human Behavior, 7, 1, 74-75.

Lockee, B., Burton, J., Cross, L. (2000). No Comparison: Distance Education Finds a New Use for "No Significant Difference." Unpublished manuscript.

Martin, E.D. and Rainey, L. (1993) Student achievement and attitude in a satellite-delivered high school science course. The American Journal of Distance Learning, 17, 1, 22-28.

McKenna, S. (1995) Evaluating IMM: Issues for researchers, Open Learning Institute Occasional Paper #17, Charles Stuart University.

Oliver, M. (1997) A framework for evaluating the use of Learning Technology. BP ELT report no. 1, University of North London.

Ory, J., Bullock, C., & Burnaska, K. (1997) Gender similarity in the use of and attitudes about ALN in a university setting. The Journal of Asynchronous Learning Networks 1.

Parlett, M. and Hamilton, O. (1976) Evaluation as illumination. In G.V. Glass (ED.) Evaluation Studies Review Annual, Beverly Hills, CA: Sage.

Patton, M.Q. (1997) Utilization-focused evaluation. Thousand Oaks: CA: Sage.

Petri, H. (1986). Motivation theory and research. Wadsworth Pub: Belmont, CA.

Popham, W.J. (1961) Tape recorded lectures in the college classroom. AV Communications Review, no. 9.

Popham, W.J. (1972) An evaluation guidebook. Los Angeles, CA: Instructional Objectives Exchange.

Reeves, T. C. (1994). Evaluating what really matters in computer-based education. In M. Wild, & D. Kirkpatrick, (Eds.), Computer education: New Perspectives (pp. 219-246). Perth, Australia: MASTEC.

Russell, T. (1999) The no significant difference phenomenon. Raleigh, North Carolina: North Carolina State University.

Schramm, W. (1962) What we know about learning from instructional television: The next ten years. Stanford Institute for Communication Research.

Scriven, M. (1972) Pros and cons about goal-free evaluation. Evaluation Comment, 3, 1-8.

Sell, D. (1995) Utilizing a Rapid Prototyping Approach in the Building of a Hypermedia-Based Reference Station. Unpublished Masters Thesis. Kent State University.

Stake, R.E. (1978) Program evaluation, particularly responsive. Western Michigan University.

Suchy, R.R. and Baumann, P.C. (1960) The Milwaukee experiment in instructional television: Evaluation report, Milwaukee Public Schools.

Tyler, R. W. (1942) General statement on evaluation. Journal of Educational Research, 35, 492-501.

Wager, Walter (1992). Educational Technology: A Broader Vision. Education & Urban Society, 24, 4, 454-65.

Dr. Bullock is the Head of the Division of Measurement & Evaluation within the Office of Instructional Resources. Dr. DeStefano, Professor, is Associate Dean for Research, College of Education.