Learning from past and present: electronic response systems in college lecture halls.

This article reviews literature from the past 33 years particular to the use of electronic response systems in college lecture halls. Electronic response systems, primarily used in science courses, have allowed students to provide immediate feedback to multiple-choice questions, and inform the instructor of student understanding. Research from the 1960s and 1970s indicates there is no significant correlation between student academic achievement and a stimulus-response method of using such systems. Recent studies have indicated there is significant student increase of conceptual gains in physics when electronic response systems are used to facilitate feedback in a constructivist-oriented classroom. Students have always favored the use of electronic response systems and attribute such factors as attentiveness and personal understanding to using electronic response systems. Ultimately, this review of literature points to the pedagogical practices of the instructor, not the incorporation of the technology as bein g key to student comprehension. Electronic response systems are viewed as a tool that holds a promise of facilitating earnest discussion. Recommendations are made that professional development focus on pedagogical practice for instructors considering the use of an electronic response system.

**********

At a recent visit to the American Educational Research Association (AERA) conference, one author attended a session focusing on innovative uses of technology in education. Among other propositions a new use for computing technology in large lecture halls was mentioned. Described as an "electronic response system," the technology was portrayed as allowing individual students in large lecture halls the opportunity to provide immediate electronic response to content questions and to inform the instructor of their level of understanding on specific concepts. Contributions from students could either be viewed by only the instructor or projected in graphical format for all to see. At this AERA session there was no time to probe deeper into the use and benefits of such a system. Adherents of educational technology realize that such electronic response systems have been popular on some campuses at least since the mid-1990s, particularly in the commercial forms of Class Talk and Personal Response System (PRS). Use of such systems in the past several years has generated research related to effects upon student achievement, student attitude, and motivation. Interestingly, this research does not pay much credence to literature more than a few years old. Yet, regarding electronic response systems, a small body of significant related literature exists from a period dating more than 30 years ago. Indeed the use of electronic response systems in large lecture courses, particularly science classes, can easily be dated to the 1960s. The intent of this paper is to first examine and synthesize pertinent literature on the use of electronic response systems from this early generation dating from 1968 to 1975, when a flurry of activity was chronicled in dissertations and scholarly journals. Following this, an examination of more recent studies will explore how the systems and the modes of their deployment have changed over time. Finally, conclusions are drawn regarding what lessons related to technology and pedagogy may be gleaned for current use of electronic response systems.

A clarification of what is meant by an electronic response system is fitting before examining the use of such systems. Although architectural variations exist among different systems, by and large the systems are remarkably similar. Spanning nearly the past four decades, electronic response systems have allowed students, typically in a large lecture hall, to immediately respond to an instructor's multiple-choice questions through the use of some distinct electronic sending unit. A multimedia presentation format of these multiple-choice questions has long been the standard; merely the method of production has changed over the years. Although the projection technology has changed from 35 mm slides to computer projection systems, from a student point of view, the result is essentially the same. From a seat in the lecture hall a student sees that a question with four or five possible answers has been projected. In some cases, an additional illustration is projected to aid in the understanding of the question. Wh ile most would make an argument for the ease of use of modem equipment, the resulting presentation of a multiple-choice question remains invariant.

Today, manufacturers are able to provide wireless and portable versions of their systems. Also standard among current electronic response systems is the ability to graph and project histograms representing whole class responses to any question. In addition administrative capacity has increased, allowing nearly effortless record keeping ability. While it is not difficult to romanticize the marvels of modem technology and overstate the significance of technological advances, it is perhaps more astonishing to consider how fundamentally comparable electronic response systems have remained. Hard-wired systems of the 1960s and 1970s mounted knobs, series of buttons, or even telephone number pads at student seats. Instructor stations of this era commonly provided a series of voltmeter type gauges that indicated the percent of students responding to each choice on a particular question. Yet the aim was the same then as it is now--to obtain instant feedback from students in large classes. It is clear that the technol ogy has changed dramatically. Less obvious is the fact that most often the underlying pedagogy has not changed at all.

Use of Electronic Response Systems of the 1960s and 1970s

As is frequently the case, new technology does not necessarily crop up but rather it emerges from existing technology. This was the case of electronic response systems in the 1960s. Such systems grew out of the military's use of filmed instruction material in the 1950s (Bessler, 1969; Boardman, 1968; Froehlich, 1963). Technically speaking, the systems used in colleges were fairly sophisticated even by today's standards. Instructors could choose systems that were labeled as either tagged or anonymous (Littauer, 1972). Tagged systems recorded answers from every seat in the classroom. An anonymous system merely provided a count of the total number of responses to each answer choice. Analyses and printouts on paper tape of student responses were even possible through the use of computers and teletypewriters (Chu, 1972).

During a time when stimulus-response learning theory and behavioral objectives were dominant (Mager, 1962), educators of this period were largely concerned with electronic response systems being able to provide quick notification of correct results to both student and instructor. Though not a component of all early electronic response systems, many did integrate a mechanism that provided students individual feedback regarding correctness of their answers. The Litton Student Response System included a responder dial that could be turned to five possible answers (A, B, C, D, and E). After turning the dial to a letter, a student would then press a response button. When a student chose a correct response, the button would vibrate. The Instructoscope lit green and red lights at individual stations to inform each student when they had chosen a correct response (Boardman, 1968).

Consistent among these electronic response systems was the provision for providing the instructor knowledge of student responses. A series of gauges typically reflected the proportion of students responding to each answer choice. Instructors often used the systems to aid in the flow of instruction. In describing the use of an electronic response system for use with computer programming students, Garg (1975) explained how he used the system to allow students to overtly inform the instructor of the appropriateness of the pace of instruction. Specifically, students were able to continually input selections such as "go faster," or "go slower." If the metaphor school as a factory can be applied to a college lecture hall, then Brown (1972) illustrated how an electronic response system allowed students to control the conveyor belt of knowledge: The instructor "was able to pace himself by moving faster when the student responses were quick and sure. He stopped to amplify, clarify, or redefine, and explain when the re sponses indicated the majority of individuals did not understand." Similarly, Casanova (1971) stated that if "class response was less than 50% correct, the same topic was reviewed again immediately."

In this vein of operant conditioning and programmed instruction, early instructional planners using electronic response systems considered two sources of stimuli. Most commonly, the multiple-choice questions were catalysts that triggered responses from students; by providing immediate feedback to students, either from individual electronic feedback integrated into the system or through the instructor, student responses were confirmed each step of the way. Correspondingly, the students might themselves be viewed as stimulus with the instructor responding by adjusting instructional pace, reiterating, or providing further review. In this latter case, students would be instructed to continually respond according to a given code during a lesson. The codes allowed the instructor to know to what degree students understood the material.

A characteristic of this time period was an emphasis on privacy of student input. Garg (1975) and Chu (1972) pointed out the importance of the systems to allow students to confidentially enter responses with recessed gauge windows or other visual barriers. This privacy issue exceeded the use of electronic response systems for test taking purposes but was a concern for everyday use.

A review of the period's literature reveals that the use of electronic response systems can be typified as tally counting of student responses largely for the instructor's edification. A few departures from this portrayal involve the use of the electronic response system as a means of promoting student discussion. Garg (1975) remarked that a class may wish to discuss the significance of votes in each category or that students who voted for minority responses should comment to support their choices. In a report authorized by the National Science Foundation, Chu (1972) reported on a project at Skidmore College aimed to use an electronic response system in as many different ways and academic disciplines as possible. The project attempted to foster the more creative aspects of teaching and learning through the use of an electronic response system. However, the report only mentioned one use of the system by a grammar instructor to overtly promote student discussion wherein the discussion itself constituted instru ction. An interesting citation on the use of an electronic response system is that of Littauer (1972) who described his own use of an electronic response system in a physics course. Littauer attempted to lessen the information load on students by distributing sheets before each lecture listing the question and answer choices. In this way he liberated students from note taking. Littauer noticed that this,

provoked a spontaneous debating session in class just before each lecture--an unforeseen development which I welcomed..[during lecture] if the answers to a certain question were coming in wrong, I could quickly abort the response period and ask the students to think for a moment. Again there would be a murmured debate, and often the correct answers would start coming in.

This case presents an intriguing situation where student discussion occurs prior to actual instruction time or as a nearly covert murmured exchange among students. It might be noted that the "liberation" supplied by Littauer had nothing to do with the degree of sophistication of the technology.

Findings and Recommendations from the '1960s and 1970s

A leading concern for integrating technology into education often is the effect upon student academic achievement. Collectively, research from the 1960s and 1970s related to electronic response systems does not support the claim that student achievement will increase as a result of using such systems. Bessler's dissertation study (1969) examined the examination scores of 664 nonmajor biology students randomly assigned to control (no electronic response system) and treatment (use of an electronic response system) sections. The data showed that no significant difference in mean achievement existed between students in control sections and students in treatment sections. Similarly, Brown (1972) tested the prediction that college students assigned to a mathematics lecture course (n = 34), augmented by the use of an electronic response system, would achieve significantly higher than those students in a regular lecture-recitation course (n = 39). Students in the experimental group received daily review and drill of the previous day's work as well as scheduled question and answer sessions all by way of the electronic response system. Brown found no statistically significant difference between the two groups. He concluded that "students learn equally as well by using the an electronic response system as by using more conventional methods." Other experiments among college chemistry, physical science, anthropology, and economics students failed to yield significant academic achievement results in favor of an electronic response system (Bapst, 1971; Casanova, 1971). Interestingly, Bessler and Nisbet (1971) considered the equivalent academic achievement among experimental and control groups to be a positive affirmation for the use of electronic response systems. Their positive interpretation viewed the electronic response system as a success because it was shown to be as effective as the more conventional lecture techniques.

Since students taught by relatively inexperienced instructors achieved as well with the response system as students taught by relatively experienced instructors, the possibility of using instructors with relatively less experience should be explored.

Bapst (1971) also investigated whether or not students would rate their instructor higher on teaching behavior criteria when their instructor was receiving constant student feedback related to comprehension and input about pace of instruction. In this study, students in both control and experimental classes used an electronic response system in the same manner. However, in the control section the instructor was unable to view any of the student input. Like the others, this study found no significant differences.

Despite a lack of evidence to support increased academic achievement, as measured by standard exams, students provided overwhelming endorsement for the electronic response systems (Bapst, 1971; Brown, 1972; Casanova, 1971; Garg, 1975; Littauer, 1972). In these studies, positive attitude toward the class, feeling of the usefulness of the system, acceptance of the system, and feeling of increased understanding were all highly supported by student survey data. In particular, such support was found in a college physics course where attendance in sections using an electronic response system was maintained at an incredible 95% throughout the semester (Littauer, 1972).

Use of Modern Electronic Response Systems

Current cases of electronic response system application can be found mirroring the stimulus-response style so characteristic of the 1960s and 1970s. Given the ease of use of current systems together with student familiarity with technology, it is common to find students still engaged in voting for an answer choice only as a means to inform the instructor of student comprehension level. Indeed, some instructors elect to use electronic response systems for attendance keeping alone (Shapiro, 1997). However, any such insular use of electronic response systems is no longer emphasized in scholarly journals. There has been a shift, portrayed in the few current writings related to electronic response systems, away from the technology being a catalyst of student achievement and attitude, toward an emphasis on effective pedagogical constructs that can be supported by electronic response systems.

While electronic response systems are still used for individual replies and for test-taking purposes, framing and responding to ideas has become a much more public event. Recent literature portrays student-to-student and student-to-instructor discussion as taking on key significance (Abrahamson, 1998; Abrahamson, 1999; Dufresne, Gerace, Leonard, Mestre, & Wenk, 1996; Poulis, Massen, Robens, & Gilbert, 1997; Shapiro, 1997). Affected by constructivist cognitive science and other epistemically compatible perspectives, investigators have highlighted the importance of collaborative discourse that allows students to negotiate meaning in science and mathematics classes. Defresne et al., described a particular system, Classtalk, designed specifically for collaborative use. After collaborative discussion, groups of students using Classtalk are able to either provide a group response or group with dissent response. The suppliers of Classtalk are so convinced of the merit of student collaboration they will not even pro vide price estimates for institutions planning on using the system for individual responses (Shapiro, 1997). Yet, there is nothing magical about modem systems that allows student collaborative work to occur. A system in the Netherlands, in place since 1966, has recently been used by physical science classes in a manner that includes ample time for students to discuss each problem (Poulis et al., 1997).

Typically, the collaborative use of Classtalk begins with a displayed question along with possible answers. However, before responding, students are prompted to discuss with other students and arrive at a consensus answer. This collaborative discourse is considered to be instruction. After answers are chosen, a histogram of responses is displayed for students and instructor. This public display of responses constitutes another layer of collaborative discourse and is a defined departure from earlier usage of electronic response systems. The limits of earlier technology only provided private information for the instructor--a display of the percentage of students responding to each choice. Such information is now considered collective property. Within the mode of interactive engagement, this display of responses becomes the catalyst for further discussion. Students are encouraged to defend their answers and to attentively listen to other students articulate their thinking (Abrahamson, 1999; Dufresne et al., 199 6).

It appears there has also been a slight shift in the types of questions posed to students through electronic response systems, particularly in physics courses. Conceptual questions that are understandable to the layperson yet focus on deep understanding of fundamental science ideas have been found to promote the most substantive discussions (Abrahamson, 1998, 1999; Shapiro, 1997). For example, deep conceptual questions similar to those in the Mechanics Baseline Test or the Force Concept Inventory (FCI) (Hestenes & Wells, 1992; Hestenes, Wells, & Swackhamer, 1992) prompt physics students, using an electronic response system, to grapple with misconceptions during discussion (Abrahamson, 1999; Dufresne et al., 1996; Falconer & Joshua, 2001). As a result, the proper crafting of questions has become crucial. Highly regarded are questions that include common sense wrong answers, thus allowing the formative nature of misconceptions to be revealed (Abrahamson 1998; Dufresne et al., 1996; Poulis et al., 1997; Shapiro , 1997).

Findings from Recent Studies

Prevalent among recent examinations of the use of electronic response systems, is evidence that students enjoy using the systems and consider the systems useful for their own understanding of subject matter (Abrahamson, 1999; Cue, 1998; Dufresne et al., 1996; Shapiro, 1997). Cue's survey of physics classes at Hong Kong University of Science and Technology evidenced students in a class using an electronic response system on a regular basis to be considerably more positive about the use of the system than those students using the system about once per week. The majority of the Hong Kong University students in the high-use class agreed strongly with statements such as "helps me to learn the subject matter of this course in greater depth" and "knowing how my classmates respond to questions in class increases my interest in the subject matter." Students in this high-use section also stated that they more regularly attended class. Student responses from the low-use section yielded nearly normal distributions to al l survey statements, using a scale of 1 to 5.

Beyond discovering that students both enjoy and value the use of an electronic response system, the issue of academic achievement remains open. On this point, recent evidence is limited but promising. Poulis et al., (1997) examined a six-year period of physics courses and provided compelling evidence that in those years when an electronic response system (combined with student discussion) is employed, examination pass rates in physics increases. For example, during a two-year period of typical lecture (n = 324), student examination pass rate for optics and Maxwell Theory averaged 57%, however, during another two-year period where the electronic response system was employed (n = 345), student pass rate for this same topics using the same tests was 70%. Other researchers point to the findings of Mazur and Hake (Cue, 1998; Abrahamson, 1998) in physics education to inferentially support the argument that electronic response systems can promote student achievement. Hake (1998) surveyed 62 introductory physics cou rses enrolling 6,542 students. After categorizing the courses into traditional and interactive-engagement (IE), Hake compared mechanics test data to discover that student achievement in IE courses exceeded that in traditional courses by nearly two-standard-deviations. Abrahamson (1999) acknowledged that certainly not all high achieving courses in the Hake study incorporated electronic response systems (perhaps only a scant few), however, among the high scoring courses is Mazur's Harvard physics course. Mazur used an electronic response system to facilitate what he termed "Peer Instruction." Emphasizing a pedagogy that specifically promoted student interaction, Mazur was able to demonstrate a doubling of gains on the Force Concepts Inventory in his lecture hall after a change to a student-centered pedagogy (from 8% to 17.3%).

Attempting to determine if a level of IE could be quantified through pointed classroom observation, Falconer, Joshua, Wyckoff, and Sawada (2001) used the Reformed Teaching Observation Protocol (Pibum & Sawada, 2000) to observe the teaching styles of 12 physics instructors. Falconer et al., correlated normalized gains of conceptual understanding of mechanics, as measured by a 14 item multiple-choice test, to the instructor's level of constructivist pedagogy, as measured by the Reformed Teaching Observation Protocol (RTOP). A correlation of .98 corroborated Hake's finding. Among the high performing courses, as measured both by conceptual gains and RTOP score, was Wyckoffs physical science class at Arizona State University. Since 1996, Wyckoff has used two different electronic response systems for a physical science course aimed at preservice elementary teachers. As with Mazur, Wyckoff has shifted the focus of communication away from herself as an instructor imparting knowledge to students and toward students a ctively negotiating and defending conceptual understanding. Key to Wyckoffs transformation of pedagogy was her participation in workshops developed by the Arizona Collaborative for Excellence in the Preparation of Teachers (ACEPT). The emphasis in these workshops was not on technology alone, but rather focused on helping participants experience and understand how critically important the use of technology was to structure and support the formation of discourse communities to facilitate the learning of science and mathematics.

SUMMARY

Electronic response systems cannot be considered emerging technology. The essential configuration allowing instructors to pose questions and students to provide informative electronic feedback has been in place since the 1960s in college lecture halls. A marked advancement among modem systems was the ability to display graphic representations of student responses. This innovation has been coupled with a general shift in how electronic response systems are used in college courses. Early use of electronic response systems emphasized an operant conditioning focus. During the 1960s and 1970s the main purpose was either for students to receive immediate yea or nay feedback regarding their answer choice or for the instructor to control the pace of lecture according to student responses. Recommended usage of electronic response systems now emphasizes student-to-instructor and student-to-student interaction. Student discussion that advances understanding of concepts and unveils misconceptions now takes on paramount importance.

In studies spanning four decades, students expressed persistently positive support for the use of electronic response systems. That students believed the systems helped them to better understand, was independent of instructor pedagogy and actual achievement data. Polls from the 1960s through the late 1990s found that the use of electronic response systems made students more likely to attend class, pressed them to think more, prompted them to listen more intently, and made them feel instructors knew more about them as students.

The assumption that electronic response systems would promote higher student achievement was not supported by research of the 1960s and 1970s. These earlier implementations took a stimulus-response pedagogic style for granted. In the 1990s investigations reporting student academic achievement were found only within the discipline of physics. The use of electronic response systems was not a distinct characteristic among high achieving physics courses, however, electronic response systems were viewed as one mechanism to elevate student interaction in large lecture halls. Among physics studies, improved student achievement was detected when the pedagogy was distinguished as constructivist in nature, thus promoting interactive engagement among students.

In brief, the following four important findings are established from past and present literature:

1. Students will favor the use of electronic response systems no matter the nature of the underlying pedagogy.

2. Academic achievement does not correlate to behaviorist use of electronic response systems, as highlighted by investigations of the 1960s and 1970s.

3. Despite "high-tech" improvements, the use of electronic response systems within a behaviorist pedagogy has not produced gains in achievement.

4. Interactive engagement has been shown to correlate to student conceptual gains in physics. Interactive engagement can be well facilitated in large lecture halls through the use of electronic response systems.

Conclusions

The only positive effects upon student academic achievement, related to incorporation of electronic response systems into instruction, occurred when students communicated actively to help one another understand. Apparently, it is more beneficial for a student, who has just arrived at a new conceptual understanding, to explain to peers how he/she struggled and arrived at his/her new rationale than it is for an instructor to simply explain the abstraction. It appears, therefore, that it is instructional method itself that is important when considering implementing an electronic response system.

Discussion

In 1912, a German chemist, Fritz Klatte, was working on acetylene, attempting to find a material which could dope airplane wings and make them resistant to the climate (Burke, 1978). In so doing, Klatte developed a milky mixture that solidified when allowed to stand in sunlight. Klatte made a note of the ingredients and filed a patent that would lapse in 1925. At the time, the mixture was considered not useful. Klatte had thrown away vinyl chloride that would later be developed into polyvinyl chloride (PVC) - the forerunner of plastics. Metaphorically, perhaps Littauer noted a similar discovery in 1972 that he did not (and perhaps could not) fully appreciate. Littauer's three-page article in the October, 1972, issue of Educational Technology is largely a how-to article regarding the implications of establishing a homemade electronic response system in a physics lecture hall. Yet, as mentioned previously in this article, Littauer devoted one paragraph to discussing an effect of student interaction he stumbled upon. Students who were provided with paper copies of questions and answer choices prior to the lecture tended to debate before class. Later during the lecture when too many students were responding incorrectly, Littauer would allow students to think for a moment and to conduct further hushed discussion until enough correct responses were received on his instructor panel. Littauer noted the seeming advantage of this type of interaction as the questions became "public property and stimulated interaction between students." Certainly it is not proposed here that educators at the time of Littauer' s article were not aware of the possibilities of student interaction. For example, Dewey had long espoused the tenets of learning from experience and social interaction (1938). But the ideas of social constructivism were likely not clear or even acceptable to most science and mathematics lecturers of the 1960s and 1970s. Indeed, convincing college instructors of the merits of an interactive setting may be no easier tod ay than 30 years ago! Learning that occurs in a communicative setting that supports students as they bring forth and vocalize their own conceptions, is not well aligned with instructional objectives when the primary use of an electronic response system is to take attendance or to present material in a programmed instruction way.

As a retrospect on past studies related to electronic response systems, it is perhaps fitting to include a buyer beware statement: An electronic response system does not come prepackaged with interactive engagement. All such systems work perfectly well, and perhaps even smoother, with pedagogy based on operant conditioning. Having students provide feedback and even interact with technology can be done independent of a constructivist pedagogy. A constructivist approach involves active articulation and critique of students' own ideas. The reports of the 1960s and 1970s reflected a traditional transmission style of instruction. Students then were expected to learn through passive listening and indicate their level of comprehension by way of electronic input. Proponents of that period might have argued that this type of activity was in accord with pedagogy pervasive of the times. Reflection upon research yielded the realization that it is pedagogical style that must alter for an electronic response system to be successful. A shift from behaviorist toward constructivist teaching entails more than just using technologically mediated responses--it necessitates the open flow of communicative discourse revealing and reconstruing student conceptions whatever their maturity. Littauer nearly unearthed this scheme of teaching physics 30 years ago, but like Klatte and his discovery of useless vinyl chloride, perhaps Littauer's discovery could only be received as an anomaly given the dominant pedagogy of his time.

One cannot assume that the existence of this type of technology will actually facilitate constructivist teaching, let alone academic achievement among students. Nevertheless, to provide the best opportunity for success, this review of the literature suggests that institutions invest as much if not more in the pedagogical development of faculty as they do in the technology. Electronic response systems show promise when used as tools to facilitate constructivist-oriented discussions in lecture halls. However, it is imperative for the instructor to understand the tenets of constructivism and to have struggled with his/her own epistemological beliefs about instruction as he/she experiments with such a method of teaching. Just as college students benefit from articulating their preconceptions of scientific and mathematical abstractions, faculty are enriched by confronting their own rooted beliefs about instruction.

References

Abrahamson, A.L. (1999). Teaching with classroom communication system: What it involves and why it works. Paper presented at International Workshop, New Trends in Physics Teaching, Puebla, Mexico. Retrieved from the World Wide Web on May 21, 2002 from: http://www.bedu.com/Publications/PueblaFinal2.html

Abrahamson, A.L. (1998). An overview of teaching and learning research with classroom communication systems. Paper presented at the International Conference of the Teaching of Mathematics, Village of Pythagorean, Samos, Greece. Retrieved from the World Wide Web on May 21, 2002 from: http://www.bedu.com/Publications/Samos.html

Bapst, J.J. (1971). The effect of systematic student response upon teaching behavior. Unpublished doctoral dissertation, University of Washington, Seattle. (ERIC Document Reproduction Service No. ED060651)

Bessler, W.C., & Nisbet, J.J. (1971). The use of an electronic response system in teaching biology. Science Education, 3, 275-284.

Bessler, W.C. (1969). The effectiveness of an electronic student response system in teaching biology to the non-major utilizing nine group-paced, linear programs. Unpublished doctoral dissertation, Ball State University, Muncie, IN.

Boardman, D.E. (1968). The use of immediate response systems in junior college. Unpublished master's thesis, University of California, Los Angeles.

Brown, J.D. (1972). An evaluation of the Spitz student response system in teaching a course in logical and mathematical concepts. Journal of Experimental Education, 40(3), 12-20.

Burke, J. (1978). Connections. Boston: Little, Brown, and Co.

Casanova, J. (1971). An instructional experiment in organic chemistry, the use of a student response system. Journal of Chemical Education, 48(7), 453-455.

Chu, Y. (1972). Study and evaluation of the student response system in undergraduate instruction at Skidmore College. Report prepared for the National Science Foundation, Arlington, VA. (Eric Document Reproduction Service No. ED076135)

Cue, N. (1998). A universal learning tool for classrooms? Proceedings of the First Quality in Teaching and Learning Conference, Hong Kong International Trade and Exhibition Center. Retrieved from the World Wide Web on May 21, 2002 from: http://celt.ust.hk/ideas/prs/pdf/Nelsoncue.pdf

Dewey, J. (1938). Experience and education. New York: Simon & Schuster.

Dufresne, R.J., Gerace, W.J., Leonard, W.J., Mestre, J.P., & Wenk, L. (1996). Classtalk: A classroom communication system for active learning. Journal of Computing in Higher Education, 7(2), 3-47.

Falconer, K., Joshua, M, Wyckoff, S., & Sawada, D. (2001). Effect of reformed courses in physics and physical science on student conceptual understanding. Paper presented at National Association for Research In Science Teaching, St. Louis, MO.

Froehlich, H.P. (1963). What about classroom communicators? Audio Visual Communication Review, 11, 19-26.

Garg, D.P. (1975). Experiments with a computerized response system: A favorable experience. Paper presented at the Conference on Computers in the Undergraduate Curricula, Fort Worth, TX. (ERIC Document Reproduction Service No. ED111355)

Hake, R. (1998). Interactive engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66(1), 64-74.

Hestenes, D., & Wells, M. (1992, March). A mechanics baseline test. The Physics Teacher, 30, 141-158.

Hestenes, D., Wells, M., & Swackhamer, G. (1992, March). Force concept inventory. The Physics Teacher, 30, 159-166.

Littauer, R. (1972). Instructional implications of a low-cost electronic student response system. Educational Technology: Teacher and Technology Supplement, 12(10), 69-71.

Mager, R. (1962). Preparing instructional objectives. Palo Alto, CA: Fearon.

Piburn, M., & Sawada, D. (2000). Reformed teaching observation protocol (Tech. Rep. No. IN00-3). Tempe, AZ: Arizona State University, Arizona Collaborative for Excellence in the Preparation of Teachers. (ERIC Document Reproduction Service No. ED447205)

Poulis, J., Massen, C., Robens, E., & Gilbert, M. (1997). Physics lecturing with audience paced feedback. Retrieved from the World Wide Web on May 21, 2002 from: http://www.bedu.com/Puplications/ PhysLectAPF.html

Shapiro, J.A. (1997). Student response found feasible in large science lecture hall. Journal of College Science Teaching, 26(6), 408-4 12.