Reading between the lines on surveys. Bottom up or top down: which way to go for research on the scholarship of teaching?

Abstract

As the demand for use of technology in education increases so too should the attention to the scholarship of teaching within these technologically mediated environments. Interested in doing just that, a team of three professors worked together to assess the value of online research components in traditionally taught face-to-face classes. Fifty-six education students taking online classes responded to a Likert style survey that included a space for comments. Reported in this article are not only the generally positive results of the study, but also, and in some ways more importantly, interesting analysis issues that the research team dealt with in relation to the scholarship of teaching and survey research.

Introduction

Integrating technology into educational environments is rapidly shifting from a novelty employed by the technologically savvy to a requirement forced upon professors in higher education (GaPSC, 2001; NCATE, 2002; Simon Fraser University, 1998; Wright, 2000). As the demand for use of technology in education increases so too should the attention to the scholarship of teaching within these technologically mediated environments. Scholars of teaching have been characterized as faculty who "frame and systematically investigate questions related to student learning--the conditions under which it occurs, what it looks like, how to deepen it, and so forth" (Hutchings & Shulman, 2000, p. 48). There is hardly a more opportune time to ask such questions than when both the student and the teacher experience the change from face-to-face instruction to teaching and learning in an online environment.

Traditionally educational research has used surveys to gather data on classroom practices (Mayer, 1999). However, Mayer (1999) questions the validity of such data when he states, "the teaching process consists of complex interactions between students and teachers that a survey may not be able to do justice to" (p. 33). Although it is recognized that adding open ended questions to surveys can permit "respondents to express feelings, ideas, or reactions without being limited to preset categories" (Thomas, 1999, p. 47), survey methods texts also emphasize the complexity of analyzing responses to open ended questions and the amount of time involved by both researcher and subject as strong disadvantages to there use (Fowler, 2002; Thomas, 1999). Yet it is the contention of these authors that the advantages listed by Fowler when he states that surveys "permit the researcher to obtain answers that were unanticipated [and] may describe more closely the real views of the respondents" (p. 91) far outweigh the disadvantages when the issue at hand relates to the quality of teaching and learning.

In this article we demonstrate that in educational settings, and indeed we believe in all research on the scholarship of teaching, pragmatic action research (Levin & Greenwood, 2001) that is paradigmatically qualitative and logically inductive (Denzin & Lincoln, 1994) is a highly effective approach to understanding and improving practice. More specifically, we show the value of incorporating a qualitative component into survey research, particularly when the survey relates to the scholarship of teaching. We believe that such a mixed method design (Tashakori & Teddlie, 1998) has the potential to yield research results that might assist others in understanding more about the online teaching and learning environment and, consequently, improve practice.

The Study

Framework

As advocates of active learning (Race, 1998) and supporters of Boyer's suggestion to make research a standard component of undergraduate coursework (Boyer Commission, 1998), we are always looking for an opportunity to incorporate those activities into our classes. Such an opportunity presented itself to one of us who was able to move two required education classes to a WEBCT-based online learning environment that allowed him the opportunity to not only incorporate, but also emphasize research-related web-based activities in classes that had been taught in a traditional face-to-face format. Davies (1997), has pointed out that "Via the Web, students can explore various types of information, judge their authenticity, compare different viewpoints, analyze and synthesize information, and construct their own knowledge" (Promoting Improved Learning sect., para 5). It has also been noted that the ability to use the Web empowers students while promoting socially relevant development (Flake, 1996) and enables students to attend to their learning style preferences (Bishop, Greet, & Cooke, 1997). Despite such rosy observations, Owston (1997) warns that,

   No medium in and of itself is likely to improve learning in a significant
   way when it is used as a tool to deliver instruction. Nor is it realistic
   to expect the Web, when used as a tool, to develop in students any unique
   skills. The key to promoting learning with the Web appears to lie in how
   effectively the medium is exploited in the teaching and learning situation.
   (p. 28)

A key to increasing the effectiveness of teaching and learning through the web is getting the students' perspective on that learning experience. In order to evaluate student's perceptions of the effectiveness of the medium and to inform the professor of changes that may need to be made in the course, a survey was developed by a team that consisted of an Instructional Technologist, an Educational Researcher, and the Physical Education professor who was teaching the courses.

The Instrument

The survey consisted of 5 demographic questions, 1 open ended comment option at the end, and 12 statements to which the respondent chose a reply from a 6 point Likert Scale where 1 represented strongly agree, 2 represented agree, 3 represented slightly agree and 4, 5, 6 respectively represented slightly disagree, disagree, and strongly disagree. The survey was administered to four sections of two different courses (n=56) during a recent fall semester using an anonymous online submission form. Thirty-seven participants were from sections of the Health and Wellness course and 16 were from a Cardio-Pulmonary Resuscitation (CPR) class. Four participants took both classes at the same time. The survey statements are listed in Table 2. Descriptive and central tendency statistics on all 12 statements indicated a preliminary finding that students agreed to some degree with all statements. See issue's website <http://rapidintellect.com/AEQweb/sum02.htm>

The Data

Mean responses were calculated for each question and it became clear that the responses were overwhelmingly positive (mean from 1.52-2.02). Moreover, the range of the means was only .5. That led us to the strong and quick conclusion that students responded positively to their online learning experience. This is helpful to know and indeed the central question the professor had posed. Had there not been a qualitative component in the instrument our analysis and interpretations might have stopped there. Analysis of the qualitative data, however, yielded an interesting result that led to a significant and deeper insight. To our surprise, when asked to, "Please make general comments on the online learning environment and its tools and resources," students most often responded with statements that thematically indicated that the online course had strengthened their capacity to learn independently and helped them become more disciplined. A few of the sample responses follow.

* I thought this online learning class was quite interesting. It makes the student set their own timelines and study techniques, thus making them more independent. I really enjoyed the class.

* Good Class! Gives students the opportunity to become more independent in their studies.

* Helps with self directed learning.

* I enjoyed the online course because it allowed me to become more responsible in my studies.

* I always knew that I had a deadline to meet, which meant that I could not afford to wait until the last minute.

A careful re-examination of the initial survey items indicated that one of the actual statements that students were asked to rate addressed independent learning (See statement number 3 in Table 2). This led us to go back to the statistical data and rank order the means of the responses to the survey statements. Statements with the most positive means (in this case that would be with lowest scores) were placed at the top of the array (See Table 3). Interestingly, the question on developing independent learning ability not only had the most positive mean (1.52) it also had the smallest standard deviation (sd=.6). Had we ended our analysis with the survey data alone we would have come to the conclusion that students overwhelming liked their online course, that they would be willing to take other online courses in the future, and that independent learning was a positive by-product of the course design. However, we would not have recognized the significance that statement 3 carried without the recognition that when given the opportunity to comment on anything in the course the majority of students commented on the value they gained from the discipline to learn on their own. See issue's website <http://rapidintellect.com/AEQweb/sum02.htm>

Course Differences

Part of our analysis of the survey data involved comparing the responses of students in the Health and Wellness class to those of the students in the CPR class. In doing so we noticed that there was a statistically significant difference at the .05 level in the responses to statements 6 through 8 between the classes. All three statements related to learning that had to do with use of the Internet for research purposes (See Table 2). The students in the Health and Wellness class agreed that they had improved their web skills with a mean across all 3 questions of 1.65. Although the students in the CPR class also agreed, they did so with a higher mean response of 2.56 indicating a weaker degree of agreement. In reflecting on this difference the professor realized that the online exercises for the Health and Wellness class were such that they could not be answered through use of the text materials. Students actually had to use the Internet to do the assignment successfully. However, it became apparent that students in the CPR class could, and in fact many did, find the answers in their text. This insight, somewhat obvious in retrospect but that did not become apparent to the professor until the data was reviewed, led him to revise his practice. Obviously, to create effective online exercises that accomplish the objective of improving students' web researching skills, exercises need to be structured in ways that cannot be answered from the text.

Discussion

This study began as a fairly straightforward examination of the hypothesis that enriching a course with the use of online exercises would be a positive learning experience for students. Preliminary analysis of data indicated that was true. One could have stopped the analysis there and produced a clear, if not overly interesting result. However, the insights gained from the qualitative data added to our understanding of students' perceptions of their learning experiences. Though we had hoped students would develop the ability to learn and work independently we were surprised to see that this was actually the case. This was an insight we might not have come to if we had not worked back and forth between the qualitative data and the statistical data.

One obvious and essential aspect of such research is gathering data on student responses to the approaches and techniques we employ in our teaching. An understanding of student responses is particularly important as we move into the uncharted territory of technologically mediated teaching and learning environments. We need look no further than typical faculty evaluation forms given to students at the end of the semester to know that data on student responses to teaching have typically been compiled using survey techniques.

Obviously, such techniques can be very effective instruments in our efforts to gather data on the scholarship of teaching. However, if we use surveys as some advocate, namely that the means for analyzing the data be predetermined (Gay & Airasian, 1996), it is our contention that, used in that way, we are limiting the possibilities for discovering new insights into our teaching and learning endeavors. Moreover, adding an open-ended qualitative component to a Likert scale survey, can add unexpected and valuable insights, which may, in turn, provide new perspectives for reassessing our quantitative data. That was our experience in the example discussed in this essay.

Conclusion

One of the problems with using survey research in the scholarship of teaching is that survey research has traditionally been conducted within the context of the positivist or postpositivist paradigm (Tashakkori & Teddlie, 1998). That is to say, the research is designed deductively and the design of the instrument and the analysis of the data are prestructured and predetermined (Fowler, 2002; Weisberg, Krosnick, & Bowen, 1996). As we noted earlier, literature on the scholarship of teaching is relatively new and research approaches to it are more appropriately set in the context of the qualitative paradigm (Denzin & Lincoln, 1994), which uses inductive logic and typically builds theory from the ground up (Corbin & Strauss, 1990). We are not arguing that surveys are not useful vehicles for gathering data on the learning experiences of students that will be useful in advancing the scholarship of teaching. Rather, what we are suggesting is that if we approach our research on teaching from a paradigmatically qualitative, logically inductive position, we can use surveys as a means of gathering data that includes open-ended, unstructured responses, which, when analyzed inductively following the dictates of grounded theory (Glaser & Strauss, 1967), may produce new knowledge that will inform practice and advance the scholarship of teaching.

References

Bishop, A.S., Greer, J. E., & Cooke, J. E. (1997). The Co-operative peer response system: CPR for students. Proceedings of the Ed-Media: World Conference on Educational Multimedia, Hypermedia, and Telecommunications. Calgary, Canada: Association for the Advancement of Computing in Education. Retrieved February 6, 2002 from http://www.cs.usask.ca/projects/aries/projects/cpr/cpr.ps.

Boyer Commission (1998). Reinventing undergraduate education: A Blueprint for America's Research Universities. The Boyer Commission on Educating Undergraduates in the Research University. Retrieved February 12, 2002, from http://naples.cc.sunysb.edu/Pres/boyer.nsf Funded by Carnegie Foundation.

Corbin, J., & Strauss, A. (1990). Grounded theory research: Procedures, canons and evaluative criteria. Qualitative Sociology, 13 (1), 3-21.

Davies, J. E. (1997). The Internet in Education. Retrieved February 12, 2002, from University of Alberta, Department of Educational Psychology Web site: http://www.ualberta.ca/~jedavies/inteduc/benefits.htm

Denzin, N. K. & Lincoln, Y. S. (Eds.)(1994). Handbook of qualitative research. Thousand Oaks, CA: Sage Publications.

Flake, J. L. (1996). The World Wide Web and education. Computers in the Schools, 12, 1-2, 89-100.

Fowler, F. J. (2002). Survey research methods. (3rd ed.). In Applied Social Research Methods Series (1). Thousand Oaks, CA: Sage Publications.

GaPSC (2001). Georgia Professional Standards Commission: Special Technology Requirement. Retrieved November 10, 2001 from http://www.gapsc.com/

Gay, L. R., & Airasian, P. (1996). Educational research: Competencies for analysis and application. (6th ed.). Upper Saddle River, NJ: Merrill.

Glaser, B., & Strauss, A. (1967). The discovery of grounded theory. Chicago: Aldine.

Hutchings, P., & Shulman, L. (2000). The scholarship of teaching: New elaborations, new developments. In D. DeZure (Ed.), Learning from Change (pp. 47-50). Sterling, VA: Stylus Publishing.

Levin, M., & Greenwood, D. (2001). Pragmatic action research and the struggle to transform universities into learning communities. In P. Reason, & Bradbury, H. (Eds.) (2001). Handbook of action research: Participative inquiry and Practice (pp. 103-113). Newbury Park, CA: Sage Publications, Inc.

Mayer, D. P. (1999, Spring). Measuring instructional practice: Can policymakers trust survey data? Educational Evaluation and Policy Analysis 21 (1), pp. 29-45.

NCATE (2002). National Council on Accreditation for Teacher Education: Standards. Retrieved February 2, 2002 from http://www.ncate.org/standard/m_stds.htm

Owston, R. D. (1997). The World Wide Web: A technology to enhance teaching and Learning? Educational Researcher, 26, 27-34.

Race, P. (1998). 500 tips for open and flexible learning. London: Kogan Page.

Simon Fraser University & TeleEducation NB, Canada, Telelearning Policy Initiative. (1998, November). Technology-mediated learning: Current initiatives and implications for higher education. Retrieved February 2, 2002 from the Telelearning Web site: http://www.telelearn.ca/g_access/news/tml.pdf

Tashakori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. In Applied Social Research Methods Series (46). Thousand Oaks, CA: Sage Publications.

Weisberg, H. F., Krosnick, J. A., & Bowen, B. D. (1996). An introduction to survey research, polling, and data analysis. (3rd ed.). Thousand Oaks, CA: Sage Publications.

Wright, M. (2000). The impacts of higher education adapting to the Web. Unidata: University Corporation for Atmospheric research sponsored by the National Science Foundation. Available: http://www.unidata.ucar.edu/staff/mwright/POV.html. [Retrieved 2002, February 6].

R.W. Schmertzing, Ed.D., is a qualitative researcher on the graduate research faculty in the College of Education who combines a longstanding interest in classroom culture with more recent interests in research in the area of educational technology. J. Stelzer, Ph.D., is an Assistant Professor in the Department of Kinesiology and Physical Education. He uses his European background to combine multicultural education research with research on applied technology to physical education. L.C. Schmertzing, Ed.D., is an Assistant Professor in the Department of Curriculum and Instructional Technology. She has focused her publishing and research on graduate students in distance learning environments.