Teaching basic information literacy skills online.

Abstract

LiONiL, an online information literacy tutorial, was introduced in a freshmen composition course at the University of Colorado at Colorado Springs in 2004. An analysis of the pilot study demonstrated that significant learning occurred, and identified pedagogical weaknesses within the tutorial and the testing instrument. Participants in the study were almost evenly divided in their preference for online vs. face-to-face instruction. LiONiL assessment instruments offer potential for individual courses and disciplines to include results in their assessment programs.

Background

Web-based information literacy tutorials offer a range of advantages to students that face-to-face instruction cannot, the most obvious being consistent self-paced instruction in a 24/7 environment. In addition computer assisted instruction can be used as an alternative for, or a supplement to, the traditional lecture mode. But can online instruction be effective, and do students feel comfortable with the web-based format?

The University of Colorado at Colorado Springs is the fastest growing institution of higher education in the state. Once, exclusively a commuter college, the number of students under age 21 has risen to 29 percent of the total enrollment. Still, 24 percent of the student body includes learners between the ages of 30 and 49. Students from all walks of life enter the University with a broad range of library experience and computer skills. Many adult learners have not stepped into a library since the disappearance of the card catalog, and their library anxiety is extremely high. Younger learners feel confident in their information finding skills using the World Wide Web but many lack the preparation for the rigors of academic research. This attitude is borne out in two recent surveys of college age students. One reported that 73 percent of the students polled were more likely to conduct research on the Internet than by going to the library (Jones, 2002). The other stated that two thirds of the students who responded to the survey feel confident that they can find valid information on the Web, and their first-choice Web resources for assignments are search engines such as Google and Yahoo (OCLC, 2002).

Three objectives compelled librarians at the University of Colorado at Colorado Springs to turn to the online environment as a mode of information literacy instruction. The long term objective is to design a process-based program of instruction emphasizing generic research concepts and skills that students can apply across the curriculum and beyond in their professional and personal lives. Such a program has the potential to raise all students to a common floor upon which librarians and faculty can build more advanced information literacy competencies or focused research skills unique to a particular assignment or discipline. The UCCS Library expects the tutorial to appeal to faculty, who in the past, have not taken advantage of traditional instructional services. Because the online format can be integrated into courses without taking up classroom time, some faculty might welcome it as an acceptable alternative. The tutorial will also provide instructional support for distance education programs. Finally, a goal of the Core Education Requirements at UCCS is to assure that students will, upon graduation, be able to understand and apply the tools and methodologies used to obtain knowledge. By including a pre and post-test in the online tutorial, measurable learning outcomes can be provided in course and discipline assessment plans.

Review of the Literature

Since the turn of the millennium a small body of literature has appeared demonstrating the efficacy of the online library instruction tutorial in first-year experience and English composition programs. Studies that compared online and in-class tutorials using similar assessment measures have all concluded that online instruction is as effective as in-class instruction. A study at the University at Albany SUNY compared a web-based tutorial to in-class instruction for 303 students enrolled in a first-year experience program. The aggregate scores for the web group and the control group were compared, revealing that significant learning took place in each. However, there was no significant difference in learning between the groups, leading researchers to conclude that the two modes of instruction were equally effective. This study did not survey student attitudes toward online or classroom instruction (Germain, Jacobson, Kaczor, 2000).

Another study at the University of North Carolina compared 125 students in an English composition course and found that there was no statistically significant difference in learning between the group that received online instruction and the group that received in-class instruction (Holman, 2000). At the University of Oswego, SUNY the pre and post-test scores for a group of students who completed an online tutorial and a group of students who received a traditional lecture in an English composition class were analyzed. The study, which also polled student attitudes about the different modes of instruction, determined that student learning and satisfaction were comparable (Nichols, Shaffer, and Shockey, 2003).

Description of LiONiL

The Library's first experience with Web-based instruction began in 2003 when two librarians, including the author, were awarded a grant from the UCCS Teaching and Learning Center to design and implement an online tutorial for English 141, Writing and Rhetoric. English 141 is a core composition course at the University and is required of most students. It offers extensive practice in writing arguments for multiple purposes and audiences, and introduces students to the reading and writing activities integral to academic research. Students learn to investigate issues, evaluate and analyze sources, integrate primary and secondary materials into their own arguments, and learn principles of research documentation. Librarians provide information literacy instruction to over half of the sections offered each semester by collaborating with instructors to tailor a 60-75 minute session that focuses on a particular assignment involving library research. However, an hour-long session is not long enough to introduce students to the resources needed to complete an assignment and cover basic skills. The tutorial would introduce students to basic skills and concepts before the librarian delivers more focused in-class instruction. If an instructor prefers, it can be integrated into a course with no follow-up visit from the librarian.

The creators of the tutorial collaborated with other librarians and English Composition instructors to design LiONiL (Library Instruction Online for Information Literacy). Librarians individually listed concepts and skills they felt were basic to library research. This list was then prioritized by the librarians as a group, and distributed within the framework of the Association of College and Research Libraries Information Literacy Competency Standards (ACRL, 2002) that form the basis for LiONiL.

LiONiL consists of six learning modules each followed by a short interactive quiz. An online pre-test and post-test are maintained on a separate server. The tutorial includes a glossary of library/technology related terms and a student satisfaction questionnaire. LiONiL features many interactive components including simulations, manipulations, and hands-on activities using the Kraemer Family Library website, the online catalog, and one periodical database. Students can complete LiONiL in 60 to 90 minutes. They may re-enter the tutorial at any point in the program and as often as they like. LiONiL is accessible to UCCS students from off-campus if their computers are connected to the University computing network.

Methodology

In November 2003 librarians, faculty in English composition and other disciplines, and students critiqued the content and interface of LiONiL. Revisions were made in the navigation and content of the tutorial based on their feedback.

LiONiL was piloted in seven sections of English 141 during spring semester of 2004. One hundred thirteen students were enrolled in the seven sections, representing about 18 percent of the total student enrollment in English 141 during this time period. In accordance with the standards of the UCCS Institutional Review Board, students were asked for permission to use their pre and post-test scores as well as their responses to the Student Satisfaction Questionnaire in a follow-up study. The four instructors who volunteered to participate in the pilot study agreed to administer the pre-test during the first week of class, and ask students to complete the tutorial before the first library related assignment. Students would take the post-test near the end of the semester in order to measure the residual effect of the tutorial. Completion of the Student Satisfaction Questionnaire at the end of the tutorial was optional.

After grading all the tests, the researchers used SPSS 12.0 to run a statistical analysis on the pre and post-test scores as well as the scores on each set of questions. They also compiled and analyzed the responses on the student questionnaire to measure the overall success of LiONiL.

Results

Sixty-five students completed both the pre and post-test. An analysis of the test scores demonstrated that statistically significant learning occurred, although the increase in scores was not as great as hoped. In addition, the analysis revealed that wording in some of the test questions was confusing and/or the pedagogy in some of the tutorial modules was unclear.

The researchers conducted a paired samples t-test to evaluate the impact of LiONiL on student learning. There was a statistically significant increase in test scores from the pre-test (M=12.60, SD=1.74) to the post-test (M=14.09, SD=1.71), t (64) = 6.036, p<.0005. The scores on the pretest ranged from 9 to 16 while the scores on the post-test ranged from 11 to 17. This would suggest that students who scored higher on the pre-test also scored higher on the post-test while students scoring lower on the pre-test also scored better on the post-test, but not as well as hoped. Two thirds of the sample (N=65) actually improved their scores on the post-test. In straight percentages, 17 percent scored at least four points higher, 22 percent scored three points higher, 6 percent scored two points higher, and 20 percent scored one point higher. Even so, a third of the sample showed no improvement. In fact, 23 percent scored the same and 12 percent scored lower on the post-test.

The researchers also used the paired samples t-test to evaluate the impact of LiONiL on each of the seventeen identical or similar questions in the pre and post-test. There was a significant difference in scores on seven questions at the 0.05 level. On two other questions scores improved, but not significantly. On four questions, the t-test revealed that scores were lower on the post-test. For three of these questions the decrease was not significant, but on the fourth one there was a significant decrease from the pre-test (M=0.953, SD=.211) to the post-test (M=0.738, SD=.442), t (64) = 3.589, p<.0005. Twenty percent of the sample that answered this question correctly on the pre-test missed it on the post-test. The final four questions were answered correctly by 95 percent of the sample on both tests.

As a result of the statistical analysis, a careful examination of the entire tutorial took place. First the design team clarified text on those screens dealing with concepts and skills with which students had the most difficulty. It was obvious, however, that even for those questions where significant learning was indicated, there was room for pedagogical improvement. For example, the scores on two questions about subject encyclopedias and primary resources improved by 25 and 35 percent, but still about one third of the total sample missed these questions on the posttest. The team added questions to three of the self-administered quizzes following each module, so that each learning objective was matched with a question. The additional questions give learners more feedback and reinforcement of concepts learned as they navigate the tutorial.

The statistical analysis of the pre and post-test questions revealed as much about the testing instrument as it did about the content of the tutorial. As a result, the tests have seen much revision. All the questions on the post-test are identical to those on the pre-test. Each question is more tightly geared to a corresponding objective in the tutorial. Several questions were rewritten to require more critical thinking. Questions that were answered correctly by 95 percent of the sample on both tests were discarded altogether. The optional Student Satisfaction Questionnaire at the end of the LiONiL tutorial was designed to elicit feedback regarding the content, design, and effectiveness of the tutorial as a learning tool. Seventy eight students answered the questionnaire. Their responses could not be correlated with their pre and post-test scores. Most of the questions were qualitative, but the findings will help the Library plan for the future of the online program. Some of the more salient points that surfaced in the data evaluation are discussed below.

First, students were asked if LiONiL increased their confidence in using library resources, sixty-five percent of the sample (N=78) stated that it did. This figure matches the percentage of students whose scores improved on the post-test. Students were also asked how they felt about learning library research skills in the online environment. While almost all (95 percent) students agreed that the online environment is an effective platform for learning basic information literacy skills, they were almost evenly divided about their personal preference for online or in-class instruction. Fifty four percent favored the online environment, and 46 percent preferred having the librarian come to class. Students who declared a preference for online instruction indicated that they enjoyed the flexibility of self-paced learning. They were able to skip over parts of the tutorial already known to them and concentrate on modules that contained new information. The anywhere/anytime access was important to them, and the hands-on activities kept these students engaged in the learning process.

Learners favoring in-class instruction enjoyed having the librarian present to explain and demonstrate concepts and skills. Furthermore, they felt more comfortable having someone there to answer questions on the spot. Many of these students indicated that they were visual or auditory learners and preferred to be shown how to do something.

The questionnaire also invited students to assess their prior knowledge for several of the concepts and skills covered in LiONiL. Students reported a high level of prior knowledge in only two of the thirteen categories listed. Their answers provided some insight into the diversity of student experience with respect to library skills, but the results may have been skewed by the fact that students were asked this question after completing LiONiL. A more accurate measure would have been gained had students been asked to assess their knowledge before they completed the tutorial. Therefore, the researchers have moved the self-assessment piece to the pre-test. The English composition instructors who participated in the study were satisfied with the outcome. Aside from some confusion with the online quizzes and when to administer the pre and post-tests, the integration of LiONiL into the course curriculum was fairly seamless.

One instructor asked her students to write a short critique of LiONiL at the end of the term. Most of her students thought the tutorial was informative and enjoyed learning at their own pace; others thought the program would be most beneficial if used in combination with a visit from the librarian.

Conclusion

This pilot study demonstrates that most students are comfortable with the online environment as a platform for learning information literacy skills. It also reveals that the promises of online instruction must be weighed carefully against the benefits of face-to-face instruction. The Librarians at UCCS believe that LiONiL can be used to enhance instruction. They realize however, that many teaching faculty prefer to skip formal library instruction or conduct it themselves. They expect that by offering a tutorial with measurable learning outcomes faculty will recognize the importance of information literacy and make a commitment to including it in their courses and assessment strategies. Librarians also expect the tutorial to open lines of collaboration with faculty to integrate an information literacy goal into their courses not only at the freshmen level, but in upper-division courses as well.

The study also illustrates how a statistical analysis of scores on individual questions as well as total test scores can define the weaknesses in an online program. The revised tutorial will again be tested in English 141 against two other groups. In addition to the group that completes the tutorial, a second group will receive in-class instruction, and a third group will have the benefit of both. All three groups will take the pre and posttests. Professors in other disciplines at UCCS have expressed enthusiasm for LiONiL. A few have integrated it into their courses, including two who teach online. A long-term goal of the Library is to offer support for the University's rapidly expanding distance education programs in which more than 750 students were enrolled during spring 2004. Librarians envision adding portals to LiONiL that will be discipline specific. Recently the Library received funding from the Beth-El College of Nursing and Health Sciences to design a tutorial. Plans are underway to seek funding for tutorials in the Colleges of Business and Education.

References

ACRE Information literacy competency standards for higher education. ALA, 2000. http://www.ala.org/ala/acrl/acrlstandards/informationliteracycompetency.htm (accessed August 1, 2004).

Germain, Carol Anne, Jacobson, Trudi, & Kaczor, Sue A. (2000). A comparison of the effectiveness of presentation formats for instruction: teaching first-year students. College & Research Libraries, 61 (1), 65-72.

Holman, Lucy. (2000). A comparison of computer-assisted instruction and classroom bibliographic instruction. Reference & User Services Quarterly, 40(1), 53-60.

Jones, Steve. The Internet goes to college; how students are living in the future with today's technology. Sep 15, 2002. Pew Internet American Life Project. http://www.pewinternet.org/pdfs/PIP_CollegeReport.pdf (accessed August 15, 2004).

Nichols, James, Shaffer, Barbara, & Shockey, Karen. (2003). Changing the face of instruction: is online or in-class more effective? College & Research Libraries, 64(5), 378-88.

OCLC. How academic librarians can influence students' web-based information choices. OCLC White Paper on the Information Habits of College Students, June 2002. http://www5.oclc.org/downloads/community/informationhabits.pdf(accessed August 15, 2004).

Suzanne Byerley, University of Colorado at Colorado Springs

Byerley, M.L.S., is an Assistant Professor and the Electronic Reference Services Librarian at Kraemer Family Library