Library instruction assessment in upper-level courses.

Abstract

Librarians can improve one-time library instruction in graduate and upper-level courses by implementing information literacy standards. Since instruction at this level is much more subject-and discipline-specific, it requires greater collaboration between librarian and professor than at the lower course levels. Assessment may be done primarily through observation and student self-assessment, and the results should be shared with the professor.

**********

Librarians and teaching faculty in higher education are facing a growing pressure to integrate information literacy into their instructional goals (ACRL, 2003). Several accrediting bodies in the United States now recognize information literacy as a key student outcome; the Middle States Commission on Higher Education, for instance, stated in their standards that, "information literacy is vital to all disciplines and to effective teaching and learning in any institution" (2002, p. 33). Instruction librarians are also becoming more involved in assessment, often because of institutional pressure for accreditation purposes (Merz & Mark, 2002). This assessment can occur on several different levels, including in the classroom, at the campus-wide level, or beyond the campus (ACRL, n.d.).

In this article, I share my own efforts in implementing information literacy standards and assessing student outcomes in graduate and upper-level one-time library instruction in hopes of making assessment more practicable to those new to it. In doing so, I follow the advice of Barbara Cambridge, an editor of Learning Through Assessment: "The best ways to alienate or intimidate persons new to assessment as a topic are to refer to complex, large-scale studies; employ jargon such as validity, reliability, and chi squares; and flaunt empirical data. On the other hand, positive ways to encourage persons new to assessment are to offer practical experiences, provide examples, and refer people to their own goals and objectives" (Gardiner, Anderson & Cambridge, 1997, para. 6).

Why, when, and how to assess

My first experience with assessment came when I began my tenure-track position, and I was asked to document my teaching effectiveness. I initially used a "Presentation Evaluation" form and tabulated the results for my annual self-report. The form asked students to rate the extent to which I, as the presenter,

1. clearly stated the purpose of the presentation,

2. introduced myself and explained how I could be contacted for further help,

3. was well organized and easy to follow,

4. spoke with expressiveness and variety in tone of voice,

5. used demonstrations and visual aids to clarify what was being said,

6. provided handouts that will help in using the library's resources.

Additionally, the form also asked students to rate the extent to which my presentation increased their understanding of how to use the library's resources and to list what was most beneficial and least beneficial about my presentation as well as what could be done to improve it. Although I generally got "good" ratings from students, I was not satisfied with the form since it did not give me any specifics about bow I might was prove my instruction. If, for example, some students indicated my presentation was a bit unorganized or hard to follow, the form gave me no clue as to why. I realized that if I wanted to improve my instruction the assessment form should be redesigned to gather more specific information about what students were learning.

However, before I could assess what students were learning, I knew from my education coursework that I had to first establish my learning objectives (Durkin, 1990). One of the first instruction sessions I taught illustrated this for me. The session was scheduled for the first day of fall semester, an upper-level research methods class, and it was the first meeting of the class. The students had not yet selected their research topics, and they were more interested in catching up on each other's gossip than they were in listening to me. One of the comments on an evaluation form summed up the students' attitude toward the session: "We've heard all this stuff before. Why do we have to keep listening to it every year?" I realized that other than showing them how to use the library I had lacked clear instructional goals. I learned that there were several steps to instruction:

1. Determine students' learning needs.

2. Establish learning goals and objectives for the session, drawing from the Information Literacy Standards of the Association of College and Research Libraries (ACRL, 2002).

3. Choose the appropriate methods, activities, and materials suited to the learning objectives.

4. Assess the student learning outcomes to see if students' learning needs were met, adjusting the objectives and curriculum as needed.

I also realized that assessment was not the same thing as evaluation or grading, although the terms are often used synonymously. Assessment compares a student's work to a standard, measuring the extent to which the student meets its criteria, for the purpose of modifying instruction, while evaluation, on the other hand, is the value assigned by a teacher to a student's individual work, usually manifested as a letter grade at the end of a course, for the purpose of reporting to others about a student's progress (Montgomery, 2002). This distinction between assessment and grading or evaluation is an important one. "Grades alone are not direct evidence of student learning. Grades alone do not tell us exactly what a student has and has not learned. The information upon which grades are based--tests, student papers and projects, and the like--are direct evidence of student learning (Middle States, Draft: August 2002, p. 72)." In fact, assigning a grade to the student's work is actually a fifth step in the instruction cycle.

Determining students' learning needs

Since instruction at the graduate and upper-level is so subject-specific, I must rely much more on the professor for input than I do for lower-level, general courses. My standard practice is to request a copy of the syllabus and class assignment from the professor. In my conversation with the professor, usually over the phone, I make it clear that I want him or her to participate in my instruction. I also ask the professor what students need to learn in order to complete their assignment and what the students' current level of expertise is.

In preparing for one class, I surveyed the students themselves about what they thought they needed to learn. At my request, the professor graciously had e-mailed my checklist to students, but only about one-third of the students responded. I quickly abandoned this practice as too time consuming; however, I found the results interesting. Most of the students requested help with conceptual, "higher order" tasks: identifying a research topic, learning about the different types of literature for their discipline (i.e. primary vs. secondary sources, peer-reviewed vs. trade vs. popular), and learning effective search techniques including how to refine a search to get better results. They did not want help with factual, "lower order" tasks: how to locate books and journals in the Library, how to access the databases off-campus, or how to find out whether the Library bad a particular journal or book.

Using the Information Literacy Standards

Information literacy instruction differs quite a bit in lower-level and upper-level courses. Instruction to freshmen and sophomores usually takes place in general education courses, and it is generally not subject- or discipline-specific. However, information literacy instruction in upper-level and graduate courses has some similar as well as unique features. Both levels require a subject- and discipline-specific approach but at different levels of depth. Graduate students frequently need to do more comprehensive searching of the literature for their field, either because they are doing a review for their thesis or because their research topic is based to a greater extent on their own interests and experience, and hence there may not be that much written in the literature. The materials they want may also be unavailable at our institution and must be requested through interlibrary loan.

Information literacy learning outcomes that apply more specifically to upper-level and graduate level students are listed as follows (ACRL, 2002). (Please note the ACRL Standard number following each outcome.)

* Students will become familiar with the literature of their discipline, including how it is organized and how it is indexed: 1.2.B, 1.2.C, 2.1.C & 2.3.C

* Students will be able to distinguish between secondary and primary sources, scholarly and trade journals, and current vs. historical sources, and so on: 1.2.D & 1.2.E

* Students will become familiar with the specialized language of the discipline, including the controlled vocabulary of the primary indexes for the literature: 2.2.C & 2.2.E

* Students will be able to appropriately utilize resources beyond the library, using interlibrary loan, for example, to obtain materials not owned by the library: 1.3.A & 2.3.C

* Students will be able to perform comprehensive searches of the literature, using the appropriate databases and sources of information efficiently: 1.4.A, 2.1.D, 2.2.A-2.2.F & 2.4.A

During instruction

Since research has shown that students learn better when they know why and what they are learning, I always make a point of discussing my instructional goals at the beginning of the session (Imel, 2002). I also supply at least one handout listing the goals, some keywords for searching, and the subject-specific sources I will be covering, including databases. I usually give a brief lecture covering the concepts to be learned, and then model the information-seeking behavior I want them to learn, which usually means demonstrating searches in two or three subject-specific databases.

During my demonstration, I try to "think aloud" about my strategies to model the process of effective searching. I talk about why I would choose this database over another one, why I might start with a keyword search rather than a subject search or a full-text search, how I can examine the records to get additional terms, and so on. Once I have instructed students and have modeled what it is I want them to learn, I then have them do guided practice for at least five to ten minutes for each database while I walk around the room, carefully watching them practice on their own. During one graduate course in human resources, I was able to identify some "problem areas" students were having. I saw that someone was using Boolean operators incorrectly in a search strategy, which prompted me to do a two-minute mini-lesson, drawing a Venn diagram on the board to illustrate. One of the students later commented that this was the most helpful part for her. Simply observing students helps me to be flexible and alter my objectives and methods when needed.

Looking hack and assessing

During the last five minutes of the instruction session, I ask students to complete my current assessment form. The first part of the form asks students to 'sum up' the overall session in terms of value to them, using a five-point scale to rate the following statements:

1. I understood the goals for the instruction and thought they matched my needs.

2. I found the instruction well-organized and easy to follow.

3. I thought there was enough time to ask questions and get help as I needed during the instruction.

4. I believe that the instruction will enable me to do library research more effectively.

5. I feel I know how to proceed with my research and get help if I need it. The second part provides me with specific information about the students' learning to modify future instruction sessions. This part of the form also has the potential to enhance students' learning by asking them to reflect on what they have learned; it asks them for short answers to the following questions:

6. What is one useful tip, fact, or technique you learned today?

7. What was covered in this session that you already knew?

8. What was not covered in this session that you wish had been?

9. Was there anything unclear or confusing in this session, and if so, what?

Although the form combines elements of both summative and formative assessments (Boston, 2002), the focus has shifted from asking students to evaluate my "performance" to asking them to reflect on what they have learned, in effect, to self-assess their learning. Although there are limits to self-assessment, it does have great value as a metacognitive strategy (Kruger & Dunning, 1999). Metacogintion--that is, thinking about one's own thought processes--has been shown to help students develop control over their learning (Imel, 2002). Students "need to develop a metacognitive approach to learning, making them conscious of the explicit actions required for gathering, analyzing, and using information" (ACRL, 2003, Use of the Standards section, para. 2). Asking them to reflect on the session reinforces that they are responsible for their own learning.

I continually revise the form. In future, I want to list my specific learning objectives for a session, asking students to assess whether the objectives were appropriate and were met. Nevertheless, the current form is an improvement over the previous one. I find that students share more information with the current form, writing more comments as well as speaking up. One student in a graduate education class suggested that I group students by ability so that I could teach mini-lessons to those who needed more help. I may try this, or I may try pairing less skilled students with more skilled ones to facilitate peer teaching during the guided practice. I also ask the professor for feedback on what he or she thought went well, what needed improvement, and what could be added for subsequent sessions. Asking for feedback makes the professor a stakeholder in the instruction. In the same education class mentioned above, a tour of the curriculum and media collections requested by the professor had met with little student response. In discussing the session after class, the professor and I both agreed that the tour should probably be dropped from any future sessions, and the students' evaluations confirmed this. By showing I am open to feedback and am willing to change, I send the message that the purpose of instruction is to help students learn and that library instruction is not the same every time.

I also try to self-assess the session, even if it is just jotting down a few quick notes on my lesson plans, which I save. My plan for next year is to keep a more formal "instruction log" to assess what worked, what did not work and why, and what I would do differently next time. Of course, discussing my instruction sessions informally with my fellow librarians is an important part of my self-assessment. Informally reporting to our Library Instruction Coordinator how a session has gone helps me do holistic assessment of my instruction. These informal 'post-mortems' also have helped me acquire some new teaching techniques. For instance, asking the professor for a list of key words or "hot topics" to use in my demonstration searches is a 'tip' I learned from our coordinator. I've also begun tabulating the results of the students' self-assessments, sending a summary to the professor. My purpose is two-fold: I believe the information may help the professor better assess the students' true information competencies, and I want him or her to see the value of information literacy instruction by reading the students' comments.

Conclusion

Assessment of information literacy competencies can be daunting to those unfamiliar with assessment jargon or techniques. Assessment is just one phase of instruction, and its main purpose is to improve instruction and student learning. It does not need to be elaborate in order to be useful; small, homegrown tools can provide good information (Carter, 2002). By working collaboratively with faculty in the planning stages and providing them with the results of library instruction assessments, librarians play a valuable role in teaching students the fine points of information literacy.

References

Association of College & Research Libraries (ACRL). (n.d.). Assessment Issues. Information Literacy. Retrieved May 17, 2003, from http://www.ala.org/Content/NavigationMenu/ACRL/Issues_and_Advocacy1 /Information_Literacy1/ACRL_Information_Literacy_Web_Site/IL_in_Classrooms /Assessment_Issues.htm

Association of College & Research Libraries (ACRL). (2002). Standards Toolkit: Information Literacy Competency Standards for Higher Education. Retrieved May 31, 2003, from http://ala.org/Content/NavigationMenu/ACRL/Issues_and_Advocacy1 /Information_Literacy1/ACRL_Information_Literacy_Web_Site/Standards_Toolkit/ The_Standards/The_Standards.htm

Association of College & Research Libraries (ACRL). (2003). Information Literacy Competency Standards for Higher Education. Retrieved May 31, 2003, from http://www.ala.org/Content/NavigationMenu/ACRL /Standards_and_Guidelines/Information_Literacy_Competency_ Standards_for_Higher_Education.htm

Boston, C. (2002). The concept of formative assessment. ERIC Digest. College Park, JD: ERIC Clearinghouse on Assessment and Evaluation. (ERIC Document Reproduction Service No. ED470206). Retrieved July 8, 2003, from http://www.ericfacility.net/databases/ERIC_Digests/ed470206.html

Carter, E. W. (2002). "Doing the best you can with what you have": Lessons learned from outcomes assessment. The Journal of Academic Librarianship 28, 36-41. Retrieved May 11, 2003, from Library Literature & Information Science Full Text database.

Durkin, D. (1990). Delores Durkin speaks out on instruction. The Reading Teacher, 43, 472-476. Retrieved February 18, 2003, from ProQuest Education Journals database.

Gardiner, L, Anderson, C., & Cambridge, B. (Eds.) (1997). Learning Through Assessment: A Resource Guide for Higher Education [Electronic excerpt]. Washington, DC: American Association for Higher Education. Retrieved May 17, 2003, from http://www.aahe.org/members_only/Lrn_excerpt.htm

Imel, S. (2002). Metacognitive skills for adult learning. Trends and Issues Alert No. 39. Retrieved May 31, 2003, from ERIC Clearinghouse on Adult, Career, and Vocational Education: http://ericacve.org/doegen.asp?tbl=tia&ID=162

Kruger, J. & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessment. Journal of Personality and Social Psychology 77, 1121-1134. Retrieved May 25, 2003, from Journals@Ovid PsycArticles database.

Merz, L. H. and Mark, B. L. (2002). Assessment in College Library Instruction Programs. Chicago: Association of College and Research Libraries.

Middle States Commission on Higher Education. (2002). Characteristics of Excellence in Higher Education: Eligibility Requirements and Standards for Accreditation. Philadelphia: Middle States Commission on Higher Education. Retrieved May 17, 2003, from http://www.msache.org/charac02.pdf

Middle States Commission on Higher Education. (Draft: August 2002). Student Learning Assessment: Options and Resources. Retrieved May 5, 2003, from http://www.msache.org/studlrng.pdf

Montgomery, K. (2002). Authentic tasks and rubrics: Going beyond traditional assessments in college teaching. College Teaching 50, 34-39. Retrieved May 12, 2003, from ProQuest Education Journals database.

Linda Neyer, University of Scranton, PA

Neyer is assistant professor/librarian. Her research interest include instructional design for distance learning and usability studies for web design.