Assessing User Needs, Satisfaction, and Library Performance at the University of Washington Libraries.

ABSTRACT

THE UNIVERSITY OF WASHINGTON LIBRARIES HAS CONDUCTED triennial faculty and student library surveys since 1992. Surveys are sent to all faculty and a random sample of graduate and undergraduate students. Results have revealed significant variation within and between user groups concerning library satisfaction, use, priorities, and importance. There were 2,749 responses to the most recent survey in 1998, including more than 1,500 completed surveys returned from faculty. These large-scale surveys, while extraordinarily valuable, have proven costly and time-consuming to design, administer, and analyze. The ARL LibQUAL+ pilot offered an opportunity to employ a different methodology and design that focused on quality of service and library support through a Web-based survey. This article discusses issues and results associated with these different approaches.

INTRODUCTION The University of Washington Libraries (UW Libraries) has utilized a number of approaches during the past decade to assess the effectiveness of service programs and library support of faculty and student research, teaching, and learning. Among the most valuable methods employed have been large-scale surveys of faculty and students conducted every three years beginning in 1992. Focus groups, usability and observational studies, targeted surveys, and interviews are also used to assess library programs and services as well as user needs. Results from the triennial surveys have played a critical role in supporting the transition to a user-centered library (Wilson, 1995) and in creating a culture of assessment (Lakos, 1998). The large representative data sets generated by these surveys have also proven to be powerful information sources in the campus political environment. Survey results and analyses can be found at the UW Libraries' Web site on user surveys: http://www.lib.washington.edu/surveys/. These surveys, though quite valuable, are expensive and time-consuming to design, administer, and analyze. Participation in the ARL-sponsored SERVQUAL (now LibQUAL+) pilot provided an opportunity to use a well-established survey tool with a different methodology, design, content, and delivery mechanism. It also afforded the chance for interinstitutional comparisons using a standardized survey instrument. Another attractive feature was the ability to gain experience with a Web-based survey that might reduce survey costs associated with printing, mailing, and data entry. This article will compare the UW Libraries' surveys with LibQUAL+ results from the University of Washington in such areas as response and representativeness of survey population, similarities and differences in results, and whether the right questions are being asked.

USER SURVEYS

Library user surveys have become widespread in academic libraries during the past twenty years. Surveys have often been used as a tool to assess service quality and user satisfaction. The Association of Research Libraries issued four Systems and Procedures Exchange Center (SPEC) kits on user surveys and studies between 1981 and 1994 (Association of Research Libraries, 1981, 1984, 1988, 1994). A substantial body of literature has been developed on surveys and service quality, led by studies and reviews from such library educators/professionals as Hernon and McClure (1990); Van House, Weil and McClure (1990); Hernon and Altman (1998, 2000); Nitecki and Franklin (1999); and Hernon and Whitman (2001). Library applications of the SERVQUAL instrument have been covered by Nitecki (1996), and Cook and Heath (1999), among others. Rapid changes in library services and operations, demands for internal institutional accountability, and assessment expectations by external accrediting agencies have contributed to further development and application of user surveys within academic libraries during the past decade. User surveys can be designed and administered in a number of ways. Self-administered surveys are often employed to reach a large number of potential respondents with a minimum of direct contact and cost. Individuals are given or sent surveys to complete and return and the responses turned into data that can be analyzed. Surveys can range from broad and comprehensive to those narrowly focused on specific services or activities. When properly designed and administered, user surveys can provide both quantitative and qualitative data directly from the target population.

UNIVERSITY OF WASHINGTON LIBRARIES' SURVEY METHODOLOGY AND DESIGN

The University of Washington Libraries began an active program of assessing user needs, satisfaction, and the impact of library services and resources in 1992. Prior to this time, user input to the UW Libraries was generally informal and unsolicited through such channels as suggestion boxes and anecdotal comments from service desks. Other opportunities for user comment came through the Faculty Senate Council on University Libraries, a biennial meeting between subject selectors and faculty liaisons on collections-related issues and some earlier in-library surveys that focused on specific activities within the library unit. The catalyst for the development of a broad-based survey of faculty and students came from the UW Libraries' first strategic plan in 1991 that called for a user-centered approach to services. Specifically, the strategic plan recommended that the libraries: "Develop and implement a study to identify user populations, their information needs and how well they are being met" (University of Washington Libraries, 1991, p. 15).

The Task Force on Library Services was appointed by the Director of Libraries in late 1991 to design and implement a user survey that would provide information on the following:

* determine who users and potential users are;

* how and why the library is used (or isn't used);

* what sources are used for library-related information;

* what faculty and students' library-related needs are; and

* how satisfied faculty and students are with the libraries.

The literature on academic library user surveys available at the time of the early 1990s revealed a wide spectrum of applications and uses (see Association of Research Libraries, 1984, 1981, 1991; Van House, Weil, & McClure, 1990). Some common characteristics of these surveys were:

* distribution within the library to users was more prevalent than mailed surveys;

* focus on physical use of the library (e.g., "what did you do in the library today?");

* concentration on specific services (especially the online catalog); and

* interest in user satisfaction.

The task force designed the initial survey in 1992 in consultation with library staff and the University's Office of Educational Assessment (OEA). The decision was made early in the design process to survey all user groups, distribute the survey through the mail in order to reach potential nonusers, and provide similar survey content for each group to enable comparisons. The survey would be sent to all faculty and a random sample of graduate and undergraduate students. While distributing the survey to all faculty would increase costs, it would also facilitate survey promotion and publicity, obtain sufficient number of responses to do analysis by academic subject areas, and foster positive political outcomes.

Survey questions were similar for faculty and graduate students, with about 75 percent consistency between faculty and undergraduates. Adequate space was provided for survey respondents to write comments. Content evolved with each subsequent survey in 1995 and 1998, and some aspects of survey design changed. Rapid changes in library services and programs during the 1990s and usefulness of the data provided by some questions were prime factors in survey revision. However, there was a core group of questions in each survey that dealt with:

* information sources needed for research, teaching, and learning;

* reasons and frequency of library use;

* campus computer network connectivity;

* use of electronic resources;

* instructional needs and effectiveness;

* library unit use;

* satisfaction; and

* services availability or satisfaction.

The initial survey in 1992 was pilot tested in March with a group of faculty and students, revised, and then mailed mid-way through the Spring quarter to 3,900 faculty and a random, nonstratified, sample of 1,000 graduate and 1,000 undergraduate students (sample size was based on an expected 50 percent return rate). An incentive (entry into a drawing for bookstore gift certificates) was offered to students who returned completed survey forms. Two weeks after the initial surveys were mailed, students were sent a second survey form, while faculty were sent a reminder notice. Completed surveys were returned to the Office of Educational Assessment (OEA) who arranged for data entry. Data were made available in SPSS format and results were available in early September 1992.

Subsequent surveys in 1995 and 1998 generally employed a similar methodology and design. Survey design work began in January of each year, pilot testing took place in March, and surveys were mailed in late April to early May. The undergraduate sample was increased to 2,000 for 1995 and 1998, and the 1998 survey also included a specialized set of questions for faculty and graduate students in the biological and health sciences, and one for faculty and students in the fine arts. Focus groups were also held prior to the 1998 survey to provide input from users on their perception of issues and concerns. The bookstore gift certificate drawing was extended to all groups beginning with the 1995 survey. Reminder notices were sent in 1995 but not a follow-up survey form. In 1998, a survey accompanied the reminder letter. Both the cover letter and survey form included the name, phone number, and e-mail address of a librarian as a contact person for questions or clarification. The few questions received generally requested another survey be sent to replace a lost one.

Sending this type of survey to nearly 7,000 faculty and students is not inexpensive. Direct survey costs (not including library staff time) in 1998 totaled $19,000, about $7 per returned survey. Survey costs in 1992 and 1995 were about $12,000. The 1998 costs were distributed in the following manner: printing 30 percent; mailing 30 percent; data entry 30 percent; other 10 percent (consultation, incentives). Staff time for the 1998 survey was estimated at approximately 500 hours, including analysis and reporting.

LIBQUAL+

The UW Libraries was one of twelve libraries that participated in the ARL-sponsored LibQUAL+ pilot administered in Spring 2000 (Cook, Heath, & Thompson, 2000a, 2000b). Survey design and methodology were handled primarily by a team from Texas A&M where a SERVQUAL-based library survey had been used several times (Cook & Heath, 1999). In addition to the twenty-two basic SERVQUAL questions which covered the standard dimensions of accountability, assurance, reliability, responsiveness, and tangibles, nineteen additional questions were added to test two additional dimensions: access to collections and the library as place. Thus, there were forty-one questions that used the SERVQUAL three-column response format of minimum, perceived, and desired. Another fourteen behavioral questions, two on frequency of library use, and an overall service quality question were also added which used just one response column. The survey also collected demographic data.

The survey team at Texas A&M determined that the survey be administered to a random sample of 600 faculty, 600 graduate students, and 900 undergraduates at each institution based on an anticipated return of 200 surveys from each group. The UW Office of Educational Assessment extracted the sample from the faculty and student databases, and e-mail address lists created for each group were sent to the UW Libraries. The UW Libraries systems office created separate mailing lists for each group. A cover letter from the director of the UW Libraries was sent by e-mail to each participant. The letter included information about the survey and the university's reasons for participation, and also provided a URL address where respondents could complete the survey. The initial message was sent May 2 and a reminder notice was sent on May 11. Almost immediately after the initial e-mail notification was sent, there was a steady stream of messages back to the director and the local survey coordinator. LibQUAL+ implementation at the University of Washington ultimately generated more than fifty e-mail messages, most coming from faculty members. The messages fell into two basic groups: technical problems trying to complete the survey, and comments, usually negative, on survey design and content.

Direct expenses were $2,000 for the UW Libraries paid as a participant in the ARL project. This worked out to be about $5 per completed survey (excluding surveys from library staff). Library staff contributed about 150 hours to the project, including responding to e-mail messages, analysis, and report writing.

SURVEY RESPONSE AND REPRESENTATIVES Survey return rates for the 1992, 1995, and 1998 UW Libraries' surveys and the 2000 LibQUAL+ survey are shown in Table 1.

Table 1. Surveys Distributed and Returned

                Faculty               Graduate Students

Year     Sent   Returned    Rate    Sent   Returned    Rate

2000      600      128     21.3%     600     131      21.8%
1998     3750     1503     40.1%    1000     457      45.7%
1995     4400     1359     30.9%    1000     409      40.9%
1992     3900     1108     28.4%    1000     561      56.1%

              Undergraduates

Year      Sent   Returned    Rate

2000       900     137      15.2%
1998      2000     787      39.4%
1995      2000     489      24.5%
1992      1000     422      42.2%

(2000 is LibQUAL+ Web-based survey; 1992,1995,1998 are
UW Libraries mail surveys)

A second survey mailing appeared effective in raising the response rate as seen in the 1992 return rates for students and for all groups in 1998. The number of faculty surveyed varied according to criteria used to define the faculty pool, but all surveys included tenure track and research faculty as well as full-time lecturers. The overall response rate as shown is slightly understated as undeliverable surveys were not subtracted from the total sent out. Undeliverable survey rates ranged from approximately .5 percent of faculty to 2 percent of undergraduate students. Response rates to the LibQUAL+ survey were substantially lower. The definition of faculty was the same as used in the UW Libraries' 1998 survey. LibQUAL+ response rates were calculated by matching the number of completed surveys against the number of e-mail addresses to which the survey message was sent. Approximately 1 percent of these messages were undeliverable.

Representativeness of Survey Respondents

The large number of responses to the UW Libraries' surveys generated correspondingly large data sets, especially for the faculty survey. As Table 2 shows, the faculty survey respondent population in 1998 was reasonably representative of the population as a whole when grouped by broad subject areas. Faculty in the Health Sciences were slightly underrepresented, while those in the Humanities/Social Sciences/Fine Arts group were somewhat over-represented compared to the actual population. Response rates by academic schools ranged from 31 percent in Business to 54 percent in the Social Science departments within the College of Arts and Sciences.

Table 2. Faculty Population and Respondents by Academic Area,
1998 and 2000

                       2000LibQUAL+   1998 Faculty   1998 Survey
                       respondents    population     respondents
                       n=128          n=3750         n=1503

Health Sciences        43.0%          48.6%          44.7%

Sciences/Engineering   25.8%          26.2%          27.1%

Humanities/Arts/
Social Sciences        31.2%          21.0%          24.4%

Other (non-
Health Sciences)                       4.2%           3.8%

Graduate student responses (Table 3) were similar to the faculty with Health Sciences respondents again lower than their percentage of the actual population while those from Humanities/Arts/Social Sciences were slightly higher. Response rates by academic schools ranged from 24 percent in Dentistry and 28 percent in Education to 62 percent in Nursing and 72 percent in Social Sciences. Health Sciences does have a larger proportion of faculty and graduate/professional students located away from the main UW campus, and this may be a factor in the underrepresentation of respondents from those areas.

Table 3. Graduate/Professional Student Population and Survey
Respondents by Academic Area

                                  1998 Graduate
                    2000LibQUAL   Population      1998 Survey
                    n=131         n=8183(*)       Respondents

Health Sciences     22.1%         30.4%           26.0%

Sciences/
Engineering         28.2%         29.4%           30.2%

Humanities/Arts/
Social Sciences     45.0%         40.3%           43.8%

Other (non-
Health Sciences)    4.6%

(*) Law was not included in the 1998 survey. 1998 graduate
population with Law is 8,785, Law is 6.9% of total.

Determining how representative undergraduate respondents are is more complicated. Undergraduates in earlier UW Libraries' surveys appeared to identify with certain academic majors (business, engineering) before they were actually accepted into those programs, thus skewing responses by academic areas. Year in school appeared to be a more reliable measure. Table 4 shows undergraduate population and respondent population by year in school. Again, the respondent population is reasonably similar to the entire population. Freshmen were somewhat underrepresented in the 1998 survey and sophomores underrepresented in the LibQUAL+ survey.

Table 4. Undergraduate Student Population and Survey Respondents
by Year

                                              1998
                             1998 Undergrad   Undergrad
             2000 LibQUAL+   population       respondents
             n=137           n=23413          n=787

Freshman     22.6%           19.0%            14.4%

Sophomore    13.9%           18.3%            17.0%

Junior       27.7%           26.3%            26.9%

Senior       35.8%           32.9%            33.9%

Other
(including
5th year)                    3.4%             7.8%

Compared to the population as a whole, and UW survey respondents in 1998, the LibQUAL+ respondent pools, although significantly smaller than those generated by UW surveys, appear reasonably representative when grouped by broad academic areas for faculty and graduate students, and by class for undergraduates (for a discussion on representativeness and response rates, see Thompson, 2000). The underrepresentation of Health Sciences and overrepresentation by those in the Social Sciences mirrors the respondent population achieved in the large-scale UW Libraries' surveys. This is probably reflective of the way faculty and students in these areas use libraries as well as the larger proportion of the Health Sciences population located away from the main Seattle campus.

SURVEY RESULTS

Results from the UW Libraries' surveys provide an effective record of changes in the way that students and faculty used library and information resources during the 1990s. These results also documented significant variations within groups (i.e., between academic areas) and between groups (i.e., faculty and undergraduates) in some areas. Information from these surveys has been used extensively by the University of Washington Libraries to revise existing programs and services and promote new ones. Survey results showed:

* high satisfaction levels;

* a shift toward remote use and increased importance of electronic resources;

* continuing importance of libraries as place for students; and

* increased complexity of finding and using information for teaching, learning, and research.

Although the University of Washington Libraries' surveys and the LibQUAL+ survey differs substantially in design and content, it is interesting to compare results where questions were similar. The large respondent pool for the 1998 UW Libraries survey can serve as a benchmark for viewing the LibQUAL+ results based on a much smaller sample. For example, UW Libraries' surveys results revealed that faculty generally viewed the libraries through a collections-related focus, while undergraduate students placed a high value on the library as a place. Even though the questions and design in these surveys differ, would LibQUAL+ results also show similar responses? Results from the UW Libraries' survey in 1998 and the LibQUAL+ survey in 2000 will be compared in the areas of overall satisfaction, opening hours, collections importance, the library as a place, and remote use of library services and resources. These areas are often barometers of service quality.

Overall Satisfaction

Responses to overall library satisfaction questions on the 1998 survey showed faculty had the highest satisfaction while undergraduate students the lowest (see Table 5). The LibQUAL+ survey phrased the questions as overall quality of services but still produced similar results although the difference in Likert scales (1 to 5 in UW Libraries, 1 to 9 in LibQUAL+) can lead to a different type of response. For LibQUAL+, the differences between undergraduate scores and graduate and faculty ones were significant at the .10 level using t-tests.

Table 5. Overall Satisfaction: UW Libraries 1998 Survey and
Libqual+ 2000 Survey by Group.

                 LibQUAL+   1998      LibQUAL         1998 Grad
                 Faculty    Faculty   Grad students   students

Very satisfied   78.0%      91.3%     80.9%           84.9%

Satisfied        21.2%      7.8%      18.3%           13.5%

Not satisfied     0.8%       0.9%      0.8%            1.6%

Mean Score        7.17       4.33      7.13            4.11

                  LibQUAL      1998
                  Undergrads   Undergrads

Very satisfied    72.9%        78.5%

Satisfied         24.0%        19.9%

Not satisfied      3.1%         1.6%

Mean Score         6.88         3.99

Very satisfied on LibQUAL+ are percentage respondents choosing 7-9
on scale of 1-9; UW 1998 survey are those choosing 4-5 on scale of
1 to 5. Satisfied on LibQUAL+ are percentage respondents choosing
443 on scale of 1-9; UW 1998 survey satisfied are those marking 3
on a scale of 1 to 5. Not satisfied on LibQUAL+ are percentage
respondents choosing 1-3 on scale of 1-9 and UW choosing 1-2.

Library Hours

The 1998 survey asked whether libraries were open when needed on evenings, weekends, summer, and interim periods. Graduate students, as the case with the previous two surveys, had the lowest satisfaction with hours while faculty had the highest, as shown in Figure 1. Undergraduate student satisfaction had slipped from 1995 when it was similar to faculty satisfaction. When asked to choose from a list of priorities, more than 37 percent of graduate students and 42 percent of undergraduates chose increased library hours as a priority compared to 17 percent of faculty. Graduate students also wrote more comments about hours than any other group.

[GRAPH OMITTED]

The LibQUAL+ question was concerned about expectations and perceived level of service related to whether the library had convenient business hours (which is a somewhat different question from that used in the UW Libraries survey). As Table 6 shows, graduate student minimum expectations for convenient business hours exceeded their perception of library performance. Faculty minimum expectations were lower, and the gap between expectations and performance, while small, was positive. Undergraduate students had both the highest perceived value (7.13) and most positive gap (.63). One likely reason for this positive response in LibQUAL+ was the Undergraduate Library's move to 24 hour opening in autumn 1998.

Table 6. LibQUAL+ Convenient Business Hours.

                   Faculty   Faculty      Grad      Grad
                   Minimum   Perceived    Minimum   Perceived
8. Convenient
Business Hours     6.45      6.56 (.11)   6.68      6.63 (-.05)

                   Undergrad   Undergrad
                   Minimum     Perceived
8. Convenient
Business Hours     6.5         7.13 (.63)

Means of minimum service level and perceived level of service
are shown along with mean gaps between the two in parentheses.

Collections

A valuable part of each one of the UW Libraries' surveys is asking users to identify their library priorities from a list of ten to twelve choices. The list of potential priorities is compiled from comments provided on the pretests as well as areas the libraries are interested in. The first survey in 1992 showed that all three groups had the same priorities in their top three choices (build collections, network bibliographic databases, and improve the online catalog), while in 1998 the top three priorities for undergraduates were different than those for graduate students and faculty. Faculty, in particular, showed an almost exclusive focus on collections/information resources-related areas as shown in Table 7.

Table 7. 1998 Survey: Library Priorities by Group.

                                             Graduate
                                   Faculty   Students   Undergraduates

Maintain quality of
  print collections                 69.6%     52.3%         28.2%

Deliver full-text to
  your computer                     60.4%     55.8%         34.4%

Deliver bib databases
  through the Web                   52.1%     40.0%         17.7%

Provide reserves electronically     18.8%     36.6%         50.8%

Add more computers
  in the library                    8.6%      20.1%         54.5%

Provide training in use
  of Web/library resources          28.3%     27.4%         47.5%

Increase library hours              17.0%     37.5%         41.9%

Preserve library materials          40.0%     34.9%         24.7%

Add group study/seminar rooms                               35.5%

Provide consultation on
  how to do library research                  13.3%         39.8%

(All priorities listed that received at least 30% from any one group)

The LibQUAL+ survey reinforced the primacy of collections and information resources for faculty. This was the only area where faculty had negative gap scores (resources added to the collection on request, fulltext delivered electronically to desktop, and complete runs of journal titles). Faculty minimum expectations were generally higher as well. Graduate student results also showed high expectations for collections-related questions and negative gaps for full-text delivered electronically and complete journal runs. Undergraduate students had lower expectations and positive gaps in all collections related areas.

Reviewing mean scores for six collections-related questions on LibQUAL+ showed higher mean scores for faculty and graduate students in minimum expectations (see Table 8). However, the differences between undergrads and faculty were not significant at the .05 or. 10 level according to t-tests. There were differences at the .05 level between grad students and undergraduates in complete runs of journal titles and comprehensive print collections, and at the .10 level for timely document delivery and interlibrary loan.

Table 8. LibQUAL+ Collections Expectations/Perceptions.

                                         Faculty   Faculty
Question                                 Minimum   Perceived

10. Resources added on request           6.23      6.10 (-.13)
17. Timely document delivery/ILL         6.41      6.82 (.41)
25. Full text delivered electronically   6.32      5.84 (-.48)
27. Comprehensive print collections      6.30      6.64 (.34)
36. Interdisciplinary needs addressed    5.89      6.44 (.55)
37. Complete journal runs                6.52      6.40 (-.12)

                                         Grad      Grad
Question                                 Minimum   Perceived

10. Resources added on request           6.04      6.27 (.23)
17. Timely document delivery/ILL         6.61      7.22 (.61)
25. Full text delivered electronically   6.20      6.16 (.04)
27. Comprehensive print collections      6.63      6.65 (.02)
36. Interdisciplinary needs addressed    6.24      6.55 (.31)
37. Complete journal runs                6.77      6.58(-.19)

                                         Undergrad   Undergrad
Question                                 Minimum     Perceived

10. Resources added on request           5.97        6.37 (.40)
17. Timely document delivery/ILL         6.31        6.88 (.57)
25. Full text delivered electronically   5.96        6.24 (.28)
27. Comprehensive print collections      6.14        6.98 (.84)
36. Interdisciplinary needs addressed    6.05        6.65 (.60)
37. Complete journal runs                6.22        6.61 (.39)

Means of minimum service level and perceived level of service are
shown along with mean gaps between the two in parentheses.

Library As Place

A consistent theme revealed through each of the UW Libraries' surveys has been the different perspectives of faculty and students on the library as a place. Faculty use of the library is primarily collections driven, while students view the library as a place to do work, including finding and using information resources. This difference shows up dramatically in responses to a 1998 survey question on reasons for visiting the library (Figure 1) as well as other questions dealing with priorities and needed services.

LibQUAL+ clearly showed similar differences between faculty and students on the library as a place (see Table 9). On seven questions related to the library as place, the differences in minimum expectations were significant at the 0.01 level except for "safe and secure space." With the exception of "secure and safe space," faculty minimum expectations were generally below 5 and gaps between minimum and perceived were larger than 1.0. While graduate student expectations were higher than those of faculty, they were still lower than those of undergraduates. Both student groups were concerned about quiet study areas.

Table 9. LibQUAL+ Library as place.

                                               Faculty   Faculty
Question                                       Minimum   Perceived

14. Comfortable and inviting location          5.08      6.12 (1.04)
21. Secure and safe place                      6.77      7.51 (.74)
22. Center for intellectual interaction        4.16      5.31 (1.15)
29. Space for group/individual study           4.57      5.94 (1.34)
30. Haven for quiet and solitude               4.74      6.02 (1.28)
40. Space that facilitates quiet study         4.69      5.75 (1.06)

                                               Grad      Grad
Question                                       Minimum   Perceived

14. Comfortable and inviting location          5.72      6.12 (.40)
21. Secure and safe place                      6.81      7.26 (.45)
22. Center for intellectual interaction        5.20      5.79 (.59)
29. Space for group/individual study           5.90      6.07 (.17)
30. Haven for quiet and solitude               6.12      6.08(-.04)
40. Space that facilitates quiet study         6.23      6.30 (.07)

                                               Undergrad   Undergrad
Question                                       Minimum     Perceived

14. Comfortable and inviting location          5.98        6.73 (.75)
21. Secure and safe place                      6.80        7.23 (.43)
22. Center for intellectual interaction        5.75        6.24 (.49)
29. Space for group/individual study           6.20        6.78 (.58)
30. Haven for quiet and solitude               6.53        6.46(-.07)
40. Space that facilitates quiet study         6.52        6.62 (.10)

Means of minimum service level and perceived level of service are
shown along with mean gaps between the two in parentheses.

Remote Use

The 1995 UW Libraries' survey revealed for the first time that, among faculty who said they used the library at least weekly, more were doing so remotely than physically visiting the library. This trend continued in 1998, and Table 10 shows degrees of remote use among all groups both in the 1998 survey and LibQUAL+. Responses to other survey questions in 1998 revealed that more than 97 percent of faculty had access to the Web through a desktop computer. While remote use is not itself a measure of service quality, this information is critical for planning and delivering electronic services and resources.

Table 10. Type and Frequency of Library Use Among Faculty and
Students Who Use Library at Least Weekly.

                                           Visit Remotely
                        Visit in Person    Using Computer

LibQUAL+ Faculty        54.5%              81.1%
1998 Faculty            47.3%              73.4%
LibQUAL+ 1998 Grad      77.0%              80.2%
1998 Grad               77.9%              66.1%
LibQUAL + Undergrad     66.7%              48.8%
1998 Undergrad          70.3%              45.5%

In general, LibQUAL+ results tended to correlate with results from the libraries' surveys which had a much larger number of respondents for each group, especially faculty. Differences between groups, especially faculty and undergraduates, that were evident in earlier UW Libraries' surveys, were also found in the LibQUAL+ results.

Subgroup Analysis

One of the benefits of a large respondent pool is the ability to do analysis on differences within the group. While there may be a set of similar characteristics that define a group, there may also be significant variation within that group. Academic user communities are not homogeneous in the way they use libraries nor in their needs for library resources and services. In addition to differences between faculty and students, there may also be significant differences between those in different academic areas or by gender or some other demographic component. These have important implications for identifying user needs, concerns, and issues that may be missed in analyzing aggregate results.

The number of respondents to the UW Libraries' 1998 survey, especially for faculty, was sufficiently large to examine the degree of variation within the group and between subject areas. The sample size and response rates of the LibQUAL+ survey generally precluded analysis by academic area and made it difficult to find differences among demographic characteristics. Increasing the LibQUAL+ sample size and the response rate would provide larger data sets that could be used to examine variation within and between groups. Factors related to the low response rate for LibQUAL+ include survey length, complexity (e.g., the three-column response), perceived redundancy, technical problems, behavioral issues associated with Web-based surveys, and how related survey content is to actual library use and issues of the respondent. While the LibQUAL+ sample appears representative of each group as a whole, it is not large enough to perform subgroup analysis.

How important is it to have a respondent pool large enough to do analysis at the subgroup level? UW Libraries' surveys have consistently shown significant differences in how faculty from different academic areas use libraries and in their needs for library resources and services. They also show surprising uniformity in areas such as connectivity and remote use where differences might be expected based on traditional use patterns. Although graduate student response numbers are lower, they were similar to faculty in the same academic areas.

Satisfaction responses, while showing some variation, generally do not differ significantly by academic area. Priorities, on the other hand, clearly do as shown in Figure 2. The 1998 results showed significant variation among academic areas in the top four overall priorities, especially for delivery of full text to the desktop and preservation.

[GRAPH OMITTED]

Variation within undergraduate responses to the UW Libraries' 1998 survey were more difficult to determine. There were some gender differences, especially in areas related to computer access and library instruction as well as differences by class year. Figure 4 shows the importance of UW Libraries and the World Wide Web to the work of undergraduates by year in school. The differences between first year and upper division students were significant at the .05 level using a simple t-test. However, LibQUAL+ survey results did not show any statistical differences in responses either by gender or class year.

[GRAPH OMITTED]

The importance of different resource types such as journals, books, bibliographic databases, and foreign language materials also showed significant variation between academic areas. When asked to rank these resource types on a scale of 1 (not important) to 5 (very important), faculty in all academic areas ranked journals as very important (ranging from 4.37 in Fine Arts to 4.97 in Dentistry).

A large number of responses also enables analysis within smaller subgroups. For example, the 1998 faculty survey had 241 responses from those in science departments. Table 11 shows mean scores by department on responses to questions dealing with type and frequency of library use and importance of resource types. Only departments with at least twenty responses were included.

Table 11. 1998 Survey (Faculty). Importance of Resource Types.

Department                   Journals    Journals   Bibliographic
(response)           Books    > 1980      < 1980      Databases

Chemistry (36)*      4.28      4.83        3.92         4.39
Geology-
Geophysics* (22)     4.54      4.91        4.27         4.18
Math-Stat (37)**     4.89      4.89        4.57         3.51
Physics (39)*        4.21      4.56        4.08         3.69
Psychology (24)      4.13      5.00        3.75         4.42
Zoology (27)         4.11      5.00        4.30         4.63
All Science (241)    4.26      4.76        4.09         4.05

(Mean scores, scale of 1 not important to 5 very important)

While current journals are important to all groups, the importance of books, earlier journals, and bibliographic databases showed variation within each group. Understanding the importance of these resource types to different subject areas is useful in allocating the collection-development budget as well as making decisions on what materials to house on-site or in storage.

There were also significant differences in the frequency of physical visits to the library by science faculty. Perhaps it is not surprising that distance from the library appeared to play an important role. Not only did the departments located closest to their primary library visit those libraries more often, the frequency of their physical visits exceeded the frequency of library use from an office computer.

A comparison between large-scale user surveys done by the UW Libraries and the LibQUAL+ survey administered to UW faculty and students shows good agreement in population representation and in broad result categories at the group level. However, the ability to do subgroup and intragroup analysis can provide valuable results and efforts to increase the number of responses if the Web-based LibQUAL+ survey should continue.

ASKING THE RIGHT QUESTIONS

Whether the survey results are statistically reliable, representative, valid, or significant, doesn't necessarily mean that they provide information that can be used to assess and improve library service quality. It is also important to examine whether these surveys are asking the right questions in the right way to the right group. Survey design is a complex and evolving process that requires substantial interaction between the surveying group and the surveyed population. At many large academic research institutions, user communities are diverse and differ in their needs for library resources and services. It is essential to recognize that these differences exist when designing and administering surveys. Undergraduate students, based on their understanding or experience, may respond quite differently from faculty to some questions, making it difficult to do cross group comparisons. It is also important to remember that surveys are just one method of acquiring user input. While surveys offer the prospect of obtaining quantifiable data from large populations at reasonable costs, they need to be employed in the right situation. Surveys should be designed from the user perspective. Questions should be short, simple, and clear to the user. Complex issues may be better addressed using other techniques. There should be sufficient motivation for faculty and students to take the time to complete a survey.

The evolution of SERVQUAL to LibQUAL+ is a positive step. The ability to move away from the twenty-two question SERVQUAL core package to a design that provides a library focus, and perhaps a simpler format, is welcome. Grounding the survey based on user-provided information on library needs and use is critical to maintain currency and relevancy (Cook & Heath, 2000). Such qualitative data obtained at regular intervals enables the library to keep on top of user issues and concerns. The library and information environment is changing rapidly. The continued growth in remote use of library services and resources and in user self-sufficiency calls for new ways to measure user needs and library performance that can be done quickly, inexpensively, and flexibly enough to catch environmental changes. The ARL New Measures Initiative plans to provide libraries with tested tools that can help provide information that will assist in meeting these challenges.

The underlying concept of developing a standard instrument to measure service quality across libraries is a powerful one and certainly one deserving institutional support. However, it cannot supplant local efforts to work closely with faculty and students to assess user needs and library collections and services. There are local issues at each institution that probably cannot be effectively addressed in a standardized survey tool.

The University of Washington Libraries expects to continue both its participation in LibQUAL+ as well as utilizing a variety of ways to assess user needs and library performance, including the deployment of locally based large-scale user surveys.

REFERENCES

Association of Research Libraries. Office of Management Services. (1994). User surveys in ARL libraries (a SPEC Kit compiled by Elaine Brekke. Systems and Procedures Exchange Center) (SPEC Kit 205). Washington, DC: ARL.

Association of Research Libraries. Office of Management Services. (1988). User surveys (SPEC Kit No. 148). Washington, DC: ARL.

Association of Research Libraries. Office of Management Services. (1984). User studies in ARL libraries (SPEC Kit No. 101). Washington, DC: ARL.

Association of Research Libraries. Office of Management Services. (1981). User surveys and evaluation of library services (SPEC Kit No. 71). Washington, DC: ARL.

Cook, C., & Heath, F. (2000). User's perceptions of library service quality: A "LibQUAL+" interview study. Unpublished paper presented at the Association of Research Libraries Measuring Service Quality Symposium on the New Culture of Assessment, Washington, DC, October 2000.

Cook, C., & Heath, F. (1999). SERVQUAL and the quest for new measures. ARL Newsletter: A Bimonthly Report on Research Library Issues and Actions from ARL, CNI, and SPARC, 207, 12-13.

Cook, C.; Heath F.; & Thompson, B. (2000a). A new culture of assessment: Preliminary report on the ARL SERVQUAL survey (66th IFLA Council and General Conference, Jerusalem, Israel, 13-18 August). Retrieved November 30, 2000 from the World Wide Web: http://www.ifla.org/IV/ifla66/papers/028-129e.htm.

Cook, C.; Heath E; & Thompson B. (2000b). LibQUAL+: One instrument in the new measures toolbox. ARL Newsletter: A Bimonthly Report on Research Library Issues and Actions from ARL, CNI, and SPARC, 212, 4-7.

Cook, C.; Heath F.; & Thompson, B. (2000). A meta-analysis of response rates in Web or Internet-based surveys. Educational and Psychological Measurement, 60(6), 821-836.

Cook, C., & Thompson, B. (2000). Reliability and validity of SERVQUAL scores used to evaluate perceptions of library service quality. Journal of Academic Librarianship, 26(4), 248-258.

Hernon, P., & Altman E. (1996). Service quality in academic libraries. Norwood, NJ: Ablex.

Hernon, P., & Altman, E. (1998). Assessing service quality: Satisfying the expectations of library customers. Chicago: American Library Association.

Hernon, P., & McClure, C. (1990). Evaluation and library decision making. Norwood, NJ: Ablex.

Hernon, P., & Whitman, J. (2001). Delivering satisfaction and service quality: A customer-based approach for libraries. Chicago: American Library Association.

Lakos, A. (1998). Building a culture of assessment in academic libraries--obstacles and possibilities (Living the Future II Conference, Tucson, Arizona, April 22, 1998). Retrieved November 30, 2000 from the World Wide Web: http://www.lib.uwaterloo.ca/~aalakos/Present/Arizona98/Ariztext98.html.

Nitecki, D. (1996). Changing the concept and measure of service quality in academic libraries. Journal of Academic Librarianship, 22(3), 181-190.

Nitecki, D., & Franklin, B. (1999). New measures for research libraries. Journal of Academic Librarianship, 25(6), 484-487.

Thompson, B. (2000). Respresentativeness versus response rate: It ain't the response rate! Unpublished paper presented at the Association of Research Libraries Measuring Service Quality Symposium on the New Culture of Assessment, Washington, DC, October 2000.

University of Washington Libraries. (1991). University of Washington libraries mission and strategic plan. Seattle: University of Washington Libraries.

University of Washington Libraries. (2000). UW Libraries assessment page. Retrieved November 30, 2000 from the World Wide Web: http://www.lib.washington.edu/surveys/.

Van House, N.; Weil, N.; & McClure, C. (1990). Measuring academic library performance: A practical approach. Chicago: American Library Association.

Wilson, L. (1995). Building the user-centered library. RQ 34(3), 297-302.

Steve Hiller, Natural Sciences Library, Allen Library South, Ground and First Floors, Box 352900, University of Washington, Seattle, WA 98195-2900

STEVE HILLER is Head, Science Libraries and Library Assessment Coordinator, at the University of Washington Libraries. Since 1991 he has worked extensively on library assessment issues at the University of Washington and has discussed the assessment process at a number of regional and national meetings and workshops.