Innovative United Kingdom Approaches to Measuring Service Quality.

ABSTRACT

THIS ARTICLE REPORTS ON APPROACHES TO MEASURING the service quality of academic libraries that are innovative in the United Kingdom. Some of them will, it is hoped, also be innovative in the United States. The discussion is also intended to draw out particular themes where there are marked similarities and differences between the two countries. After a brief introduction to the UK national background of quality measurement, the article deals with four topics--measurement frameworks, better use of statistics, benchmarking, and measuring user satisfaction--before offering some suggestions about likely future developments.

THE UNITED KINGDOM QUALITY MEASUREMENT BACKGROUND

Despite widespread adoption of quality frameworks such as ISO 9000 or the various "Quality Award" systems in commerce, public services in the United Kingdom have, for the most part, not followed this lead. Yet there is growing pressure from national government and customers for accountability interpreted as the high quality services sought by customers. This disjunction is perhaps partly responsible, along with natural tendencies for a socialist government, for a growing culture of government bureaucratic enforcement of performance measurement for public services. This is typified by the setting of compulsory government-prescribed performance indicators, required comparison between similar organizations (e.g., different local councils or different schools), and compulsory publication of results to both customers and national government.

As an example of this, we can look at recent draft proposals for twenty-five compulsory performance measures for public libraries. The public library service is the legal responsibility of the district or county council. But the legal framework controlling public libraries is set by national government. The 1964 Public Libraries Act required the provision of "comprehensive and efficient" public library services. But the definition of this was vague until the 1990s. In 1993, five compulsory performance measures were set, and each library authority must publish its results annually together with comparative results. From 1998, authorities were further required to submit a formal annual plan to the central government. The plan, among other requirements, must indicate how the authority will improve performance on the standard measures. It is now proposed (summer 2000) to extend the set of measures to twenty-five. A new concept is also proposed of a "target" level of performance and an "intervention point" for each measure. Frequently the intervention point is set at the level currently achieved by 50 percent of authorities while the desired level is that achieved by the top 25 percent, so there is a clear agenda of improvement as well as measurement. Some examples are given in Figure 1.

Figure 1. Examples of Twenty-Five Draft Public Library Performance
Measures.

     Measure            Target                 Intervention Point

* User satisfaction    95% good+              90% good+
  with helpfulness
  of staff

* Active borrowers     45% of relevant        30% of relevant
                       population             population

* Opening hours        45 hours per           45 hours per week
                       week for large         large branch
                       branch

* Reservation          50% in 7 days          35% in 7 days
  turnround

* Bookspend            3,500 [pounds          1,970 [pounds sterling]
                       sterling] per 1000     per 1000
                       population             population

* IT workstations      0.7 per 1000           0.35 per 1000
                       population             population

The strong role of national government in the United Kingdom reflects the political framework where most power and tax resources are held by the national government and the regional or local governments have their powers, and most of their funding, set by national government. In the United Kingdom, the national government can close a poorly performing school or remove the right of a local authority to run its schools if there is evidence of poor management. This is arguably the reverse of the U. S. situation, where the national federal government has only the powers ceded to it by the states, and local services are locally funded and controlled. So here is a first key difference between the United States and the United Kingdom. But there is also a key similarity in the generally growing pressure for more and better performance measurement and pressure for improved service quality.

ACADEMIC LIBRARY QUALITY MEASUREMENT FRAMEWORKS

Since higher education in the United Kingdom is predominately publicly funded through government agencies, it is no surprise that there is here, too, a sharp-edged and bureaucratic framework for the measurement of performance and service quality. Academic quality is competitively audited through three processes covering research, teaching, and quality assurance.

The level of core research funding for each university depends on a four-yearly "Research Assessment Exercise." For each of some seventy subject areas, universities and colleges are invited to submit a report on the productivity of the last four years and a plan for the next four. The supporting evidence required includes details of publications, project-specific research grants obtained, and other data. The submissions are peer-reviewed by committees of subject experts who assess the quality of research performed and likely future productivity against a seven-category scale ranging from "poor" to "major international significance." The resulting funding is based on the number of researchers and their overall performance as a group. The funding per researcher is zero for the two lowest categories, and from one to some five times the minimum amount for the higher categories.

The quality assurance process follows an audit model. The institution sets out its processes. An inspection panel then, once every five years, visits and checks whether the quality assurance processes are effective and makes suggestions for improvements.

The area which has the most direct effect on university libraries is the process for "Teaching Quality Assessment" (TQA). This is again currently based on periodic review (every six years) by a visiting inspection team for each subject area taught. There are approximately forty-two subject areas. Each subject visit to each institution results in a published report incorporating gradings against six "aspects" of teaching quality. The gradings go from one (fail)to four (excellent) for each aspect.

Low rating on any aspect results in the threat of closure of the courses concerned and withdrawal of funding for them unless satisfactory remedial action is taken within twelve months. In theory, there is no overall grade but, of course, in practice, institutions and compilers of league tables cannot resist totaling the scores to give an overall grade (e.g., "twenty-three out of twenty-four" or 72 percent) and then aggregating scores for all the subjects reviewed to give some kind of overall teaching quality comparative metric. It seems likely that very few prospective students, parents, or employers ever read the individual reports. Any public relations outcome, good or bad, is also suffused within many other factors (mainly prejudice!). But the system has undoubtedly focused minds very hard on achieving holistic quality of the student experience. Over time, average scores have risen. Cynics ascribe this entirely to growing expertise at "playing the system." There is no doubt about some of this, as well as fewer tactical errors by institutions making claims they could not substantiate, but there has actually been objective improvement, too.

Improvement can readily be demonstrated by reference to the library aspects of Teaching Quality Assessment. Library services are reviewed under a broader "Learning Resources" aspect, which also covers teaching facilities, laboratories, any departmental libraries, and so on. The guidance to assessors on reviewing libraries boils down to three questions:

1. Is there an overall learning resources strategy consistent with the course aspirations?

2. Are library services available, accessible, and appropriate in terms of ... stock, study space, induction, opening hours, and user support?

3. Is there effective liaison with subject staff?

The answer is based prominently on student feedback, also on assessors' own judgments, evidence offered, and conversations with library and teaching staff. The assessors "triangulate" the answers from each source and are particularly sharp about inconsistencies and whether the "learning resources" answers fit with what they have been told about curriculum, student workloads, and so on.

The Standing Conference of National and University Libraries (SCONUL) has been involved in shaping the process through lobbying the (successive) agencies concerned over the last eight years and offering advice and guidance (sometimes accepted) to improve on the early poor and patchy handling of library issues. Specifically, the official guidance to assessors reflects and is supplemented by a "SCONUL aide-memoire" which fills out for assessors the three basic questions and suggests what kinds of answer might be acceptable. There has also been a significant local effect. The impending arrival of a subject review significantly increases academic keenness to talk seriously and systematically to library colleagues, and libraries can build on this by using the SCONUL aide-memoire as a kind of script for these discussions. SCONUL has helpfully provided libraries with a further aide-memoire which fleshes out the kind of answers we believe assessors will be looking for. These answers are not prescriptive or number-based. The core is to look at how effective liaison, resourcing, and monitoring takes place. Finally, in this area, SCONUL continues to monitor how the process works and what the reports say for evidence of progress or backsliding.

In terms of library-specific measurement frameworks, the most complete is "The Effective Academic Library" (Joint Funding Councils Ad-hoc Group on Performance Indicators for Libraries, 1995). This was drawn up in response to a recommendation in the 1993 Follett Report (Joint, 1993) that "a coherent and generic" set of performance indicators for academic libraries should be developed. It might be argued that this was a classic piece of buck-passing from a report which made its major contributions in respect of obtaining national funds for library buildings and the development of electronic library services.

"The Effective Academic Library" takes a broad approach, incorporating a mixture of numerical and other indicators, formulated sometimes nationally and sometimes locally. In all, the report suggests thirty-three indicators split into five facets of performance. The five facets are:

* Integration (relevance to institution)

* User satisfaction

* Delivery (meeting targets; output)

* Efficiency

* Economy

An initial consultation with vice-chancellors (who, of course, mostly passed on the task to their librarians) produced many long replies and no consensus. If there was a typical response, it ran something like "There are too many indicators here.... please add one on...." After a year's pause, the agencies which fund higher education passed the issue to the Committee of Vice-Chancellors and Principals, who passed it on to SCONUL. Later sections of this article will describe the outcomes, but it might be argued that at least the beginning of a "coherent and generic" set of indicators has now been established.

There has been work in the United Kingdom on two other issues which bear on the framework question. One is the notion that there are multiple stakeholders with different performance measurement requirements. For example:

* End-customers: students

* Service purchasers: academic departments, institutions

* Funders: funding councils, government, the taxpayer

* Guardians of quality: QAA, professional bodies

* Service managers

* Staff

John Crawford of Glasgow Caledonian University has been particularly active in carrying out research to identify the key issues for different groups and assess how far they overlap. This is an important insight, often overlooked and often responsible for fruitless debate about which are the right indicators to use. Which are right depends, of course, on the audience and the purpose.

An acknowledged omission from "The Effective Academic Library" were any indicators for electronic services. There is no need here to recap the difficulties in achieving this. Suffice to note that everyone wants indicators, and no one has satisfactory answers. SCONUL has tackled this issue by relying on the work of funded researchers on United Kingdom and European Community projects. These are currently coming to fruition through the EQUINOX project. EQUINOX combines a suggested set of twelve electronic indicators with a software package designed to demonstrate the linkages between each indicator and library (and institutional) objectives. The full set of indicators are shown on the Web site. Examples include: percentage of target population reached by electronic services; number of "sessions" per head; cost per session/ document delivered; and percentage of activity which is electronic. Currently there seems to be some diverging of the paths between the European researchers and their equivalents in the United States. It is hoped that this gap can be closed again. In any event, SCONUL libraries (as well as libraries in some other European countries) are currently testing the EQUINOX products.

BETTER USE OF THE STATISTICS

Like their American cousins, United Kingdom academic libraries have been collecting statistics for many years. But it has to be admitted that, in 1995, there were still many failings. The statistics had achieved little recognition outside libraries. Analysis, presentation, and interpretation of the data were all poorly developed. Several attempts at a conceptual framework had failed to achieve acceptance. Overall, impact had been limited, particularly given the effort that has been spent over the years. The SCONUL Advisory Committee on Performance Indicators (ACPI), which has responsibility for this area, determined a number of steps to try to move forward, including:

* use of a professional statistical agency: Library and Information Statistics Unit (LISU), University of Loughborough;

* empirical testing of theory: the Cranfield Study;

* getting "official" recognition--HELMS (Higher Education Library Management Statistics);

* electronic submission of data; and

* joint work with university IT directors.

The United Kingdom is fortunate to possess a grant-funded specialist agency devoted to library statistics--LISU. For SCONUL, LISU has taken over data input and storage, created a ten-year database, and cleaned the data, filling some gaps and correcting obvious errors. It has then begun publication of an annual discursive "trends" volume (Standing Conference of National & University Libraries, 2000, for the latest issue), offered a customized statistical comparison service to individual libraries (allowing libraries to have created, to their individual specification, a selective set of results for selected comparator institutions), and is a useful source of expert statistical advice and data in electronic form. Figure 2 shows the kind of table which is included in the "trends" volume, giving comparisons over time and between broad groups of institutions. Figure 3 gives an example of the kind of local data that can be generated.

[GRAPHS OMITTED]

At Northumbria, the library has become concerned that use of conventional study seats is gradually falling off, leaving the facility with a growing waste of space. A two-week survey showed that, during this period, utilization of study seats barely extended beyond 50 percent at any time, and the average was around 30 percent. It happens that one of the SCONUL data series is based on counts of occupation of study seats on specified sample days of the year. Figure 3 shows Northumbria's results against the national average. Both show a gradual lowering of occupancy over seven years. The Northumbria decline is more rapid. This helps to confirm that it would not be imprudent for the library to consider removing some study seats in order to create badly needed shelf-space--or IT seats--or possibly return to the university some unneeded space.

The second strand in better use of statistics was to undertake some empirical testing of the alternative measures about which debate sometimes takes place: whether to use gross student numbers or numbers of students and academic staff as a divisor in ratios, and whether to introduce weighting of any kind; which output measures discriminate most usefully? John Blagden, former chair of ACPI, obtained research funds to employ a research assistant for one year to test the quality of the SCONUL data and explore the discriminatory power of various measures proposed in "The Effective Academic Library" and other international sources. The project was successful in answering many of the arguments and in generating a genuinely small set of proposed indicators which, after review by a group of university heads, have led to a new annual publication, "UK Higher Education Library Management Statistics" (HELMS), aimed at university administrators (Standing, 1999). This publication incorporates a number of new features for the United Kingdom. All institutions, not just members of certain library organizations, are included in principle--though not all choose to supply data. The library data are brought together with relevant data supplied by the official government agency for data about universities and colleges--the Higher Education Statistics Agency (HESA)--and is presented using templates supplied by HESA. Contact with HESA and university heads has led to the use of the term "management statistics" rather than measures or indicators and to the separation of two sets of "contextual data" about the libraries and the institutions from the management statistics. There are currently seven "Library Management Statistics" (FTE = full-time equivalent):

Input Measures:
  Total library expenditure per FTE user
  Expenditure on information provision per FTE user
  Expenditure on staffing per FTE user

Output Measures:
  Seat hours offered per week per FTE user
  Loans per FTE user
  (In the future: Stock on loan; electronic services; user education)
  Interlibrary loans as a percentage of all loans

There are six items of library contextual data, designed mainly to give an idea of scale of operation:

1. Number of libraries

2. Space occupied

3. Size of collection

4. Number of seats

5. Number of workstations

6. Total library expenditure

Finally, there are five items of institutional context data, designed mainly to give background on size and emphasis between teaching and research:

1. Number of FTE students

2. Percentage of postgraduate students

3. Percentage of part-time students

4. Number of academic and research staff

5. Government research funding as a percentage of all government funding

There is some evidence that this new approach has attracted the interest of some university heads. Depending on the circumstances, this interest may, of course, be felt as beneficial or otherwise. But it must be progress to feel that at least some of the decisions and judgments are partly based on data in which we have reasonable faith.

In the second year, graphic presentation has been added. In the future, we hope to improve coverage of institutions and to increase the output indicators to cover areas such as library instruction and electronic services. Figure 4 gives an example of the graphic presentation.

[GRAPH OMITTED]

Within the graphics, initial attempts have also been made to relate inputs to outputs, not a common feature of library statistics. Figure 5 plots an output (loans per year per user) against an input (total expenditure per user). It is often assumed that there will be some correlation. Initial inspection of this graph does not offer any confirmation.

[GRAPH OMITTED]

Loans are only one output, and the graph produces more questions than answers. But this is surely the kind of analysis which is long overdue alongside the ritual demonstrations that resources are not keeping up with costs and user numbers.

BENCHMARKING

The SCONUL Benchmarking Pilots Project, 1997-99 (Town, 2000), has been a successful attempt to apply standard benchmarking approaches to libraries, rather as the Association of Research Libraries is building on SERVQUAL. Led by Stephen Town of Cranfield University, the project has set up six volunteer self-selected groups of two to five partners, each exploring a particular area of library service. Two have focused on advice desks, two on library skills training, and one each on "counter services" and the library environment. The project has based its work on detailed analysis and comparison using a wide variety of measurement techniques. Figures 6 and 7 give much simplified and compressed overviews of the methods and outcomes relating to aspects of two of the pilots: enquiries and physical space.

Figure 6. Case Study Example 1.

* Counter Services: Enquiries

- Critic al Success Factor: provide accurate answers to personal
queries

- End product: Correct answer

- Processes

- Benchmarks and method of testing:

  * Is service clear to customer? (Visit)

  * Answers accurate? (Mystery shopper)

  * Referral process? (Staff questionnaire)
Figure 7. Case Study Example 2.

* Library Environment: Physical Space

- Critical Success Factor: space works appropriately

- End product: Comfortable customer

- Processes

- Benchmarks: e.g.

  * Customer satisfaction (Customer survey)

  * Good planning (Visit / Checklist)

  * Amount of space (Library questionnaire / SCONUL statistics)

More details will be available in the "SCONUL Benchmarking Manual," due to be published in December 2000. The manual is the major outcome of the project. It is designed to offer a practical "how to do it" guide based on standard methods, modified for United Kingdom higher education libraries. The benchmarking model used envisages a loose seven-stage process comprising: (1) Defining, (2) Partnering, (3) Agreeing, (4) Measuring, (5) Collecting, (6) Analyzing, and (7) Acting. In practice, the seven stages are rarely as distinct as shown and may sometimes be carried out in parallel.

Added to the overall model are case study reports from the pilot projects, which give an excellent insight into the issues and benefits of benchmarking.

USER SATISFACTION

The final United Kingdom initiative to be covered is the SCONUL User Satisfaction Project, 1998-99. Led by yet another ACPI member, Jacqueline Whiteside of Lancaster University, this project was the third or fourth attempt at a standard nationally used user satisfaction assessment method. The aim was to devise a popular method which would be easy to use and would also generate a database of comparative data which might help to establish whether a local user satisfaction rating of 70 percent equaling "good" indicates a good, bad, or indifferent situation. The innovative aspects included:

* working with a commercial agency--Priority Search Ltd.;

* new data collection methods using a digitizer tablet and light pen to read questionnaires;

* use of specialist software to analyze data and present the results; and

* innovative graphic presentation of the results.

The questionnaire is composed entirely of tickboxes to permit machine reading, with sets of questions on how successful users were in using particular services, how satisfied they were, how important particular services were, whether expectations had been met (an echo here of the SERVQUAL methodology), use of other libraries, and satisfaction and importance overall. Figure 8 shows a specimen output from the pilot project.

[GRAPH OMITTED]

Each bar represents the result for one of the pilot group of libraries, with an indication of the number of questionnaires and the average score achieved. It is interesting that the method achieves statistically significant results with quite small numbers of questionnaires in each library. The horizontal scale plots positive and negative results against a notional -100 to +100 scale. The vertical line shows the average score for the whole group--fortunately slightly on the positive side of neutral. It can be seen that there are marked differences in ratings and why, therefore, the results are anonymous. The shading of the bars shows whether the difference from the mean is statistically significant or not. The project report will itself provide a useful broad benchmark of scores for different services. The report can point to an apparently useful standard method for further testing and a database of results for comparison. One additional result of interest is that people who have used other libraries tend to be about 10 percent more critical than those who have not.

The technology and methods are already in use in a significant number of United Kingdom libraries. In local surveys, a recommended technique is to ask respondents to rate the relative importance of two randomly generated statements about possible improvements in library services. The statements are generated as a result of focus groups with users rather than by a priori guessing by library staff or researchers. The computer then randomly generates pairs of statements (e.g., "more study seats" versus "longer opening hours on weekends"; "more catalog terminals versus more user education"). Figure 9 shows the kind of matrix a respondent is asked to complete.

Figure 9. Sample Response Form.

Library users'         This side    This side    Both sides
ideas                  very much    quite slot   just as
                       more         more         important or
                       important    important    unimportant

Provide more
help/guidance in
using the library
catalogue

Open longer at
weekends

Provide more
copies of key
texts

Open earlier in
the morning on
weekdays
during the
semester

Provide more
dedicated
catalogs
terminals in the
library

Library users'         This side    This side    Library users'
ideas                  quite slot   very much    ideas
                       more         more
                       important    important

Provide more                                     Purchase
help/guidance in                                 more
using the library                                new books
catalogue

Open longer at                                   Provide more
weekends                                         formal training
                                                 in using
                                                 electronic
                                                 resources

Provide more                                     Allocate more
copies of key                                    staff to shelving
texts                                            books/journals

Open earlier in                                  Help/guidance
the morning on                                   in using the
weekdays                                         library catalogue
during the
semester

Provide more                                     Ensure the
dedicated                                        library catalogue
catalogs                                         is accurate and
terminals in the                                 up to date
library

How Could the Library Provide You with a Better Service?
Important: Place a Cross in Only One Box on Each Line.

The outcome is a list of desired improvements in a prioritized order--not just an unquantified wish list, something which Northumbria has applied usefully in review of our IT facilities.

CONCLUSIONS AND THE FUTURE

In the last five years, there has been innovative progress on a number of fronts, some more successful than others. At the very least, some of the long-standing roadblocks have been moved or shifted to one side. For the future, the key objective is to continue this work, taking advantage of opportunities that arise from developments in other sectors or countries. The draft plan for SCONUL includes:

* continue with a search for better presentation, interpretation, and publicity;

* review the items collected and the use made of them by a sample of libraries (a new 1999/2000 project);

* encourage use of the customized comparison service from LISU;

* encourage adoption of the benchmarking method;

* more use of standard instruments, including the User Satisfaction Method;

* incorporate measures for electronic services;

* renew the search for a satisfactory overall framework for performance measurement (EAL2? UK Balanced Scorecard?); and

* perhaps, if ARL is willing, LibQUAL+: the UK Pilot?

It is stimulating for a United Kingdom librarian to see the potential of cross-Atlantic collaboration. It is hoped that some of the work reported here will strike a chord with librarians in the United States and elsewhere.

REFERENCES

Creaser, C. (1999). Summary of trends for SCONUL members. Loughborough: LISU [internal distribution to SCONUL].

Joint Funding Councils Ad-hoc Group on Performance Indicators for Libraries. (1995). The effective academic library: A framework for evaluating the performance of UK academic libraries. Bristol: Higher Education Funding Council for England.

Library and Information Statistics Unit, University of Loughborough. LIST: Library and information statistics tables for the United Kingdom [free leaflet].

Proceedings of the 3rd Northumbria International Conference on Performance Measurement in Libraries and Information Services: "Value and Impact," Friday 27 to Tuesday 31 August 1999. (2000). Newcastle: Information North. ISBN 0 906433 35 5. Retrieved from the World Wide Web: http://equinox.dcu.ie/.

SCONUL/HCLRG. (2000). UK higher education library management statistics, 1998-99. London: SCONUL/HCLRG.

SCONUL. (Forthcoming early 2001). User satisfaction project report. London: SCONUL. Retrieved March 28, 2001 from the World Wide Web: http://www.sconul.ac.uk.

Standing Conference of National & University Libraries. (2000). Annual Library Statistics 1998-99. London: SCONUL.

Town, J. S. (Ed.). (2000). SCONUL benchmarking manual. London: SCONUL.

Ian Winkworth, University of Northumbria, Library Building, Ellison Place, Newcastle-upon-Tyne, NE1, England

IAN WINKWORTH is Director of Learning Resources at the University of Northumbria at Newcastle, England. He is Chair of the Advisory Committee on Performance Indicators for SCONUL, the (United Kingdom) Standing Conference of National and University Libraries. He has been active in researching, applying, speaking and writing about library performance measurement for many years. Mr. Winkworth was founding Chair of the Northumbria International Conference on Library Performance Measures, which in 1995, 1997, and 1999 attracted a large audience of participants from around twenty-five countries, and in 2001 will take place in Pittsburgh, Pennsylvania. He is currently Co-Director of a nationally funded consortium project of the hybrid electronic and print library, HyLiFe.