Performance, Processes and Costs: Managing Service Quality with the Balanced Scorecard.

ABSTRACT

A GERMAN PROJECT, SPONSORED BY THE German Research Council, uses the Balanced Scorecard as a concept for an integrated quality management system. Performance indicators across four equally significant perspectives--users, finances, internal processes, and potentials (innovation)--are combined to produce a "balanced" evaluation of the library. The project is a joint effort of the University and Regional Library Munster with the Bavarian State Library Munich and the State and University Library Bremen. The three libraries are among the largest in Germany, each with special activities and operating conditions. Thus the project takes a broad view of management issues in academic libraries. Work started in June 1999 and will be finished in autumn 2001. The results will be published in a handbook including software that will enable academic libraries to establish an integrated controlling system and to collect and evaluate performance as well as cost data for management decisions.

QUALITY MEASURES

The mission of libraries is generally to provide and deliver information for the needs of a specified population. Other tasks--e.g., legal deposit rights, preservation of rare materials, or special collections in a nationwide program--are, in most cases, subservient to the main purpose.

Therefore, the best testimony for a library's quality would be the influence of the library's products and services on the information literacy of its population. For academic libraries, that would be the library's impact on the educational process and the research results in the university.

Libraries have tried to find and test quality indicators that might prove the direct effect of their activities on the academic population (compare Hiscock, 1986; Self, 1987; Wells, 1995; de Jager, 1997). Some such indicators that were proposed include: students' success compared to library use; years of studying time compared to library use; and number and/or impact factor of research publications compared to library use.

But the direct influence of the library remains doubtful. If frequent library users get better marks, this might well be attributed to their general application and industry, using every means of information more intensively than others do. And faculty have many ways of finding information for their research, the library being only one of them.

In order to show their value for education and research, libraries have therefore developed more indirect measures of evaluation, such as studying the use of their collections and services; the speed of delivering information and services; the accuracy of delivery; the costs of the library's products and services; the adequacy of processes; and the satisfaction rate of the population served.

When libraries substitute these measures with more direct outcome measures, they assume that high use (library visits, issues, reference transactions) indicates benefit to users' information needs, that quick and reliable delivery will heighten this benefit, that cost-efficiency and well-organized processes will set resources free to enlarge and improve services, and that user satisfaction indicates good performance.

THE STAKEHOLDERS' VIEWS

Libraries have developed sets of statistical data, performance indicators, cost analysis data, and user and staff surveys in order to assess the quality of their products and services. They must, however, keep in mind that there are certainly different views as to what service quality in libraries actually means. Quality concepts usually name as the library's "stakeholders" the population served, the institution, financing authorities (which must not correlate with the institution), staff, and the general public. The two most interested stakeholder groups are the population the library is set up to serve and the institution to which it belongs.

The users' view as to library service quality concentrates on the fulfillment of their special needs. In other words, the library is good if I get the material I need at once or at least with quick delivery, if I get correct information and help the moment I need it, if I always find a seat and well-functioning equipment in the library, and if I feel well in the library.

Service quality in this sense could be assessed with data like: opening hours, availability of requested titles, delivery time for books out of closed stacks or by ILL, percentage of material in open stacks, queuing times at reference desks or computer stations, and seating occupancy. Data out of satisfaction surveys could corroborate the aforementioned indicators.

The institution, especially if it provides funding, will see library quality on another scale--i.e., the library is good if it helps to shorten studying time, produces graduates that quickly find a job, supports research in an effective way, helps to raise the image of the institution, and if it is cost-effective overall. The last issue will often be the most important when resources are scarce.

Indicators for these issues might be the market penetration of the library, high use statistics, acquisitions expenditure per member of the population, library costs per student, and user satisfaction. There are, of course, other concepts of service quality--e.g., from the point of view of the library's staff or the responsible ministry of science.

DATA FOR THE PROJECT

The current process of reforms in the academic sector favors financial autonomy of universities. Universities will work with an overall budget and will be able to decide independently on its use. Mechanisms of input-oriented regulation are replaced by performance indicators supporting allocation of budgets. Such indicators are, for instance, "number of graduates per term," "length of study time," and "proportion of research projects funded externally." Indirect service institutions, like the central administration, the computer center, and the library, are included in this trend and must prove the quality and cost-effectiveness of their services for education and research.

In previous years, libraries have developed, tested, and standardized methods for the evaluation of their products and services. The project at Munster relied especially on handbooks, standards, and projects in which the library had cooperated earlier.

For Statistics

ISO / DIS 2789. (2000). Information and Documentation--International Library Statistics (Two different standards): Deutsche Bibliotheksstatistik Teil B: Wissenschaftliche Bibliotheken (revised version 2000).

For Performance Measurement

ISO 11620. (1998). Information and Documentation--Performance Indicators for Libraries (Two different standards): Poll, Roswitha, & Boekhorst, Peter te. (1996). Measuring Quality: International Guidelines for Performance Measurement in Academic Libraries. Munchen: Saur.

EQUINOX: Library Performance Measurement and Quality Management System (Electronic Library Performance Indicators). http://equinox.dcu.ie.

For Cost Analysis

Ceynowa, Klaus, & Coners, Andre. (1999). Kostenmanagement fur Hochschulbibliotheken. Frankfurt am Main: Klostermann (for a short description of the cost analysis project, see Poll, 2000).

In addition, the library has implemented its experience in staff satisfaction surveys, regional surveys of library operating data, and process evaluation by commercial firms. Thus, a large collection of data is available for the evaluation of services. Table 1 shows data that could be used for assessing the quality of the lending service.

Table 1. Possible Data for Assessing Lending Service Quality.

Active users                                   40.999
Issues per year                                990.987
Availability of requested titles
           in the collection                   87%
           for direct use (not lent out)       63%
Time of document retrieval in open stacks      3 minutes
Book processing time                           25 days
Cost of one issue (staff costs, operating
costs, building costs, ...)                    1,80 DM
User satisfaction with lending system
(from 1 = very satisfied to 5 = dissatisfied)  1,9
Satisfaction of lending staff with their job
(very satisfied/satisfied)                     95%

Comparison between data from satisfaction surveys and more "objective" performance indicators showed that there may be vast gaps. In the user satisfaction survey of 2,000 users, responses indicated that, on average, 60 percent of the material users wanted was not available (it was either lent out or in in-house use). An availability study showed a rate of only 37 percent. Though it is quite understandable that disappointed users overrate the frequency of failure, the example shows that several methods must be used to get relevant management data. The quantity, diversity, and complexity of management data collected by libraries stresses the need for an integrated system that connects strategy, evaluation, and action.

THE BALANCED SCORECARD

The tool chosen for the management system is the Balanced Scorecard (see Kaplan & Norton, 1992, 1996), a concept originally developed for the commercial sector. The concept "translates" the planning perspective of an institution (mission, strategic vision, and goals) into a system of performance indicators that covers all important perspectives of performance--i.e., finances, users, internal processes, and improvement activities.

The system thus integrates financial and nonfinancial data, input and output data, the external perspective (funding institutions, users), and the internal perspective (processes, staff), goals and measures taken, and causes and results.

The basic model of the Balanced Scorecard, adapted to the conditions of academic libraries, deviates from the original model in placing not the financial, but the user perspective, foremost. Libraries do not strive for maximum gain but for best service.

[ILLUSTRATION OMITTED]

The indicators chosen for the user perspective correspond to the fundamental goals of reaching as large a portion of the population as possible and of satisfying their information needs by the services offered: (1) market penetration (percentage of the population registered as actual users); (2) user satisfaction rate; (3) opening hours compared to demand; (4) cases of use (issues, in-house use) per member of the population (use of electronic resources to be included as soon as possible); and (5) immediate availability--percentage of immediate loans over total number of loans (including reservations and ILL).

The last indicator shows whether the collection covers all topics asked for by users and whether there are sufficient copies. Two indicators assess the use of electronic services offered by the library and the growing portion of that use coming from outside the library: (1) the percentage of the population using electronic library services, and (2) percentage of remote accesses to electronic library services of all accesses. The indicators for the financial perspective answer the question regarding whether the library is functioning in a cost-effective way. The goals comprise low costs per instance of use or per product and a high proportion of the total budget spent on the print and electronic collection. These indicators include:

* total costs of the library per member of the population;

* total costs of the library per case of use;

* acquisitions expenditure compared to staff costs; and

* percentage of staff costs per library service/product to total staff costs.

A last indicator shows the allocation of resources to the electronic library:

* percentage of acquisitions expenditure spent on electronic media.

For the perspective of processes, the underlying goals are to organize all processes in a way that, in spite of budget restrictions, allows space for investment into new developments and improvement of service. The indicators pick out background activities as examples of process organization:

* acquired media per staff year (staff persons in the processing department counted as FTE);

* average media processing time; and

* number of stages involved in providing a product/service (for every library service).

Again, one indicator was chosen to show the allocation of resources to the electronic services:

* percentage of all staff costs spent on electronic services and provision of electronic media.

The last perspective, named "potentials," describes the capability of the library to cope with the challenges of the future and its ability to change and improve. The institution's support for the library is indicated by the budget it allocates to the library; its expenditures for Information and Communication Technology (ICT) prepare the infrastructure for technological development and, the main factor for all development, the staff, is represented by two indicators for teaching and engagement:

* library budget as a percentage of the institution's budget;

* percentage of current expenditure for information and communication technology;

* number of formal training hours per staff member; and

* number of short-time illnesses per staff member.

STRATEGY WITH THE BALANCED SCORECARD

One great advantage of the Balanced Scorecard is that it can visualize relationships of cause and effect among target values, evaluation data, and actions taken. Figure 2 shows the planning process from the definition of goals and target values, and the choice of adequate indicators, to the actions that the library takes to achieve the target values.

[ILLUSTRATION OMITTED]

As the mission of academic libraries is, in many aspects, identical, the indicators system of the project described here might be used as a reference model for benchmarking purposes. Individual variations in libraries can be expressed by different target values and operational actions. Thus, a library whose main task is to provide basic information for students will further the use of electronic media by offering multimedia learning material. A special research library, however, would perhaps offer its scientific journals in electronic form to achieve the same result. In spite of such differences, benchmarking would be possible.

The implementation and continuous use of the Balanced Scorecard demands a large set of data. The project has developed a special tool named Library Audit based on a system of data analysis, Online Analytical Processing (OLAP), that allows the multidimensional and flexible analysis of data collections. The library in Munster has already filled Library Audit with extensive data regarding the library's products and services. Benchmarking data from other libraries are added continuously. Many of these data will not be used in the strategic evaluation of the Balanced Scorecard, but the large data pool can be useful for many operational problems.

The number of indicators for the Balanced Scorecard has been purposely kept small in order to avoid a flood of data without direct relevance for strategic management. When choosing the indicators for the Balanced Scorecard, the project libraries were focusing on the concept of the hybrid library that combines electronic and traditional library services in a comprehensive function. Structuring and implementing a scorecard model for a library demands a clear formulation of mission and strategic goals--a duty that has not yet been performed by every academic library.

The most important issue in the integrated controlling concept is not to look at different quality aspects separately, but to keep them all in view. The following shows the steps of measuring quality in collection building:

1. The costs per document processed are low. Does that mean that there are backlogs?

2. Processing time proves quick and adequate. Processes are well organized, but perhaps there is no time for claiming overview orders?

3. Claiming is done regularly and in good time. Maybe staff is overworked and absence rates are rising because of illness?

4. Illness rates are quite normal, and a staff satisfaction survey shows high satisfaction with the job.

Everything looks fine, but collection use is declining, and a user survey shows dissatisfaction with the collection. Apparently much well-organized labor has been spent on the wrong material. The example shows that service quality has many aspects--the Balanced Scorecard attempts to integrate them. The project will be finished in 2001 and will result in a handbook including the software Library Audit. A first direct outcome is an initiative in Nordhrein-Westphalia, where seventeen university libraries consented to use a set of "ten core data" that relies on the Balanced Scorecard project. The core data are grouped as to input, services, and usage.

Input

* Acquisition expenditure per capita (members of the population served)

* Proportion of acquisition expenditure spent on electronic documents

* Library costs per capita

Services offered

* Opening hours per week

* Immediate availability of the loan collection

* Percentage of PC-places of all user working places

* Processed accessions per employee man-year (this is the only indicator showing the efficiency of background processes).

Usage

* Market penetration

* Loans per capita

* User satisfaction rate

The objective of the "ten core data" initiative is to give a concentrated view of a library's performance and to facilitate benchmarking between libraries of similar mission and structure. Such concentrated sets of data for the quantity, quality, and costs of the library will be indispensable for representing library services to institutions, funders, and the general public.

REFERENCES

de Jager, K. (1997). Library use and academic achievement: A study of the relationship between academic performance and usage of libraries at the Underdale site of the South Australian College of Advanced Education. South African Journal of Library and Information Science, 65(1), 26-30.

Hiscock, J. E. (1986). Does library usage affect academic performance? Australian Academic and Research. Libraries, 17(4), 207-213.

Kaplan, R. S., & Norton, D. P. (1992). The balanced scorecard--measures that drive performance. Harvard Business Review, 70(1), 71-79.

Kaplan, R. S., & Norton, D. P. (1996). The balanced scorecard: Translating strategy into action. Boston, MA: Harvard Business School Press.

Poll, R. (2000). Cost analysis and cost management as counterpart to performance measurement. In Proceedings of the 3rd Northumbria International Conference on Performance Measurement in Libraries and Information Services. Newcastle upon Tyne: Information North.

Self, J. (1987). Reserve readings and student grades: Analysis of a case study. Library and Information Science Research, 9(1), 29-40.

Wells, J. (1995). The influence of library usage on undergraduate academic success. Australian Academic and Research Libraries, 26(2), 121-128.

Roswitha Poll, University and Regional Library Munster, Krummer Timpen 3-5, Munster, D48143, Germany

ROSWITHA POLL is chief librarian of the University and Regional Library Munster. From 1991 to 1993 she was chair of the German Association of Academic Librarians and, since 1997, chair of the German Standards Committee for Information and Documentation. She chaired the IFLA group for the handbook on performance measurement in libraries and is now convener of the ISO working group for the International Standard of Library Statistics, member of the ISO group for performance measurement, and partner in the EU project EQUINOX (performance indicators for the digital library). She is working in national and international groups on collection preservation, quality management, statistics, and cost analysis in libraries.