Measuring outcomes: applying cost-benefit analysis to middle-sized and smaller public libraries. (Public Libraries).
THE RECENT DEMAND FOR MORE ACCOUNTABILITY from public libraries has made it essential that true cost-benefit analysis be applied to their operations. With funding from the Public Library Association, the authors developed a cost-benefit analysis methodology and applied it to five large public library systems. The present article describes their ongoing research to modify their methodologies to make them viable for application to public libraries of much smaller size.
OUTCOMES MEASUREMENT IN PUBLIC LIBRARIES
Like it or not, American public libraries have entered the age of accountability. This shift is transforming library statistics and measurements--what statistics are gathered, how they are gathered, and how they are interpreted and applied. To put the matter simply, library assessment, like public school assessment and higher education accreditation, is shifting from measuring outputs to measuring outcomes. The shift marks a transformation in viewpoint. Input-output measurement methodology--the established system of library accounting--follows an industrial production model. The library represents a black box. On one side of the black box, boards and administrators drop in "inputs," including financial resources to purchase staff, materials, and support services. From the other side of the box emerge "outputs," which find their principal expression in raw or adjusted counts of circulation and visitation. A whole reportorial culture emerged to address library inputs and outputs. A few professors gained considerable reputation by defining appropriate inputs and outputs (Zweizig & Rodger, 1982; Van House, Lynch, McClure, Zweizig, & Rodger, 1987). State libraries hired staff whose principal tasks were (and are) to collect library input and output statistics, ensure their internal consistency, and pass them to officials at the state and federal level. The federal government set up a section in the Department of Education (DOE) to gather the state compilations and turn the forwarded statistics into an annual publication that belatedly aggregated inputs and outputs. (1)
Meanwhile, the Public Library Data Service (PLDS) collected its own sets of input and output statistics using categories and presentation tables often different from DOE (Public Library Association, 2001). Two input-output reporting families now lived side by side, and, because of their different methods for parsing the library world, the two sets of measurements intersected and supported each other only incidentally. Within the framework of input-output statistics, the libraries that circulate the most books and count the most visitors while spending the least amount of money per circulation and visitor are "the best libraries." Implicit in this measurement is the notion that all circulations and all visitations are equal, and that the largest numbers produced at the lowest costs represent hallmarks of efficiency and even quality. Intriguingly, this point of view did not have its strongest exponent until after a decade of life in the networked-computer Information Age and years after civic and political leaders were pushing public libraries to exercise many nontraditional service roles to improve the quality of life in their constituent cultures (Hennen, 2002). Hardly any of these service innovations, however, found their way into traditional counting mechanisms.
The difficulty with these statistical appliances is that they measure what libraries do, not the benefits their constituents derive from them. Politicians, taxpayers, and major donors care about how much the public benefits from the resources provided to libraries, not how many volumes circulated during the last month. When it comes to outcomes, all circulations are not equal (e.g., some represent reading; others represent browsing to find something to read). All visitations do not represent equal consumption of services or equal value to the library customer (e.g., stopping by to use the restroom or copier represents a different benefit from that derived by the prospective entrepreneur whom staff help to get the statistics needed to start a new business). In the age of public-sector accountability, these differences raise questions: What is the worth of a library in the networked-computer age? How do shifts in use patterns reflect changes in customers' valuations of library services, and how would customers prefer that library resources be added or reallocated? What benefits are conferred on different types of library customers by their variant uses of public libraries? And, how can those benefits be measured?
At least two different professional groups in the year 2000 organized meetings that mark a growing trend toward moving library measurement culture from outputs toward outcomes. The first of these was a gathering of invited participants at a February 2001 conference hosted by the National Information Standards Organization (NISO). The subject was "Issues for Libraries: Measuring the Information Age (NISO, 2001)." Hampered by hideous weather and the seasonal flu bug that deterred travel by many scheduled participants, this conference addressed how networked computing was changing libraries and how library measurements had to change because of this shift. Along with standards, best practices, and electronic service measures, outcome measures played a prominent role in the conference agenda. Participants left Washington, DC, with examples of several different projects that were attempting to measure service outcomes or benefits. One of these was the St. Louis Public Library cost-benefits methodology. The second meeting was a gathering of recipients of fall-2000 grants made by the Institute of Museum and Library Services (IMLS) (2000). The meeting began with the statement of the meeting's legal context: the passage of the Government Performance and Results Act (1993). This legislation required "every government agency to establish specific performance goals for each of its programs, preferably with performance indicators stated in objective, quantitative and measurable terms (Shepherd, 2000)." Following the mandate of this legislation, IMLS consultants helped grant recipients devise strategies and methodologies by which they would measure the impact and/or benefits of the federal funds they were receiving. A revision of this seminar was repeated in the fall of 2001 for that year's grant recipients. (Even the granting agencies have entered the age of accountability. Can foundations and charitable trusts be far behind?) Neither the NISO conference nor the IMLS training advocated a dismissal of library input-output measurements. Nor do the authors of this paper. Like other advocates of outcome-based measurement, however, they do believe that the library community can build a strong case for its continued economic legitimacy by measuring the benefits that libraries provide their constituents (Weil, 2000; Rudd, 2000). The social sciences provide a number of these outcome-based measurements. Cost-benefit analysis (CBA) is one of these measurements. CBA has been used by economists to measure the benefits of education, pollution control, and locks and dams--to name only a few applications. The St. Louis CBA project applies the tool to measure library outcomes.
THE FIRST CBA PROJECT (CBA I), 2000-2001
Responding to a call from large urban library directors and the push of the St. Louis Library Board to "Prove it" (no matter what "it" was), the authors of this paper set out in the mid-1990s to measure the value of public library services. The purpose of this project was to develop a conservative, transportable methodology that large urban libraries could adapt to their own institutional settings. In making estimates of this value, the St. Louis Public Library researchers utilized the economics-measurement tool of cost-benefit analysis (CBA). With funding from the Public Library Association, the researchers tested the application of this methodology on the operations of the St. Louis Public Library (SLPL). This study demonstrated the feasibility of using CBA to measure library service outcomes. Next, with a grant from IMLS, the SLPL researchers refined the methodology and applied it to four additional library systems: Baltimore County Library System, Birmingham Public Library, Phoenix Public Library, and King County (Seattle) Public Library. They also replicated the earlier study of St. Louis Public Library. The study demonstrated the robustness and sensitivity of CBA methodology in the library setting. The methodology's reliability could be seen when the reanalysis of St. Louis produced results for general users that were comparable with the earlier study. The sensitivity could be seen when the study showed its ability to detect substantial differences in valuation among libraries based not only on service consumption but also on constituent demographics and ability to pay (Holt, Elliott, Watts, & Holt, 2000).
LAUNCHING CBA II
With these successes, the St. Louis researchers have set out to apply the same methodology to middle-sized and smaller libraries. This study, which the researchers call CBA II, will proceed through 2002 and 2003. The project replicates many of the first project's goals and adds others as well.
Like the first study of large urban libraries, this project is designed to develop, apply, and disseminate a methodology to value and communicate the economic benefits of library services, this time in mid-sized and smaller public libraries. Mid-sized and smaller public libraries are those with population services areas ranging from 50,000 to 150,000.
Adaptability is central to this project. In CBA II, the primary goal is to adapt the large-library study methodology to the often very different mission and variant funding of mid-sized and smaller libraries. To ensure institutional adaptability of the methodology, it will be developed in concert with nine very different mid-sized and smaller libraries located in three states in different regions of the United States. After development, the methodology will be tested at each of the nine libraries. The result will be an adaptable, conservative, transportable methodology that will meet the policy needs and cost constraints of mid-sized and smaller libraries.
The project will have two phases. In 2002, the researchers will work with nine institutions in three states to develop and test a practical, conservative, cost-feasible, transportable methodology that mid-sized and smaller urban libraries can use to estimate the value (i.e., the direct return on annual taxpayer investment) provided by their individual organizations. Through 2003, the researchers will disseminate the valuation methodology (i.e., what it is, how it was developed, how individual institutions can undertake their own valuations, and how they can use the tool to communicate their value in the community) through paper and electronic publications, conference presentations, and delivery of training.
After selection of the institutional participants, researchers will meet with the nine test-site libraries to discuss their mission statements and to categorize their library services and users into a service/user matrix. This matrix makes explicit the relationships among the components of a typical library's mission. By identifying major classes of a library's customers (e.g., households, educators, etc.) the matrix is customer-focused. By arraying customers against the library's portfolio of services (e.g., reference and reader's advisory, adult materials, children's materials, etc.) a library's service and user categories become visually explicit. Each of the cells of the matrix represents a stream of benefits from a library service to a particular class of customer.
When arrayed in this way, the matrix becomes the basis for a series of value measurements focusing on how much of which services the library's customers use and customers' valuation of the services. These measurements rely on customer responses in a telephone interview based on the benefit methodologies described below. A simple library service/user matrix and the matrix's explanation can be found in Holt, Elliott, & Dussold's "Framework" (1996). As with other aspects of this study, the service/user matrix is designed conservatively. By intent, some worthwhile but hard-to-measure functions (e.g., the library as a safe place for children, as a neighborhood center, or as a family recreational center) will be ignored. Such benefits are so hard to value that such contestable estimates would obscure the primary focus of the study. The large-library project demonstrated for five major libraries that measurable direct benefits were more than sufficient to prove the libraries' value to their communities. If so, why add explicit suspect measurements to the value-estimating formulae? Furthermore, a major thrust of the CBA II project is to reduce the cost of the methodology so that it is affordable for a much wider range of libraries to use. To reduce the cost of the survey, the measurements in CBA II will omit minor services as well as separately reported benefits to small specialized user groups that were included in CBA I. The service categories and user groups that will be dropped are less important in mid-sized or smaller libraries.
Measurement of Direct versus Indirect Benefits
Benefits can be classified as direct or indirect, individual or collective. Users of library services receive benefits directly, such as the recreational enjoyment from reading a novel or the strategic advantage enjoyed by a business that researches a new market for its products. Libraries provide many indirect benefits also. Enhanced reading skills of a young participant in a summer reading program may be passed on to her progeny. The community as a whole may benefit from a more informed electorate. Individual users can cite specific benefits that accrue to them through the use of specific library services. For example, a household that checks out and views a video receives direct and individual benefits. Collective benefits accrue to all members of the neighborhood, however, if the very presence of the local library or library branch instills a shared sense of community and pride. Recognizing this difference, this study will estimate a lower bound of the value of library services by focusing on the direct individual benefits provided by the library. To establish this lower bound, the project design focuses on creating a transportable model for estimating direct benefits provided by mid-sized and smaller libraries. The explicit calculation of indirect or collective benefits to nonusers is excluded from the study design.
Multiple Estimations of Benefits to Produce a Conservative Range of Benefits
Using sample surveys, this study will employ two methods of contingent valuation to estimate direct benefits to patrons from using library services. One is a service-by-service approach using the economist's tool "consumer surplus." The other approach measures the value of the library as a whole through users' "willingness to pay."
Consumer surplus will be used to measure the value that library users place on separately valued library services. Consumer surplus measures the value that consumers place on the consumption of a good or service in excess of what they must pay to get it. Although library services typically are "free," many substitutes for library services are available in the marketplace. For example, library users can buy novels rather than borrow them from the library's collections. The willingness of library users to purchase such substitutes if the library service were not available is one indicator of the value that the user places on the particular library service. Such estimates can be made for each service used by each library customer surveyed. These calculations can be summed to provide an estimate of total direct annual benefits for all library users measured in dollars. This approach offers several merits. Respondents are comfortable with the queries' scenario: most households are accustomed to (or have at least considered) purchasing books, newspapers, magazines, encyclopedias, or even Internet service. Most do not hesitate to respond about what additional purchases they would make if necessary to replace most library services (with the possible exception of staff help). For the library that wishes insight into the comparative contributions of different services, this approach offers a well-grounded method for obtaining detailed estimates. In CBA II queries, the researchers plan to vary the order of services randomly and provide respondents with a running total of their announced purchases to enhance the validity of these estimates.
Contingent valuation measures, though controversial, have been used extensively, even in high-stakes judicial proceedings, to value environmental conditions. The Exxon Valdez damage suit and Superfund (CERCLA) litigation have employed estimates using contingent valuation. (2) Contingent valuation requires a respondent to value a scenario depicting a counterfactual state of the world relative to the existing state of the world. Two alternative approaches are described in the economics literature. In the willingness-to-pay approach (WTP), interviewers ask respondents how much they would be willing to pay to have something that they currently do not have. In the willingness-to-accept approach (WTA), interviewers ask respondents how much they would accept to give up something that they already have.
Typically, WTA estimates of benefits are considerably higher than WTP estimates. Also, most experts view WTA estimates as less reliable. In CBA I, interviewers asked respondents how much they would accept to vote to close their public library. The WTA responses were consistent with the literature in that the WTA estimate for those who did respond was much higher than the WTP estimate. More importantly, over 80 percent of household respondents at each of the five sites refused to answer the WTA question. The method provided no reliable quantitative measure of the value of library services at any of the five study sites. Probes of WTA refusals, however, provided exceptionally insightful anecdotal comments regarding cardholders' views of the library as an irreplaceable community asset. WTA will be dropped in CBA II, however, saving both time and money. In CBA I, WTP produced the most conservative estimates of value and had a consistently small refusal rate. As in the large-library study, the CBA II study will ask library patrons how much they would be willing to pay rather than forego library usage or, if libraries did not exist, how much they would pay (in taxes) to enjoy the library privileges they have today.
The two major project goals will be accomplished by working through six measurable objectives. These are to:
1) Modify the analytical framework (service user matrix) and survey instrument from CBA I to address the major services and user group(s) that characterize mid-sized and smaller public libraries.
a. Reduce the number of services investigated in the earlier instrument. Focus on those emphasized by smaller libraries and those that produced the greatest contribution to benefits in the earlier large-library study. This step will reduce the length of the survey and cost per respondent completing the survey.
b. Restrict survey queries to provide benefit estimates for consumer surplus and willingness-to-pay approaches. Eliminate willingness-to-accept and value-of-time methods of measuring benefits.
c. Reduce the number of user groups by querying only general users (households) and, possibly teachers. Eliminate service providers and business users as targeted survey subpopulations. These subpopulations are likely to be too small to add substantially to the public's valuation of smaller libraries. Focusing on general users and not reporting separate estimates for benefits related to teachers will reduce the number of survey completions required for reliable statistical inference by two-thirds. This change will cut the cost of each library survey.
2) Develop and test programming applicable to most PC software systems that will embed the survey instrument, check for response validity, create a database of responses, and perform most of the calculations that will estimate a lower bound for a library's annual benefits.
a. Development of such computer software will substantially reduce the consulting cost to individual libraries in producing benefit estimates from the surveys.
b. Successful accomplishment of objectives I and 2 should permit an individual library to implement the methodology in a statistically valid manner for about $15,000 in external costs ($10,000 in survey costs plus $5,000 in other costs). In contrast, the very comprehensive version like that undertaken in the large-library study might be expected to cost, say, $40,000 ($30,000 in survey costs plus $10,000 in other costs).
3) Demonstrate the methodology for nine mid-sized or smaller libraries (three in each of three states) by completing 500 or more telephone interviews with patrons drawn in a random sample from the active cardholder database for each library.
4) Report results to the participating libraries. Assist them in interpreting the results and communicating those results to internal and external constituencies.
5) Evaluate the demonstration in achieving the following outcomes:
a. Reduction in cost to apply the methodology to an individual library so that the methodology is affordable for a wide range of mid-sized and smaller public libraries.
b. Provision of a conservative, meaningful lower bound for benefits and return to taxpayer investment in each library.
6) Disseminate the methodology to other mid-sized and smaller libraries. Develop a training model by which other public libraries may learn about and apply the transportable valuation methodology to estimate their own returns on taxpayer investment.
IMPACT WITHIN ORGANIZATIONS AND ACROSS ORGANIZATIONS
Successful accomplishment of the project goals and objectives outlined above will permit mid-sized and smaller urban libraries to engage in cost-benefit studies. As demonstrated in the large-library study, a library's execution of such a cost-benefit analysis can impact substantially the institution's management practices and external relations. Participants in the large libraries' CBA I study reported the following:
* The construction of the service user matrix leads library personnel to a greater understanding and appreciation of the library and its services from a customer perspective.
* The results of a CBA study help board members and administrators see the relationship between the socioeconomic characteristics of communities and the value they place on library access and service.
* The results of the cost-benefit study help executive directors make more informed budgetary decisions. The study informs resource allocation by quantifying the benefits of particular services for comparison against their costs.
* The results of the cost-benefit study are informative to library staff, help to boost staff morale by demonstrating the value of the library to the community, and impress upon executive directors and administrators the importance of staff training to effective customer service.
* The results of the cost-benefit study are very valuable to the library in its external public relations. The study quantifies the library's value to the community in a manner that is persuasive to external audiences such as local governments, donors and foundations, and taxpayers.
* The concept of the return on taxpayer investment that is part of the study implicitly incorporates the opportunity to assess the benefits of private-public financial partnerships. Private-sector gift-and-grant programs magnify library service benefits to local patrons beyond those paid for by taxes.
*In some cases, participation in the cost-benefit study may cause the library to reevaluate the effectiveness of its practices in maintaining its cardholder database. Unless cardholding records are updated annually, they do not reflect the library's actual user membership.
The applicability of the proposed methodology to other educational and cultural institutions is an open question. The central building blocks of the methodology all appear to be applicable to other publicly supported institutions, such as museums or performance arts organizations. The use of a matrix to display mission as an array of services versus subgroups of users is appropriate to a variety of public service venues. Contingent valuation as a means of estimating benefits is also widely applicable. The use of consumer surplus is more problematic, as there may not be readily available market substitutes for services of some public institutions. Return on taxpayer investment and return on invested capital are easily applied to almost any publicly supported institution. Nevertheless, the project's first purpose is to refine and demonstrate the methodology in the context of mid-sized and smaller libraries. The methodology may be extended at a later date to serve other institutions, including museums and historical societies.
Steps in Project Research
Each step of the CBA II work plan outlined below follows the project's research objectives:
1) Construct matrices of patrons and services for each participating library based on the library's mission. In meetings with the administrative staffs of each of the nine test-site libraries, the researchers will facilitate the classification of library services and patrons into major categories.
2) Consolidate the matrices from the nine libraries into a common framework. Staff from each of the libraries will critique the common framework to ensure its applicability to each of the individual libraries.
3) Design survey instruments, develop interviewing software, and select samples of library patrons to estimate benefits for each of the libraries by using measures of consumer surplus and contingent valuation (WTP). Since the research plan replicates many elements of the large-library study, a comparison of consistency in measuring bigger and smaller systems can be accomplished.
4) Ensure compliance with human-subject guidelines. All research will be conducted in conformity with Federal guidelines for human subject research as applied by the Human Subjects Research Committee at Southern Illinois University at Edwardsville; in compliance with all appropriate state laws protecting the privacy of library transactions; and the highest standards for user privacy articulated in the principles documents of the American Library Association.
5) Complete 4,500 telephone surveys, 500 for each test-site library. This process has several steps. First, computer services staff at each of the nine sites will draw a random sample of 2,500 cardholders who have used their cards within the last twelve months. Second, the director of each library will send a personal letter to each of these cardholders inviting their participation in the survey. Third, university telephone interviewers, trained by the project staff to use the project's survey instrument, will call those who have not declined the invitation to participate. A sample of 500 completed interviews will allow the extrapolation of sample statistical results to estimate the benefits to all cardholders from the tax investment of each public library.
6) The principal researcher, Dr. Holt, and the project's principal consultant, Professor Elliott, will write the project report. Prior to any public announcement, the results of the survey will be shared with the directors and participating staff of each library.
7) Develop executive summaries and visual aids for each library that convey clearly, but simply, the conclusions of the study.
8) The researchers will ask the director of each of the participating libraries to complete an assessment form that evaluates the project and its products, They will be asked to make this assessment based on the project's value as a managerial tool for understanding and communicating the mission of their library, informing budget decisions and strategic planning, and assisting communications with constituencies and the general public. Directors and test-site library staffs also will assess the project and products for cost, practicality, and transportability.
9) St. Louis Public Library project staff will host a post-project workshop to disseminate the methodology developed in this study to the staff of test-site libraries. Funds from the grant will support the instruction and materials for the workshop.
Participants in the post-project workshop will evaluate the applicability of the methodology to their own libraries and how the materials might be modified to make it easier for other libraries to use them. Several months after the conference, participants will respond to surveys asking whether they have plans to implement the methodology and, if so, when, how, and the expected use of the results.
The principal researcher and the project's principal consultant will prepare articles for electronic and paper publication and solicit appearances at national conferences.
ANTICIPATED FINDINGS FROM CBA II
The methodology of the second (current) study is very similar to that of CBA I. (3) In this section of the paper, the authors report the findings from the first study and suggest differences and similarities in findings they expect in CBA II.
1) CBA I clarified the usefulness of recognized CBA methods of contingent valuation as a basis for calculating a dollar estimate for all five cities. The contingent-valuation methodology is clearly applicable in a large public library setting. The study demonstrated that cost-benefit methodology is a tool well adapted to measuring the direct benefits of library services. The successful application of CBA methodology in the first project will allow the researchers in CBA II to make applications of the methodologies to nine libraries with a greater income range and greater variation in services than was accomplished in CBA I. The researchers expect the methodology to hold up but the range of benefits to vary considerably.
2) Recognizable methods of cost-benefit analysis used in many other kinds of CBA studies were used to measure the direct benefits of library services to each class of patrons. Using data obtained during twenty-five-minute user surveys, the project team calculated direct benefits for general users, teachers, and business users. In carrying out CBA II, the researchers will calculate benefits only for general users and using telephone surveys considerably shorter than twenty-five minutes. The benefits will be stated even more conservatively than in CBA I, but the methodology will be less expensive to apply.
3) When subjected to standard statistical tests for reliability, the study proved to be reasonably valid and reliable. The tests indicated that the survey produced a replicable valuation of services based on voluntary responses by those surveyed. The research team expects CBA II results to be statistically valid and reliable.
4) Based upon their answers to similar questions, the study demonstrated that different user groups receive different levels of benefits from library expenditures. The general user was asked consumer-surplus (CS), willingness-to-pay (WTP), and willingness-to-accept (WTA) questions. Teachers were asked about their professional use of the library with consumer-surplus and willingness-to-accept questions. Business users were also asked consumer-surplus and willingness-to- accept questions. The researchers also initially attempted to query caregivers. Anticipating that the representation of business users and caregivers would be even smaller than in CBA I and to reduce survey costs, CBA II researchers made the decision to focus on general users without reporting separate results for other special subpopulations of cardholders.
5) As in CBA I, the conclusions of CBA II will be defensibly conservative.
a. The study will capture benefits to cardholders only. No benefit estimation will be attempted for walk-in or virtual visitors who did not hold cards.
b. Nor are there explicit estimates of benefits to third-party beneficiaries. Only direct library users will participate in the surveys, and the focus will be on the direct benefits to cardholders and their families. In CBA I, interviewers' probes of the cardholders' WTP and WTA responses indicated some cardholder consideration of the value of the library to the community. The researchers expect that same consideration will occur in WTP responses in the new research work.
c. The consumer-surplus estimates are conservative because they are based on very conservative pricing of library services and direct benefits were calculated only if those surveyed stated that they will make a purchase to replace library services. In CBA II consumer-surplus queries, the researchers plan to vary the order of services randomly and provide respondents with a running total of their announced purchases to enhance the validity of these estimates.
d. Finally, the library's physical capital assets will be valued at replacement cost excluding depreciation. Thus, the study will calculate returns to invested capital as the ratio of conservatively measured annual benefits to overvalued capital assets. Measuring the numerator conservatively and inflating the denominator results in a conservative measure of the rate of return to invested capital.
6) Annual local taxes spent for library operations yield substantial direct benefits. Each library returns more than $1 of benefits for each $1 of annual taxes. In the first study, SLPL returned more than $2.50 in benefits per tax dollar; Baltimore County Public Library returned more than $3 in benefits per tax dollar; Birmingham Public Library returned at least $1.30 in benefits per tax dollar; Phoenix Public Library returned over $10 in benefits per tax dollar; and King County Library System returned more than $5 in benefits per tax dollar
CBA researchers do not yet have sufficient data to anticipate the benefits return for any particular library or library type in CBA II.
7) Each library studied in CBA I yielded a good annual return on invested capital. SLPL returns a minimum of 22 percent; Baltimore County Public Library returns a minimum of 72 percent; Birmingham Public Library returns a minimum of 5 percent; Phoenix Public Library returns over 150 percent; and King County Library System returns a minimum of 94 percent.
a. Shortly after completing the IMLS CBA study and before publicizing its results, Phoenix Public Library participated in a city-wide bond referendum that will expand its capital assets by 20 percent over five years. The referendum passed with more than 75 percent of voter support. The overwhelming strength of this majority confirms the public's (and cardholders') perception of the high social rate of return to the public's investment in library assets, consistent with the results of the CBA study.
b. The measurement of return on invested capital and return on annual taxpayer investment are both summarized in the seminar casebook, Libraries Are Valuable ... Prove It (Holt, Elliott, Watts, & Holt, 2000).
8) The methodology of CBA I detected differences in benefit streams flowing from different levels of investment. The CBA methodology is sufficiently fine-grained to detect differences in levels of benefits that flow from different levels of support for various areas of library activity. St. Louis Public, for example, had higher levels of benefits from children's services than did King County, which invests a lower percentage of its annual taxpayer investment in youth services. Not surprisingly, differences in cardholder subpopulations (e.g., households, teachers, business users, etc.) in different systems also affect CBA outcomes. Even without the study of library user groups like teachers and business, the CBA II researchers expect that different-sized benefits streams will flow from different levels of investment by the study-site libraries.
9) In spite of these differences in benefits streams, consistency proved to be the theme of the benefit levels of the studies, especially when calculations were made for categories of library services. In the case of all five libraries, when benefits were calculated, they did so in the following order: 1) Materials for adults, on average 35 percent; 2) staff interactions, on average 30 percent; 3) materials for children, on average 20 percent; and 4) library technology, on average 15 percent. Of these, the most problematical was technology, because comments that those surveyed made during their exchanges with interviewers often indicted that they were placing technology benefits implicitly into other categories (e.g., electronic newspaper and magazine databases were thought of as adult materials, not technology). In CBA II, the researchers expect the same consistencies as discovered in the first study.
10) CBA has considerable value as a communications tool. Not unexpectedly, the first persons to utilize the CBA findings were the directors of the systems in which the economic analysis was accomplished. They addressed the CBA findings to diverse audiences. Some used the results to orient staff to the value of their work. Others used them to communicate with individual donors or to the general public through publicity releases (Holt, Elliott, Watts, & Holt, 2000, Sections 2 and 4).
11) Quality of library databases is critical for successful completion of the survey. The most problematical element in the first study was the quality of library-user databases. No library that has not taken considerable care in creating or maintaining its user database should undertake a CBA study of the type described in this report. These techniques require random sampling of an accurate census of active library cardholders. The database used to generate the census must be up-to-date to avoid low response rates and response bias in completing the telephone surveys. In several of the study sites, missing or outdated telephone numbers in cardholder fields lowered the completion rates, and, in other cases, the researchers had to ask that the participating library systems obtain missing telephone data for cardholders before the telephone surveys could be started.
12) Population demographics can affect survey outcome. Phoenix, known for its seasonal residents and diverse ethnicity, presented this study's most serious challenge in implementing the survey design.
a. Approximately 30 percent of the cardholders who were active at some time during the previous twelve months had moved or changed phone numbers.
b. The response rate to an April survey of general users in Phoenix was only 18 percent. Data for the general user survey were weighted in proportion to the frequencies of cardholders by library branch to correct for any possible response bias.
c. Phoenix households were surveyed again in October to obtain statistically adequate samples of teachers and business users. The response rate again was only 18 percent. To obtain a sufficient number of educator responses, a list of Phoenix public school teachers was matched against a sample of Phoenix cardholders. (4)
In CBA II, researchers will exercise the advantage of the 2000 census reports that will make demographic patterns easier to determine. They still recognize demographic differences as a primary factor in CBA outcomes.
13) At the end of CBA I, the study team cautioned against comparing the benefit estimates across the five libraries studied. The benefit measures are designed conservatively to provide a defensible lower bound to the annual benefits of each library, not an unbiased estimate of each library's annual benefits. For this reason, comparisons across libraries are fraught with problems.
14) Nevertheless, some observations were apparent. For example, average or median family disposable income is correlated with benefits per household across cities: suburban King County and Baltimore County households reported higher benefits per household than the central cities of Birmingham, Phoenix, and St. Louis.
15) With sufficient information, it is possible to measure the nature and extent of economic benefits received by each class of patron for each type of service used. Classes of patrons can be identified by cardholder type and/or by self-identification. No matter what are the means of differentiation, care has to be taken because user types tend to overlap. In CBA II, however, to reduce survey costs for smaller libraries, the survey design will not identify separate classes of users and no such comparisons among user groups will appear.
16) Some CBA measures are more useful than others. As the CBA literature predicts for the whole range of activities, consumer-surplus and willingness-to-pay benefits estimates of library services were more accurate than willingness-to-accept measurements. The researchers also found that the cost-of-time measure that had been considered at the beginning of the project was less useful than other CBA study methods. This methodology, therefore, was not reported in the study results. It will not be included in the CBA II study.
17) CBA I measured the benefits from both public and private dollars. Return on taxpayer investment calculations, in addition to tax-dollar benefits, can assess the benefits of private contributions, foundation grants, and grants from different levels of government. This measure will not change in CBA II.
18) The study produced a replicable methodology, but one that is not without high expense. The biggest expense was the cost of surveys, and this expense was based on the amount of detail that the research team was attempting to capture. Based upon the experience in this project, the researchers recognize that they need less detail to produce reliable results. The costs of future CBA studies at smaller, less complex library systems therefore can be reduced.
CBA AS AN OUTCOME MEASUREMENT
The CBA II research team anticipates that many of the outcome relationships identified in the previous study will hold true in the new study. If these findings do hold up, then the research team will have applied cost-benefit analysis to fourteen different public libraries, large, medium-sized, and smaller, in eight different states. The study already has been replicated successfully when the initial PLA-funded case study was replicated in the first multilibrary IMLS-funded study. Furthermore, the CBA II team expects to be able to conduct the middle-sized and smaller library research far more cheaply per site than the large library study. From a cost standpoint, this project should bring the measurement methodology into the budget range of many more institutions than was possible using the methodology developed in CBA I. Admittedly, CBA II will not measure all of the benefits that libraries confer directly upon all classes of users. The original intention, however, was not to find every benefit but to estimate a conservative lower bound of benefits.
Outcome measurement will become a valued and even necessary tool for library administrators. It will provide a standard, easily understood statement of how their users benefit from a library's services. Museums already have such a tool in their applications of economic impact analysis. IMLS and NISO both recognize the need for libraries to have such outcome-measurement tools. Cost-benefit analysis, now applied not only in the United States but in Norway and New Zealand as well, is recognized increasingly as a valuable outcome-measurement tool. Considerable work remains to be done to perfect the tool's wide applicability to public libraries. Much already has been accomplished.
(1.) See the Web site for the National Center for Education Statistics. Library Statistics Program. Located at http://nces.ed.gov/surveys/libraries/; Bassman, Lecampagne, Korb, and Chute, 1988.
(2.) For a summary of environmental applications and the controversy surrounding CVA, see Portney (1994).
(3.) CBA I is summarized in varying degrees of detail in Holt, Elliott, & Moore, 1998. Holt & Elliott, 1998. Holt & Holt, 1999; and Holt, 1996.
(4.). For additional information on the Phoenix study, see Holt, Elliott, Watts, & Holt, 2000, Appendix D.
Bassman, K., Lecampagne, C., Korb, R., & Chute, A. (1998, September). How does your public library compare? Service performance of peer groups. Statistics in Brief. National Center for Educational Statistics. Located at http://nces.ed.gov/pubsearch/ pubsinfo.asp?pubid=98310.
Hennen, T.J.,Jr. (2002) Hennen's American public library ratings. Located at http://www.haplrindex.com/ratings.html.
Holt, G. E. (1996). Something more than sound bites: Communicating value to library constituencies. Bottom Line, 9(3), 36-39.
Holt, G. E., & Elliott, D. (1998, November 1). Proving your library's worth: A test case. Library Journal, 123(18), 42-44.
Holt, G., Elliott, D., & Dussold, C. (1996, Summer). A framework for evaluating public investment in urban libraries. Bottom Line, 9(4), 4-13.
Holt, G. E., Elliott, D., & Moore, A. (1998, March (April). Placing a value on public library services. A St. Louis case study. Public Libraries, 98-108.
Holt, G., Elliott, D., Watts, A., & Holt, L. E. (2000). Libraries are valuable ... Prove it! St. Louis: St. Louis Public Library.
Holt, G. E., & Holt, L. E. (1999,June). Assessing the value of children's library services. School Library Journal, 45(6), 47-48.
Institute of Museum and Library Services. (2000, October). Perspectives on outcome based evaluation for libraries and museums. Washington, DC: IMLS. Repeated October, 2001.
NISO. (2001, February 15-16). Issues for libraries: Measuring the information age. Washington, DC: NISO.
Portney, P. R. (1994) The contingent valuation debate: Why economists should care. Journal of Economic Perspectives, 8(4), 3-17.
Public Library Association, Public Library Data Service. (2001). Statistical report 2000. Chicago: American Library Association.
Rudd, P. D. (2000, October). Documenting the difference: Determining the value of libraries through measurement. In Institute of Museum and Library Services. Perspectives on outcome based evaluation for libraries and museums. Washington, DC: IMLS, pp. 16-23.
Shephard, B. (2000, October). Introduction. In Institute of Museum and Library Services. Perspectives on outcome based evaluation for libraries and museums. Washington, DC: IMLS, pp. 2-3.
Van House, N., Lynch, M.J., McClure, C. R., Zweizig, D. L., & Rodgel, E.J. (1987). Output measures for public libraries: A manual of standardized procedures. 2d ed. Chicago: American Library Association.
Weil, S. E. (2000, October). Transformed from a cemetery of bric-a-brac. In Institute of Museum and Library Services. Perspectives on outcome based evaluation for libraries and museums. Washington, DC: IMLS, pp. 4-15.
Zweizig, D. L. & Rodger, E.J. (1982). Output measures for public libraries: A manual of standardized procedures. Chicago: American Library Association.
GLEN E. HOLT is Executive Director of St. Louis Public Library (SLPL). Before heading SLPL, Holt served as a university professor and administrator, while providing policy analysis and planning consulting to more than 100 clients. Former clients include the Danforth Foundation, the Lilly Endowment, the City of Chicago, the Northwestern Railroad, Des Moines Public Library, and Buffalo and Erie County Public Library. He has published often, including thirty-six pieces on library topics, many of which are based on applied research studies done at SLPL.
DONALD ELLIOTT is Professor, Chair, and Graduate Program Director of the Economics and Finance Department at Southern Illinois University, Edwardsville. Elliot is the author of several articles and numerous reports on the economic impact of public institutions and policies. These articles and reports have addressed the measurement of regional benefits arising from individual campuses and systems of higher education and from environmental policies. He is former president of the Illinois Economic Association and has served as a consultant to federal, state, and local government agencies on matters of public finance.
Glen E. Holt, Executive Director, St. Louis Public Library, 1301 Olive St., St. Louis, MO 63103
Donald Elliott, Department of Economics and Finance, Southern Illinois University/Edwardsville, Edwardsville, IL 62026-1102
|Printer friendly Cite/link Email Feedback|
|Date:||Jan 1, 2003|
|Previous Article:||Public opinion and the funding of public libraries. (Public Libraries).|
|Next Article:||Cooperative game theoretic models for decision-making in contexts of library cooperation (1). (Library Cooperations).|