ARE YOUR PROGRAMS WORTHWHILE?

A well-thought-out evaluation process will help you maximize the value of your programs and products.

Let's get personal: Consider your favorite association program. Is it the newsletter or journal? Or perhaps the gala awards ceremonies created by presidential fiat 20 years ago? Did any of these programs begin because they "seemed like the right thing to do," or because "we had the money"? Are they now simply self-perpetuating flying machines, kept aloft because no one put on the brakes or considered the critical question: Why?

Now let's consider how nonprofit programs and products differ from their for-profit kin in many ways. The most germane is that nonprofit programs that are benefits of membership lack the ultimate means for evaluation: the marketplace. If we give products and services away (or advertise them as benefits in exchange for the price of membership) and we don't evaluate them, then how do we know if we're doing the right thing? If we don't compare one program to another in terms of relative value and ask vital questions about how the program relates to an organization's mission or what the cost is per member served, we are failing to meet our fiduciary responsibilities. And if we don't ask who should be involved, we may be missing opportunities to intensify our impact or make better use of our resources through appropriate partnerships. Finally, if we don't question the value of our programs (Why should we do this? How else might we achieve the same goals?), we may miss timely opportunities to provide program s with a natural death, to save money, or to institute better programs.

In the deadline-mad, short-staffed, and overburdened world of the average association, program evaluation often sounds like a good idea...for next year. And though I am a convert to the gospel of evaluation, allow me a confession: The American Society for Bone and Mineral Research, Washington, D.C., wasn't doing it either when I became executive director in 1998.

Before describing ASBMR's evaluation process and its outcome, some words of caution: Programs begun in the full flush of enthusiasm often acquire a life of their own. It's hard to consider axing some programs or even questioning their importance. Many executive directors feel that doing so would be professional suicide. Often it is financial difficulty or new leadership that leads associations to embark on program evaluation.

The evolution of evaluation ASBMR, founded in 1977, has grown from a handful of members to 3,500. We went from operating in the red to generating a revenue of $3.5 million. With such phenomenal growth, ASBMR leaders have been able to add important programs across the years with remarkable consensus. Yet, as then-President Michael Rosenblatt noted in December 1998--at the first ASBMR Council meeting I attended--after 20 years of life, it was time for ASBMR to determine which programs to continue.

Thinking in terms of the need to renew, modify, or end programs was new for a council used to adding new programs that took on lives of their own. However, having recently redefined ASBMR's mission at our first strategic retreat in the summer of 1998, the organization was ripe for focusing on how that mission was to be implemented.

Philip Osdoby, an ASBMR councilor at the time and a longtime ASBMR member, explains why evaluation didn't seem important earlier in the organization's development: "We didn't have as many programs and therefore could be less structured," he notes. "But I also think it was the retreat that brought to light the concept that none of the programs should necessarily be an immortal mandate, but should always be reviewed no matter how successful and important they seem to the society."

In late 1998, the council--a board of directors composed of 14 researchers and physicians in bone and mineral research--was ready to consider a new way of doing business. And by the time of ASBMR's June 1999 strategic retreat, the council agreed that given the increasing competition for resources and the expansion in proposed programs, the time had come to evaluate our programs.

Then-President David Goltzman notes that the evaluation process was necessary "to determine whether ongoing programs were still needed and relevant and cost-effective." Also "to help prioritize them and to facilitate phasing out of programs, while allowing some budget savings for the introduction of new ones."

The council agreed that all new programs would be evaluated after one year. So as not to subject small-budget programs to resource-intensive evaluations, we would evaluate only those ongoing programs costing at least $15,000 or 5 percent of ASBMR's total operating budget. To minimize the strain on staff and volunteers, we would limit the evaluations to no more than four or five programs a year. And we determined that regular evaluations every three years or so would be appropriate for ongoing programs.

Our annual strategic retreats tend to look at the big picture, targeted trends, or the future. Realizing that we needed a mechanism for evaluation, the council directed me to develop a process. Indebted to the National Center for Nonprofit Boards' publication Evaluation and the Nonprofit Board by Peter Szanton (1998, NCNB), I shaped a process suitable to our needs. In January 2000, I presented a process that detailed what would be evaluated, when, how, and by whom. The council approved the process, and with a deadline of the June 2000 council meeting for evaluation group reports and recommendations, the clock started ticking.

Addressing concerns

While the council shared the belief that evaluation was important, some ASBMR members were wary. It was important to address concerns that hidden agendas lay behind the process. The process needed to be as objective as possible and applied across the board. It also needed to be introduced as a sign of ASBMR's maturity: a formal process necessary to good management.

I knew that we had a moral imperative to evaluate our programs. After all, the money spent on programs didn't belong to the council or staff or individual members. Was ASBMR's money being well spent? Osdoby phrased it more eloquently and persuasively to critics: "If anything, evaluation provides a formal vehicle to get additional support for good programs while modifying or eliminating programs that are not working or no longer serve the needs of the Society (or need new leadership)."

Pinpointing programs

ASBMR's newsletter was far from the most expensive program we had, but the cost in staff time, production, printing, and mailing added up. Quarterly newsletters could not showcase current events. No one seemed certain about what our newsletter's primary focus should be: informing the members and supporters about upcoming events? Persuading readers to advocate for particular shared causes? Providing a forum for current debates or snapshots of behind-the-scenes activity? Moreover, we didn't know who actually read the newsletter or whether members and supporters appreciated the publication.

The newsletter was initiated long before we launched our Web site or began sending regular broadcast e-mails to our members. Certainly we had quicker and cheaper ways to communicate. Did we still need the newsletter? At the same time, could we afford to eliminate any membership benefit if competing societies were adding newsletters?

The membership directory was another product targeted for early evaluation. In the early years of the organization, the directory was a compilation of typewritten names and contact information. By 1999, the bound book was expensive to produce and just as expensive to ship (half of ASBMR's membership is international). Fundraising for its support wasn't easy, and we wondered whether a paper directory was really more desirable than an online version. We didn't think we could afford to offer both.

A third product for evaluation was the prestigious text, The Primer on Metabolic Bone Diseases and the Disorders of Mineral Metabolism. This comprehensive text was the brainchild of Murray Favus, who with a dedicated editorial board had volunteered countless hours to produce four editions of this much-used volume between 1990 and 1999. Evaluating this product was the most difficult--was it the best use of resources to meet our educational goals? Was it appreciated and used by members? Should we continue to publish it every three years? Corporate sponsors had generally covered the cost of distribution, while the text's publisher had done an excellent job working with the editors on its production.

Also selected for review was a research grant program begun in 1997, in collaboration with the National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS), of the National Institutes of Health, Bethesda, Maryland. Furthering research is central to ASBMR's mission, but was this strategy fulfilling its promise?

Identifying who evaluates

Selecting the right chair for the evaluation group was key to the process, but equally as important were the members who had oversight of each program, members who were involved in each program, and members who weren't involved in each program. In addition, each evaluation task force needed a staff liaison to help keep it on course, provide backup information, and contribute to the process. When possible, we selected for each task force an individual who wasn't an ASBMR member but who knew the program and programs with similar aims.

Staffing the Primer evaluation effort was challenging because some individuals invited to participate answered immediately that they could not be objective. Others questioned the existence of a hidden agenda or why the program's founder wasn't on the task force.

As a parent, I know that evaluating your child objectively is practically impossible. Yet surely a parent's input is critical. We knew we needed a task force member who was intimately involved in the program, and we knew that the Primer's first and only editor would need to be a critical resource. His willingness to provide history and good counsel, and the task force members' willingness to research other primers, share their experiences, consider changes, and seek member input, guaranteed a good process.

"Obviously we tried to be as objective as possible," explains Osdoby, who chaired the Primer evaluation group. "It was not necessarily easy, but we had the advantage that the program is a success. [Also, it's] obvious that nothing is ever perfect. You wish to make sure that the high quality persists and is not totally dependent on one or two individuals."

Under the microscope

With the goal of objectivity in mind, it's important to select the right members and allow the president to appoint the task force, involving the appropriate committee chair and the executive director or senior staff.

The process should be as standardized as possible for consistency and best results. The ASBMR evaluation form (see sidebar, "Sample Program Evaluation Form") was designed to apply to all programs and be used by all evaluation groups. Group members need to be guided by the necessary data. They need to examine the program budget, both the actual figures and the budgeted numbers. They need to consider the indicators of the program's growth or decline, the stability of its costs, and staff and volunteer resources. Last, the group needs to compare the program to other programs with similar goals. Is it a higher quality program? Is it more successful? Is it more or less ambitious in its goals? These questions often lead to another: Could the programs be combined, or the ASBMR program improved?

Once the task force is appointed, a preliminary conference call brings together the task force members and the staff liaison. The chair outlines an agenda, while staff ensures that each member has a copy of the description of the process and its rationale, the ASBMR program evaluation form, and contact information for all task force members.

Frequently, the preliminary discussion focuses on the specifics of the program or product, what information is available about it, and what information is needed for its evaluation. Sometimes programs have clearly delineated goals, budgets, and timelines. Various documents (memos, board or council minutes, procedural manuals, budgets, auditors' reports, survey data) provide all the necessary data needed about the current status and evolution of the product or program. More often, pulling this together--particularly for longstanding programs--requires digging. Staff members are often assigned responsibilities related to understanding the program's history. Evaluation group members may disagree about the program's initial or current goals.

When it is clear that the evaluation group requires more data, the members focus on how to acquire it: Surveying the membership through broadcast e-mail or Web-based surveys? Holding focus groups at our annual meeting? Interviewing others involved in the program's launch and evolution? Researching other associations to learn about how their programs differ?

It's important to end the first conference call with some clear assignments of tasks and timelines. Strong chairs are likely to follow up with informal summations. Staff can help by issuing reminders about impending deadlines and offering assistance.

The members may gather together by conference call, or in person when possible, but most continue their work by e-mail. The staff liaison may help with member surveys or setting up conference calls. The chair drafts the group's report, using the ASBMR evaluation form as a template.

Revealing results

Three evaluation groups reported at the June 2000 ASBMR Council meeting. The longest-standing program, the Primer, required more time for evaluation. The Primer evaluation presentation was postponed until September 2000.

Recommendations ranged from a straightforward vote of approval with minor suggestions to dramatic recommendations that saved ASBMR significant dollars. For example, the ASBMR Council voted to maintain the paper directory but to save money by decreasing the white space, resulting in significantly fewer pages. In addition, the group recommended that it be mailed with our Journal of Bone and Mineral Research to decrease postage costs. We discovered that many members, including task force members, didn't realize that their contact information was already online--it is included in the online directory of the Federated American Societies for Experimental Biology, of which ASBMR is a member. The evaluation group recommended c-mailing this information to members. We have done this, and also advertised this information in our newsletter as well.

Similarly, another evaluation group recommended continuing our partnership with NIAMS in funding research grants, and gathered data from current and past grantees to delineate their success (publications, presentations, additional grants, professional career advancement). The benefits were tangible and intangible, and the recommendations included that ASBMR promote the program more widely within the organization as well as to students and post-doctoral fellows.

The newsletter evaluation group was chaired by former President Jane E. Aubin. This group considered the newsletter's cost relative to its value, along with its history, audience, content, and availability on the ASBMR Web site. The perceived pass-along value of the newsletter left the group reluctant to axe it. Its cost at approximately $4 per member per year made it appear to be a relatively inexpensive member benefit as well, Aubin observed. As a result of its deliberations, the evaluation group called for continuing a paper newsletter for the next year while the council "discusses its intent for the newsletter" and discusses funding mechanisms for increasing its scope and length. As of January 2001, we began including the newsletter--mailed with the Journal in 2000--within the pag of the Journal itself. This has considerably decreased production costs and allowed for an increase in newsletter publication from quarterly to monthly. Regular reminders to members tell them to look for the newsletter in its n ew home.

In September 2000, Osdoby presented the Primer evaluation to the council. The report was the most extensive document completed. The document noted that the Primer, "approaching 100,000 [distributed copies]... has become the most widely circulated source of information on basic and clinical information regarding metabolic bone diseases and mineral disorders."

"When you have a program as successful as the Primer, you want to do it justice and you do not want to either dampen enthusiasm or try to fix what is not broken," Osdoby points out. "So we really tried to provide input that would make the Primer better by listening carefully to the needs of the editors and their experiences."

While the evaluation group members were unanimously in favor of continuing to produce the Primer, they also recommended the inclusion of several junior faculty on the editorial board to ensure continuity and provide insight for updates." Other recommendations included seeking ASBMR member input regarding Primer content and considering whether to continue free distribution to medical residents (a practice not shared by far wealthier associations, we learned). In addition, the group questioned whether the Primer might be published in book format every three or four years or on the Internet or on CD-ROM to minimize production costs and enhance the text's regular revisions and links to references. (Our contract with a for-profit publisher did not allow for Internet publication.) Distributing the book with the Journal might reduce shipping costs, too. Finally, the group called for consideration of self-publishing.

At the January 2001 ASBMR Council meeting, the council approved a self-publishing feasibility study and business plan to investigate the pros and cons as well as the costs and implementation guidelines.

ASBMR is now in its second year of evaluations. Among the programs being evaluated are our Web site and several aspects of our advocacy program. The lessons we've learned from the process are many. (See sidebar, "Understanding the Costs.") We implemented cost savings that we might not otherwise have pursued. We enjoyed expanded member involvement in our governance and offered improved services at the same time. Now we are embarking on extensive membership surveys to gather additional data.

Program evaluation isn't flashy, but it's critical to good management, worth the investment of time and money, and--like many good deeds--eminently satisfying as well.

Joan R. Goldberg is executive director of the American Society for Bone and Mineral Research, Washington, D. C., and a group executive of Smith, Bucklin & Associates, Inc. E-mail: joan_goldberg @dc.sba.com.

UNDERSTANDING THE COSTS

Evaluating programs is not without risks and costs. We didn't budget for direct costs the first year, nor did we know how much time the evaluations would require from staff and volunteers.

ASBMR learned the following:

* Staff support is necessary. Do you have sufficient resources or will this stretch your staff too thin?

* Political fallout is a risk. Do you have the allies you need to take this on at this time?

* There are real costs for staff time, conference calls, surveys, and shipping materials. Be sure to factor these in before you start.

* The process is subjective. Try to make it as objective as possible by selecting a good balance of individuals and a respected chair, but acknowledge that no evaluation can be 100 percent objective.