Benchmarking in sponsored programs administration: using the web to analyze nationwide data collection results. (Shop Talk).

Introduction

The Society of Research Administrators International (SRA), the National Association of College and University Business Officers (NACUBO), and the Higher Education Practice of KPMG Consulting, Inc. have jointly undertaken the development of a national benchmarking program. This program has two purposes: (a) to provide participating institutions tools for quantitative analysis of their activities and comparison data and (b) to provide the sponsored programs administration community with performance and practice benchmarks to aid training and development programs.

Two rounds (FY 1998 and FY 2000) of complete data collection focus on institutional sponsored research competitiveness, administrative efficiency, productivity, and organizational practices. The resulting database includes a nationwide sample of academic and non-profit institutions, representing over 40% of total U.S. academic research expenditures. Data are available to participating Institution using a Web-based reporting and analysis tool. This reporting system allows participants to customize and generate institution-specific peer comparisons in a variety of tabular and graphical formats. This brief describes the measures and refers participant institutions to the Web page that allows them to make online comparisons.

Results from the FY 1998 and FY 2000 national surveys are widely available. Visit the SRA International Benchmarking Web page or the Research Management study area at www.higheredbenchmarking.com.

The Need for Customized Reporting

As a result of experience gained during the first round of data collection and feedback, the study leaders moved to the World Wide Web to make the data collection process more efficient and to provide participants with more flexibility and control over the reporting process. A powerful reporting tool is available for the study participants. The following sections describe the data elements and illustrate how the tool can be used to analyze comparative data.

Data Elements and Variables

The following sections describe the various data elements, pre-defined variables, and predefined comparison groups in the system. Refer to the survey definitions for the description of each element and its inclusions and exclusions. See Table 1 for key data definitions.

Indicators and Variables. The performance indicators are organized around the four themes: (a) sustaining or enhancing sponsored projects activity and funding, (b) containing the costs and improving the efficiency of sponsored projects administration, (c) improving administrative services to faculty, and (d) maintaining and improving institutional and sponsor accountability. Table 2 lists the demographic comparison groups for analysis.

Examples of Results

Figures 1 and 2 illustrate the types of analyses that can be done using the Web-based reporting tool. These examples were produced using data from the FY 1998 and early data from the FY 2000 survey. The graph in the first figure plots a hypothetical participant's FY 1998 and FY 2000 data on the number of proposals submitted per 100 faculty FTE reported for each year. The participant shows higher performance than the mean values of all participants and also for comparison groups of the NSF top-100 universities in the sample.

The graph in the second figure plots direct data, not ratio measures. It compares the participant's staffing levels in post-award financial administration with the mean staffing levels of other participants and in the NSF top-100 comparison group. This slide shows a relatively large post-award financial administration staff compared to the means of comparison groups.

The FY 1998 survey yielded some interesting results with regard to competitiveness and cost and efficiency. Generally, the survey results confirmed the conventional wisdom that sponsored research administration at larger, more research-intensive institutions appear to be more cost effective, having generally higher median levels of proposals and projects per sponsored program administration employee (FTE) or operating dollar. Since the survey did not address the issue of decentralized staff and cost, this is not surprising.

However, the survey also showed that the so-called success rate (the ratio of proposals submitted to the number of awards received in the year), for institutions with smaller research programs was better than the more research-intensive institutions. This would seem to indicate that researchers in these institutions tend to focus their proposal-preparation effort on those projects where they have a higher probability of success in competition, and not to shotgun proposals.

Michael Warnock, who developed a methodology for performance ranking on 18 areas of his own institution against a peer group, used the benchmarking results to make a successful business case for five additional staff, and improved performance in the area that they had targeted as weak. (See the Web site.) Programs not participating in the study could, if they so desired, use the survey questions to calculate their own performance ratios and then compare them to aggregate results published on the website using the same or similar methodology. Of course, the goal of the SRA Benchmarking Task Force is to encourage members to participate in the upcoming survey as the best way to be able to effectively use the survey data. Greater numbers of participants create more reliable results. An online audio presentation for SRA members on the past summary results will be scheduled in the spring.

For Further Information

The Web site will also be the location of a series of presentations intended to demonstrate the utility of benchmarking in program management and operational improvement. The FY 2000 data collection closed 31 December 2001. Web-based analysis opened to participants in January 2002. SPA general members may view selected findings similar to the figures in this paper at www.higheredbenchmarking.com.

Table 1

Data Elements from the 2000 Benchmarking Survey

Data Element                  Data Definition

Sponsored, Programs Staffing  FTE staffing for the institution's
                              central sponsored programs
                              administration including both
                              pre-award and post-award financial
                              management functions

Sponsored Programs Costs      Direct operating budgets of the
                              functions included in above

Workload                      Number of proposals and awards
                              (competitive, as well as non-
                              competitive), number of active
                              projects, dollar value of active
                              projects, number of funded Pl's

Research Staffing             Total number of faculty FTEs
                              eligible to participate in
                              research and the number of
                              principal investigators or co-Pls

Funding                       Expenditures from externally
                              sponsored sources over the
                              preceding five year period,
                              identification of the top three
                              sponsors, and a breakdown of one
                              year's awards by source of funds
                              (Federal, State, Private)

Organization                  Organizational structure and
                              reporting lines for sponsored
                              programs administration
                              (pre-award and post-award)

Decentralization              Distribution of sponsored programs
                              administration functions and
                              responsibilities across different
                              organizational units and levels
Table 2

Available Demographic and Comparison Groups

Public Control

Private Control

Land Grant Institutions

Independent Research Institutes, including stand-alone
Medical Schools

Institutions with Medical Schools

Minority Institutions

NSF Top 100 (Participating institutions that are among the
top 100 in overall research and development expenditures) (a)

NSF Top 101-200 (Participating institutions that are among
the institutions ranked from 101-200 in overall research and
development expenditures)

Self-selected peers Participants may construct a peer group
of other participants

Source: (a)National Science Foundation (2000, Table B-32).

References

National Science Foundation-SRS (2000) Table B-32. Academic Research and Development Expenditures -FY 1999. Washington, DC: National Science Foundation.

William S. Kirby has over 30 years experience in the management of sponsored research, consulting, executive training, and the application of technology. Since 1997 his efforts at KPMG Consulting, Inc. include university research administration and benchmarking issues with a focus on assessment, strategy, and operations. Prior to KPMG, he was Director of NSF's grant management and policy operations and a senior consultant at the Federal Quality Institute. He is a member of SRA's Benchmarking Task Force and Co-Chair of SRA's Evaluation Task Force for 2001-2002. Mr. Kirby has authored numerous publications and won the 1993 Rod Rose Award for the most outstanding article in The SBA Journal.

Paul G. Waugaman is a Principal of the Technology Commercialization Group, LLC, and Chair of SRA's Evaluation and Benchmarking Task Force. He has extensive experience in technology evaluation and management, and benchmarking. Over his 40-year career in research administration, he served as the chief management officer at two NIH institutes and led the technology transfer efforts at North Carolina State University and the Wake Forest University School of Medicine. Mr. Waugaman has published on performance benchmarking and best practices in technology transfer, the management of sponsored research, and the nature of academic-business research relationships.

This paper was originally part of a Contributing Paper presented at the October 2001 SRA International Annual Meeting in Vancouver, CN. The authors are employed by private firms with practice areas serving the research enterprise. Correspondence about the project should be addressed to William S. Kirby, KPMG Consulting, Inc. or to Paul G. Waugaman, TCG, LLC. E-mail: [e-mail] or [e-mail]