Look for the silver lining: when performance-based accountability systems work.
This article examines MFR reform in all fifty United States, based on analyses in two time periods, 1998 and 2000. Using a criteria-based approach to evaluate and rank MFR systems, this article seeks to uncover the common management factors that enable governments to meet the criteria for effective MFR. This means that we identify states that did well and poorly according to the criteria and we explore in some depth why they achieved or failed to achieve the criteria, first reviewing important governance variables and then describing the common management practices of states that performed well. We call these management practices integrative facilitators, the practical actions that ensure that systems work as theorized: creating, disseminating, and using valid performance information.
The concepts discussed in this article treat MFR as a capacity tool, reflecting an advance from early efforts to establish performance accountability based on measurement of government outputs (ICMA 2000). Although precise measurement offers many benefits, particularly at the operational level, this initial focus often precluded the consideration of other elements of performance. Most significantly, although the focus allowed increasingly precise explanation of what was measured, it did not allow discussion of why measures reflected good or bad performance. Specifically, the focus on measurement did not permit analysis of potential capacity to perform or the extent to which the objectives measured ultimately reflected priorities and objectives that were present throughout design and implementation processes (Ingraham, Joyce, and Donahue, forthcoming). Both of these qualities were problematic for decision makers and others who wished to more tightly link accountability to performance (see, e.g., Heinrich 1999).
MFR systems are a potential solution to this problem because they are intended to link strategic planning to resource allocation and to consider both in measuring public performance. Furthermore, an effective MFR system produces information that is appropriate to decision makers' needs and channels such information into critical decision forums. On the other hand, MFR systems come with their own baggage: They present an extreme case of the classic implementation problem in which legislators set broad direction and bureaucratic actors shape the reality of the program in place. In the case of MFR, members of public organizations are asked to implement a system that evaluates their effectiveness and places their performance in a much more public eye. For the system to be effective, there must be very serious commitment to purposes, processes, and outcomes, as well as to increased transparency--characteristics markedly absent in the implementation of results-based reform at the federal level (Radin 2000).
MFR IN THE STATES
The American states provide excellent laboratories for analyzing MFR systems and the impact that they are likely to have on government. States have long been fertile ground for comparison and learning about policy and other differences (Beyle 1999; Brudney, Herbert, and Wright 1999; Dye 1966; Jacoby and Schneider 2001). In addition, states have increasingly relied on the tools of results-based reform in the last decade. The Government Performance Project (GPP) surveyed all fifty states on their MFR practices in 1998 and again in 2000; in 2000 we found every state implementing some form of MFR system (Moynihan 2001). In almost all states, the rise of MFR systems is grounded in legislation or in administrative requirements (Melkers and Willoughby 1998, 2001; Snell and Grooters 1999).
A CRITERIA-BASED APPROACH TO ASSESSING MANAGEMENT SYSTEMS
The GPP and its associated project, the Federal Performance Project (FPP), are multiyear analyses of management capacity in federal, state, and local governments. The GPP explores the "black box" of management by examining the extent to which effective management and performance may be linked to the creation of the capacity to perform and variation in government outcomes. To this end, the GPP examines five distinct management systems: financial management, human resources management, information technology management, capital management, and managing for results. A set of criteria was established for each system.
Criteria-based assessments provide an innovative and useful method for evaluating government systems in a way that facilitates comparisons across different units of governments (Ingraham and Moynihan 2001). Thus far, criteria-based models have been utilized largely by practitioners for individual organizational assessments (1) or some form of policy analysis or evaluation (Easton 1973; Rossi and Freeman 1989; Weimer and Vining 1992), but not as a systematic mode of comparison across different governments, (2) and rarely for academic analyses.
We argue that criteria-based assessments can be used profitably for scholarship on public management because they offer an understandable, transparent set of standards that can assess and compare a high number of governments. These standards provide the basis for rigorous data collection. The GPP triangulated qualitative and quantitative methods to increase data validity (Miles and Huberman 1994). For all states, data was collected using interviews by staff from Governing magazine of state government officials and stakeholders, an in-depth written survey containing closed-ended and open-ended questions completed by government representatives, and content analysis of MFR public documents (the latter two elements completed by faculty and staff at the Maxwell School). Another benefit of the criteria-based approach is the ability to facilitate comprehensive and in-depth analyses. Typically, empirical analyses of government management trade comprehensiveness for sample size and standardization for case detail. In contrast, the criteria-based approach enabled the GPP to examine the main public management systems in all fifty states in some depth.
There are risks associated with this approach. The most obvious is the assumption that the "right" criteria, identifying desirable variables, are selected. It is therefore critical that the process of selecting criteria be thorough and legitimate, that the process incorporate a variety of perspectives, and that the criteria are amenable to change if evidence demands. The GPP criteria for each management system were selected on the basis of previous public management research, consultation with panels of elected and administrative officials and public management experts and academics, and a yearlong pilot study prior to full fielding of the GPP survey.
Another potential criticism is that research findings using criteria-based evaluations will be circular. Positive aspects that researchers identify in high-performing governments are actually implicit in the criteria, and any research findings could essentially be simple restatements of the criteria. There are several ways to respond to the potential difficulties. First, the criteria-based approach is not simply about achieving the criteria, but also about the ways in which the criteria are met. There is not likely to be a single approach. For instance, the section of this article on explaining variation in state grades explains the influence of governance and other high-level variables on the criteria achievement. The rest of this article examines how management practices are used for the same goal.
Furthermore, the in-depth description and simple categorization of management practices in high-performing governments against a set of common criteria is informative to others. This is valuable because management practices, unlike high-level governance or socioeconomic variables, are actually amenable to change and adaptation by managers; they are policy tools. Findings on management practices therefore constitute usable policy advice (Bardach 1998). Such findings are based to a high degree on what governments are doing, and there is no guarantee that governments pursuing widely agreed-upon criteria will select the same practices, or even the right practices. Future research should test these choices. Knowledge of these management practices must therefore be of interest to academics seeking to answer what works in performance management and to practitioners seeking to make more useful the performance information systems that are the dominant model for performance management.
Criteria-Based Grading of State Governments
Every round of GPP analysis saw summary grades published for each management system in each government. The grades were based on separate assessments undertaken by researchers from Governing magazine and the Maxwell School. Each set of researchers developed a grade for each state before ultimately meeting to compare findings and to decide a final set of grades (correlations between the academics' and journalists' sets of grades for the 2000 state MFR analysis were .787). Cases with marked differences between each set of initial grades were accounted for by different research sources illustrating an aspect of the MFR system unavailable to the other team of graders.
Research drew on a large amount of qualitative and quantitative data, with each type of information acting as a check on the other. For the 2000 grading process, researchers at the Maxwell School developed and used quantitative scales for each of the MFR criteria and subcriteria and based on closed-ended responses to survey questions and content analysis of public documents (see appendix). (3) Rater bias was also considered and corrected. (4) Maxwell researchers also considered qualitative evidence based on survey responses. Researchers at Governing relied on the GPP survey, interviews with state officials and stakeholders, and summaries of each MFR system prepared by the Maxwell researchers. The final grades therefore incorporate quantitative scales but also qualitative evidence not easily amenable to reducing into quantitative form. Much of the qualitative information dealt with the criterion that was most difficult to document: that performance information be used for decision making.
All state governments were graded against the criteria in 1998 and 2000 (see table 1), providing a detailed data bank of information on MFR practices at the state level for these points in time. Although most states have similar MFR requirements, the quality of the systems implemented varies widely. As table 1 demonstrates, few states were judged to have strong, functioning MFR systems in either 1998 or 2000. A far larger number had some part of a system in place but were struggling to improve its effectiveness. States with the lowest grades tended to have an MFR system "on the books" but were not emphasizing implementation. In virtually all cases, evidence of systematic use of performance information for decision making was limited.
ESTABLISHING CRITERIA FOR MFR
Table 2 presents the MFR criteria, a concise definition of what MFR actually means, as well as the necessary components of an effective system.
The criteria essentially call on governments to develop a performance information system and to make use of that system. In creating a performance information system, governments should use strategic planning and valid performance data, should use the input of government stakeholders, should ensure coordination of MFR efforts, and should communicate performance information to employees and the public. Perhaps most importantly, governments are called on to use performance information in the making of decisions. (5)
To those that disagree with the criteria as discussed here, the findings of this article will be of limited interest. However, evidence from the GPP (discussed later) and others (Brudney, Herbert, and Wright 1998; Melkers and Willoughby 1998; Snell and Grooters 1999) suggest that the model of managing for results discussed here (and illustrated in figure 1) has become the dominant model of performance management in state government. The following two sections further discuss the underlying logic of the MFR model in the context of these criteria.
A Simple Model of Managing for Results
The GPP data collection and analysis for MFR proceeded from conceptual assumptions outlined in the criteria in table 2 and illustrated in figure 1. Figure 1 represents a simple, well-established consensus model of how a performance management system improves governmental decision making and performance.
[FIGURE 1 OMITTED]
For the purposes of MFR analysis, as well as for the integrative facilitator points discussed in this article, two elements of the model are key. First, performance information lifts the focus of managers from inputs and processes to outputs and suggests that integrating performance information into decision venues is critical to improving performance. MFR is integrative in this regard because it links multiple providers of performance information to multiple users of performance information in diverse decision venues. The second way in which MFR is integrative is in linking across management systems. Conceptual explorations of the "black box" of management identified the frequency of "stovepiped" management systems, each pursuing inward-looking policies without consideration of overall management capacity and high-level management goals (Ingraham and Donahue 2000). MFR, through its production and dissemination of high-level performance information, offers governmental actors common goals to pursue and an understanding of how present management structures could be adjusted to pursue these goals, improving the ability to make informed decisions about capacity.
Figure 1 also illustrates the fact that the lifeblood of an MFR system is performance information, the means of transforming inputs from the environment and internal stakeholders into specific goals. MFR systems are designed to take information from the environment through consultation with the public, stakeholders, public representatives, and analyses of the external environment in the strategic planning phase. The possible breadth of the environment means that "every public administrator has to supply some criterion of relevance to the external environment" (Gawthrop 1984, 45) when considering phenomena related to the organization. Katz and Kahn (1966) call this process coding; it simplifies the potential information pool into a few meaningful categories. MFR systems provide a means by which the organization codes, interpreting and refining information from the external environment and internal stakeholders into a series of information categories such as strategic goals, objectives, performance measures, and targets--performance information that can then be presented to decision makers. Performance information also provides a language for communicating with the external environment, transmitting strategic goals and performance measures via public documentation, speeches, Web sites, and other means.
Figure 1 also illustrates the linkages between strategic planning and performance measurement, suggesting that valuable performance information implies the connected used of both approaches. Experience suggests that past separation was to the detriment of each (Ansoff 1976; Toft 1989). Without the linkage, the potential for goal conflict, confusion, and inaccurate measurement rises (Heinrich 1999). Strategic planning without performance measurement fails to link goals to actions or identify implementation issues, failures that generate a lack of credibility among stakeholders (Wildavsky 1973; Langley 1988). Performance measurement without broader strategic guidance fosters measurement without a sense of overall purpose: a frequently highly technical exercise undertaken out of habit or administrative compliance with little practical relevance for decision makers.
What is the purpose of all this activity? One justification for results-based reform is the accountability to the external environment and elected officials (Gormley and Weimer 1999). However, state government managers report that the more immediate rationale for results-based reform is improved internal decision making and improved public performance (Melkers and Willoughby 2001). High-quality information and the ability to communicate it to the right decision-making venue in a useful and timely way are as necessary to these interim objectives as to broader accountability purposes (Ingraham and Donahue 2000; Macintosh 1994).
In a capacity-building system, the communication of performance information is intended to act primarily as a stimulus to the decision-making process, provoking, informing, and improving the quality of decisions (Scott 1961). The most basic and fundamental way that MFR achieves this goal is, as illustrated by figure 1, by the provision of information to decision venues in government. This provision is expected to lead to decisions, actions, and allocations of resources that enable governments to fulfill strategic goals in more efficient and effective ways.
The state of Washington provides a clear example: "What we are trying to achieve in Washington is a system whereby all types of decisions are routinely informed by performance and planning information, in addition to traditional factors, such as competing priorities, organizational capacity, financial reality, and stakeholder and public opinion." (6) As with many other states that emphasize the resource allocation aspect of MFR, Washington State undertakes agency strategic planning and performance reporting as part of the budget process. However, the governor has emphasized alternative decision venues for performance information and the value of sharing examples of positive decisions or-innovations as a result of performance information use. In 1997, the governor issued an Executive Order for Quality Improvement that encouraged process improvement. Support for process improvement came from a subcabinet for management and quality improvement consisting of fifteen cabinet directors and chaired by the governor's chief of staff and a new post of special assistant for quality improvement. Agency improvement programs are nominated quarterly for the Governor's Award for Service and Quality Improvement. The programs and the responsible team members are recognized publicly: "As of April 2000, agencies have shared over 1,350 quality improvements with the Governor.... The success of this recognition program is evident by the increasing competition to receive the awards and reports that teams are now setting a goal of 'going to the mansion.'"
The Adoption of the MFR System
Evidence from American state governments suggests wide acceptance (if not always actual implementation) of the simple MFR conceptual model presented in figure 1. In reporting to the GPP, states repeatedly emphasized strategic planning and performance measurement as related activities intended to feed into multiple decision venues and improved quality of decisions. Virginia, for example, described its Performance Management System as "comprised of four, linked processes: strategic planning, performance measurement, program evaluation, and performance budgeting ... these processes are designed to work together to manage the performance of state government." Florida explained its Performance Accountability System as the "framework to ensure the critical link is maintained between the strategic plan, budget, and performance measures." Louisiana's management processes move from "planning to budgeting to implementation to evaluation (or accountability), back to planning and so on. All processes are linked; each builds upon the one that precedes it and contributes to the one that follows. No matter where you enter the circle, you will eventually move through all the processes." Texas described its system as "an integrated comprehensive system of statewide and agency strategic planning, performance measurement, performance-based budgeting, and performance reporting, assessment, evaluation and auditing." States with more limited experience in MFR do not seem any less in pursuit of the MFR criteria illustrated in figure 1, but simply less far along the road to implementing it. For example, Alabama described its system in ways similar to other states. That state's Strategic Plan and Performance Measurement System "connects the strategic goals to specific actions and performance measures by the agencies" and "communicates strategic objectives clearly, links objectives to annual budgets, provides a common methodology and framework for all agency performance efforts."
EXPLAINING VARIATION IN STATE GRADES
Original research for this article focused on what Lynn, Heinrich, and Hill (2000) describe as the discretionary organization and management level of governance or what the GPP has referred to as management capacity, essentially management practices that are assumed to be directly related to performance. This section initially deals with the broader question of why states achieve high grades. Findings are based on quantitative cross-sectional analyses that use GPP data and seek to explain how higher-level variables shape state management capacity. In addition to socioeconomic and historical factors, we identify variables at higher levels of Lynn, Heinrich, and Hill's logic of governance: citizen preferences and interests; variables that shape public sector decisions, activity, and influence; and the structure for formal authority. Table 3 tracks the effect of these variables on MFR state grades and the overall GPP grade each state received.
Variables tangentially related to governance, including the length of time a state has been in existence and socioeconomic factors, have mixed results. Older states tend to perform better, but there is limited support for well-established economic controls such as income, income equality, and education, although Knack (2000) finds that more populous and more diverse states tend to have higher overall GPP grades.
At the level of citizen preferences and interests, Knack (2000) and Kim, King, and Zeckhauser (2002)--drawing on Putnam's (1993) hypothesis that social capital improves government--test the effect of social capital. Both report a positive and significant relationship for the overall state GPP grade. The relationship with MFR is also positive, but only Kim, King, and Zeckhauser report it as significant. Knack finds that states with many interest groups but little diversity between groups tend to have a better MFR system. This suggests that interest-group activity pressures government to undertake MFR but conflict between groups will cause government to shy away from formalizing preferences through goals. Kim, King, and Zeckhauser find that an entrepreneurial business climate leads to higher MFR grades, because states that are probusiness are also likely to introduce privatesector practices into government. Berry's (1994) analysis on the diffusion of strategic planning complements this finding, noting that government agencies are more likely to adopt strategic planning if they work closely with the private sector.
For variables relating to public sector decisions, the only significant result is the positive relationships between more professional legislatures on the one hand and both MFR systems and overall government capacity on the other. Finally, at the level of structure of formal authority, Kim, King, and Zeckhauser (2002) confirm that states with stronger governors tend to have stronger MFR systems but not better overall grades. This aligns with GPP findings that MFR reforms--or at least their objectives--appear to garner the greatest support from executive branch leadership, and with Berry's (1994) emphasis on role of governors in initiating strategic planning, especially in the early parts of their terms. Berry also finds that states are more likely to pursue strategic planning in times of strong fiscal health, a finding that coalesces with the widespread adoption of the MFR practices during the prosperous 1990s.
Having reviewed high-level governance variables that shape management capacity, the rest of this article deals directly with management practices designed to enable MFR reforms to work as intended: the integrative facilitators.
The integration across systems and processes of government, fundamental to building performance capacity, was easily sketched in figure 1 and is widely accepted as the de jure system of performance management. However, ensuring that this system works as designed is difficult to achieve, and there is a degree of thought and effort that makes the de jure MFR process a reality that separates high-performing from low-performing states. To work effectively, each part of the system illustrated in figure 1 must contain appropriate information, link to different parts of the system, and be viewed as legitimate by governmental actors.
The basic goal of MFR--improved decisions through the use of performance information--faces a number of prominent dangers in this respect: If the MFR system fails to provide information, or the information is perceived as lacking value, or if the system fails to link information to decision-venues or is viewed as another compliance exercise, its overall utility is diminished. In overcoming these dangers, the effective creation and distribution of performance information depends to a great extent on factors we call integrative facilitators, factors that the high-performing states employed to make their MFR process work. Integrative facilitators are the practical actions that ensure that systems work as theorized. For MFR systems, such connectors need to ensure that strategic goals link to performance measures, that performance information is formulated in a way to be useful for decision venues, and that performance information actually reaches desired venues. Ideally, this means that MFR systems cross executive-legislative lines and are used by senior decision makers. Ultimately, however, such use depends on the willingness of the actors at each step in the system, which is clearly a factor of both bureaucratic and political will (Radin 2000).
Unlike the high-level governance variables described in the previous section, both the MFR criteria and the integrative facilitators deal with MFR management capacity, but they deal with different aspects of that capacity. A short methodological comment that identifies the conceptual distinction between the facilitators and the criteria is therefore appropriate. The criteria are standards of capacity to be pursued; the facilitators are actions to enable these standards to be reached. Given the potential overlap between the criteria and facilitators, it is important from a methodological standpoint to be clear about how this conceptual distinction blurs in practice. With the exception of vertical integration and guidance of agency efforts influencing the adoption of an additional criterion of coordination, the integrative facilitators do not have an equivalent criterion. (7) Measures used to satisfy the MFR criteria did not deal with the integrative facilitators of comprehensiveness, a balance between top-down and bottom-up approaches, and political oversight and commitment. Although the integrative facilitators are not independent variables in the same way as the socioeconomic or governance factors discussed earlier, we claim a conceptual distinction between integrative facilitators and the criteria.
Data and Methodology
Gill and Meier (2001) point out that both public managers and academic analysts are more interested in the experience and lessons of high performers than of the average or poor performers and that these high performers may substantively differ not only from the average agencies but from the above-average agencies. We focused on states that performed well against the GPP criteria: states that received a B+ or higher grade in the GPP assessment of MFR practices. The GPP multimethod approach of collecting information and assigning grades to all state governments matches Gill and Meier's standard for identifying high performers: "The designation of the best agencies should rely on systematic methods that combine quantitative analysis with qualitative assessments, rather than anecdotes" (Gill and Meier 2001, 16).
Following Yin's (1994) recommendation to select cases of two contrasting theoretical conditions to provide a form of theoretical replication, we also examined states that struggled to satisfy these criteria: states that received a D grade or worse. Data was collected on the MFR practices of all fifty state governments for two separate points in time, 1998 and 2000, providing a developmental perspective to emerge over time (Agranoff and Radin 1991) and an overall case-study selection pool of one hundred cases. Sampling from the universe of state governments permits a systematic and generalizable analysis not available from a single or small set of case-studies. There is a reasonable level of consistency in states that performed well--eleven states received a B+ over the two years; six of these received the grade in both 1998 and 2000. Five states received a D or lower, but only one state received a D for both years. A total of sixteen states was studied.
Drawing on data collected as part of the GPP and follow-up analysis (interviews of state officials, an open--and closed-ended written survey, and content analysis of MFR documents), we identified a series of management characteristics logically associated with enabling an MFR system and examined the frequency of such characteristics among states with high and low grades. We judged that a factor was present to the extent that a state could (a) demonstrate awareness of a practice, (b) articulate it as a management practice that was desirable to achieve, and (c) demonstrate actual implementation of the practice. Although the research presented here presents these factors as dummy variables, they are in fact continuous and vary considerably between states. We note simply that reduction of continuous variables into more limited categories is a frequent practice in qualitative analysis seeking to assert the relative presence or absence of a conceptually distinct variable (Miles and Huberman 1994). With this caveat in mind, an advantage of the high-low approach is that it allows a clearer categorization of presence or absence of key practices. High performers had multiple sources of evidence that demonstrate an awareness of the complexities of MFR and focused on actual implementation. States with low grades were, overall, not cognizant of or attempting to achieve the factors discussed.
We present the general patterns of results in table 4. The table demonstrates the finding that management characteristics present in states that perform well according to the GPP criteria tend to be absent among states that perform poorly. Each integrative facilitator is discussed in turn.
Comprehensiveness of the MFR System
States that scored well tend to promote goal setting and the use of performance information throughout government at both the statewide and the agency levels. Such states recognize the limitation of adopting a purely agency-level or statewide approach and describe MFR as a single unitary system that incorporates statewide and agency goals. Missouri, for example, introduced statewide goals--"Show Me Results"--because agencies needed more central direction in developing their strategic plans and the governor wished for the budget, the statewide plan, and agency plans to focus on the same set of goals.
A comprehensive approach implies a common results framework and language among decision makers throughout the planning, performance measurement, and decision phases in the MFR process illustrated in figure 1. Eschewing a common approach reduces the potential for these three stages of the MFR process to link together. For example, Iowa, recognizing a disconnection between the legislative and executive approach to MFR, recently created a joint legislative and executive working group "to reach consensus on a common conceptual framework and language through which to pursue MFR." The disconnection had led to agencies preparing their budget requests in two formats, a traditional line-item format and a format that linked requested allocations to results. The legislature funded a Budget Redesign project to offer a model that eliminated duplication and consolidated performance information within the budget process.
Vertical Integration of Goals
Vertical integration emphasizes consistency between different levels of performance goals, that is, among the center, agencies, and programs. However, vertical integration of goals proves to be the exception rather than the norm in actual MFR practice. High-performing states have begun to recognize that the multitiered nature of an effective MFR system poses challenges and are making an effort to ensure vertical integration of goals. In relation to figure 1, this means ensuring that low-level goals match high-levels goals in strategic planning, that high-level planning is relevant to program planning, and that performance targets reflect strategic goals. Six states feature a legislative requirement for explicit links between statewide plans and agency plans (Snell and Grooters 1999).
One such state is Iowa, which uses the term "alignment" to describe the coordination of strategic planning at different levels--statewide, agency, program, and individual employees. Guidelines for statewide strategic planning, agency strategic planning, budgeting for results, and process improvement are published together and provided to all department heads with the stated purpose of reinforcing the point "that these processes are all part of the way that Iowa State government does business and that they are very interrelated."
Another example of an effort to foster vertical integration is in Oregon, where a Benchmark Blue Book intends to display the connections between the high-level outcome goals enumerated by the Oregon Benchmarks and state programs. In Texas, the governor's update of the strategic plan is distributed at the beginning of the biennial planning process with the idea that agencies align strategic plans and budget proposals to reflect the governor's goals. Agency heads receive a letter asking that these goals be included in the agency plans. In addition "agencies are required to align with the Governor's vision and other elements and are required to specifically identify which goals and benchmarks their strategies impact and to discuss the nature and degree of that impact. Agency budget requests also must include this alignment and must specifically include the goal and benchmark codes for each requested item of appropriation."
Balancing Top-Down and Bottom-Up Approaches to Results
Building an MFR system that includes congruent statewide and agency-level goals requires central coordination (which offers policy coordination and political accountability) with high levels of discretion at the agency level (which allows for the development of detailed MFR information and taps into substantive agency expertise). In relation to figure 1, efforts to balance a top-down and bottom-up approach mean that primarily senior decision makers determine high-level strategic planning goals and that their decisions are informed by substantive expertise of lower levels of the organization. Lower-level strategic goal setting and performance measurement are roles for agency-level staff--given their expertise and knowledge of higher-level goals--but subject to central review.
Central coordination is generated through strong central administrative agencies and committed political leadership, whereas agency discretion provides agencies with primary planning authority for its substantive area. The complexity of substantive policy areas, the breadth of government-wide responsibilities, the need for detailed performance information, and the advantages of agency ownership of goals point to the necessity of giving agencies freedom in developing goals and performance measures. An MFR system that is centrally driven may become a compliance exercise and fail to infiltrate the culture at the agency level. Washington State noted that "our view is that driving this [MFR] change into the management culture of agencies, in addition to connecting measures and planning to the budget process, will help state government more fully realize the benefits of performance measurement. Real improvement in state government performance is created in every single state agency, not in the budget office."
Close examination of the processes of government suggests the bottom-up-top-down divide is a false dichotomy. A strong MFR process finds an appropriate balance between these, as well as between the roles of elected officials and those of career administrators. Most of high-performing states get this mix right. Elected officials provide a statewide strategic vision that is based on political goals and informed by lower-level planning efforts; they also provide the political demand for quality that can make the difference between an MFR system that works and one that fails. Agencies, informed by broad statewide goals, create their own plans and measures. As described in the next section, central administrative agencies facilitate the flow of communication and focus on creating linkages between high-level goals and agency objectives.
An example is Missouri's strategic planning process. There is a clear and defined set of roles with specified responsibilities between political and administrative actors. The executive team includes the governor and is tasked with defining priority results, providing directions to the subcabinet, communicating results to citizens and stakeholders, and overseeing and approving of subcabinet recommendations. Subcabinet teams are made up of department directors or deputy directors and focus on key substantive areas, developing plans to improve the "Show Me Results" process, measuring and reporting performance of the "Show Me Results," and overseeing research teams. Research teams are made up of planners, policy analysts, researchers, and program managers, and one or more teams are assigned to each subcabinet team. Research teams support the subcabinet, providing the substantive answers for policy questions. The strategic planning process identifies strategic questions that the MFR process is intended to answer, with a clear understanding of who is responsible for each question.
Clear Guidance for Agency Efforts
MFR legislation in many states details exactly the type of information to be featured in agency strategic plans or budget proposals. This may seem to be an overly narrow prescription for strategic planning and performance measurement, particularly from an agency perspective. However, this approach is not inconsistent with allowing a high amount of discretion for agencies in developing their plans. Requiring specific information standardizes the type of information generated by agency efforts, while allowing agencies to decide the content of that information.
Requirements for and support in producing specific types of information are featured in the strategic planning and performance measurement phases of the MFR process illustrated in figure 1. The purpose of central guidance is to increase the potential for use in the decision-making phase because standardization should make information more understandable, transparent, and comparable to users dealing with large amounts of data. The types of information that agencies are required to produce include mission statements, core values or philosophy, goals, lower-level objectives, strategies for implementation, output and outcome performance measures and targets, timelines for implementation and an internal-external assessment (or Strengths-Weaknesses-Opportunities-Threats analyses). Some states, including Texas, Florida, and Louisiana, require discussion of the validity and reliability of each individual performance measure presented. Louisiana requires that every performance measure be subject to a process of critical questioning designed to improve confidence in the information presented and to weed out measures that fail to provide satisfactory answers to key questions.
Requiring specific types of information does not, of course, automatically enable agencies to produce such information. In meeting their legislative and administrative reporting requirements, however, agencies in high-performing states found frequent support and guidance. This support typically comes from a central administrative body with a reservoir of management expertise. Support often takes the form of training for agency officials, although interviews with agency officials emphasize the importance of having their own experts and training processes internal to the agency. It is notable that among high-performing states central agencies tended to be very active, but defined their role in the limited terms of supporting other decision makers rather than making decisions themselves. In practice such support means institutionalizing an MFR system, ensuring the regular flow of information that is valuable to decision makers, and making the information easily accessible to them. Survey evidence demonstrates that where central agencies attempted to move beyond a support role and attempt to control the actual strategic agenda, reported use of performance information decreased (Moynihan and Ingraham 2001).
Although many states use consultants to develop and improve their MFR systems, high-performing states tend to ensure a store of in-house expertise to support agencies. Missouri chose in-house capacity after early use of consultants led to "conflicting terminology and approaches. This set some of our agencies back in their understanding of the Missouri model. Strategic plans were therefore not as strong and consistent as we would have liked. In the past several years, however, Missouri has developed extensive training for its agencies that has resulted in a much greater understanding of the strategic planning model and stronger strategic plans."
Information processed through a central agency in an age when employee involvement and managerial flexibility are key themes may seem antiquated. However, legislation in seventeen states provides that a central agency oversee or facilitate agency planning (Snell and Grooters 1999). One role of central agencies is to communicate information downward to ensure that agencies understand the statewide vision and operationalize it in a way consistent with political intention. As well as actively communicating statewide goals to the agency level, Utah emphasizes the potential for a central review to ensure the consistency of goals with the Utah Tomorrow statewide plan. Another role for central agencies is to facilitate the flow of information upward to senior officials. MFR systems produce huge amounts of information, with potential for high variance in quality and format. Without a means of processing and standardizing information to make it more understandable, elected officials will not find it useful.
In some instances it makes more sense to move the discretion down to the point where the information is generated rather than to bring the information to senior decision makers unlikely to view it. Even in this setting, with information communicated laterally--between agencies or within agencies--central agencies play a role in coordinating the flow of information. Electronic systems increasingly collect, store, and disseminate detailed amounts of information, offering advantages in terms of timeliness, information storage capability, and reduced transaction costs. The Department of Planning and Budgeting in Virginia has developed a Web-based system, Virginia Results,, that allows immediate access to regularly updated agency information that is disaggregated at increasingly detailed levels of agency activity.
Leadership and Commitment
Many respondents to the GPP emphasized the key role of leadership in making MFR work, preferably with support in both the executive and legislative branches (Joyce and Tompkins 2000). The experience of high-performing states confirms the importance of leadership, political oversight, and commitment in building support for MFR systems. When elected officials are interested in using MFR to pursue better governance and are willing to take on quasi-administrative oversight roles, the system tends to work. Governors in the highperforming states offered a range of positive examples of personal involvement in and leadership of the MFR process. For example, in Iowa, the governor met with department heads who act as enterprise planning teams four times a year. Each team has responsibility for one of the governor's six statewide strategic goals. Meetings review goals, data availability, "what is working well, and what barriers need to be addressed and how to address them."
The role of committed leadership is critical to all aspects of MFR: in choosing whether to pursue MFR, promoting its value and importance to other governmental actors, and institutionalizing MFR as a system (a role that executive leadership tends to delegate to central agency officials, as described in the previous section; lngraham, Sowa, and Moynihan 2002). In relation to the actual MFR process illustrated in figure 1, political leaders should be involved in the strategic planning phase, but their most challenging role is to ensure visible use of MFR in decision making. Perhaps the most important contribution that elected leaders can make in this regard is encouraging and enabling agency-level leadership and managers to use performance information, as survey evidence suggests that this is likely to lead to greater overall use of performance information in making decisions about governmental activities (Moynihan and Ingraham 2001).
In high-performing states where working MFR systems were not in place, new governors made MFR a political priority, actively promoting its virtues. In Missouri, the late Governor Carnahan raised the focus on MFR by calling for a handful of regularly updated and publicly communicated "Show Me Results" on key aspects of life in Missouri. He also created a commission on management and productivity to oversee this process and foster a greater emphasis on the "goals of improving government efficiency and productivity and making service to the public, rather than bureaucratic activity, the measure of program success." In states where a strong MFR system is already in place, the role of the governor is to build new vision into and further institutionalize the existing framework. Indeed, one of the great dangers to constructing an MFR system is the threat of discontinuity due to electoral change. Over the past ten years, Texas has shown that it is possible to continually renew and improve an MFR system even under the stewardship of governors with differing political beliefs.
Where succession poses a problem, administrative continuity becomes critical for leadership to fulfill its MFR roles. Kentucky observed that "our primary obstacle that has stood in the way of better use of performance measures in Kentucky is the lack of opportunity for gubernatorial succession. Succession gives the Executive Branch, at all levels of state government, more comfort and confidence that performance measurement systems put in place are meaningful and will lead to better decision-making." In Virginia, extraordinary administrative leadership with an ability to integrate gubernatorial goals with existing institutional strengths compensates for the lack of continuity created by a gubernatorial office limited to a single four-year term.
Missouri also describes the discontinuity of all types of elected officials--the transition of a new governor, legislative term limits and turnover, and leadership changes in constitutionally independent departments--as the most prominent potential challenge to MFR. Conversely, among states that received lower grades, lack of political support for the MFR process was a recurring theme. South Dakota commented that "without political will, formal strategic planning and performance or outcome-based budgeting processes start to fall apart."
An additional point about commitment and will is necessary. The consistent identification of specific leaders or teams of leaders, both political and bureaucratic, in effective MFR systems was a marked characteristic of high-performing states. The qualities of leadership associated with effective MFR systems were quite specific: The willingness to defend the performance management process publicly, to utilize negative information as well as "good news" in constructive and public ways, and to commit to a consistent oversight of the implementation process were commonly identified. Interviews at the agency level in selected states also confirm the recognition of leadership as a vital factor. Agencies where MFR not only is implemented but also is a distinct part of the management culture feature leaders who commit to the idea of managing by clear strategic goals and evaluating performance. These insights into leadership deserve additional analysis in future research.
Many of the broad patterns described here sound much like the criteria for effective managing for results described earlier. We caution the reader, however, that these commonalities overlay much more subtle patterns and balances. Clear and firm central coordination and direction is of little use if there is not commitment to the process at the agency level or if the capacity to gather critical information simply does not exist. Leadership is a complex blend of teamwork, political will, and stamina. Communication--clear and consistent communication-is critical to every step of the process. The elements of MFR symbolism--increased political accountability, efficient government, open government--are double-edged swords. When it works, MFR has enormous potential for improving the effectiveness of government. When MFR is not effective or exists only "on the books," it becomes a source of increased cynicism about the ability of government to change. The high-performing states analyzed fully understood the level of commitment and effort that effectiveness demands. States that did not do well lacked this understanding.
Although a great deal has been written about MFR, the need to step back and examine its rapid diffusion and that diffusion's likely impact remains. Effective collection and utilization of information can be critical to integrating across governmental systems and to building capacity for performance--or to alerting decision makers to places where they need to build capacity. Information collection can also be simply another burden in administrative life. It is necessary to consider the conditions under which information should be deemed effectively utilized. The factors we have identified attempt to make performance information accessible and useful, but appear to be a precursor to the use of information for management purposes, not a guarantee of use. Most states can offer anecdotal evidence of performance information use, reporting highest levels of use among agency-level managers, but admit that systematic use is difficult to demonstrate. Even in the high-performing states, we found little evidence of good evaluative information crossing into legislative decision making. The presence of all the positive factors we identified, it appears, does not ensure full use of results-based data.
Despite the caution expressed earlier, there is a silver lining. As more states and other governments embrace the concept and the processes contained in MFR, guidance emerges from its analysis and from lessons learned from the states that are doing well. These lessons include:
* Identify the role and responsibility of the many actors in the MFR process ahead of time, most crucially for decision making. Elected officials, central agencies, and line agencies have different yet crucial roles to play. Elected officials create the major strategic governance goals; line agencies operationalize strategies in terms of specific objectives, information collection, and early evaluation; central agencies coordinate efforts, communicate findings, and review strategic goals, measures, and actual performance.
* Lead the system. Leadership can come from a variety of places and may look different in every setting, but someone in a leadership position must commit to making the system work. Further, this analysis reveals, the leadership commitment must be long term and consistent. Absence of leaders' commitment to managing for results was a common characteristic of unsuccessful systems.
* Recognize that effective results-based government balances accountability systems, blending and balancing political accountability with managerial accountability and performance measurement accountability. All three are necessary to success.
* Recognize the value of simplicity. Many systems gather information that lends itself to complicated metrics that don't lead to useful information. Effective systems gather information that will be used and are clear about where and when the information will be used. Finally, this analysis points to an important, but somewhat unrelated conclusion: It is clearly possible for governments to learn in meaningful and constructive ways from other governments, rather than from other sectors The most successful governments we analyzed had learned many of their lessons the hard way: by getting it wrong the first time That part of the learning process does not have to--and should not be--replicated.
APPENDIX: ELEMENTS OF THE QUANTITATIVE ANALYSIS
The following list provides a summary of the factors used in a quantitative assessment of a state's MFR system, derived from closed-ended written survey questions and a content analysis of MFR documents
* Formal MFR system in place
* Published statewide plan
* Length of time statewide plan in place
* Agency plans in place
* Agency plans publicly in place
* Common format for agency plans
* Central guidance for agency plans
* Percentage of agencies with agency plans
* Frequency of revisions
* Communication of strategic goals and performance results to the public
* Involvement of staff in setting strategic goals
* Involvement of public or stakeholders in setting strategic goals
* Documentation of performance measures in performance reports
* Verification of the accuracy of performance measures
* Use of IT systems to coordinate performance information
* Use of performance information by elected officials and managers
* Use of performance measures and targets for contracted out services
Content analysis of MFR documents (statewide plan, budget, and sample of agency plans-education, transportation, and corrections-and performance reports) for the following factors:
* Vision statement, mission statement, core values, medium long-term goals, short-term goals, performance measures, performance targets
* Linking of responsible actor, unit, program, or organization to goals
* Assessment of key external factors
* Level and type of goals and measures
* Clarity of goals
* Consistency of measures with strategic goals
* Cross-year comparison of performance indicators
* Comparison of actual performance with performance target
* Consistency between goals in agency documents and statewide documents
Table 1 GPP State Grades for Managing for Results 1998 and 2000 State 1998 MFR Grade 2000 MFR Grade Alabama F D+ Alaska C- C- Arizona B- C+ Arkansas D C- California C- C- Colorado C C+ Connecticut D+ C- Delaware B B Florida B C+ Georgia C+ B- Hawaii C- C Idaho C- C- Illinois C B- Indiana C B- Iowa B+ A- Kansas C C+ Kentucky B B+ Louisiana B B+ Maine C C+ Maryland B- B Massachusetts C C Michigan B B+ Minnesota B B Mississippi C D+ Missouri A- A- Montana C C Nebraska B- B- Nevada C C New Hampshire D+ D New Jersey B- B- New Mexico D+ C New York D+ C- North Carolina B- B North Dakota D C- Ohio C+ B Oklahoma D+ D+ Oregon B+ B Pennsylvania B- B Rhode Island C C South Carolina B- B South Dakota D D Tennessee C B- Texas B+ A- Utah B+ B+ Vermont B- B Virginia A- A- Washington B+ A- West Virginia C C Wisconsin C C Wyoming C C+ Table 2 Government Performance Project Managing for Results Evaluation Criteria for State Government 1. Government engages in results-oriented, strategic planning. Strategic objectives are identified and provide a clear purpose. Government leadership effectively communicates strategic objectives to all employees. Government plans are responsive to input from citizens and other stakeholders, including employees. Agency plans are coordinated with central government plans. 2. Governmnet develops indicators and evaluative data that can measure progress toward results and accomplishments. Government can ensure that data is valid and accurate. 3. Leaders and managers use results data for policy making, management, and evaluation of progress. 4. Government clearly communciates the results of its activities to stakeholders. Table 3 Governance Factors Leading to Improved Management Capacity Knack 2000 Overall GPP MFR Grades Variable Grades 1998 1998 Historical Mature state Socioeconomic factors Income per capita - - Income inequality - - Education + + Population size [+.sup.a] + African American population [+.sup.a] + Citizen preferences and interests Good government groups Number of interest groups + [+.sup.a] Diversity of interest groups - [-.sup.a] Political culture b b Social capital [+.sup.a] + Entrepreneurial climate Tax burden Variables relating to public sector decisions. activity, and influence Political competition b b Political ideology Democratic governor + + Professional legislature The structure of formal authority Strong governor Kim et al. 2002 Overall GPP MFR Grades Variable Grades 1998 1998 Historical Mature state [+.sup.a] [+.sup.a] Socioeconomic factors Income per capita Income inequality Education Population size African American population Citizen preferences and interests Good government groups - - Number of interest groups Diversity of interest groups Political culture Social capital [+.sup.a] [+.sup.a] Entrepreneurial climate [+.sup.a] [+.sup.a] Tax burden + + Variables relating to public sector decisions. activity, and influence Political competition - - Political ideology + - Democratic governor Professional legislature [+.sup.a] [+.sup.a] The structure of formal authority Strong governor + [+.sup.a] Grades 1998 1998 Note: +, positive effect on grades; -, negative effect on grades. [sup.a.Significant]. [sup.b.Tested] and found to be not significant and not included in final model. Table 4 Metamatrix of Integrative Facilitators in High- and Low-Performing MFR Systems State, Grade Vertical and Year Year Comprehensive Integration B+ or higher Iowa 98/00 X X Missouri 98/00 X X Oregon 98 X X Texas 98/00 X X Utah 98/00 X X Virginia 98/00 X X Washington 98/00 X X Florida 00 0 0 Kentucky 00 X X Michigan 00 0 X Louisiana 0 X X D or less New Hampshire 00 0 0 North Dakota 98 0 0 South Dakota 98/00 0 0 Alabama 98 0 0 Arkansas 98 0 0 Balance Guidance between Political State, Grade, for Agency Top-Down Oversight and and Year Efforts and Bottom-Up Commitment B+ or higher Iowa X X X Missouri X X X Oregon 0 X X Texas X X X Utah X X X Virginia X X 0 Washington X X X Florida X X 0 Kentucky 0 X X Michigan X X X Louisiana X X X D or less New Hampshire 0 0 North Dakota 0 0 0 South Dakota 0 0 0 Alabama 0 0 0 Arkansas 0 0 0 Note: X, existence of a factor; 0, absence of a factor.
(1) The Baldridge criteria, the European Foundation for Quality Management, and Balanced Scorecard methodologies are clear examples of criteria-based approaches used for assessing public (and private) organizations.
(2) A partial exception to this statement is the use by the World Bank and the International Monetary Fund of standardized approaches to evaluating aspects of governance, for example, civil service systems, public expenditure management, and transparency. The standards themselves are publicly accessible, but country evaluations based on these standards or county-by-county comparisons are usually not released externally.
(3) Scales were first developed separately for survey responses and content analysis. A number of different weightings were created to reflect different scenarios, put greater emphasis on individual state practices, and reweigh particular criteria or subcriteria. The benefit of examining different weightings is to provide a sense of the overall robustness of scores, to highlight a state's relative strong and weak points, and to provide a check on each of the different scales. Another benefit is that it does not reward states that have the most performance information or the greatest number of types of performance information. This prevents the possibility of endogeniety problems between the scales and the integrative facilitator comprehensiveness described later in this article.
(4) Moynihan and Ingraham (2001,9) describe the process of calculating intercoder reliability and correcting for disagreement in detail: "The coding process for analyzing surveys and documents included the following steps:
* A coding scheme and state codebook for the conversion of qualitative data from documents into quantitative data was designed. The state codebook was an improved version of the city codebook used in previous GPP analyses of city MFR documents.
* An instruction book furnishing standard definitions for variables and clarification of the coding scheme and codebook was designed.
* Coders were trained to develop a common understanding of the survey and documents, and consistent application of the coding scheme.
* A coding pretest of three sample states was performed.
* The coding scheme and codebook was revised based on feedback from the pretest.
* The instruction book for the revised coding scheme and codebook was expanded.
* The documents were coded, using two coders working separately for each document.
* Inter-coder reliability was computed.
* A process for resolving coding discrepancies was developed and employed.
* Intercoder reliability was assessed using percentage pairwise agreement, the portion of the total number of observations for which the coders were in complete agreement (Bullock and Tibbs 1987). The percentage pairwise agreement is 82.89%."
(5) Since the pilot round of the GPP, the MFR criteria have been adjusted slightly to incorporate the idea of coordination between plans, This change arose directly from evidence of the pilot and first round of state surveys, where it became apparent that MFR was frequently uncoordinated to a problematic degree. Because the coordination criterion arose directly from observing variation in state management practice, it is unsurprising that it has a close equivalent in the integrative facilitators of vertical integration and guidance for agency efforts.
(6) This and all other quotations featured in the article come from the Government Performance Project Managing for Results written survey, unless otherwise attributed. For further information on the GPP, visit www.maxwell.syr.edu/gpp/.
(7) The overlap between the criterion "agency plans are coordinated with central government plans" (criterion I d) and the integrative facilitators vertical integration and guidance of agency efforts creates circularity between the criteria and two of the integrative facilitators. However, this potential endogeniety, problem does not appear to shift the outcome of the results. Criterion Id makes up only 6.25% of the quantitative scales employed to assess state MFR practices, and the scales themselves were only part of the information used for grading purposes. If we were to examine quantitative scales based on the criteria but without criterion Id, there is no variation in terms of the rank order of states. We maintain the facilitators because they were not included in the original MFR criteria (which applied to the 1998. bat not the 2000, analysis), but were added only after they were identified as a desirable management practice.
Agranoff, Robert, and Beryl Radin. 1991. The comparative case study approach in public administration. Research in Public Administration 1:203-31.
Ansoff, H. Igor, ed. 1976. From strategic planning to strategic management. London: John Wiley.
Bardach, Eugene. 1998. Getting agencies to work together: The practice and theory, of managerial craftmanship. Washington, D.C.: Brookings institute.
Behn, Robert. 200 I. Rethinking Democratic Accountability. Washington, D.C.: Brookings Institute.
Berry, Frances Stoke. 1994. Innovation in public management: The adoption of strategic planning. Public Administration Review 54:322-30.
Beyle, Thad. 1999. The governors. In Politics in the American states: A comparative analysis, ed. Virginia Gray, Russell L. Hanson, and Herbert Jacob. 7th ed. Washington, D.C.: CQ Press.
Brudney, Jeffrey L., Ted Herbert, and Deil S. Wright. 1999. Reinventing government in the American states: Measuring and explaining administrative reform. Public Administration Review 59, no. 1:19-30.
Bullock, R.J. and Tibbs, M.E. 1987. The case meta-analysis for OD. Research for Organizational Change and Development 1:171-228.
Dye, Thomas. 1966. Politics, economic and the public: Policy outcomes in the American states. Chicago: Rand McNally.
Easton, Allan. 1973. Complex managerial decisions involving multiple objectives. New York: Wiley.
Gawthrop, Louis C. 1984. Public sector management: Systems and ethics. Bloomington: Indiana University Press.
Gill, Jeff, and Ken Meier. 2001. Ralph's pretty-good grocery versus Ralph's super market: Separating excellent agencies from the good ones. Public Administration Review 61, no. 1:9-17.
Gormley, William T., Jr., and David L. Weimer. 1999. Organizational report cards. Cambridge, Mass.: Harvard University Press.
Heinrich, Carolyn J. 1999. Do government bureaucrats make effective use of performance management information? Journal of Public Administration Research and Theory 9, no. 3:363-93.
Ingraham, Patricia W., and Amy E. Donahue. 2000 Dissecting the black box revisited: Characterizing government management capacity. In Government and performance: New perspectives, ed. Carolyn J. Heinrich and Laurence E. Lynn Jr. Washington, D.C.: Georgetown University Press.
Ingraham, Patricia W., Philip G. Joyce, and Amy E. Donahue. Forthcoming. Government performance: Why management matters. Baltimore, Md.: John Hopkins University Press.
Ingraham, Patricia W., and Donald P. Moynihan. 2001. Comparing management systems and capacity: The benefits of a criteria-based approach. Paper presented at the 5th Annual IRSPM Conference, April 2001, Barcelona, Spain.
Ingraham, Patricia W., Jessica E. Sowa, and Donald P. Moynihan. 2002. Public sector integrative leadership: Linking leadership to performance in public organizations. Paper presented at the Research Workshop on the Empirical Study of Governance, February 21-23, 2002, Texas A&M University, College Station, Tex.
International City/County Management Association (ICMA). 2000. Comparative performance measurement: FY 1999 data report Washington, D.C.: ICMA.
Jacoby, William G., and Saundra K. Schneider. 2001. Variability in state policy priorities: An empirical analysis. Journal of Politics 63:544-68.
Joyce, Philip G., and Susan S. Tompkins. 2000. Using performance information for budgeting: Clarifying the framework and investigating recent state experience. Paper presented at the 2000 Symposium of the Center for Accountability and Performance of the American Society for Public Administration, George Washington University, Washington, D.C.
Katz, Daniel, and Robert L. Kahn. 1966. The socialpsychology of organizations. New York: Wiley.
Kim, Mark T., David C. King, and Richard J. Zeekhauser. 2002. The management performance of US states. Retrieved on June 30, 2003, from http://ksgbome.harvard.edu/~. dl(mg.academic.ksg/ Grading_States_vl.PDF.
Knack, Stephen. 2000. Social capital and the quality of government: Evidence from the U.S. states. Policy Research Working Paper 2504. Development Research Group, World Bank, Washington, D.C.
Langley, A. 1988. The role of formal strategic planning. Long Range Planning 21, no. 3:48.
Lynn, Laurence E., Jr., Carolyn J. Heinrich, and Carolyn J. Hill. 2000. Studying governance and public management: Challenges and prospects. Journal of Public Administration Research and Theory 10, no. 2:233-61.
Macintosh, Norman B. 1994. Management accounting and control systems: An organizational and behavioral approach. Chichester, England: Wiley.
Melkers, Julia E., and Katherine G. Willoughby. 1998. The state oftbe states: Performance-based budgeting requirements in 47 out of 50. Public Administration Review 58, no. 1:66-73.--. 2001. Budgeters' views of state performance-budgeting systems: Distinctions across branches. Public Administration Review 61, no. 1:54-64.
Miles, Michael B., and A. Michael Huberman. 1994. Qualitative data analysis. 2d ed. Thousand Oaks, Calif.: Sage Publications.
Moynihan, Donald P. 2001. The state of the states in managingJbr results. Retrieved June 30, 2003. from http://www.maxwell.syr.edu/gpp/pdfs/The State of_the_States In MFR.pdf.Moynihan. Donald P., and Patricia W. lngraham. 2001. When does performance information matter to performance information use? Putting the factors in place. Paper presented at the Sixth Public Management Conference, School of Public and Environmental Affairs, October 14-18, Indiana.
Putnam, Robert. 1993. Making democracy work: Civic traditions in modern Italy. Princeton, N.J.: Princeton University Press.
Radin, Beryl A. 2000. The government performance and results act and the tradition of federal management reform: Square pegs in round holes. Journal of Public Administration Research and Theory 10, no. 1:111-35.
Rossi, Peter H., and Howard E. Freeman. 1989. Evaluation: A systematic approach. Newbury Park, Calif.: Sage.
Scott, William G. 1961. Organization theory: An overview and an appraisal. Academy of Management Journal 4:7-26.
Snell, Ron, and Jennifer Grooters. 1999. Governing-/or-results: Legislation in the states. Draft Report presented to the National Council of State Legislatures.
Toft, Graham S. 1989. Synoptic (one best way) approaches to strategic management. In Handbook of strategic management, ed. Jack Rabin, Gerald J. Miller, and W. Bartley Hildreth. New York: Marcel Dekker.
Weimer, David L., and Aidan R. Vining. 1992. Policy analysis: Concepts and practice. Englewood Cliffs, N.J.: Prentice Hall.
Wildavsky, Aaron. 1973. If planning is everything, Maybe it's nothing. Policy Sciences 4:127-53.
Yin, Robert K. 1994. Case study research: Design and methods. 2d ed. Thousand Oaks, Calif.: Sage Publications.
Donald P. Moynihan
Texas A&M University
Patricia W. Ingraham
|Printer friendly Cite/link Email Feedback|
|Author:||Moynihan, Donald P.; Ingraham, Patricia W.|
|Publication:||Journal of Public Administration Research and Theory|
|Date:||Oct 1, 2003|
|Previous Article:||Transaction costs and institutional explanations for government service production decisions.|
|Next Article:||Strategic planning environment, process, and performance in public agencies: a comparative study of departments in Milwaukee.|