Accountability by numbers: how the Lone Star State's auditor introduced a balanced scorecard management system.

Everywhere you look, good business management seems to be in short supply. Recent examples abound of poorly run companies such as Qwest, Sunbeam, Andersen, Tyco, WorldCom, Enron and most of the late dot-coms. So what causes such conspicuous failures? Not lack of ideas (there are plenty of those). Most experts agree the major cause of business meltdown is management error, which occurs across the enterprise spectrum, not just at high-profile companies.

How is it at your business? Has a project you thought was on time and on budget recently blown both measures and caught you off guard? Can you identify trouble spots right now? How would you know about them, and what changes could you make if you did? CPAs who consult to companies, operate in industry or are managing partners can follow this case study to learn how to implement a balanced scorecard oversight system, which organizes performance data clearly while showing how operational areas are linked.

A MEASURED RESPONSE

About seven years ago, Lawrence F. Alwin, CPA and Texas state auditor, decided the State Auditor's Office (SAO) needed a better way to track how it was meeting its mission to provide lawmakers and agencies with the most useful possible financial operating data for taxpayer-funded projects. The SAO's monthly status reports were thorough: They analyzed staffing, summarized time use for all employees and evaluated budget and progress performance for about 150 projects. In fact it took four people three days to compile the data into four-inch-thick binders and distribute them. The problem: Each division handled its performance data a little differently, so deciphering the information stalled meetings and knocked projects off schedule.

"We weren't getting the right data at the right time for good decision making" Alwin says.

In 1996 I was one of Alwin's audit directors when a Harvard Business Review article, "Using the Balanced Scorecard as a Strategic Management System" by Robert Kaplan and David Norton, caught my eye. I suggested to Alwin the SAO might benefit by using the process it described, which links objectives, measures, targets and initiatives to collectively depict a business strategy and how to achieve it. Based on the idea that one benchmark can't show true performance, it recommended using an array of indicators to express an organization's strategic goals and progress toward them. For example, it linked measures for process (on-time delivery) and finance (revenue growth) with others such as customer feedback and employee knowledge.

Alwin agreed the approach could be useful. His first step toward instituting it was to choose which information categories SAO monthly status reports would systematically address: financial performance, audit performance, customer reaction and overall results. Audit performance would track three areas: the audit projects completed at the end of each month, the number likely to meet the next month's deadlines and the reasons some audit reports were late (see "Performance Measurement Ground Rules," page 64). It would give managers more time to concentrate on solving operational problems.

SETTING GOALS

As Alwin saw it, the SAO's new measurement system's essential goals were to

* Generate accurate reports. SAO measurements had to yield clear, accurate, timely data that were parallel from agency to agency.

* Establish visible accountability. Alwin assigned responsibility for producing results to specific managers, and he sharpened the benchmarks.

* Motivate informed corrective action. Managers would use monthly executive meetings to share ideas to correct problems based on what they had learned. To meet those goals, Alwin had to

* Designate staff to accomplish objectives.

* Improve employee efficiency (minimize the gaps, overlap, friction and errors that occur when people aren't clear about their job duties).

* Make knowledge more transparent (give people fast feedback to help them find ways to work better).

* Eliminate process redundancies to reduce costs (see "Before and After," page 71).

He then focused on five steps for implementing the goals. They were

* Prepare a strategic plan (action outline). The strategic plan's implementation actions would be tracked by performance measures that fit the SAO's goals. (For more information on making or updating such a plan, see "Strategic Planners Lead the Pack," JofA, Dec.01, page 26.)

* Identify critical business objectives for each balanced scorecard "perspective" or area view (in effect, windows into different aspects of an organization's performance) and link each one to the goals of the strategic plan. Name an individual "owner"--typically a member of the management team--for each objective.

* Develop a limited number of measures--cycle time, quality or volume, for example--to evaluate progress toward objectives. It's important to not have too many measures, which can cause "data fatigue."

* Set measurable goals (targets or budgets). Measures such as "reduce employee turnover by 5% in 12 months" or "cut administration expenses by 3% in nine months" clarify priorities. Targets have to be realistic as well as challenging. Setting the bar too high would demoralize workers and cause the system to fail.

* Collect and discuss data at least monthly. This is where measurement turns into management: At regular meetings, owners explain why they got the results they did; if targets have been missed, they say what steps will get them back on track.

PEOPLE, TIME, MONEY AND ACCOMPLISHMENTS

To get the balanced scorecard system under way, the SAO had to delineate its goal and the actions to reach it. A private company might pick maximizing revenue growth or capitalizing on a market niche as a strategy to fulfill a goal to "become the most profitable" in its sector, for example. A public-sector profitability strategy could be to offer more services without increasing program costs-that is, stretch finite resources further (see "What to Track," page 67).

Alwin convened a cross-section of about 20% of the SAO staff (from receptionists to auditors, twenty-somethings to near retirees) to state their goal, to update the agency's strategic plan and to identify the data most relevant to its mission. Their goal: Provide the most accurate, timely financial information to Texas legislators to effect better public-policy outcomes. He asked participants: "What should we do to fundamentally improve that service?"

In a series of one-day meetings over a six-week period, staff members offered more than 400 answers. Alwin took a month to review them, noting which themes participants brought up again and again. The process homed in on four strategies (see exhibit 1, below): The SAO would provide independent audit services (AS) as chartered, provide educational services (ES) to teach agency staff how to avoid problems and improve accountability, offer management advisory services (MAS) to help agencies improve operations and manage the state's compensation structure and address other human resources issues (SCO).

After the first group's strategic planning input was complete, Alwin next worked with an implementation team of SAO managers to adapt its mission's four supporting scorecard areas: financial, customer, process and knowledge (see exhibit 2, page 63). At the, top of the scorecard, the implementation team added a fifth category--mission--to represent the public-sector bottom line: public-policy outcomes related to SAO data.

To measure its information's utility, the SAO collects data on how legislators use its audit reports in actual hearings and debates. A related measure tracks the dollar value created through audits, including cost savings, cost avoidance and identified efficiency gains (the public-policy outcomes). For example, a sportsman or woman who wants to go fishing now fills out a two-part form, mails it in with payment and waits for the state to enter the data and issue and mail a fishing license. Using the scorecard to track the licensing cycle shows wait time as an inefficiency. One possibility to reduce it would be to use a system where a sportsman or woman enters driver's license data at a service center, makes payment and obtains a license on the spot (getting better service for less money).

The implementation team initially was puzzled about how the financial area view (making money) fit into the public sector, but then it reasoned that because state agencies operate on a budget and can't generate earnings, they create value if they increase what they do with those funds. Accordingly, the SAO chose to track how efficiently it spent taxpayer dollars.

The data tracked in the knowledge area view show what staff skills are required to get a task done. It reveals the match (or gap) between competencies the SAO needs and those employees possess. For example, during the late-1990s boom, this category showed a decrease in the number of CPAs on staff. Because auditing requires CPA knowledge, the data signaled human resources to fill the gap and the financial manager to make money available for salaries.

The SAO'S internal process area view tracks whether projects get done on time and whether audits meet high standards, and it encourages people to experiment with ways to improve efficiency. Finding solutions to problems indicated by process data can be simple. In an actual private-sector example, one company's productivity measure showed a drop in output from the day to night shifts. Ironically, it turned out the night shift produced less because the day supervisor, before leaving, locked needed supplies in a cabinet and took the key home.

The customer viewpoint captures feedback from important clients or users--in the SAO'S case, the agencies it audits. Before the scorecard implementation, the SAO didn't collect this information. Under the old system, an agency director calling the state auditor to express annoyance about audit findings could be construed as an impromptu performance report. In the new system, the SAO seeks to know what the agencies think of its information quality, reporting objectivity and how they view its balance and fairness, professional independence, expertise in relevant subjects and staff comportment.

The SAO now generally gets high marks, but previously it had poor communication with One agency, for example. During scorecard development, Alwin asked the agency director, "How can we serve you better?" The director replied she needed information sooner, so bad news--if any--wouldn't catch her by surprise. As a result, Alwin instituted more meetings to keep all the directors better informed about audits in progress.

Next, the team designed a system of organizational measures to track performance within each area and indicate appropriate targets (see exhibit 3, page 64). The five areas incorporate a total of nine measures, two for mission, one for customer feedback, one for internal processes, one for knowledge and four for financial.

Once the SAO had chosen the measures it thought would most comprehensively track people, time, money and accomplishments as well as report performance trends and focus everyone on results, it selected Canadian software developer Panorama Business Views (pbviews) to supply its scorecard software. The resulting system translates SAO data into charts that are accessible to all managers and are displayed on-screen during monthly meetings. Depending on how well results have met targets, the underlying data for each measure determine whether a box turns green (on track), yellow (having problems) or red (in trouble). At monthly meetings a red box makes it easy to decide what needs attention.

THE AUDIT SCORECARD

After the SAO set the officewide scorecard, Alwin and the agency managers needed drill-down capability for the AS, ES, MAS and SCO strategies. Because 85% of SAO data come from audits, the audit part of the scorecard system was especially important (see exhibit 4, page 66).

Frank Vito, CPA and director of audits, oversaw that area of the design in a process parallel to the overall development. "We needed metrics using both lead and lag measures," he says. (Lag checks data monthly, quarterly or even less often; lead checks more frequently so corrections can occur earlier.) Over a 16-month period, Vito and several managers developed an audit scorecard and measures to reach three goals related to the five areas. They were

* Eliminate unnecessary steps.

* Select audit goals similar to SAO goals (connecting audit results to strategy).

* Assemble the most representative data for expressing audit performance so managers could easily spot work trends and problems.

A SYSTEM PEOPLE CAN USE

Linking performance data in one multilevel intranet has been good for the SAO, Alwin says. The system shows how the SAO meets targets in each area, so team members can discuss how individual audits are proceeding. Managers have single-click access to detailed analysis, and this succinct, visual information lets them focus on results, solving problems, developing staff and operational planning (see exhint 5, below).

[ILLUSTRATION OMITTED]

The SAO did face challenges in implementing the scorecard system, however. The staff members had to

* Learn to trust the data. Ultimately, the support system convinced them. The scorecard protocol checks units of measure for accuracy and consistency, and the system administrator regularly reviews the data for accuracy. In addition, there is comprehensive guidance to ensure that data are reported in exactly the same way. For example, one SAO data-entry instruction is: "Enter the dollar amount of invoices budgeted for billing for the month. Take the value from the Statement of Revenue and Expenditures Report, cell F11."

* Understand they'd have less work at the end of the implementation, not more. That developed with time.

* Trust that a red box at the front of the room would not embarrass them. The fact that meeting participants mutually or cooperatively concentrate on solving problems has allayed anxiety and helped keep the process collegial.

En route the SAO learned some other lessons:

* Get manager buy-in. Managers are the link between an organization's mission and its day-to-day tasks, and most of them would rather walk on nails barefoot than change. The SAO helped develop buy-in by giving people thorough individual training, monthly training updates, participation in developing the system and clearly written reference guides.

* Make sure a project leader understands his or her accountability. Clear responsibilities and supervisory input were essential for helping staff focus on solving problems.

* Manage day-to-day implementation. Because they best understand their areas, Alwin had line managers (business leaders accountable for results) develop their own measures to incorporate into the scorecard project.

* Edit data volume. In the past the SAO had collected a great deal of performance data. The new goal was to focus on only the most relevant. Alwin insisted they let go of familiar but no-longer-meaningful measures.

* Automate data reporting. The technology had to be powerful enough to display important cause-and-effect relationships. Individual divisions needed to see in time how their performance affected each other and the end results to identify problems and make corrections nimbly.

* Lead by example. Alwin used the scorecard as a framework for meetings as soon as he could. His managers quickly did the same.

ACOUNTABILITY THAT WORKS

The SAO is in its third full year of officewide balanced scorecard management, and the benefits are clear. The agency reports information about its performance trends and outcomes in a timely way that helps clarify goals and expectations. Simply put, the office now measures only what really matters, its staffmembers know what they need to do their jobs better and accountability is a part of everyday management. "Now we have real-time, accurate management information that lets us focus on business, not on chasing data," Alwin says. "Change challenged us, but as Frank Zappa said, 'Without deviation, progress isn't possible' Our system proves its worth every day. It's that simple," Alwin adds with satisfaction.

EXECUTIVE SUMMARY

* WHEN THE TEXAS STATE AUDITOR'S OFFICE (SAO) decided its management system could better monitor how taxpayer dollars were spent, it adopted a balanced scorecard approach to track its*performance. As a public-sector entity, the SAO sought a "profitability" strategy of offering more services to citizens without increasing program costs (it aimed to stretch finite resources further).

* BASED ON THE IDEA THAT ONE BENCHMARK can't show true performance, a balanced scorecard uses an array of indicators related to an organization's strategic goals and to progress toward them. For example, it will link measures for process (on-time delivery) and finance (revenue growth) with others such as customer feedback and employee knowledge.

* THE PRIMARY GOALS for instituting a performance measurement system were to ensure accurate and timely reporting of SAO financial data, establish individual manager accountability for clearly defined results and use the score card reporting system to make informed decisions about how to efficiently use its budget.

* A CROSS-SECTION OF ABOUT 20% of its workforce collaborated on identifying the four key strategies: to provide independent audit services, to teach managers to avoid problems, to help agencies fix problems and improve operations and to make sure the state's compensation structure offered appropriate salaries.

* THE PARTICIPANTS IDENTIFIED the most important data to track and developed focused measures--cycle time, quality or volume, for example--to evaluate progress toward objectives. Too many measures can cause "data fatigue."

* THE RESULTING OFFICEWIDE, MULTILEVEL information system reports data in a visual, easy-to-understand format that gives all managers single-click access to detailed analysis and information. Meetings now are about project results, solving problems and operational planning.

Performance Measurement Ground Rules

In "What Gets Measured Gets Done,"J of A, Jun. 02, page 53, Edward Gregory and Roslyn Myers reported on the CPA Performance View, which teaches five ground rules for developing good performance measures. How does the SAO scorecard stack up?

Ground rules

* Measures should be linked to the goals and strategy of the company.

* Everyone from top management to frontline personnel must participate in developing and gathering the performance measures.

* Employees must understand the "why" of the measures before being held accountable to them.

* Employees should be held accountable only for what they can control.

* Employers should offer staff an incentive (a bonus, time off or promotion, for example) to support the new system of gathering, monitoring and improving the measures.

SAO scorecard

* Measures throughout the organization flow from the SAO's strategies. They are displayed in the reporting software used by the office.

* From state auditor to receptionist, a cross-section of employees participated in creating the measures by which their performance would be assessed.

* For each measure a detailed description of who, what, why and how is created. This information is a mouse click away in the reporting system software program. During the design, participants added new measures only where they saw holes in the measurement framework.

* Each measure has an "owner," the employee responsible for those operational results.

* The agency developed an organizational performance incentive (OPI), ar annual bonus for officewide goal achievement that rewards employees who have used the system particularly well or enhanced it.

What to Track

Start an organizational overhaul with a strategic plan. Be sure your entity's mission is clear to your executive team and all your employees. Answers to the following questions will help your organization pinpoint the performance data to track.

* Why do we exist? In the private sector the question is: What do we do to make money? In the public sector it's: What results and services are we paid to provide?

* How does any function or service we provide help us achieve our mission?

Think about your customers (clients or service users). To establish who they are and what they want or expect from you, ask the following

* How do we establish and maintain effective working relationships with our customers?

* What do our customers value?

* What must we deliver to satisfy our customers?

Think about how you run your business and how your employees do their work.

* What systems or processes are critical to achieving our mission?

* What internal processes must we excel at to promote quality and working relationships with our customers?

* What business processes deliver value to our customers and our stakeholders?

* Which processes have the greatest impact on customer satisfaction and mission or goal achievement?

* Do our employees have the knowledge and skill they need to accomplish our goals?

* What must we do to develop employee skills and promote a motivated workforce that will continue adding value to our customers?

* Public sector: How do we use our appropriations and other funding sources to achieve our mission?

* Private sector: What are our long-term financial goals? What are the important short-term goals we must meet to achieve our long-term goals and our mission?

* How will we allocate our budget so that employees and processes focus on achieving our goals and mission?

Before and After

Before adopting a balanced scorecard, the SAO used 15 different end-of-period reports (some including as many as 17 different measures) to assess how it was doing. This process was scrapped in favor of nine measures focusing on key activities. The new approach yields faster decision making, reduced operating costs and better business results. Prior to implementing the scorecard, projects were assessed at the end using "lag" measures such as "completed on time" or "completed within budget." After implementing the scorecard, progress on audits is evaluated throughout using "lead" measures that enable managers to make midcourse corrections. For example, projects meeting a fieldwork checkpoint target date (a lead measure for meeting release date targets) are more likely to complete their projects on time.

Before

* Problems might stay buried until the end of projects.

* Meeting participants often debated the meaning of the data in the reports.

* Performance data were scattered across the organization; there were 15 reports in different formats, some with as many as 17 data points.

* Cost of producing the monthly report: $3,600.

* Strategy was in the background; there were no well-defined project measures to drive performance toward specific goals,

* Four employees processed the reports.

* The focus was on processes.

* Fiscal benefit delivered to the state: $58 million.

* Performance data often were reported late or at the last minute.

After

* It is easy to spot problems midproject.

* Meeting discussions focus on problem solving and performance improvement.

* All critical performance data are collected in one system and one easy-to-use format--focus is on nine organizational measures.

* Cost of producing the monthly report: $1,600.

* Daily operations are linked to strategy: Fiscal impact, customer feedback and report release status are always available to the project manager.

* Two employees process the reports.

* The focus is on the results.

* Fiscal benefit delivered to state: $250 million.

* Performance data are available by the eighth of each month.

AICPA Measures and Management Resources

Developing and monitoring performance benchmarks for clients' businesses is a practice opportunity for CPAs. See the AICPA Web page, www.alepa.org/performanceview.

Books

AICPA practice guides include CPA Performance View Services: A Practitioner's Guide to Providing Performance Measurement Engagements, an easy-to-use handbook for PM engagements; and CPA Performance View Services: A Financial Manager's Guide to Leading Performance Measurement Initiatives, guidance for financial managers, cpa2biz.com.

Conferences

AICPA/APQC National Performance Measurements Conference;July 17-18, 2003, Denver. A forum for learning about performance measurement that offers guidance on how to align processes with goals and improve the value of CPA services. www.aicpa.org and cpa2biz.com.

Exhibit 1: Basic Scorecard

The SAO planning team identified four strategies key to its mission to provide the most accurate, timely financial information to Texas legislators: It would

* Offer more educational services to related agencies (ES).

* Improve audit services and reporting (AS).

* Offer management advisory services (MAS).

* Make sure statewide compensation levels were appropriate (SCO).

This would enable the SAO to better support taxpayer-funded projects.

[ILLUSTRATION OMITTED]

Exhibit 2: Organizational Areas

A private-sector organization's goal is to make money (financial), and it manages customer relationships, internal processes and knowledge to achieve this (left column). A public-sector entity such as the SAO has mission as its goat, which requires it to manage customer relationships, internal processes, knowledge and financial responsibilities (right column).

[ILLUSTRATION OMITTED]

xhibit 3: Organization Scorecard

The SAO performance measurement system translates data into visual
images that are displayed on-screen during meetings. The data linked to
a box determine whether it turns green (is meeting targets), yellow (is
having some minor problems) or red (is in trouble). The top row shows
the agency performance "index" is 92.8%, below its target of 100%.
Where indexes exceed targets in row 2, two views are green. "Weight"
assigns impact value; the weights in row 2 total 100%. The five views
incorporate nine measures, shown in row 3: two for mission, one for
customer feedback, one for internal processes, one for knowledge, four
for financial. Clicking on any line lets a user drill down to the next
level of data. The "owner" for SAO overall performance is Larry Alwin.
Other owners are as named.

SAO overall
performance

Actual    --
Budget    --
Index    92.8%

Owner: Alwin, Larry

Row 2:
Organizational
areas

Mission

Actual    --
Budget    --
Index    103.8%
Weight    30.0%

Owner: Alwin, Larry

Customer

Actual    --
Budget    --
Index    118.7%
Weight    10.0%

Owner: Alwin, Larry

Internal process

Actual   --
Budget   --
Index    84.1%
Weight   30.0%

Owner: Alwin, Larry

Knowledge

Actual   --
Budget   --
Index    94.1%
Weight   10.0%

Owner: Alwin, Larry

Financial

Actual   --
Budget   --
Index    76.0%
Weight   20.0%

Owner: Alwin, Larry

Row 3:
Measures for
each area

Fiscal impact

A        $555,067,182.01
B          13,333,333.34
Index             200.0%
Weight             10.0%

Owner: Alwin, Larry

Useful information

Actual   215.00
Budget   231.00
Index     93.1%
Weight    90.0%

Owner: Alwin, Larry

Customer feedback

Actual   494.53
Budget   416.50
ndex     118.7%
Weight   100.0%

Owner: Alwin, Larry

Service delivery status

Actual    --
Budget    --
Index     84.1%
Weight   100.0%

Owner: Alwin, Larry

Employee feedback

Actual     3.53
Budget     3.75
Index     94.1%
Weight   100.0%

Owner: Wilson, Dennis

AAS workforce

Actual   147.97
Budget   142.90
Index    103.5%
Mission   25.0%

Owner: Wilson, Dennis

Support workforce

Actual   90.19
Budget   90.60
Index    99.5%
Weight   25.0%

Owner: Wilson, Dennis

Budget compliance

Actual    3.76%
Budget    2.0%
Index    12.0%
Weight   25.0%

Owner: Kinton, Craig

Budget summary

Actual   --
Budget   --
Index    87.6%
Weight   25.0%

Owner: Alwin, Larry

xhibit 4: Audit Projects Scored

Audit services (AS) are the most important work of the SAO. Here data
are sorted and displayed by operational areas. Row 2 shows the four
operational areas at the SAO (AS, ES, MAS, SCO); row 3 indicates the
two AS subdivisions: audit and review and special investigations unit.
Row 4 links the five area views to the audit and review box. The
performance story here is that two divisions are somewhat below
expectations (yellow); one is meeting its goals (green); and the SCO
division isn't up to date because data are missing (gray).

SAO overall
performance

Actual    --
Budget    --
Index    92.8%

Owner: Alwin, Larry

Row 2: Audit
services area

SAO overall
performance
Assurance services (Audit)

Actual   --
Budget   --
Index    --

Owner: Vito, Frank

SAO overall performance
Educational
services

Actual    --
Budget    --
Index    97.8%

Owner:

SAO overall performance
Management advisory
services

Actual    --
Budget    --
Index    101.2%

Owner: Kerr, Deborah

SAO overall performance
State compensation
office

Actual   --
Budget   --
Index    --

Owner: Vito, Kelli

Row 3: AS
subdivisions

SAO overall
performance
Audit and review

Actual   --
Budget   --
Index    87.6%

Owner: Vito, Frank

SAO overall performace
Special investigations
unit

Actual    --
Budget    --
Index    127.0%

Owner: Weber, John

Mission
Audit and review

Actual   --
Budget   --
Index    91.3%
Weight   20.0%

Owner: Vito, Frank

Customer
Audit and review

Actual   --
Budget   --
Index    92.4%
Weight   20.0%

Owner: Vito, Frank

Internal process
Audit and review

Actual   --
Budget   --
Index    86.4%
Weight   20.0%

Owner: Vito, Frank

Knowledge
Audit and review

Actual   --
Budget   --
Index    65.8%
Weight   20.0%

Owner: Vito, Frank

Financial
Audit and review

Actual    --
Budget    --
Index    102.0%
Weight    20.0%

Owner: Vito, Frank

Exhibit 5: SAO Scorecard Electronic Briefing Book

Users can display information in a notebook format per the screen capture below. Measures on the left side have links to documents and related reports. The measure's "owner" can comment at right. Where yellow or red indicates missed targets, the owner describes how he or she will improve results. The bar chart shows results across the year. The description says what is being measured, how and by whom as well as reasons why the measure shows up in green, yellow or red.

[ILLUSTRATION OMITTED]

DEBORAH L. KERR, PhD, is chief strategy officer of the Texas State Auditor's Office. She writes and lectures on performance management topics. Her e-mail address is [e-mail].