Searching for strategy: Value for Money (VFM) audit choice in the new public management era.
Each year, the Office audits selected ministry or agency programs and activities, with the major ones generally audited every five to seven years. In deciding which areas to audit, the Office considers such factors as the results of previous audits, the total revenues or expenditures at risk, the impact of the program or activity on the public and the costs of performing the audit in relation to the benefits (Office of the AGO 2018).
This study explores the VFM inquiries of the AGO between 1997 and 2014 as a case to uncover trends of consistency of approach under the flourishing "new public management" philosophy of cutting costs and raising efficiencies. It asks three broad questions: does the frequency and intensity of VFM audits correlate with the financial importance of the ministries? Do political controversies affect the choice of what will be subject to a VFM audit? Finally, do Auditors General make a personal difference in what ministries will be audited through a VFM framework? The period 1997-2014 was chosen for review because it covers the first audits under the Progressive Conservative government led by Premier Mike Harris, elected in late 1995, and the end of the Liberal government under Premier Dalton McGuinty. It also corresponds to the unequivocal arrival of the "new public management" philosophy of cutting costs and raising efficiencies in the administration of the state (Evans and Smith 2015; Gendron, Cooper, and Townley 2001; Richards and Smith 2006) which made VFM all the more relevant to governance.
This study begins by reviewing the literature on VFM and provides background on Ontario's Auditor General's approach to VFM. It then reports on the findings of our empirical analysis, which a pattern of over-auditing of some ministries while important ministries are consistently under-audited, which challenge the notion that the audits are strategically chosen depending on risk, on size, or on impact. The study then considers other possible explanations and outlines a research agenda which should be of interest to scholars and practitioners alike.
Perspectives on VFM auditing
Remarkably few scholars have sought to track the workings of Auditors General, though there have been indications that this data could be helpful to a deeper analysis of the work of this important agent of Parliament. Most of those who research and write on audit have devoted studies on the impact of VFM far more than on its actual practice. Scholars of VFM have divided into three broad camps. The first and earliest either contested or defended the legitimacy of VFM either as instruments to improve the management of public affairs or as a control on democratic government. The second group, more contemporary, examines VFM through an impact lens. The third camp specializes in better understanding the technical aspects of VFM.
VFM and legitimacy
Michael Power, in his landmark The Audit Society: Rituals of Verification (1997; see also 2000), argued that modern society increasingly demanded that auditors function as an institutional check on those entrusted with decision-making and spending. In terms of politics, he pointed to the late 1980s/early 1990s when government VFM audits in the United Kingdom were applied with increasing frequency as demand to verify government action and practices grew louder. Power distinguished three core characteristics that made VFM distinct from other forms of assessment. First, there had to be a certain level of independence from the subject under investigation to avoid bias, methodologically collected evidence, and a series of recommendations based on the findings. Second, the results of an audit had to be based on solid financial information to legitimize any advice on possible reforms. Third, audits had to ensure accountability: they had to identify exactly which parts of the organization had not lived up to their mandates. For all his attention to method, however, Power did not test his characteristics on which ministries were actually covered by the AG. Admittedly, this was never his focus: he likely assumed that the process was consistent.
Power's claims of the spread of audit were challenged by a few authors (Bowerman, Raby, and Humphrey 2000; Humphrey and Owen 2000; Lindeberg 2007.) on the grounds that the practice was not sufficiently widespread to justify his claims. The issue of the comprehensiveness of the VFM is compelling, and given the concerns expressed at the beginning of this century, must be tested: to what degree are government operations actually audited for VFM? Given the general concern about government efficiency, why are government ministries not systematically audited on their performance? What is the logic of auditing some and not others?
VFM as power tool
In the Canadian context, the nature of the Auditor General attracted a fundamental debate among political scientists. Donald Savoie (1995), Sharon Sutherland (1986) and Denis Saint-Martin (2004) considered VFM audits as political activities that essentially challenged democratic government. David Good labelled the federal Auditor General as a "watchdog/' overseeing the political government spenders and guardians (Good 2014). Because of this oversight responsibility, Good considered the Auditor General as independently powerful, easily misconstrued as separate from Parliament (Good 2014).
Hatherly (1999) argued that audits typically fell into two models, accountability or hierarchical, that also raised political choices. In the accountability model, the relationship between auditors, the audited and stakeholders could be seen as lateral, based on an exchange between the auditor and the group being audited. The power dynamic in this model allows for feedback between the auditor and the audited, but leaves the stakeholders as external actors since there are few mechanisms for interaction or dialogue, especially regarding feedback on quality (Hatherly 1999). The hierarchical model was based on regulatory control stemming from the stakeholder's lack of willingness or ability to oversee the group in question. Pressures from external standards set expectations of the audited, while internal procedures regulate its actions. Hatherly argued that the real-world application of the horizontal accountability model tended to naturally evolve into the hierarchical model. Internal regulations became increasingly influenced by the external standards set by the auditor, not the equitable, bilateral relationship that governs the accountability model. As this happened, stakeholders became increasingly removed from the process, which increased the power awarded to the auditor to ensure accountability and oversight of the audited. The audit took on a more corporatist approach that sought to establish auditee compliance through report publication rather than an assessment of auditee performance and function.
VFM in technical areas
The third school has focused on trying to theorize the process of VFM audits. Jeppesen et al. (2017), for instance, have identified four strategic options in their study of four Nordic national audit offices: a performance auditing strategy; a financial auditing strategy; a portfolio strategy; and a hybrid strategy and found that most had adopted a portfolio strategy. This study was based on interviews with principals, and did not document the subjects of VFM audit.
Very early on, Roberts and Politt (1994) argued that VFM enhanced democratic accountability, but fell short of the standards expected in a program evaluation or review. Keen (1999) posited a model that showed how aviditors mixed intuitive and analytical thinking, empirical data, and behavioural observations in audits. Otto Brodtrick's 2004 study of Auditors examined tangentially what they examined and depicted VFM choices mainly as the product of routine and rationality. Based on interviews with auditors and VFM auditees, Alwardat, Benamraoui, and Rieple (2015) pointed out that while audit procedures in the UK did lead to better managerial practices, the VFM processes were not well institutionalized, which they felt could lead to a number of conflicts. Also in the Canadian context, Mayne (2004, 2006, 2010) has published provocative articles challenging various aspects of VFM in practice.
In her many surveys of line ministry managers, Danielle Morin shed light on the role of aviditors. She showed that VFM audits improved public organization and that the process was necessary for effective and efficient governance (Morin 2004, 2014; Morin and Hazgui 2016). The Auditor General was seen as a necessary check on government that could ultimately streamline and improve organizational processes (Morin 2004). Moreover, through the VFM, the Auditor General was seen as playing a fundamental role in alerting the public to weaknesses in the management of public organizations (Morin 2003; Van Loocke and Put 2011). Morin's study confirmed findings in Scotland, as well as local governments in Finland and Norway, which revealed that VFM audits are regarded by the auditees as an important and efficient tool for controlling and reforming the public sector (Morin 2008; Lapsley and Pong 2000; Johnsen et al. 2001). In the Canadian context Taft (2016) concluded that recommendations by Auditors have been more frequently accepted and implemented over the past twenty years. Still, the assumption was that VFM audits rigorously covered all aspects of spending. In another original contribution on the work of Auditors General in the Westminster system, Talbot and Wiggan (2010) examined how public sector audit offices self-reported on their performances. They found that while audit offices make useful contributions in analysing their work, they do not report a concern for what ministry or program is actually audited (see also Leclerc 1996; Sharma 2007; Sterck 2007; Leeuw, 2011). In sum, the literature is heavily engaged in formulating theories and scrutinizing practices. Yet no study has documented exactly what ministries are audited in VFM approaches.
VFM audit in Ontario
Typical of other provinces and the government of Canada, Ontario was an early adopter of VFM audits (Morin 2003; Desautels 2001). It conducts VFM audits to report on "whether government programs are being well managed and whether they provide good value for the money spent on them" (Office of the AGO 2015). Its reports cast judgment on whether government programs are administered with "due regard for economy, efficiency, and environmental impact" and if effective performance measures are in place to establish program accountability (McGlashan 2011). The Auditor General Act allows the Office of the Auditor General to go beyond government ministries and audit broader public sector organizations, crown corporations, municipalities and agencies receiving provincial funds. The Office employs approximately a hundred people and spends an annual budget of around $16 million (Office of the AGO 2015).
Such a large mandate would justify the assumption that all ministries are systematically audited to determine the degree to which they deliver value for money. There exists no evidence to support this supposition. Moreover, the AGO does not justify its audit choices beyond what it declares on its website.
Three hypotheses can be identified to test the scope of the AGO's auditing strategy. First, it would be logical to assume that the AGO is more intensely focused on larger ministries that spend greater proportions of public funds. Thus the Ministries of Health and Long-Term Care and Education, which collectively made up sixty-four per cent of the 2014 provincial budget, would be expected to proportionately receive the most audit scrutiny, in terms of both frequency and intensity. A competing hypothesis would be that the AGO makes choices as a response to political pressure for special studies to be conducted, in other words where risk has been perceived by exogenous actors. These can emanate from media revelations, the Public Accounts Committee of the Ontario legislature or from the government itself (legislation allows the premier of Ontario or the cabinet to ask the Auditor General to conduct special audits). If this is the case, VFM audits would be expected to spike in sectors with spending controversies. A third hypothesis would be that VFM audit choices are influenced by individual auditors, driven by their own intellectual and professional interests and perceptions of risk. In the period under study between 1997 and 2014, the AGO was led by two men with strikingly similar backgrounds: Erik Peters (1993-2004) and Jim McCarter (2004-2013). Bonnie Lysyk began her tenure as Auditor General in late 2013, so only the final year covered in this study involves her mandate. All three were chartered accountants who had worked in both the public and private sectors; McCarter and Lysyk also completed an MBA.
To identify the targets of VFM audits, AGO reports were compiled by ministry and categorized into eight sector categories: health, transportation, justice, community and social services, finance, education, energy and environment, and "other" for the remaining audits ("other" is a large catch-all category of one-off studies on ministries, crown corporations, colleges and universities, school boards and children's aid societies that do not reappear consistently). These groupings were chosen in order to correspond with budget categories in the province. In each sector category, the aim was to gauge VFM audits based on the frequency (number) of scrutiny during the selected period, the comprehensiveness of audits per sector, and policy recommendations. For each year under inspection, ministerial expenditures were compiled into the eight categories applied to the VFM publications. To control for inflation and fluctuating budgets, the proportion of allocations for each category was calculated, giving a percentage figure. Audit frequency was measured as a percentage of the overall yearly VFM efforts.
Audit intensity (or comprehensiveness) was measured by the length (in page numbers) of each audit report and recorded. The operating assumption is that programs and services typically receiving greater page allocations were audited in more detail; certainly a longer report would reflect the depth and detail of an audit investigation. Another dimension of this analysis required a deconstruction of the VFM report contents in order to reveal the kinds of objectives provided in notable publications. Thus a content analysis was applied to the AGO's scope to determine if there were any recurring elements in their evaluation criteria, or if risk was of particular concern.
Media scans were conducted on VFM coverage for each sector category to test media influence and determine whether there was a relationship between the AGO's interests and what was topical in mainstream news stories. The assumption was that if the AGO was influenced by political trends in the media coverage, there would be a corresponding increase in VFM reports in that sector. A limitation to this method comes from the sole reliance on online media sources. Internet use by the press was more limited in the mid-to-late 1990s, possibly resulting in fewer online uploads by mainstream news organizations. As a result, this study is likely to capture more stories found covering the AGO as the years progressed.
Finally, to measure the influence of individual Auditors General in terms of audit choices, the results were divided between the tenures of Peters and McCarter.
The number of yearly VFM audits remained relatively consistent throughout the eighteen-year period under study (Table 1), ranging from a high of 38 in 2013 to a low of 26 in 2001, with an average of 29 in the Peters years and 33 in the relatively larger ministry led by McCarter. The health category received the most attention from the AGO's VFM reports under both Auditors, with an overall average of 6.4 reports a year, typically constituting more than twenty per cent of the AGO's VFM output per year (Table 2). The aggregation of "other" categories that fell outside the eight broad budget categories used by the government of Ontario consistently came in second place, taking over eighteen per cent of the VFM audit output. Next in prevalence was community and social services with four reports per year. It constituted an average of over thirteen per cent of the output over the years examined. Transportation received the least attention, averaging less than two VFM audits a year, or just above five per cent of the output (Table 2). The second lowest frequency of VFM audits was the massive ministry of education, receiving less than three reports a year, less than ten per cent of the output.
Hypothesis 1: budget weight
The results demonstrate that the AGO's interests are not determined by funding allocations. The health sector reflected this clearly: it consumed an average of 37.8 per cent of the provincial budget during this period, but was the object of 20.3 per cent of the yearly VFM reports (Table 2). In contrast, there seemed to be more intensity in the reports produced by the AGs on health care ministries. The average report over the period under study was 92.6 pages long, reaching a high of 171.5 pages in 2008 and only 40 pages in 1997 (Table 3). These significant fluctuations in report length minimize any obvious trends and do not support the budget weight hypothesis. On average, VFM reports in all ministries over the period under study were 56 pages long.
In striking contrast to the health sector, the justice sector received far more attention from the AGO than its budget would justify. It averaged 11.5 of the total reports, yet only received an average of 3.7 per cent of the budget (Table 2). Interestingly, the VFM audits were fairly comprehensive of this sector, totalling approximately 54 cumulative pages per year (Table 4). Similarly, the environment, energy and natural resources sector was audited five times more per year (11.5 per cent) than would be expected by a 2.3 per cent budget allocation. Audit comprehensiveness was also quite high at an average of 58 pages. Conversely, the finance sector, which made up 14.7 per cent of the budget, comprised only 10.1 per cent of the yearly VFM reports. These publications were also less comprehensive, with an average page count of 44 pages. One of the more surprising results was the audit count of the education sector which was expected to garner significant attention to coincide with its 20.4 per cent annual budget allocation. VFM reports averaged to half as many, hovering at around 9.3 per cent. These reports were lacking the comprehensiveness of larger, more detailed studies, and averaged 39 pages. Thus, the logical assumption that ministries with greater funding would receive a proportionate amount of regulatory attention is not consistent with the evidence. This study's findings do not support the hypothesis that there is a relationship between budget and VFM frequency per sector as there does not appear to be a correlation between the two.
The closest relationship between VFM audit frequency and the proportion of the budget was evident in the community and social services sector. It received 13.3 per cent of the VFM audits, as well as receiving an average of 13.9 per cent of the average provincial budget.
If the AGO's choice to audit programs was based on financial importance, then it would be expected that the proportion of VFM audits should more closely match the budget allocation for that sector. While it is clear that healthcare, as the largest expenditure in the government of Ontario, would have commanded more attention than other ministries, it did not do so in proportion to its weight in government. The health care sector is significantly under-audited.
It is apparent that the choice of VFM audits is not driven by how much it costs the treasury to fund a program. Other factors must drive the AGO's decision. Is risk a consideration? Possibly, but it stretches credulity. Are the ministries of the Environment, Energy and Natural Resources inherently more risky? Strikingly, few programs in these ministries are concerned with direct-to-client services. More research is needed in determining how choices are made if indeed they are based on risk, and this is more likely to be realized following a sustained research program of interviews.
Hypothesis 2: the impact of political controversy
The influence of political controversies on the AGO's selection process is evident in the division between new/special and follow-up reports. This division is consistently equal in all sector categories except for health, finance, and the "other" aggregate. There was an average of 3.4 (53 per cent) new/special reports on the health sector and 3.0 (47 per cent) follow-up reports out of a total average of 6.4 yearly audits. The remaining five sector categories showed an equal division. The finance and "other" sector categories had the same split between new/special reports (56 per cent) and follow-ups (44 per cent). It is possible that, due to the high rate of program output by the health and finance sectors, the AGO is limited in its resources to conduct more follow-up reports. However, this does not explain why the education sector has an even division despite having a similarly high policy output. Interestingly, the "other" aggregate category had more new/special reports than follow-ups. This may simply be due to the AGO's reduced interest in smaller ministries and programs, so it produces new/special reports more often than follow-ups.
The impact of political controversy may shed light on the perceptions of risk of the AGO. It seems that issues in the media drive at least a portion of AGO VFM audits, certainly far more that those triggered by the perceived administrative risks. There was no shortage of controversies in this period affecting a number of ministries: Government Services (IT project cost-overuns); Ministry of Transport (highway 407); the Ministry of Energy (the Mississauga and Oakville Power Plants; costly Green Energy initiatives); the Ministry of Health (the ORNGE and E-Health controversies) to name but the more prominent. Government debt was driven from $101,9B in 1995-96 to $268B in 2013-14, making Ontario the world's most indebted sub-sovereign borrower. But there again, in terms of audits, controversies hardly seemed to move the AGO significantly in terms of its audit choices.
Looking at the neglected sectors can help reveal patterns in the AGO's interests. Several ministries were altogether ignored. For example, there were no VFM reports conducted on the energy sector and its affiliated ministries and programs in 1999, 2001, 2003, 2010, and 2012, or nearly thirty per cent of the period under investigation. Similarly, there were no audits of the ministry of the Environment in 1998, 2001, and 2013, while Natural Resources was also neglected in 1999, 2001, 2003, 2010, and 2012. These omissions are remarkable given the important programs at work throughout this time.
The media influence on the AGO's selection process was also evident in another way. The difference between new/special reports and follow-ups may reveal a propensity for the AGO to quickly shift its focus to high-profile issues rather than to take stock of previous recommendations to track their progress. If there is a bias toward new/special reports, then the VFM is actively pursuing policy issues raised by the media rather than responding to political pressures to review established programs and policies. When VFM reports precede news coverage, one could assume that this reflects the AGO's favoured position in the media as a source of salacious news stories regarding government "waste of tax dollars." However, when the reverse is true, it may reveal the influence that the media has on the AGO's selection process for audits, contributing a novel explanation for the AGO's scope unseen in the literature.
By favouring high profile ministries, the AGO has secured its role as the most credible player in assessing programs and expenditures (Hatherly 1999). Coupled with its strong media presence, the office wields significant power not only to highlight issues in accountability, but in some ways to also enforce its message through public backing. For example, the November 2012 report on the Education Sector Collective Agreement for 2012-2014 had a significant impact on the media's spin of the conflict. With the AGO ultimately deciding that the government had acted in accordance with managerial and efficiency standards, public opinion shifted against the teacher's union (Hammer and Alphonso 2013).
An impact such as this one should not be taken lightly. The AGO acts a public watchdog for pointing out government shortcomings and areas requiring increased accountability, and the media is quick to tout its support for this role. In the news scans conducted for this research paper, no negative media reports on the AGO were identified. All the same, this link between political controversy and VFM choice needs a great deal more attention from researchers in a comparative framework. The evidence would seem to show that the AGO is far more reactive in making its choice of audits (and therefore more political, in a limited sense) than it is proactive in anticipating risk.
Hypothesis 3: the impact of the individual Auditor General
There seems to be little substantial difference based on the tenure of each Auditor General as to the choices made. For instance, VFM audits in the years of Erik Peters featured 5.5 reports each on the health sector and the mixed bag of "others." The same trend could be observed in the work performed by the Auditor General's office under Jim McCarter. Flowever, the education category attracted relatively more attention under McCarter, as did environment and energy. Erik Peters paid relatively more attention to social services and justice.
Given the general consistency, there is little evidence that personal Auditor General choice was a factor. The small differences are more driven by controversies or government requests for special inquiries. For instance, the ministry of Child Services was particularly under fire at the turn of the century while energy issues were more prevalent at the end of the period under study (the issue of gas-generated energy plants, in particular, drove a number of AGO studies). This snapshot is limited because there were only two individuals at the helm of the AGO's office for most of this period. All the same, the study invites comparison with other jurisdiction over a lengthier period.
As decentralization of government progressed, the AGO's role as the preeminent accountability watchdog has become practically uncontested. Audit choices thus become a key concern and the problem must be asked: what motivates the AGO's choice of targets in conducting VFMs? Are they made based on strategic management concerns, as the AGO claims on its website? Using the data available, this study posited three key hypotheses as the bases for choices: the professional views of the individual AGOs; the budgetary importance of the various ministries/agencies; or triggered by political controversies. The evidence shows no purely rational incentive to audit ministries based on budget allocations as VFM audit proportions did not align with budget proportions. Moreover, there is no evidence that a change in the chief Auditor had a material impact in the choices made.
There is some evidence that political controversies did influence the AGO's selection process when it comes to ministries with high policy turnover, but the relationship is more correlational than causational. The health, finance, environment and energy, and "other" aggregate sector categories were unevenly reviewed between new/special and follow-up reports. Lastly, the influence of the media on which programs were scrutinized more intensely is not consistent across all sectors, but does present itself sporadically, such as the VFM report on ORNGE Ambulance Air Services in 2012. There is evidence to support the hypotheses that the AGO selects programs and ministries on which to conduct its VFM audits based on political and incidental factors, but it is limited to a few sector categories. These are reaction-audits, certainly not based on anticipated risk.
This study leaves open the possibility that other hypotheses could explain audit choice. It could be, for instance, that the AGO has focused on issues other than programs (for example, governing issues, regulatory initiatives, outsourcing, and partnerships) that may explain the profile of audited ministries. Audits may also have been triggered by internal concerns that have not been captured by the media or by AGO reports. Finally, it could be argued that VFM audit selections are more driven by a history of concerns (that is, that some ministries or programs are more vulnerable to failure and thus a regular concern of Auditor General of the province). If so, these concerns have not been articulated by the Auditor General of the province, nor by the Public Accounts Committee of the Ontario legislature. Finally, the "other" category may need to be refined for a closer examination and other distinctions.
Regardless, the enormous gaps in what gets audited show that Westminster, at least in the Ontario case, is still far from realizing the comprehensiveness of government audit, let alone an "audit society." This policy conundrum may require a case study approach to document and analyse the sectors under investigation by the AGO in order to reveal trends. Of the three central hypotheses of this paper, the literature appears to only support the rational argument for the AGO's interests as something of a risk-reducing mechanism for government (Power 1997; Hopwood and Miller 1994). Furthermore, any links between media coverage and the AGO's reports have proven to be minimal and tend to reflect the perspective that the AGO influences mainstream discourse, not vice versa. In the words of Sharon Sutherland, the media is "following the auditors' script" (Sutherland 1986). While this may often be true in the case of revealed "scandals" in misappropriations of money, there is little evidence that auditors follow media leads unless a major scandal is uncovered.
This study reveals that the Auditor's Office in Ontario has not followed a particular risk-based or administrative-innovation strategy in its focus on VFM except to track its operations on a fixed formula that has been consistent for the entire period under study. The Auditor's Office in Ontario sticks to "what's been done before" of what ministries will be examined, and to what degree. It has proven to be remarkably path-dependent and generally unresponsive to the budget importance of various ministries, or to their news-worthiness. As such, many ministries are left relatively under-evaluated, while others are systematically over-examined.
These findings should interest both practitioners of audit as well as scholars (see Funkhouser 2011). At a minimum, it calls on auditors to justify the reasons for their choices of audit and to demonstrate that, over time, they are indeed contributing to making audit as universal as possible. For scholars the propositions of this article call for more research. How far back does the pattern identified in this relatively narrow study go? How does it compare with other jurisdictions? Many new lines of inquiry on the office of Auditors General in the Westminster system are thus urgently needed.
Sana Adi is a graduate of Ryerson University's Master of Public Policy and Administration program, class of 2015. Patrice Dutil is Professor in the Department of Politics and Public Administration, Ryerson University, Toronto (http://patricedutil.com). The authors thank the journal's editor and anonymous reviewers for their helpful comments.
Alwardat, Y. A., A. Beriamraoui, and A. Rieple. 2015. "Value for money and audit practice in the UK public sector." International Journal of Auditing 19: 206-217.
Bowerman, M., H. Raby, and C. Humphrey. 2000. "In search of the audit society: Some evidence from health care, police and schools." International Journal of Auditing 4: 71-100.
Brodtrick, O. 2004. "How does the Auditor General's office see itself?" Canadian Public Administration 47 (2): 225-42.
Desautels, D. 2001. Reflections on a Decade of Serving Parliament. Report of the Auditor General of Canada to the House of Commons: Ottawa, Canada.
Evans, Bryan, and Charles W. Smith. 2015. "The transformation of Ontario politics: The long ascent of neoliberalism." In Transforming Provincial Politics: The Political Economy of Canada's Provinces and Territories in the Neoliberal Era, edited by Bryan M. Evans and Charles W. Smith. Toronto, Canada: University of Toronto Press, pp. 3-18.
Funkhouser, M. 2011. "Accountability, performance and performance auditing: reconciling the views of scholars and auditors." In Performance Auditing: Contributing to Accountability in Democratic Government, edited by Jeremy Lonsdale, Peter Wilkins and Tom Ling. Cheltenham: Edward Elgar Publishing Limited.
Gendron, Y., D.J. Cooper, and B. Townley. 2001. "In the name of accountability: State auditing, independence and new public management." Accounting, Auditing and Accountability Journal 14 (3): 278-310.
Good, D.A. 2014. The Politics of Public Money. Toronto, Canada: University of Toronto Press.
Hammer, Kate, and Caroline Alphonso. 2013. "Ontario teachers divided over return to extracurriculars." The Globe and Mail, 24 February, https://www.theglobeandmail.com/news/national/education/ontario-teachers-divided-over-return-to-extracurriculars/ article9013649/
Hatherly, D.J. 1999. "The future of auditing: The debate in the UK." The European Accounting Review 8 (10): 51-65.
Hopwood, A. G., and P. Miller. 1994. Accounting as social and institutional practice (Vol. 24). Cambridge, UK: Cambridge University Press.
Humphrey, C., and D. Owen. 2000. "Debating the 'power' of audit." International Journal of Auditing 4: 29-50
Jeppesen, K.K., T. Carrington, B. Catasus, A. Johnsen, K. Reichborn-Kjennerud, and J. Vakkuri. 2017. "The strategic options of supreme audit institutions: The case of four Nordic countries." Financial Accountability and Management 33: 146-170.
Johnsen, A., P. Meklin, L. Oulasvirta, and J. Vakkuri. 2001. "Performance auditing in local government: An exploratory study of perceived efficiency of municipal value for money auditing in Finland and Norway." The European Accounting Review 10: 583-99.
Keen, J. 1999. "On the nature of audit judgements: The case of value for money studies." Public Administration 77: 509-525.
Lapsley, I., and C.K.M. Pong. 2000. "Modernization versus problematization: Value-formoney audit in public services." European Accounting Reviezv 9 (4): 541-67.
Leclerc, G. 1996. Accountability, performance Reporting, Comprehensive Audit - An Integrated Perspective. Ottawa, Canada: Canadian Comprehensive Audit Foundation.
Leeuw, F. 2011. "What the critics say ... on the effects, the lack of effects, and perverse effects of performance audits." In Performance Auditing: Contributing to Accountability in Democratic Government, edited by Jeremy Lonsdale, Peter Wilkins and Tom Ling. Piscataway, NJ: Transaction Publishers. 231-247.
Lindeberg, Tobias. 2007. "The ambiguous identity of auditing." Financial Accountability and Management 23 (3): 337-50.
Mayne, J. 2004. "Reporting on outcomes: Setting performance expectations and telling performance stories." Canadian Journal of Program Evaluation 19 (1): 31-60.
--2006. "Audit and evaluation in public management: challenges, reforms, and different roles." The Canadian Journal of Program Evaluation 21(1), 11-45.
--2010. "Performance Auditing: Cozy, comfortable and in need of challenge." Optimum Online 40(3), 1-6.
McGlashan, L. 2011. Performance Auditing at The Office of the Auditor General of Canada: Beyond Bean Counting. Ottawa, Canada: Library of Parliament.
Morin, D. 2003. "Controllers or catalysts for change and improvement: Would the real value for money auditors please stand up?" Managerial Auditing Journal, 18(1), 19-30.
--2004. "Measuring the impact of value-for-money audits: A model for surveying audited managers." Canadian Public Administration 47 (2): 141-64.
--2008. "Auditors General's universe revisited - An exploratory study of the influence they exert on public administration through their Value for Money audits." Managerial Auditing Journal 23 (7): 697-720.
--2014. "Auditors general's impact on Administrations: A pan-Canadian study 2001-2011." Managerial Auditing Journal 29 (5): 395-426.
Morin, D, and M. Hazgui. 2016. "We are much more than watchdogs: The dual identity of auditors at the UK national audit office." Journal of Accounting and Organizational Change, 12 (4), 568-89.
Office of the Auditor General of Ontario. 2015. History of Our Office. Toronto, Canada. Office of the Auditor General of Ontario.
--2018. "What We Do" http://www.auditor.on.ca/en/content/aboutus/whatwedo.html Last verified on 30 January 2018.
Power, M. 1997. The Audit Society: Rituals of Verification. New York: Oxford University Press.
--2000. The Audit Society--Second thoughts. International Journal of Auditing 4 (1), 111-19.
Richards, D., and M. Smith. 2006. "The tensions of political control and administrative autonomy: from NPM to a reconstituted Westminster model." In Autonomy and control: Coping with agencies in the modern state, edited by Tom Christensen and Per Laegreid. London: Edward Elgar Publishing, pp. 181-202.
Roberts, S., and C. Politt. 1994. "Audit or evaluation? A National Audit Office VFM study." Public Administration 72: 527-549.
Saint-Martin, D. 2004. "Managerialist advocate or 'control freak'? The Janus-faced Office of the Auditor General." Canadian Public Administration 47 (2), 121-40.
Savoie, D.J. 1995. "Who is auditing the Auditor-General?" The Globe and Mail, 25 August.
Sharma, N. 2007. "Interactions and interrogations: Negotiating and performing value for money reports. Financial Accountability and Management 23: 289-311.
Sterck, M. 2007. "Impact of performance audit." In Budgeting, Accounting and Auditing for Results--Towards Integrated Financial Management, edited by V. Conings, M. Sterck and G. Bouckaert. Leuven: Steunpunt Bestuurlijke Organisatie Vlaanderen. http://steunpuntbov.be/eng/pub/s0405014.htm
Sutherland, S.L.1986. "The politics of audit: The federal Office of the Auditor General in comparative perspective." Canadian Public Administration 29 (1): 118-48.
Taft, J. 2016. "From change to stability: Investigating Canada's Office of the Auditor General." Canadian Public Administration 59 (3): 467-485.
Talbot, C, and J. Wiggan. 2010. "The public value of the National Audit Office." International Journal of Public Sector Management 23 (1): 54-70.
Van Loocke, E., and V. Put. 2011. "The Impact of Performance Audits: A Review of the Existing Evidence." In Performance Auditing: Contributing to Accountability in Democratic Government, edited by Jeremy Lonsdale, Peter Wilkins and Tom Ling. London: Edward Elgar Publishing, pp. 175-208.
Table 1. Number of VFM Audits by Auditor Peters McCarter Year Audits Year Audits 1997 32 2005 29 1998 30 2006 28 1999 28 2007 36 2000 28 2008 37 2001 26 2009 33 2002 32 2010 35 2003 22 2011 34 2004 33 2012 28 2013 38 Total 231 298 Average 28.9 33.1 Total 566 Average 31.5 Lysyk Year Audits 2014 37 Total 37 Average 37 Total Average Table 2. Proportion of Annual Budget (%$) and Proportion of Total VFM Reports (%A) 1997-2014 Per Category Health Trans. Justice Year %$ %A %$ %A %$ %A 1997 31.8 15.6 -- 6.3 3.1 9.4 1998 33.6 23.3 4.9 3.3 3.3 6.7 1999 34.2 17.9 2.6 7.1 3.7 10.7 2000 35.6 21.4 2.3 3.6 3.9 14.3 2001 37.3 18.5 2.3 6.9 4.2 17.2 2002 38.0 21.9 2.3 3.1 4.3 15.6 2003 39.6 13.6 2.4 4.5 4.4 18.2 2004 39.6 18.2 2.2 3.0 4.0 12.1 Avg./year 36.2 18.8 2.7 4.7 3.9 13.0 2005 40.0 17.2 2.3 6.9 3.7 6.9 2006 39.1 17.9 4.1 3.6 3.6 7.1 2007 40.5 22.2 3.1 11.1 3.6 13.9 2008 39.5 24.3 3.6 2.7 3.8 13.5 2009 42.1 24.2 2.1 12.1 3.9 6.1 2010 37.5 28.6 2.0 2.9 3.3 11.4 2011 37.1 14.7 1.9 5.9 3.5 5.9 2012 37.9 32.1 1.9 3.6 3.2 14.3 2013 38.8 18.4 2.0 5.3 3.2 5.3 Avg./year 39.2 22.2 2.6 6.0 3.5 9.4 2014 38.7 16.2 2.2 2.7 3.3 18.9 Avg. 37.8 20.3 2.6 5.3 3.7 11.5 Social Services Finance Education Year %$ %A %$ %A %$ %A 1997 16.5 25.0 15.3 9.4 15.5 3.1 1998 19.5 20.0 17.2 6.7 17.3 10.0 1999 16.8 25.0 18.7 7.1 20.0 7.1 2000 15.5 14.3 18.1 7.1 20.5 7.1 2001 15.4 20.7 18.6 10.3 19.8 6.9 2002 14.2 6.3 16.7 9.4 18.5 6.3 2003 13.4 18.2 15.6 9.1 19.8 13.6 2004 13.0 9.1 17.4 12.1 18.8 3.0 Avg./year 15.5 17.3 17.2 8.9 18.8 7.1 2005 13.0 10.3 15.8 6.9 19.5 10.3 2006 13.2 10.7 14.0 7.1 19.4 7.1 2007 12.0 5.6 12.7 13.9 19.8 5.6 2008 12.6 16.2 13.3 2.7 19.8 13.5 2009 13.3 12.1 11.8 12.1 19.9 9.1 2010 12.6 11.4 13.2 14.3 20.0 14.3 2011 12.0 14.7 10.1 17.6 24.6 11.8 2012 11.7 3.6 12.6 17.9 25.0 14.3 2013 12.4 10.5 11.5 5.3 24.5 13.2 Avg./year 12.5 10.6 12.8 10.9 21.4 11.0 2014 12.2 5.4 11.2 13.5 25.3 10.8 Avg. 13.9 13.3 14.7 10.1 20.4 9.3 Env. & Energy Other Year %$ %A %$ %A 1997 11.1 9.4 6.9 21.9 1998 1.5 13.3 3.0 16.7 1999 2.0 7.1 3.2 17.9 2000 3.2 17.9 2.7 14.3 2001 2.0 0.0 3.6 10.3 2002 2.1 15.6 3.9 21.9 2003 1.1 4.5 3.8 18.2 2004 1.4 15.2 3.7 27.3 Avg./year 3.1 10.4 3.9 18.6 2005 1.3 6.9 4.7 34.5 2006 1.3 21.4 5.1 26.9 2007 1.5 11.1 5.5 16.7 2008 1.5 13.5 5.9 13.5 2009 1.5 9.1 5.4 15.2 2010 1.3 5.7 6.2 11.4 2011 1.8 14.7 9.3 20.6 2012 2.4 7.1 5.4 3.6 2013 2.2 23.7 5.3 15.8 Avg./year 1.6 12.6 5.9 17.6 2014 2.1 10.8 4.9 21.6 Avg. 2.3 11.5 4.9 18.2 Table 3. New/Special (N/S) VFM Reports and Follow-Up (F) VFM Reports 1997-2014 Per Category Health Trans. Justice Year N/S F N/S F N/S F 1997 2 3 1 1 1 2 1998 2 5 0 1 2 0 1999 3 2 1 1 2 1 2000 5 1 1 0 2 2 2001 2 3 1 1 3 2 2002 3 4 0 1 3 2 2003 2 1 0 1 1 3 2004 3 3 1 0 1 3 Total 22 22 5 6 15 15 Avg./year 2.8 2.8 0.6 0.8 1.9 1.9 2005 4 1 2 0 1 1 2006 3 2 0 1 1 1 2007 4 4 2 2 3 1 2008 6 3 1 0 4 1 2009 4 4 2 2 0 2 2010 5 5 0 1 0 4 2011 2 3 0 2 2 0 2012 5 4 1 0 4 0 2013 5 2 2 0 0 2 Total 38 28 10 8 15 12 Avg./year 4.2 3.1 1.1 0.9 1.7 1.3 2014 2 4 0 1 3 4 Grand Total 62 54 15 15 33 31 Avg. 3.4 3.0 0.8 0.8 1.8 1.7 Social Services Finance Edu. Year N/S F N/S F N/S F 1997 4 4 1 2 1 0 1998 3 3 1 1 1 2 1999 3 4 1 1 1 1 2000 2 2 1 1 1 1 2001 3 3 2 1 1 1 2002 1 1 2 1 1 1 2003 2 2 0 2 2 1 2004 2 1 2 2 0 1 Total 20 20 10 11 8 8 Avg./year 2.5 2.5 1.3 1.4 1.0 1.0 2005 1 2 2 0 1 2 2006 2 1 0 2 2 0 2007 1 1 3 2 1 1 2008 4 2 1 0 3 2 2009 3 1 3 1 2 1 2010 1 3 4 1 2 3 2011 2 3 4 2 2 2 2012 0 1 2 3 2 2 2013 2 2 0 2 3 2 Total 16 16 19 13 18 15 Avg./year 1.8 1.8 2.1 1.4 2.0 1.7 2014 2 0 4 1 2 2 Grand Total 38 36 33 25 28 25 Avg. 2.1 2.0 1.8 1.4 1.6 1.4 Env. & Energy Other Year N/S F N/S F 1997 2 1 2 5 1998 3 1 1 4 1999 0 2 3 2 2000 3 2 4 0 2001 0 0 1 2 2002 3 2 3 4 2003 1 0 3 1 2004 2 3 7 2 Total 14 11 24 20 Avg./year 1.8 1.4 3.0 2.5 2005 1 1 7 3 2006 4 2 1 6 2007 3 1 2 4 2008 1 4 4 1 2009 1 2 4 1 2010 1 1 1 3 2011 4 1 3 4 2012 1 1 0 1 2013 5 4 3 3 Total 21 17 25 26 Avg./year 2.3 1.9 2.8 2.9 2014 3 1 8 0 Grand Total 38 29 57 46 Avg. 2.1 1.6 3.2 2.6 Table 4. Total Page Count Per VFM Report 1997-2014 Per Category Social Year Health Trans. justice Services 1997 40 31 19 76.5 1998 51.5 1 35.5 65.5 1999 75 22.5 42.5 58 2000 67 7 41 38 2001 76 34 76.5 63.5 2002 116.5 2 81.5 45.5 2003 48 10 79 51.5 2004 96.5 25 43.5 65.5 Total 570.5 132.5 418.5 464 Avg./year 71.3 16.6 52.3 58 2005 72 47 25.5 28.5 2006 85 6 25 53 2007 11.5 59 65 25.5 2008 171.5 24.5 119 98.5 2009 154 47 10.5 64 2010 167.5 9 37 36.5 2011 58 14 36 51 2012 150.5 25 96 9.5 2013 128.5 47 13.5 71 Total 998.5 278.5 427.5 437.5 Avg./year 110.9 30.9 47.5 48.6 2014 97 11.5 129 96 Grand Total 1666 422.5 975 997.5 Avg. 92.6 23.5 54.2 55.4 Env. & Year Finance Education Energy Other 1997 11.5 18 35.5 31 1998 17 23.5 59 20 1999 15 26.5 5 22.5 2000 15 19 50 53 2001 39.5 32 0 33.5 2002 69.5 34 115.5 116.5 2003 14 55 19.5 86.5 2004 55 9.5 85.5 149 Total 236.5 217.5 370 512 Avg./year 29.6 27.2 46.3 64 2005 35 26 27.5 147 2006 7.5 21 89 50 2007 77.5 21 82 69 2008 17 78.5 54 111 2009 60.5 35.5 33.5 90.5 2010 74.5 54 33 37 2011 70.5 51 89 79 2012 50 51 31.5 6.5 2013 6.5 75.5 139 87.5 Total 399 413.5 578.5 677.5 Avg./year 44.3 45.9 64.3 75.3 2014 148.5 69 101.5 287.5 Grand Total 784 700 1050 1477 Avg. 43.6 38.9 58.3 82.1
|Printer friendly Cite/link Email Feedback|
|Author:||Adi, Sana; Dutil, Patrice|
|Publication:||Canadian Public Administration|
|Date:||Mar 1, 2018|
|Previous Article:||Federal policy ideas and involvement in Canadian urban transit, 2002-2017.|
|Next Article:||Open data and its institutional ecosystems: A comparative cross-jurisdictional analysis of open data platforms.|