Assessing financial risk at DND.

The last 12 years have been challenging for the Department of National Defence (DND). Significant budget cuts following the 1991 Defence White Paper were the catalyst for an era of devolution and decentralization in the department and have led to substantial changes in the way it operates. For instance, as part of the Federal Government's Modern Comptrollership initiative, the Chief Financial Officer (CFO) of the DND, Major-General T.M. Hearn, CMA, has implemented a new financial risk assessment model. This model is designed to ensure that business cases prepared by the department reflect a reasonable estimate of their expected costs, including a quantitative assessment of financial risk. At the same time, they are designed to offer a clear and concise understanding of a business case, improving accountability and transparency.

[ILLUSTRATION OMITTED]

[ILLUSTRATION OMITTED]

DND has a long history of validating business cases, but a new management philosophy was adopted during the era of downsizing--self-audit. Managers developed and submitted business cases without the arm's-length review of the senior full-time financial officer (SFFO). Over the past few years, it's become evident in both Canada and the U.S. that the management philosophy of self-audit doesn't work well in the stewardship of public resources because the fundamental control of profit generation isn't there to constrain public managers.

[ILLUSTRATION OMITTED]

Early in 2003, the CFO (the SFFO within DND) developed a way to conduct arm's-length reviews of departmental business cases so that they reflected reasonable estimates of expected costs. The system gives a quantitative assessment of the business case's financial risk and examines the full cost of the business case, including acquisition and life cycle costs. Using this model, one can see which option is best from a financial perspective, once risk-adjusted life cycle costs are considered. This procedure has now been applied to business cases ranging in value from several hundred thousand dollars to $17.6 billion.

This costing validation process is integral to fulfilling the expectations of the Federal Government's Modern Comptrollership initiative. The intent of the validation isn't necessarily to change the decision, but to review the decision for analytical consistency, conformity with the intent and spirit of statutes and policies, and finally, for financial risk assessments.

Orientation

The intent of the costing validation report is to be a stand-alone document that doesn't require the reader to refer to source documents. It typically includes an orientation (background, definition of options, assumptions, constraints), a financial risk assessment, a restatement of risk-adjusted financial, and a conclusion and recommendations.

The orientation gives a synopsis of the key points of the business case under review and the options available for solving it. The assumptions referred to identify any constant factors, significant cost drivers and any models used to derive costs--for example, whether the real price of crude oil is expected to increase. Constraints are any issues that may affect the degree of due diligence required.

Financial risk assessment

The critical component of the costing validation is the financial risk assessment, something that is rarely done as well as should be expected from a due diligence perspective. It involves examining three concerns:

a. Data Integrity Risk: the data provided is verifiable and meets accepted reporting standards;

b. Program Risk: everything which should be in the analysis, like potential changes in resource or currency prices, has been included; and

c. Integration Risk: potential collateral impacts to other capabilities, such as the cost of delivering other items, have been considered.

Through this process, the business case addresses broad uncertainties and expected real price adjustments.

Depending on the type of risk assessed, contingency or escalation factors are assigned to the costs as well.

Contingencies are applied to account for uncertainty, and escalation factors are applied to account for anticipated real price adjustments.

Data integrity risk

The first step in the risk assessment is to conduct a data integrity confidence level gap analysis. This is critical to effective decision making because if the fundamental data underlying a decision is flawed, executives need to factor this consideration into the decision-making process. The gap analysis provides a visual account of the delta between specific project costs and the sources of data underlying the cost estimates compared to accepted sources of data. The gap analysis is the basis for a costing validation report and includes three primary measurements: data integrity confidence level, data integrity risk factor, and a criticality assessment. An example of a data integrity confidence level gap analysis is provided in Table 1.

Determining the data integrity confidence level involves categorizing the financial information available into logical cost elements. The purpose of this exercise is to determine the source of the costing information and assess how reliable the cost estimates are. Cost data can be divided various ways: functionally, categorically or in a hybrid form. Functional divisions delineate milestones in a project. For example, definition, acquisition, implementation and modification would be considered mile-stones. Categorical divisions would include such areas as personnel, operations and maintenance, infrastructure and overhead, to name a few. Hybrid divisions blend the two. If a hybrid approach is used, care is taken to ensure that there is no double counting within an assessment.

The data integrity confidence level assessment is a subjective one, done on a five-point scale (see Table 2) from very low to very high. As a guide, very high data integrity confidence level standards would include the following:

a. specific detailed identification of the asset or liability to confirm the identity and associated costs of the item being considered, including an appropriate contingency factor;

b. engineering or other technical documentation for the item that provides a concrete assessment of costs, including an appropriate contingency assessment; or

c. other documentation identifying a professional assessment by a competent authority, including an appropriate contingency assessment.

Assessment

Once the data integrity is assessed, the amount of financial risk associated with it is assigned. This is necessarily subjective. Sources such as independent industry studies or norms would be used to mitigate any lack of data, but these are only guidelines. Some examples of the sources considered are NASA Cost Estimating Guidelines, RAND Corporation Research Papers, Conference Board of Canada publications or the Working Council for Chief Financial Officers. Risk factors can be quantitative or qualitative in nature. The impact of the risk is assigned based on a five-point spread ranging from very insignificant to very substantial with contingency factors (10% - 50%, in 10% increments) being applied to the cost element. These contingency factors have been developed through experience and correlate with the NASA Cost Estimating Guidelines. The criticality factor of the risk is then assessed on a five-point scale. It is a measure of the importance of each line item to the success of the project. Consideration of timelines, availability of funds and milestones should be considered when assessing this factor. This measure is independent of the others in its application and helps identify Key Factors for Success (KFS) in the cost validation process. An example of a data integrity confidence level gap analysis from a hypothetical cost validation for replacing 40-year-old CC-130 (Hercules) aircraft is provided in Table 1.

Program and integration risks

Program risks that could arise in the acquisition or operation of the capital asset being considered are then reviewed. These include currency exchange risks or real changes in resource prices. The risks are typically modeled on economic and/or operational information, or the best practices of other organizations. For example, research models developed by public policy research institutes in Canada or the U.S. may be used to model specific risk situations. The overall risk factor is assessed on the same five-point financial impact scale as the data integrity risk.

Integration risks, on the other hand, are determined by applying contingency factors to related costs to account for potential increases in the cost of delivering other items. These risks are also assessed on the same five-point scale.

Quantification of Risks

Once the risks have been categorized they are then quantified. This quantification represents the risk-adjusted or expected cost for each cost element. Totaling the risk-adjusted cost for each element yields the total risk adjusted cost. Using the previous example of the replacement of the CC-130 (Table 1) would result in the following assessment of each cost element:

a. National Procurement plan is a program risk because it doesn't account for the uncertainty associated with equipment failures. Using a RAND Corporation Study an estimate can be used to quantify this uncertainty as 7.7% compounded annually;

b. Maintenance data provided by a contractor is a data integrity risk and would be assigned a 20% contingency because it would be assessed as an indicative measure of the likely costs, unless other users could corroborate the data;

c. Personnel costs, for which authorized positions could not be identified (but the rank levels and salary costs are), represent a data integrity risk and would be assessed a 10% contingency, as the cost estimate would be considered substantive and this treatment would be consistent with departmental experience; and

d. No integration risks are identified in Table 1.

Restatement of risk-adjusted financial information

Once all the risks have been quantified, the cost of the program is restated, identifying all the risk-related costs and presenting a total risk-adjusted cost (Table 3). The alternatives identified in the business case are re-examined using the risk-adjusted cost (expected value) to determine the ranking of the alternative from an expected value perspective. The standard measures used are constant year dollars (a base dollar amount not adjusted for projected inflation), net present value (NPV), accounting rate of return and internal rate of return. Constant year comparisons depict the cash flow for each alternative on an annual basis using a base year for comparative purposes. The NPV comparison presents the cost of each option in today's dollars. The accrual accounting rate of return measures the relative cash flow of each alternative against the status quo. Ideally, the relative cash flow should be greater then the discount rate used in the NPV calculation. The internal rate of return assesses the break even discount rate, which should be less then the discount rate for the NPV calculation. The discount rate is considered to be the current yield on Government of Canada 10 Year Bonds. The alternatives are ranked against each other for each metric and the preferred option from a financial perspective is determined.

Conclusions and recommendations

The conclusion follows the restatement of risk-adjusted financial information and presents a summary of the key issues. Suggestions are made to the CFO as to whether, in the opinion of the author, the financial information presented in the business case should be accepted as a reasonable estimate of the costs. If not, specific recommendations are provided for amendments to align the business case with the findings of the costing validation. Typically, the CFO strives for a reconciliation of +/-1% between the costing validation and the financial information presented in the business case.

The costing validation report gives risk-adjusted financial assessments to the department's CFO for his consideration before he certifies that the costs are reasonable. There's a very real risk inherent in this process--that the subjective assessments of the analysts will not be consistent or reasonable. To mitigate this risk, the CFO is told where the assessments are subjective and the method used to derive the assessment. This helps him quickly decide if he agrees with the assessment and provides an explanation of the process to any other interested parties.

This process ensures that there's due diligence in DND and assures departmental decision makers that the proposals presented to them are financially viable and reasonable. Executives' reliance on intuition in decision making has decreased because the financial risk associated with the business case is clearly presented. As the federal government builds on its modern comptrollership initiatives, systems like this offer necessary support to strategic decision making and create greater transparency in the process.

Table 1: Hypothetical example of a data integrity gap analysis to
replace CC-130 (Hercules) Aircraft

Cost Element    Standard Expected      Info Provided

National       * NP figures for     * National
Procurement /  CC-130 (Hercules)    procurement plan
Maintenance    maintenance          figures do not
               represent actual     address aging fleet
               plan                 of aircraft

               * Replacement        * Maintenance data
               airframe figures     is provided by the
               based on actuals     manufacturer

Personnel      * Position numbers   * Not provided

               * Rank & occupation  * Provided

               * Cost factors       * Provided
               manual used for
               standard cost for
               rank and occupation

 Data Integrity        Cost         Risk Weight     Criticality
Confidence Level  Categorization

* Low             * Indicative    * Substantial    * Substantial

* Low             * Indicative    * Substantial

* High            * Indicative    * Insignificant  * Very
                                                     insignificant

* Very high       * Substantive   * Very
                                    insignificant

* Very high       * Substantive   * Very
                                    insignificant

Table 2: Financial risk assessment model

Data Integrity  Appropriate       Cost           Explanation
  Confidence    Contingency  Categorization
    Level         Ranges

Very Low        >40%         Estimate        Very substantial
                                             amount of additional
                                             detail is required
                                             to raise the data
                                             integrity level to
                                             acceptance

Low             31-40%       Indicative      Substantial amount
                                             of additional detail
                                             is required to raise
                                             the data integrity
                                             level to acceptance

Medium          21-30%       Indicative      Significant amount
                                             of additional detail
                                             is required to raise
                                             the data integrity
                                             level to acceptance

High            11-20%       Substantive/    If data integrity is
                             Indicative      considered to have a
                                             contingency of 15%
                                             or below, it is
                                             substantive. Above
                                             this level it is
                                             indicative; a minor
                                             amount of additional
                                             detail is required
                                             to align with
                                             protocol

Very High       0-10%        Substantive     Data integrity is
                                             very high,
                                             commensurate with
                                             protocol

  Risk Weight       Explanation       Criticality       Explanation

Very substantial  Very high impact  Very substantial  Very high impact
                  on ability to                       on resulting
                  assess costs or                     risk to assess
                  achieve savings                     costs or achieve
                                                      savings

Substantial       High impact on    Substantial       High impact on
                  ability to                          resulting risk
                  assess costs or                     to assess costs
                  achieve savings                     or achieve
                                                      savings

Significant       Medium impact on  Significant       Medium impact on
                  ability to                          resulting risk
                  assess costs or                     to assess costs
                  achieve savings                     or achieve
                                                      savings

Insignificant     Low impact on     Insignificant     Low impact on
                  ability to                          resulting risk
                  assess costs or                     to assess costs
                  achieve savings                     or achieve
                                                      savings

Very              Very low impact   Very              Very low impact
insignificant     on ability to     insignificant     on resulting
                  assess costs or                     risk to assess
                  achieve savings                     costs or achieve
                                                      savings

Table 3: Example of risk adjusted restatement of financial information

                           Financial risk adjustments ($CDN in
                                         billions

   Cost Element            Base case       Option A     Option B

Business case           $            3.6  $       4.9  $       2.1

Data integrity
issues

National procurement                 0.8          0.3          0.1

Personnel                              -          0.2          0.1

Total data integrity    $            0.8  $       0.5  $       0.2
issues

Project risks

Fuel adjustments                     0.6          0.8          0.1
(modeled)

Total project risks     $            0.6  $       0.8  $       0.1

Integration risks

Impact to other                        -            -          0.1
operations

Total integration       $              -  $         -  $       0.1
risks

Total risk related      $            1.4  $       1.3  $       0.4
adjustments

Risk adjusted cost      $            5.0  $       6.2  $       2.5
forecast

Percentage change                    39%          26%          17%

Overall financial risk  Very substantial  Substantial  Significant
assessment

For a copy of the references used for this article, please contact the editor, Robert Colman, at [e-mail].

By Major M.C. Lionais, CMA

Major Mike Lionais, CMA, ([e-mail]) is a logistics officer currently employed as a senior strategic cost analyst with the Canadian Forces.