Interpreting the performance of business economists during the great recession.

Ii is generally believed that the recession of 2007-09 was not foreseen by business economists. Is this perceived view accurate? We explore this issue by examining business economists' published statements about economic conditions. We compare these qualitative forecasts with the Beige Book. We conclude that both sets of data are similar and that business economists are responsive to information about the economy and adjust their predictions quickly Business Economics (2012) 47, 148-154.

doi:10.1057/be.2012.2

Keywords: business forecasts, great recession, beige book, payroll data

It is generally believed that the recession of 2007 09 was not foreseen by 1110sf business economists. While Nouriel Roubini did forecast the recession in advance he had been predicting such a decline for years and his timing was obviously not correct. Is the perceived view about the accuracy of other business forecasters accurate? We explore this issue by examining business economists published statements about economic conditions.

These published statements were usually ot a qualitative rather than numerical nature and had to be converted into quantitative data by a method described below in order to be evaluated. The forecasts covering the period January 2007 through December 2008 were obtained from the Wall Street Journal Eastern Edition by searching the ProQuest ABI Inform Complete article database. (1) This search uncovered articles ranging from opinion pieces regarding the world economy to summaries of the Federal Open Market Committee (FOMC) meetings. Only those articles with direct statements made by businesses or Wall Street Journal authors concerning the current or future state of the U.S. economy were included in our database. We excluded statements made by officials of the Federal Reserve System. Other U .S. government and international institution forecasts were included only if discussed by business or Wall Street Journal authors in the context of the business environment. Altogether we obtained 231 forecasts.

1. Evaluation Methodology

Because a Large majority of the forecast statements in our sample were qualitative in nature. it was necessary to convert them into quantitative data in order to evaluate them. (2) We used a method for scoring qualitative in Formation employed by Goldfarb and others [2005] to evaluate qualitative forecasts from the Great Depression. This proCedure converts qualitative statements into quantitative data.

The individual forecasts provided qualitative assessments about both the current state and the future direction of the economy. These two types of assessments were analyzed separately. We scored each statement using an optimism-pessimism scale that ranged from - 1 to + 1 with gradations or steps of 1/4

For example% + 1 was assigned to any quotation indicating that the economy was doing very well and would expand strongly. Statements that the economy was already in a deep recession, or would be in a particularly had recession, were given a score of - 1. Any citation indicating the economy's direction was unclear was given a 0. Table 1 presents the relationship between the scores and the paraphrased statements.

Table 1, Criteria for Scoring Qualitative Forecasts

General Mindset  Score    Condition Diagnosed or Forecast

Optimism             1  The economy is strong or will expand
                        very strongly

                   3/4  The economy is growing

                        normally or will definitely continue
                        to grow

                   1/2  The economy is growing at a "modest"
                        pace or will do well barring
                        unforeseen events

                   1/4  There is some risk of a recession or
                        downturn < 30%, or the economy will
                        still grow

                        but slower than usual

Neutral              0  It is unclear where the economy is
                        or where it will go because the
                        signs are mixed

Pessimism          1/4  The economy is visibly slowing,
                        "decelerating," or there is quite a
                        bit of risk of a recession,

                        >30% but <60%

                  -1/2  The economy is "sluggish'7, barely
                        growing, or there is >60% risk of
                        recession

                  -3/4  The economy is declining, will
                        contract, or there are mild
                        recession conditions

                    -l  Recession conditions are here or
                        imminent and it is worse than any
                        recession in recent history

Separate scores were assigned to statements about the current situation and to projections about the future condition of the economy. We thus could separately examine the economists' assessment of the current situation and their forecasts of the future on the aforementioned scale. Two time series of these scores were then constructed by averaging the scores of the individual forecasts made in each month--one time series for the statements about the present and another about the future.

Validation of the scoring procedure

Goldfarb and others [2005] validated their methodology by comparing their scores for the forecasts made with the monthly analyses of economic conditions published in the 1929-30 issues of the Federal Reserve Bulletin. Similarly. we verify the appropriateness of our scoring of the business economists' qualitative statements by comparing our results with an analogous scoring of the qualitative statements in various issues of the Federal Reserve Beige Book. Compiled periodically by each of the 12 Federal Reserve Banks, the Beige Book is a qualitative analysis of economic conditions within each Banks region. (3) Released two weeks before each FOMC meeting, the Beige Book contains information about the state of the US. economy that the FOMC considers in setting monetary policy. The Beige Book is important in this decision context because it becomes available before certain official U.S. macroeconomic data are released.4 Although this information is anecdotal, previous research showed that the Beige Book generally provides valid information about the current direction of the U.S. economy [see Fettig and others 1 999; Balke and Yucel 2000; Ginther and Zavodny 2001; Balke and Petersen 2002; and Armesto and others 2009].

A comparison of the Beige Book information with our scores poses several possible technical problems. The first involves timing. While our data constitute a time series of monthly observations, the Beige Book is not published monthly because the FOMC meets on'ly eight times a year. Thus, there are missing observations for some months.

Second, economic conditions are described differently across the Federal Reserve districts. Some branches tend to be much more conservative in their descriptions while others use more extreme language to describe similar events. Thus, the scores assigned to each Banks statements may not be consistent. Finally, it has also been noted that some districts are better than others at accurately gauging the state of their regional economy: some seem to focus only on the city where the district bank is located [Armesto and others 20091. Despite these potential discrepancies, most analyses that scored the Beige Book statements and compared them with actual macroeconomic data fOUnd that this source generally tracked the U.S. economy well. Therefore, we can be relatively confident thai by using the same scoring scale and adjusting for missing dates. the beige Book should be an excellent benchmark for evaluating the business economists statements about the U.S. economy.

The closing date ['or collecting data for the Beige Book was used to determine the date to which the average scores was attributed. For example February 26, 2007 was the closing date for collecting data that were in the March 2007 report. Consequently, the date assigned for the inFormation in the March Beige Book was February 2007.

In scoring the statements in every Beige Book, particular attention was paid to the primary sentence that described the state of each regional economy. The scorings were identical to those applied to the Wall (5) freer Journal articles, but with particular attention paid to key words, such as "modest growth and 'sluggish growth.' The Beige Book score assigned to each month was the average of the scores obtained from the statements of 12 Regional Banks.

Evaluation procedure

The evaluation procedure compared the scores ot the business statements with the Beige Book scores to determine whether they were consistent. A distinction was made between current and future period Forecasts. All Wall Street Journal statements referring to the current situation were assumed to be assessments of the economic conditions prevailing in the previous month: all statements about the future were assumed to concern the very near term. For example. a statement made in April 2007 about current conditions would be assigned to March 2007. It was compared with the Beige Book score, which, presumably measured the actual economic conditions that prevailed in that month.

It was assumed that a business forecast made in April 2007 referred to conditions expected in April. May. and June of' 2007. Given this assumption the Wall Street Journal forecast score for month I should be compared with the actual conditions that prevailed in those three months. Consequently, the scores assigned to forecasts made in month t were compared with the average of the Beige Book scores for months t, t + 1, and t + 2. (6), (7)

2. Results

Assessment of current conditions

Table 2 presents the comparison of the Wall Street Journal current assessments with the Beige Book. On average throughout 2007 and 2008 the Wall Street Journal current assessments are more pessimistic than the comparable Beige Book assessments. Although the business current assessments of the economy are more pessimistic, both series have strong downward trends (see Figure 1) and the majority of scores for the Beige Book and the Wall Street Journal have the same sign. However. the business scores show more month-to-month volatility. Overall, it appears that the business economists had a reasonable understanding of the actual conditions of the U.S. economy as it evolved throughout 2007 and 2008.

This view is confirmed when movements in the business scores are compared with a series considered a coincident indicator of turning points in the U.S. economy: Nonfarm payroll employment (PAYEMS). Figure 2 presents the movements of the business scores and the growth rates ol the payroll data. (8) Both series show similar trends, hut there is more volatility in the Wail Street Journal assessments. (9)

Table 2.  Assessment of Conditions: Scores of Forecasters and Beige
Book, December 2006-November 2008

                Dee-06  Jan-07  Feb-07  Mai-07  Apf-07  May-07  Jun-07

Score of          0.38    0.07    0.04    0.38    0.13   -0.50   -0.25
Representative
Forecaster

Score of Beige    0.38      NA    0.38      NA    0.54.   0.60      NA
Book

                Dec-07  Jan-08  Feb-08  Mar-08  Apr-08  May-08  Jun-08

Score of         -0.25    0.54   -0.61   -0.68   -0.38   -0.63   -0.88
Representative
Forecaster

Score of Beige    0.17      NA   -0.15   -0.35      NA   -0.17      NA
Book

                InJ-07   Aug-07  Sep-07  Oct-07  Nov-07

Score of          0.45    -0.08    0.13   -0.17   -0.26
Representative
Forecaster

Score of Beige    0.44     0.33    0.25    NA      0.13
Book

                Jul-08   Aug-08  Sep-08  Oct-08  Nov-08

Score of         -0.07    -0.38   -0.63   -0.75   -0.92
Representative
Forecaster

Score of Beige   -0.10    -0.15   -0.40      NA   -0.60
Book

Note: The assessment in row 1 of conditions prevailing in a
particular month is based on the individual's statement in
the subsequent month (that is--a statement made in a May
report about current conditions refers to the situation
prevailing in April).

Forecasts

The Wall Street Journal forecasts have a very similar relationship to both the average Beige Book scores and to nonfarm payrolls as in the case oF the current. assessments. Using our scoring procedures. the forecasts of the business economists are more pessimistic than the Beige Book assessments of actual conditions that prevailed in the periods being Forecasts as shown in Table 3. The differences between the Wall Streek Journal forecasts and the average Beige Book scores in Table 3 are similar to those in Table 2 although slightly larger and even more pessimistic. Throughout 2007. the Wall Street Journal data posted negative or near zero forecast scores while the average Beige Book score remained positive up until February 2008, the first report alter December 20O7 which the National Bureau of' Economic Research has identified as the beginning oF the 2008-09 recession. Again both series exhibit strong negative trends as seen in Figure 3. The much higher degree of volatility in the Wall Street Journal forecasts is also replicated in the PAYEMS comparison in Figure 4. (10)

Table 3. Evaluation of Forecasts: Scores of Forecasters and Beige Book,
January 2007-December 2008

                Jan-07  Feb-07  Mar-07  Apr-07  May-07  Jun-07  Jul-07

Score of          0.13   -0.07    0.15   -0.08   -0.25   -0.06    0.00
Representative
Forecaster

Average Score       NA    0.46      NA    0.57    0.52      NA    0.39
of Beige Book
for t, t+1, t+
2

                Jan-08  Feb-08  Mar-08  Apr-08  May-08  Jun-08  Jul-08

Score of         -0.53   -0.30   -0.88   -0.10   -0.53   -0.33   -1.00
Representative
Forecaster

Average Score      NA    -0.25   -0.26      NA   -0.14      NA   -0.13
of Beige Book
for t, t +1, t
+2

                Aug-07  Sep-07  Oct-07  Nov-07  Dec-07

Score of          0.04   -0.13   -0.15   -0.19   -0.37
Representative
Forecaster

Average Score     0.24    0.19      NA    0.15    0.01
of Beige Book
for t, t+1, t+
2

                Aug-08  Scp-08  Oct-08  Nov-08  Dec-08

Score of         -0.40   -0.63   -0.82   -1.00   -0.90
Representative
Forecaster

Average Score    -0.27   -0.50      NA   -0.55    -0.5
of Beige Book
for t, t +1, t
+2

There may be an explanation for some of this volatility in the business scores. For examples the July 2007 scores show a j imp in optimism and interrupt a generally negative trend. The GDP data for the second quarter of 2007 had just been released and showed that U.S. GDP had increased by 4 percent. a very strong growth rate. The U.S. financial market was also relatively calm in the summer of 2007. Although some investment banks continued to have serious problems, these seemed to be manageable without government assistance.

There was also more optimism in the July 2008 business SCOreS. At that point, it appeared that the economic decline would be short-lived and that the Fannie Mae and Freddie Mac bailouts would be all that was necessary to move the economy forward. (11) Even though the job problem was recognized. federal stimulus money from the tax rebate was working through the economy and was expected to stabilize the situation. This was probably the reason that the Beige Book and the Wall Street Journal had such similar scores in July 2008 and why these scores were optimistic relative to PAY EMS.

Finally, the relative volatility and pessimism in the Wall Street Journal Scores may also be due to the greater variety of people making statements. as well as variations in the number of statements collected per month from the Wall Street Journal articles. Statements were made by sources ranging from financial businesses, to professional forecasting firms to shipping companies. This variety of statements may also partially explain the greater degree of pessimism in the Wall Street Journal scores. Whereas the language in the various Beige Books was relatively conservative, the Wall Street Journal sources may have used more extreme words or more exaggerated statements. In any event, it is possible to draw some conclusions from this analysis of the business economists statements.

3. Conclusions

In order to evaluate the predictive accuracy of the business economists' qualitative statements it was necessary to convert them into quantitative scores. They were then compared with the scores that were assigned to the qualitative statements in the Beige Book, which served as the benchmark for this evaluation. Both sets of scores had similar time trends. Both the business current assessments and the forecasts became more and more pessimistic as the financial crisis progressed in 2007 and 2008. Both tracked the scores of the Beige Book statements. Based on this evidence, we conclude that the current assessments and forecasts of business economists captured the underlying trend of the economy during 2007-08.

We now return to the question that was posed originally. While the economists understood the underlying trends, there is no evidence that they predicted the Great Recession in advance. (12) While the economists scores became negative in mid-2007, a negative score does not necessarily indicate a prediction of a recession. The scores assigned to the economists' statements are based on an optimism/pessimism scale with a negative score measuring a pessimistic outlook and not a recessionary state, as described in Table 1. Based on the criteria in Table 1, a value exceeding -0.50 would indicate an assessment or forecast of a recession. This level was first exceeded in January 2008. only one month after the date identified by' the National Bureau of Economic Research as the beginning of' the Great Recession.

Thus, this sample of forecasts suggests that. business economists are very responsive to the latest information about the state of the economy and adjust their predictions quickly. On the other hand, there is also evidence that the forecasts are more volatile than warranted because the economists sometimes react too quickly to information that reverses their previous views.

Acknowledgments

We wish to thank Robert Goldfarb and Tara Sinclair for their comments on an earlier draft. All remaining errors are the responsibility of the authors.

REFERENCES

Armesto, M.T. R. Hernandez-Murillo. M.T. Owyang. and J. Piger. 2009. "Measuring the Information Content of the Beige Book: A Mixed Data Sampling Approach." Journal of Money, Credit and Banking. 41(1): 35-55.

Balke, N.S., and D. Petersen. 2002. "Flow Well Does the Beige Book Reflect Economic Activity4? Evaluating Qualitative Information Quantitatively. Journal of Money, Credit and Banking. 23(1): 114-36.

Balke. N.S., and M. K. Yucel 20OO. Evaluating the Eleventh District's Beige Book Federal Reserve Bank of Dallas Economic and Financial Review. 2000(4): 2-9.

Evans. K.. A. Trolanovski. and S. Reddy. 2008. "U.S. News: Exports, Stimulus Plan Prop up the Economy: GDP Rose 1.9% But Job Woes Loom" Wall Street Journal (Eastern Edition) (August 1): A:3.

Fettig. D., A.J. Rolnick, and D.E. Runkle. 1999. The Federil Reserve 's Beige Book: A Better Mirror than Crystal Ball. Minneapolis Federal Reserve The Region. http://www.minneapolisfed.org/publications_papers/pub_displav.cfm?id=3568 (accessed March 22, 2010).

Ginther. D.K.. and M. Zavodn . 2001. "The Beige Book: Timely Information on the Regional Economy." Federal Reserve Bank of Atlanta Economic Review, 2001(3): 19-28.

Goldfarh. R.S., H.O. Stekler. and D. Joel. 2005. "Methodological Issues in Forecasting: Insights From the Egregious Business Forecast Errors of Late 1930." .Journal of nomic Methodology. 12(4): 517-42.

Stekler. H.O, and R. Talwar. 2011. "An Analysis of' Quantitative Forecasts During the Great Recession."Mimeo

(1.) The search term "recession" was used for the year 2007 and the term "economic forecast" was used for both 2007 and 2008

(2.) The forecasts that are examined here do not include any quantitative forecasts from professional consulting firms.

(3.) The Beige Book has been published since 1970.

(4.) The real GDP estimates are not available until a month after tile quarter ends: state employment statistics are only available with a month delay; and gross state product is not available until two years later [Balke and Yuccl 2000; Balke and Petersen 2002; and Ginther and Zavodny 2001].

(5.) Each of the district Federal Reserve branches has their own methods for putting together their reports. but they generally comprise surveys that include, but are not limited to, local businesses, banks, and trade associations [Balke and Petersen 2002; Armesto and others 2009].

(6.) It is important to note that the Beige Book is never released in three consecutive months. Since this is the case, the scores were averaged for the months available over a three month period. For example, in early 2007 data were collected and released on February 26 and April 1 6 hut not in March. Therefore the scores for these two dates were averaged and assigned to February 2 as an average three-month score. March was given no score ill this case.

(7.) To compare the Beige Book scores to the Wall Street Journal scores, the Beige Book dates needed to he adjusted because the [J'all Street Journal scores were all assigned to the last date of every month. The Beige Book scores were originally assigned to the date when their data collection ended. hut the data collection end date is not consistent from survey to survey. If that date was on or before the 15th of a month that score was considered to apply to the last date of the previous month and if it was after the 15th, that score was assigned to the end of the same month.

(8.) The unsealed, continuously compounded annual rates of change for the series 'AIl Employees: Total nonfarm (PAY EMS were used. Available at http://alfred.stlouisfed.org/serics'?seid=PAYEMS. The Wall Street Journal assessments were compared to the unrevised growth rates first assigned to the months under study.

(9.) We only found three business current assessments for May 2007, which may account for the differences between the business scores and both the Beige Book scores and the payroll movements.

(10.) Forecasts were compared to the revised estimates for the continuously compouded annual growth rate of nonfarm payrolls (PAYEMS).

(11.) Evans and others [2008]; July 2008. http://timeline.stlouisfed.org/index.efm?p=timeline

(12.) A study of' business economists' quantitative forecasts from 2007 and early 2008 showed that they predicted that the economy Would slow down but not decline [Stekler and Talwar 2011].

* Kathryn Lundquist graduated in May 2011 from the George Washington University with a master's degree in International Trade and Investment Policy and a strong interest in Forecasting. She currently works as an international trade analyst covering aluminum at the L .S. International Trade CommisSion.

Herman 0. Stekler is a Research Professor of Economics at the George Washington University. His specialty is economic Forecasting with an emphasis on evaluation. He has written extensively on that subject.