An overview of forecasting error among international manufacturers.
This article summarizes the most recent survey results in regard to forecasting practices and forecasting error of sales for Global Manufacturing Research Group (GMRG) participants. GMRG includes over 200 manufacturing companies from Hungary, Lebanon, Italy, Taiwan, and the United States. Participants in GMRG reported average forecast errors for sales of around 20%. A number of findings concerning use of forecasts, methods of forecasting, factors considered in the sales forecast, primary authority for the forecast, and variation in forecasting error by country are detailed. The practices of GMRG participants may be of interest to forecasters in like-minded companies.
The Global Manufacturing Research Group (GMRG) is an organization of academic researchers interested in international manufacturing research. GMRG has administered surveys of manufacturing practices to companies worldwide; this paper summarizes results of the third survey, collected in 2003-2004, that pertain to forecasting accuracy, including methods of forecasting used, who is responsible for the forecast function, factors considered in the sales forecast, and how the forecasts are used. The international comparisons of forecast error and reliance on various forecasting techniques are of particular interest.
Companies that participated in the survey were located in Italy, Lebanon, Hungary, Taiwan, and the northwestern United States, and included firms from the following manufacturing industries: electronics, machinery, cables, plastic furniture, plastic containers, plastic packaging, food products, textiles and building materials. The companies and countries were not randomly selected for inclusion but participated based on location, availability of management information, willingness to participate and products manufactured.
The 235 companies in the sample ranged in size from 6 employees to 9,500 employees, with a median of 142 employees, and with sales ranging from $50,000 to $16.3 billion and a median of $29 million in sales. Many, but not all, companies reported exporting their products; on average, they reported that about 45% of their sales were exports. The companies were also mainly domestically owned (78% of ownership, on average, was domestic).
The companies in this sample relied to a greater degree on management opinion than on quantitative (e.g. regression) or qualitative (e.g. survey) techniques in forming their forecasts. "Forecasts" refer to total sales of the company. The variable of interest is the average percent forecast error over the past two years.
SUMMARY OF PREVIOUS WORK
Wacker and Sprague (1995; 1998) used previous GRMG survey results to examine forecasting accuracy. They first used a sample of UK manufacturers to investigate the effects of institutional factors, such as technology culture and forecasting methods used, on forecast accuracy (1995). They found that companies with newer technology tended to have lower forecast error, and companies that measured forecast error had more accurate forecasts. Companies that had high forecast error modified their forecasts more frequently. In addition, companies for which sales planning was the primary purpose of the forecast tended to have more accurate forecasts. Companies in which top management was involved in the forecasting procedure had lower forecast accuracy.
A subsequent study by Wacker and Sprague (1998) used GMRG survey results from seven countries, including Germany, Japan, Mexico, New Zealand, Spain, Sweden, and the United States, to compare the relative effectiveness of management behaviors that affect forecast accuracy. This study found country differences in forecasting that were partially explained by Hofstede's cultural values dimensions. For example, companies in countries with high individualism tended to be more technology oriented and top management less involved in forecast development than companies in collectivism countries. The use of quantitative techniques did not improve forecast accuracy, which confirmed results of their previous study.
Sanders and Manrodt (1994) surveyed U.S. firms to see if quantitative techniques were being more widely used than in the past, and to see what role judgment played in the forecasts. They found that companies continued to rely more heavily on judgmental methods than on quantitative methods, even though respondents were familiar with all of the quantitative techniques except Box-Jenkins. Furthermore, they found that large companies were more likely to use quantitative methods than were small companies, a result previously reported by Dalrymple (1987).
A recent study of sales forecasting investigated whether management judgment provides the most accurate forecasts, and whether such forecasts are unbiased and efficient (Lawrence, O'Connor & Edmundson, 2000). They found that these judgmental forecasts were not, in most cases, more accurate than the naive forecast, and the forecasts suffered from both inefficiency and bias. The authors listed a number of reasons why contextual information failed to improve forecast accuracy, including recent random movements being misinterpreted as true changes in the series, and having forecasts serve as targets; that is, managers who are rewarded for exceeding the sales target may favor low forecasts. Wacker and Sprague (1998) found that the use of subjective external factors caused forecasts to be less accurate, while the use of subjective internal factors improved forecast accuracy.
Lawrence and O'Connor (2000) tested the accuracy, bias and efficiency of judgmental sales forecasts made by 10 manufacturers to see if forecast revisions would improve forecast performance. Even though a significant amount of forecast revision took place, it appeared to be of little value in improving the forecasts. In fact, the problem appeared to be excessive revision of forecasts, which led to overshooting.
Other studies that have examined forecast accuracy of subjective methods include Fildes (1991) and Goodwin and Fildes (1999). In general, they found the judgmental forecasts to be less than optimal. However, when it comes to quantitative forecasts, Barnett, Starbuck and Pant (2003) state that simple models tend to forecast more accurately than complex ones. Their own study of Moore's Law found that forecasts were more accurate when they covered shorter periods and industries were more concentrated.
This paper will now examine the results of the 2003-2004 GMRG survey, including the magnitude of forecast error, the methods used in forecasting, who is responsible for the forecasts, and how the forecasts are used.
MAGNITUDE OF FORECASTING ERROR
The companies were asked questions about the methods used to forecast sales, including both formal and informal practices. The average forecasting error of the 218 companies that responded to the question, "What has been the approximate average percent forecast error over the past two years?" was 22.3%. Most of the companies reported forecast error of 50% or less with the exception of a few outliers; almost half of the sample reported forecast errors between 15 and 30%. There was no apparent relationship between the size of the company, either by sales or number of employees, and size of forecast error. The correlation coefficient, r, for sales and forecast error was -.04, and for employees and forecast error was -.07. A summary of forecast error by country is shown in Table 1.
The percentage forecast error is fairly consistent across countries at about 20%, with the exception of Taiwan, which reported an average forecasting error of 30%. Wacker and Sprague (1995) reported an average percentage error of 28% in their sample of UK manufacturers.
RELIANCE ON VARIOUS FORECASTING TECHNIQUES
Respondents were asked to estimate their reliance on various methods of forecasting, including quantitative methods (e.g., regression), qualitative methods (e.g., surveys), and management opinion, by rating on a scale of 1 (not at all) to 7 (to a great extent). These companies relied most on management opinion, with an average rating of 5.3, and used quantitative and qualitative methods to a lesser degree (average of 3.6 and 3.9, respectively). This is consistent with the findings of Sanders and Manrodt (1994). The use of quantitative and qualitative techniques was related; companies that used quantitative techniques to a great extent also tended to use qualitative techniques to a great extent as well (r = .34). The use of quantitative or qualitative techniques appeared to be independent of whether the companies relied heavily on management opinion or not.
There was no relationship between the degree to which each method of forecasting was used and the measure of forecast error, i.e., whether a company used quantitative methods very little or to a great degree did not seem to affect the size of forecast error they reported. The correlation coefficient, r, between use of quantitative methods and forecast error was .06; between use of qualitative methods and forecast error was -.06; and between use of management opinion and forecast error was 0. This result agrees with Wacker and Sprague (1998), who found that the use of quantitative techniques in forecasting did not improve accuracy.
A summary of reliance on various forecasting techniques by country is shown in Table 2. Companies in Lebanon relied on quantitative methods to a greater degree than companies in the other four countries, with U.S. companies reporting the lowest reliance on such methods. Taiwanese companies rated qualitative methods the highest of the group, while management opinion had the highest weight in Italy (5.8 out of 7). American companies also relied on management opinion to a large degree.
Not surprisingly, there was a weak but positive correlation (r = .27) between use of qualitative methods (e.g., surveys) and size of company, with larger companies tending to make more use of surveys. Otherwise, there was no relationship between size of company and use of quantitative methods or management opinion. This is different than the results of Manrodt and Sanders (1994), who found that companies with high sales used various quantitative techniques to a much greater degree than small firms, a result also found by Dalrymple (1987).
PRIMARY AUTHORITY FOR FORECASTING
The companies were asked to identify the position of the person who had primary authority for producing the company's sales forecasts. A summary is shown in Table 3.
Most companies (about 70% of the sample) identified either a president or vice president as being the primary authority for producing the sales forecast, with about 24% reporting a department or division head and 6% reporting a group or section manager as primary authority. Forecast error did not vary much by who was reported as the position responsible for the forecast. Wacker and Sprague (1995) found that the involvement of top management tended to decrease forecast accuracy in their sample of British firms. Their later study (1998) found that involvement of top management had no effect on forecast accuracy. Here, having top management involved in the forecast did not significantly affect accuracy, although the lowest forecast error was reported by group or section managers.
The companies were also asked to identify the function of the person who had primary authority for producing the sales forecast; results are summarized in Table 4.
Forecasting error varied by function of who was primarily responsible for the sales forecast. Of the functions listed in Table 4, survey respondents listed "planning" most frequently as the position primarily responsible for the forecast, followed by "sales" and "administration." "Planning" had one of the highest average reported forecast errors; "production" and "engineering" had some of the lowest average forecast errors. "Finance" and "accounting" had the lowest and highest reported errors, but there were too few survey respondents identifying with those functions to be meaningful.
The table above also shows that when "sales" was primarily responsible for developing the forecast, average percentage error was 19.3%, or about average for the entire sample. This is in concordance with previous findings of Wacker and Sprague (1998), who found that forecast accuracy is not necessarily improved when sales/marketing is primarily responsible for developing the forecast.
FACTORS CONSIDERED IN THE SALES FORECAST
Companies were asked to rate five factors in terms of how important they were considered in the sales forecast: current economic conditions, customer information, supplier information, results of market research, and current order backlog. The same scale that was used to evaluate the extent to which the companies used various forecasting techniques was used in rating these factors (1 = not at all, 7 = to a great extent). Table 5 summarizes the factors considered in the sales forecast by country.
For most of the countries, customer information was used extensively, with an average rating of 5.8 on a 7 point scale, with the exception of Lebanon, where current economic conditions counted most heavily (rated 6 on a 7 point scale). Results of market research had the lowest average rating (4.1), closely followed by supplier information (4.2). American companies in particular rated market research low (3.1) and customer information high (5.5). Taiwanese companies tended to rate all factors consistently high, in the range of 5.2 to 6.0.
PURPOSE OF FORECAST
Wacker and Sprague (1995) found that forecast accuracy is enhanced when the forecast is used for sales planning; however, only 3% of the firms in their sample developed the forecast with that as its primary purpose. Respondents in the 2003-2004 GMRG survey were asked to evaluate how the forecast was used for planning, budgeting, and other decision making activities using the same 7-point scale described in Table 5. Results are detailed in Table 6.
How the forecast is used influences the level of aggregation of the forecast, among other things. Zotteri, Kalchschmidt, and Caniato (2005) explain that "for short-term production planning probably a very detailed demand forecast is required, while for plant design or budgeting a rather aggregate forecast will be used." In this GMRG sample, the most important use of the sales forecast was in production planning (5.6 on a 7-point scale), followed closely by budget preparation (5.5) and sales planning (5.4). In Taiwan especially, the forecast was important in sales planning (6.0), while in Lebanon, it was important in budgeting (6.0) and production planning (6.1). Subcontracting decisions had the least weight overall (4.2), especially in Hungary (3.9), Italy (3.4) and the U.S. (3.8). Once again, the Taiwanese companies rated all factors consistently high, ranging from 5.1 to 6.0.
TIME HORIZON OF FORECASTS
The average time horizon of the survey respondents was 8.4 months into the future; there did not appear to be any relationship between the time horizon of the forecast and the percentage of forecast error. Most of the companies (54%) used months as the smallest time period into which the forecast time horizon was divided, with the rest using days (9%), weeks (36%) or years (1%). Typically, the companies modified their forecasts about 3 to 4 times per year.
Participants in the Global Manufacturing Research Group (GMRG) averaged forecast errors of around 20%. The amount of forecast error was relatively consistent by position of primary authority for the forecast, but forecast error tended to be higher for primary authorities in planning or marketing, and lower for those in production and engineering. Survey respondents rated customer information as the most important factor in making a forecast, and this result was fairly consistent across the five countries in the sample. The most important use of the forecast was in production planning, and again, this result was consistent across the countries.
Management opinion was relied upon to a greater degree than quantitative or qualitative methods in forecasting, and was especially true for American and Italian companies. This result is consistent with other studies (Manrodt & Sanders, 1994; Dalrymple, 1987). This may be due to companies' perceptions and experience that quantitative techniques do not necessarily improve forecasting accuracy, even though some studies have found management opinion to be lacking in terms of bias and efficiency.
These results are not representative of all companies; however, the firms that participated in this survey are those that keep track of a wealth of information including forecasting accuracy, and their practices may be of interest to forecasters in like-minded companies.
Barnett, Michael L., William H. Starbuck & P. Narayan Pant (2003). Which dreams come true? endogeneity, industry structure and forecasting accuracy. Industrial and Corporate Change, 12(4), 653-672.
Dalrymple, Douglas J. (1984). Sales forecasting practices--results from a United States survey. International Journal of Forecasting, 4(3), 51-59.
Fildes, Robert (1991). Efficient use of information in the formation of subjective industry forecasts. Journal of Forecasting, 10, 597-617.
Goodwin, Paul & Robert Fildes (1999). Judgmental forecasts of time series affected by special events: does providing a statistical forecast improve accuracy. Journal of Behavioral Decision Making, 12, 37-53.
Lawrence, Michael & Marcus O'Connor (2000). Sales forecasting updates: how good are they in practice. International Journal of Forecasting, 16, 369-382.
Lawrence, Michael, Marcus O'Connor & Bob Edmundson (2000). A field study of sales forecasting accuracy and processes. European Journal of Operational Research 122, 151-160.
Sanders, Nada R. & Karl B. Manrodt (1994). Forecasting practices in U.S. corporations: survey results. Interfaces, 24(2), 92-100.
Wacker, John G. & Linda G. Sprague (1998). Forecasting accuracy: comparing the relative effectiveness of practices between seven developed countries. Journal of Operations Management, 16, 271-290.
Wacker, J. G. & L. G. Sprague (1995). The impact of institutional factors on forecast accuracy: manufacturing executives perspective. International Journal of Production Resources, 33(11), 2945-2958.
Zotteri, Giulio, Matteo Kalchschmidt & Federico Caniato (2005). The impact of aggregation level on forecasting performance. International Journal of Production Economics, 93-94, 479-491.
Joanne Tokle, Idaho State University
Dennis Krumwiede, Idaho State University
Table 1: Forecast Error (%) by Country, GMRG Participants, 2004 Country Sample size % Forecast Error Hungary 65 19.0% Italy 32 18.5% Lebanon 20 22.8% Taiwan 60 30.0 United States 41 19.2 Total 218 22.3 Table 2: Techniques Used to Forecast Sales, by Country, on a Scale of 1 (not at all) to 7 (to a great extent) Country Quantitative Methods Qualitative Methods Hungary 3.1 4.1 Italy 3.1 2.7 Lebanon 5.1 4.7 Taiwan 4.2 4.8 USA 2.9 3.2 Total 3.6 3.9 Country Management Opinion Hungary 4.7 Italy 5.8 Lebanon 5.2 Taiwan 5.4 USA 5.7 Total 5.3 Table 3: Primary Authority for Producing Forecast and Average Percentage Error, By Position Position Number % Forecast Error President/CEO/Managing Director 79 21.3 Department/Division Head 51 21.8 Vice President/Director 74 24.6 Group/Section Manager 13 18.4 Table 4: Primary Authority for Producing Forecast and Average Percentage Error, By Function Function Number % Forecast Error Administration 41 16.1 Production 8 10.4 Sales 52 19.3 Finance 2 10.0 Planning 78 30.6 Engineering 22 15.1 Marketing 11 25.0 Accounting 3 31.7 Table 5: Factors Considered in the Sales Forecast (1 = not at all, 7 = to a great extent) Country Current Customer Supplier Economic Information Information Conditions Hungary 4.2 6.0 4.0 Italy 5.0 5.6 3.5 Lebanon 6.0 5.3 4.8 Taiwan 5.5 6.0 5.2 USA 5.1 5.5 3.4 Overall 5.0 5.8 4.2 Country Results of Current Order Market Backlog Research Hungary 4.1 5.6 Italy 3.3 4.1 Lebanon 4.8 4.1 Taiwan 5.2 5.2 USA 3.1 4.9 Overall 4.1 5.0 Table 6: Extent to which the Company's Sales Forecast is Used For Planning and Decision Making (1 = not at all, 7 = to a great extent) Purpose Hungary Italy Lebanon Budget Preparation 5.7 5.0 6.0 Production Planning 5.8 5.0 6.1 Subcontracting Decisions 3.9 3.4 4.3 Material/Inventory Planning 5.3 4.3 5.5 Sales Planning 5.6 4.4 5.9 Human Resource Planning 5.1 4.2 5.0 New Product Development 4.1 4.0 5.0 Facilities Planning 3.8 4.2 4.4 Equipment Purchase Planning 4.6 4.1 5.2 Purpose Taiwan USA overall Budget Preparation 5.4 5.3 5.5 Production Planning 5.6 5.5 5.6 Subcontracting Decisions 5.1 3.8 4.2 Material/Inventory Planning 5.7 4.8 5.2 Sales Planning 6.0 5.2 5.4 Human Resource Planning 5.5 4.3 4.9 New Product Development 5.7 3.9 4.6 Facilities Planning 5.5 4.3 4.5 Equipment Purchase Planning 5.2 4.5 4.7
|Printer friendly Cite/link Email Feedback|
|Author:||Tokle, Joanne; Krumwiede, Dennis|
|Publication:||Journal of International Business Research|
|Date:||Jul 1, 2006|
|Previous Article:||Convergence in major euro-zone stock markets: evidence from monthly data.|
|Next Article:||Thailand: Southeast Asian tiger or historical underachiever.|