Using confidence intervals in supply chain and operations research.
Confidence intervals ... should be used for major findings in both the main text of a paper and its abstract. (Gardner and Altman, Statistics in Medicine, 1986, Vol. 292, p. 746)
The usefulness of hypothesis testing hypothesis testing
In statistics, a method for testing how accurately a mathematical model based on one set of data predicts the nature of other data sets generated by the same process. methods has long been the subject of debate in the scientific community (Boring 1919; Kaiser 1960; Morrison and Henkel 1970; Hunter 1997; Wilkinson and the Task Force on Statistical Inferences. 1999), with both the tenor and tone of these criticisms becoming more pronounced in recent years (Harlow, Mulaik and Steiger 1997; Kline 2004). This controversy has resulted in much debate in the social sciences regarding the appropriate use of hypothesis testing. In an attempt to provide further clarity to this highly charged subject, we make the important distinction between informative and noninformative hypothesis testing and explain why, whenever possible, confidence intervals should replace the use of hypothesis testing.
Interestingly, business researchers have been mostly apathetic ap·a·thet·ic
Lacking interest or concern; indifferent.
apa·thet with regard to the hypothesis testing controversy. As a consequence, article-acceptance decisions in leading business journals, including those in well-regarded supply chain and operations management Operations management is an area of business that is concerned with the production of goods and services, and involves the responsibility of ensuring that business operations are efficient and effective. outlets, continue to rely almost exclusively on the results of hypothesis testing methods. In fact, and not withstanding the sage advice from Gardner and Altman (1986), many supply chain management researchers remain unfamiliar with the use and interpretation of confidence intervals. For instance, our of 234 empirical articles published in 2005, 2006 and 2007 in the Journal of Operations Management, Production and Operations Management and the Journal of Supply Chain Management, only 26 articles (11. 1 percent) reported a confidence interval confidence interval,
n a statistical device used to determine the range within which an acceptable datum would fall. Confidence intervals are usually expressed in percentages, typically 95% or 99%. . This founding is typical across the various business disciplines. Bonett and Wright (2007) reported general management journals, the Academy of Management Journal and Administrative Science Quarterly Administrative Science Quarterly, founded in 1956, is one of the most eminent academic journals in the field of organizational studies. It is published by Cornell University.
People claimed to have been involved as founders include James D. , in which only one out of the more than 130 empirical articles published in 2003 and 2004 reported a confidence interval.
This continued reliance on hypothesis testing is the result of a number of misconceptions regarding populations, population parameters and the type of information provided by tests of hypotheses (Bonett and Wright 2007). In this article we will: (1) briefly review the basic concepts of a population and a population parameter which provide the necessary foundation for the interpretation of confidence intervals; (2) outline limitations of statistical hypothesis testing One may be faced with the problem of making a definite decision with respect to an uncertain hypothesis which is known only through its observable consequences. A statistical hypothesis test, or more briefly, hypothesis test : (3) provide five examples to illustrate how confidence intervals may be used in place of hypothesis testing methods and (4) close with five publication guideline suggestions designed to foster a more effective use of confidence intervals in supply chain research. We begin our discussion with some basic definitions and ideas regarding populations and population parameters.
POPULATIONS AND POPULATION PARAMETERS
Research involving the planning, implementing and controlling of the operations of such business functions as the supply chain often requires the use of inferential in·fer·en·tial
1. Of, relating to, or involving inference.
2. Derived or capable of being derived by inference.
in statistical methods to answer the questions of interest. Answers to these research questions typically involve an understanding of the characteristics of some large se of units. The units might be firms, suppliers, customers, employees, products, packages, orders or items in inventory, just to name a few. The set of units under investigation is called the study population. Some examples of study populations in supply chain research are: all Fortune 1,000 firms, all customers in the firm's database, all employees who work in a network of firms, all packages shipped by a firm during some period of time, all orders received by the firm during some period of time and all items of a particular type in a firm's inventory.
The researcher will typically be interested in particular characteristics of the unites that make up the study population. For instance, the researcher may want to study the amount of communication between a firm and its suppliers in a study population of all Fortune 1,000 companies. Examples of other characteristics include the length of time a supplier has worked with a firm, time-to-delivery expectations of customers, employee knowledge of inventory management, product quality, package delivery time, order size and age of inventory items. These characteristics may be measured using various scales or questionnaires.
It is often the case that a summary statistic of the measurements of all units in the study population can provide useful information for research or decision-making purposes. Commonly used summary statistics are means, proportions, standard deviations, correlations and slopes. The value of a summary statistic for the entire study population is called a population parameter.
Researchers may not have the time or resources to measure every unit in the study population and be able to compute the population parameter of interest. In situations such as these, the researcher can instead measure a random sample of units from the study population and use inferential statistical methods to make certain types of statements about the population parameter value. A random sample of size n is selected in such a way that every possible sample of size n from the specified study population has the same chance of being selected.
Most inferential methods can be classified into hypothesis testing methods and interval estimation In statistics, interval estimation is the use of sample data to calculate an interval of possible (or probable) values of an unknown population parameter. The most prevalent forms of interval estimation are confidence intervals (a frequentist method) and credible intervals (a methods. The difference between these two methods can be illustrated in the case of a simple linear regression Simple linear regression
A regression analysis between only two variables, one dependent and the other explanatory. model where variable x predicts variable y and p is the unknown slope parameter. If the researcher was able to measure x and y for every unit in the study population, the value of p could be computed and the researcher would be able to report its exact value. When p has been estimated from a random sample, the researcher will not be able to determine the exact value of P but instead will only be able to make specific types of vague and less-than-certain statements regarding its value.
In a hypothesis testing application, the researcher might want to test the null hypothesis null hypothesis,
n theoretical assumption that a given therapy will have results not statistically different from another treatment.
n [H.sub.0]: [beta] = 0, and if rejected, conclude that either [beta] > 0, or [beta] < 0. Concluding that [beta] > 0, for instance, is a vague statement about the value of p. Furthermore, the researcher cannot be certain that [beta] > 0 but must admit, with probability [alpha], that [beta] might be [less than or equal to] 0. In hypothesis testing applications, the researcher has control over the value of [varies] and typically sets a to be [varies] small value such as .05 or .01.
Alternatively, the researcher could use the information in a random sample to specify a range of possible values for p, called a confidence interval, with some specified degree of confidence. For instance, the researcher might report a 95 percent confidence interval for P equal to (2.1, 4.5). Like the hypothesis testing result, the confidence interval also is a vague statement about p because only a range of possible values for p is given. Furthermore, the researcher cannot be certain that P is within this range but can be only 95 percent confident. Bonett and Wright (2007) provide more details on how confidence intervals may be interpreted using a degree-of-belief definition of probability rather than the relative frequency definition used in many introductory statistics texts.
Confidence intervals are almost always preferred to hypothesis tests because confidence intervals provide more information than hypothesis tests. Confidence intervals provide information about the magnitude of the population parameter. Confidence intervals may also be used to test certain types of hypotheses. For instance, in our example where the 95 percent confidence interval for [beta] is (2.1, 4.5), the researcher also could reject [H.sub.0]: ([beta] = 0 and conclude that [beta] > 0.
LIMITATIONS OF HYPOTHESIS TESTING
Well aware of the limitations of hypothesis testing, Graybill and Iyer (1994) recommended "that traditional statistical tests of hypotheses for a parameter, say [theta Theta
A measure of the rate of decline in the value of an option due to the passage of time. Theta can also be referred to as the time decay on the value of an option. If everything is held constant, then the option will lose value as time moves closer to the maturity of the option. ] (where one rejects or does not reject the null hypothesis) never be used if a confidence interval for [theta] is available because confidence intervals always provide more information than tests" (p. 35). We note that [theta] may represent a single parameter, a difference or ratio of parameters, or a linear or nonlinear A system in which the output is not a uniform relationship to the input.
nonlinear - (Scientific computation) A property of a system whose output is not proportional to its input. function of multiple parameters. Unfortunately and failing to heed Graybill and Iyer's (1994) advice, business researchers routinely rely almost exclusively on hypothesis-testing methods to address research questions and report results as being "significant" when the null hypothesis has been rejected or "nonsignificant non·sig·nif·i·cant
1. Not significant.
2. Having, producing, or being a value obtained from a statistical test that lies within the limits for being of random occurrence. " when the null hypothesis has not been rejected. In fact, a "significant" result is often a necessary criterion for acceptance in such well-regarded journal outlets as the journal of Operations Management, Production and Operations Management and the Journal of supply Chain Management. However, a "significant" result simply means that the null hypothesis has been rejected and does not imply that an interesting or meaningful result has been obtained. Alternatively, a "nonsignificant" result should not be interpreted as evidence that the null hypothesis is true.
Bonett and Wright (2007) classify tests of hypotheses into informative and noninformative hypotheses. Informative hypotheses provide useful information but non-informative hypotheses do not. To highlight this important distinction, we provide several examples of both informative and noninformative hypotheses. For instance, three examples of informative hypotheses are listed below where [theta] denotes a population parameter (or some function of two or more population parameters) and h is some specified value. The first example of an informative hypothesis is a directional one-sided hypothesis,
[H.sub.0]: [theta] [less than or equal to] h
[H.sub.1]: [theta] > h
the second example is a directional two-sided hypothesis,
[H.sub.0]: [theta] = h
[H.sub.1]: [theta] > h
[H.sub.2]: [theta] < h
and the last example is a finite-interval hypothesis.
[H.sub.0]: |[theta]| [less than or equal to] h
[H.sub.1]: [theta] > h
In each of these examples, a statistical hypothesis testing procedure will provide useful information. For instance, in a directional two-sided hypothesis, if the null hypothesis is rejected the researcher will conclude that [theta] > h or will conclude that [theta] < h. This is an informative test because the researcher would typically not know with certainty, before conducting the test, if [theta] > h is true or if [theta] < h is true.
Although informative tests of hypotheses are useful, especially in applied decision-making settings, informative tests of the type given above can be obtained from confidence intervals. For instance, if a one-sided lower limit for [theta] is greater than [h.sub.1] then [H.sub.1]: [theta] > h may be accepted. Or if a two-sided confidence interval for [theta] has a lower limit that is greater than [h.sub.1] then [H.sub.1]: [theta] > h may be accepted and if the two-sided upper limit is less than [h.sub.1] then [H.sub.2]: [theta] < h may be accepted. Finally, using a two-sided confidence interval to test a finite interval hypothesis, if the two-sided confidence interval for [theta] has a lower limit that is greater than-h and an upper limit that is less than [h.sub.1] then [h.sub.0]: |[theta]< h may be accepted.
The confidence interval approach is preferred to informative hypothesis testing because confidence intervals provide more information than the hypothesis tests. Specifically, confidence intervals provide information about the magnitude of the population parameter so that instead of simply concluding, for example, that [theta] > [h.sub.1] the confidence interval would provide a range of plausible values of [theta].
As explained by Bonett and Wright (2007), not all hypothesis tests are informative. In fact, a number of frequently used hypothesis tests are noninformative. For instance, F-tests in ANOVA anova
see analysis of variance.
ANOVA Analysis of variance, see there , ANCOVA ANCOVA Analysis of Covariance and regression models are examples of noninformative tests. The basic type of a noninformative hypothesis test is a test of null A character that is all 0 bits. Also written as "NUL," it is the first character in the ASCII and EBCDIC data codes. In hex, it displays and prints as 00; in decimal, it may appear as a single zero in a chart of codes, but displays and prints as a blank space. hypotheses such as
[H.sub.0]: [[theta].sub.1] = [[theta].sub.2] = ... = [[theta].sub.k]
[H.sub.0]: [[theta].sub.1] = [[theta].sub.2] = ... = [[theta].sub.k] = 0.
For [H.sub.0]: [[theta].sub.1] = [[theta].sub.2] = ... = [[theta].sub.k], the alternative hypothesis alternative hypothesis Epidemiology A hypothesis to be adopted if a null hypothesis proves implausible, where exposure is linked to disease. See Hypothesis testing. Cf Null hypothesis. states that there is at least one pair of parameter values that are not idential. For [H.sub.0]: [[theta].sub.1] = [[theta].sub.2] = ... = [[theta].sub.k] = 0, the alternative hypothesis states that there is at least one parameter that is not exactly equal to zero. Statistical tests of this type of hypothesis are noninformative because we know in advance, with near certainty, that the null hypothesis is false and the alternative hypothesis is true. Tests of noninformative hypotheses are common in supply chain management research. In addition, and unfortunately these methods are also overemphasized in many business statistics textbooks. As a consequence, we advise supply chain researchers to pay very careful attention to the admonition Any formal verbal statement made during a trial by a judge to advise and caution the jury on their duty as jurors, on the admissibility or nonadmissibility of evidence, or on the purpose for which any evidence admitted may be considered by them. given by Casella and Berger (2002, p. 525) that The classic ANOVA test is a test of the null hypothesis [H.sub.0]: [[theta].sub.1] = [[theta].sub.2] = ... = [[theta].sub.k], a hypothesis that, in many cases, is silly, uninteresting (jargon) uninteresting - 1. Said of a problem that, although nontrivial, can be solved simply by throwing sufficient resources at it.
2. Also said of problems for which a solution would neither advance the state of the art nor be fun to design and code. , and not true." The classic ANOVA test is but one example of the noninformative tests described by Bonett and Wright (2007).
The F-test reported in a multiple regression analysis In statistics, a mathematical method of modeling the relationships among three or more variables. It is used to predict the value of one variable given the values of the others. For example, a model might estimate sales based on age and gender. with k predictor variables is another widely used noninformative test. This F-test is a test of the null hypothesis
[H.sub.0]: [[beta].sub.1] = [[beta].sub.2] = ... = [[beta].sub.k] = 0. which is noninformative because we know in advance (with near certainty in any real application) that none of the [[beta].sub.i] values are exactly equal to zero and therefore we know that [H.sub.0] must be false. Another common example of a noninformative test is the [X.sub.2] goodness-of-fit test with k degrees of freedom routinely reported in structural equation models as a test of [H.sub.0]: [[theta].sub.1] = [[theta].sub.2] = ... = [[theta].sub.k] = 0, where [[theta].sub.i] is a model parameter. We know with near certainty that none of the [[theta].sub.i] values are exactly equal to zero and thus a statistical test that rejects [H.sub.0] does not tell us anything we did not already know. In general, F-tests with numerator numerator
the upper part of a fraction.
see additive genetic relationship.
numerator Epidemiology The upper part of a fraction degrees of freedom > 1 and [X.sup.2] tests with degrees of freedom > 1 are used to test noninformative hypotheses.
As in the other business disciplines, supply chain management researchers often compound the problem and routinely misinterpret mis·in·ter·pret
tr.v. mis·in·ter·pret·ed, mis·in·ter·pret·ing, mis·in·ter·prets
1. To interpret inaccurately.
2. To explain inaccurately. the results of noninformative tests. For instance, when the statistical test lacks power to reject [H.sub.0], the researcher often interprets the result as evidence that the null hypothesis is true. Alternatively, when the statistical test rejects [H.sub.0], the researcher will claim that a "significant" result has been obtained with an added implication that the findings have important scientific or practical value. It is the widespread misinterpretation of hypothesis testing results, coupled with the noninformative nature of certain popular statistical tests that has led to recommendations to ban the reporting of statistical tests in scientific journals (Gardner and Altman 1986; Hunter 1997).
In response to the growing concerns about the inappropriate use of hypothesis testing in the social sciences, the American Psychological Association commissioned a group of a leading statisticians Statisticians or people who made notable contributions to the theories of statistics, or related aspects of probability, or machine learning: A to E
http://www-gap.dcs.st-and.ac.uk/~history/Mathematicians/Tukey.html. (I960, p. 429), "Probably the greatest ultimate importance, among all types of statistical procedures we now know, belongs to the confidence procedure which, by making interval estimates, attempt to reach as strong conclusions as are reasonable." The hypothetical examples given in the following section illustrate how confidence intervals maybe used to provide more useful information than hypothesis test results.
A random sample of 275 Certified Purchasing Managers (CPMs) was obtained from a study population of 11,468 CPMs who are members of a particular professional group. Each of the 275 CPMs was contacted and asked to report the percent change in number of primary suppliers used by their firm over the last 2 years and to also report the quality of the relationship with the suppliers on a 1-10 scale with larger numbers indicating a higher quality relationship. The typical method of analyzing the data from this study would involve a test of the null hypothesis of a zero correlation between percent change in the number of suppliers and the quality of the relationship with the suppliers. If the null hypothesis of a zero correlation can be rejected, the hypothesis testing results might be reported as: "A highly significant negative correlation Noun 1. negative correlation - a correlation in which large values of one variable are associated with small values of the other; the correlation coefficient is between 0 and -1
indirect correlation between percent change in number of suppliers and quality of relationship with suppliers was found, t(273)= - 5.003, p < .001." This hypothesis testing result simply indicates that the population correlation is less than zero.
An alternative analysis could report the following confidence interval result: "A 95% confidence interval for the correlation between reported percent change in number of suppliers and supplier relationship in the study population of 11,468 managers ranges from -0.395 to -0.178." This confidence interval result provides all the information needed to reject the null hypothesis but also provides additional information regarding the magnitude of the unknown population correlation. Specifically the confidence interval suggests that the absolute value of the population correlation between percent change in number of suppliers and supplier relationship could be as small as 0.178 or as large as 0.395.
A random sample of 250 suppliers of stamped metal products was obtained from a study population of about 6,400 suppliers. The CEOs of the 250 sample firms were contacted and asked to answer a few questions as part of a university-sponsored research project in which all information would be treated confidentially. The CEOs were asked to name the firm's largest customer and rate on a 1-7 scale the extent of use of concurrent engineering ([x.sub.1]) and computer aided design (application) Computer Aided Design - (CAD) The part of CAE concerning the drawing or physical layout steps of engineering design. Often found in the phrase "CAD/CAM" for ".. manufacturing". ([x.sub.2]) in their firms. Each firm's largest customer was contacted and asked to rate its satisfaction (y) with the supplier on a 1-7 scale. Hypothesis testing results might be reported as: "A multiple linear regression Linear regression
A statistical technique for fitting a straight line to a set of data points. analysis was conducted with satisfaction as the outcome variable and extent of concurrent engineering and extent of computer aided design as the two predictor variables. The overall test for the predictive ability of the two explanatory variables was highly statistically significant, F(2, 247) = 4.48, p=.012." This is a noninformative test because we know in advance that the alternative hypothesis, which states that at least one of the two explanatory variables has a nonzero non·ze·ro
Not equal to zero.
Not equal to zero. regression coefficient Regression coefficient
Term yielded by regression analysis that indicates the sensitivity of the dependent variable to a particular independent variable. See: Parameter.
regression coefficient in the population, is almost certainly true.
A confidence interval for the population squared multiple correlation might be more useful and could be reported as: "We are 95% confident that differences in extent of concurrent engineering and computer-aided design are associated with 0.2% to 8.3% of the variance of customer satisfaction in this population of 6,400 stamped-metal suppliers." The confidence interval results show that the explanatory variables are at best weakly associated with customer satisfaction, even though the hypothesis testing results were "significant."
Confidence intervals for the unstandardized regression coefficients also provide useful information. For instance, the effect of concurrent engineering could be reported as: "The 95% confidence interval for the concurrent engineering regression coefficient in the study population of 6,400 suppliers range from 0.08 to 0.36. This result suggests that any 1-point increase in the concurrent engineering rating is associated with a 0.08 to 0.36 increase in customer satisfaction scores." The test that the population regression coefficient equals zero would be "significant" at the .05 level but the confidence interval for the population regression coefficient suggests that the effect of concurrent engineering on customer satisfaction is very weak.
A random sample of 100 firms was selected from a study population of 1,200 component-assembly firms. The 100 sample firms were classified into one of the four stages of supply chain integration proposed by Stevens (1989) and the 2-year percent change in sales was determined for each of the 100 firms. A typical statistical analysis for a study such as this would use a one-way ANOVA and these results might be reported as "The one-way ANOVA revealed highly significant differences in 2-year percent change in sales across the four stages of supply chain integration, F(3,96) = 54.5 p< .00.1." This result is noninformative because it simply indicates that the population means are not identical across the four types of firms. The F-test does not indicate how the population means are ordered nor does it provide any information about the magnitudes of the differences among the population means. A more informative analysis would report simultaneous confidence intervals for all six pairwise comparisons as shown in Table 1.
TABLE 1 Simultaneous Confidence Intervals for Six Pairwise Comparisons Comparison 95% lower 95% upper limit limit Stage 2 - Stage 1 -1.32 1.26 Stage 3 - Stage 1 2.49 4.11 Stage 4 - Stage 1 4.59 6.27 Stage 3 - Stage 2 1.95 3.84 Stage 4 - Stage 2 4.06 5.94 Stage 4 - Stage 3 2.20 3.99
Suppose that a difference in population means < 1.5 is considered to be small and unimportant in the context of this study. Note that the confidence interval includes zero for the Stage 2 - Stage 1 comparison and that the population mean for Stage 2 firms is at most 1.26 greater than the population mean for Stage 1 firms. This result suggests that the advantage of moving from Stage 1 to Stage 2 may be small and unimportant. The confidence intervals for the other comparisons indicate that Stage 4 firms in the study population have greater mean sales growth than Stage 3 firms, Stage 3 firms have greater mean sales growth than Stage 2 firms, and that these differences are not trivial because the lower limits are all > 1.5.
A study population of about 4,500 small manufacturing firms was identified and a random sample of 150 firms was selected from this study population. Each of the 150 firms was contacted and asked about their use of the just-in-time (JIT JIT - dynamic translation ) inventory strategy in 2006 and in 2007. Although JIT can lower a firm's storage and carrying costs Carrying costs
Costs that increase with increases in the level of investment in current assets. , the researchers wanted to know if the sharp increase in gasoline prices from 2006 to 2007 affected the use of JIT during this period. Of the 150 firms, 36 reported that they used JIT in 2006, 20 reported that they used JIT in 2007 and 16 reported that they used JIT in both 2006 and 2007. The McNemar test is the appropriate statistical test for this repeated measures design and the results might be reported as: "The McNemar test was computed and a significant decrease from 2006 to 2007 in the population proportion of firms using JIT was found (z = - 3.27, p < .001)." An alternative analysis would report a 95 percent confidence interval for the difference in population proportions. The results might be stated as: "We are 95% confident that the proportion of firms using JIT in 2007 is 0.23-0.34 less than the proportion of firms using JIT in 2006 in our study population of 4,500 small manufacturing firms."
A measure of the degree to which returns on two risky assets move in tandem. A positive covariance means that asset returns move together. A negative covariance means returns vary inversely. structure models (also known as causal models or structural equation models) are being used with increasing frequency in supply chain management research. As noted in Bonett and Wright (2007) a particular causal hypothesis will imply that some of the model parameters should be very small while other model parameters should be meaningfully large. Suppose that a particular causal hypothesis implies that variable A (e.g., amount of communication between the firm and supplier) and variable C (e.g., length of relationship between the firm and supplier) with both variables B and C causing variable D (e.g., quality performance of firm). This hypothesis implies that there will be meaningfully large path coefficients from variables A to B, A to C, B to D and C to D. The causal hypothesis also implies that the path coefficients from variables A to D and variables B to C should be small and unimportant. In current practice, the researcher fits a model that includes the path coefficients from variables A to B, A to C, B to D and C to D. The path coefficients from variables A to D and B to C are constrained to equal zero. The [x.sup.2] goodness-of-fit-test in this example has 2 degrees of freedom. A typical, but inappropriate, interpretation of a [x.sup.2] goodness-of fit test for this model might be stated as "The chi-square goodness-of-fit test (2) = 4.55, p> .10, was nonsignificant and thus provides support for our hypothesized causal model A causal model is an abstract model that uses cause and effect logic to describe the behaviour of a system. See also
This data analysis approach has two serious limitations. First, the [x.sup.2] test is actually a test that the population path coefficients from variables A to D and B to C are exactly zero. A "nonsignificant" [x.sup.2] test does not imply that these two population path coefficients are zero. These coefficients are almost certainly nonzero in the population and the [x.sup.2] test will become significant" if the sample size is large enough even if the path coefficients are small and unimportant. Second, a nonsignificant [x.sup.2] test (which researchers interpret as a "good fit") does not provide information about the magnitude of the population path coefficients from variables A to B, A to C, B to D and C to D. Even when the [x.sup.2] test suggests a "good fit," these four path coefficients could be small and unimportant. The goodness-of-fit indices popularized by Bender and Bonett (1980) do not provide the type of information needed to assess the size and importance of the path coefficients.
Bonett and Wright (2007) suggest that a better way to assess the hypothesized model involves the computation of simultaneous Bonferroni confidence intervals for all six path coefficients. If the simultaneous confidence intervals suggest that the two path coefficients that should be small are small and the four path coefficients that should be large are large, then this would provide strong support for the hypothesized model. It may be difficult to specify "small" and "large" values of the path coefficients because their values depend on the variance of the predictor variables. However, standardized path coefficients less than about 0.2 could be considered small and standardized path coefficients greater than about 0.5 could be considered large in most applications. In practice, a causal model will often be partially supported because not all path coefficients that should be large will be large and not all path coefficients that should be small will be small. This approach to causal model assessment is consistent with the notion of exposing theories to "risky tests" described by Edwards (2008).
The following recommendations address some common weaknesses in articles that frequently appear in prestigious business journals, including those in production, operations management and supply chain journal outlets. If supply chain, production and operations researchers follow these basic recommendations, the quality of their research will improve. More specifically, and consistent with Carter, Sanders and Dong (2008), our recommendations are designed to help provide the necessary "tipping point The point in time in which a technology, procedure, service or philosophy has reached critical mass and becomes mainstream. See network effect. See also tip and ring. " for the results of studies reported in these outlets to have a greater scientific and practical impact.
Inferential statistical methods allow the researcher to use information in a random sample to make certain types of statements about the study population from which the random sample was taken. It is important then to clearly define the study population so that there is no confusion about the study population for which the statistical results apply. The study population will often be a subset of a larger population, called the target population, which is the population of theoretical interest. In these cases, authors will want to generalize generalize /gen·er·al·ize/ (-iz)
1. to spread throughout the body, as when local disease becomes systemic.
2. to form a general principle; to reason inductively. the results from the study population to the target population using logical arguments. Unless the study population is clearly defined, it will be difficult for the authors to argue convincingly that the results of the study population can be extended to some target population.
Clearly describe the sampling procedure and explain how the sample was obtained. If the sample is a convenience sample and not a true random sample, make this fact clear to the reader. Random samples can be difficult to obtain and many supply chain researchers avoid this difficulty by using convenience samples and then tacitly assume that the convenience sample is a random sample of some "hypothetical" target population to which the inferential statistical results can be generalized. When convenience samples are used, it is the responsibility of the researcher to provide the reader with information that would support the claim that inferential statistical results will generalize to the hypothetical target population. These arguments can be extremely difficult to make and supply chain researchers will find that it may be easier to take random samples where the results of the statistical inferential methods (assuming assumptions of the methods are satisfied) may be generalized to the study population without explanation.
Always report and interpret a confidence interval for all important effect-size parameters and not just the point estimates of these parameters. Bonett and Wright (2007) explain why the current practice of reporting only point estimates of effect-size measures can be very misleading.
When studying the relation between two variables, if the variables are measured in a metric that is well understood (time, dollars, distance, etc.), report a confidence interval for an unstandardized measure of the effect size (e.g., difference in means or unstandardized regression coefficient). If the metric of the variable is not well understood (e.g., scores from author-constructed questionnaires) a confidence, interval for a standarized measure of effect size (e.g., standardized difference in means, correlation coefficient Correlation Coefficient
A measure that determines the degree to which two variable's movements are associated.
The correlation coefficient is calculated as: , standardized regression coefficient) should be reported along with or in place of the understandaridized measure. General methods for computing confidence intervals for standardized differences in means are described by Bonett (2008).
Clearly define all variables used in the statistical analysis. It is helpful to report large-sample estimates of means and standard deviations for every quantitative variable so that the reader will be better able to interpret the population parameters that are described by the confidence intervals (it may be possible to obtain these large-sample). For instance, in a linear regression analysis the reader will not be able to meaningfully interpret a confidence interval for a population slope coefficient in the absence of accurate estimates of the population standard deviations of the variables.
If a questionnaire is used to measure some attribute, report the psychometric psy·cho·met·rics
n. (used with a sing. verb)
The branch of psychology that deals with the design, administration, and interpretation of quantitative tests for the measurement of psychological variables such as intelligence, aptitude, and properties of the questionnaire. Measure of test-retest reliability are often more informative than the more commonly reported measures of internal consistency In statistics and research, internal consistency is a measure based on the correlations between different items on the same test (or the same subscale on a larger test). It measures whether several items that propose to measure the same general construct produce similar scores. such as Cronbach's alpha Cronbach's (alpha) has an important use as a measure of the reliability of a psychometric instrument. It was first named as alpha by Cronbach (1951), as he had intended to continue with further instruments. . Furthermore, researchers should not assume that the name of the questionnaire (e.g., "Customer Satisfaction Survey") accurately reflects the attribute measured by the questionnaire. Ideally, several forms of validity evidence such as convergent validity Convergent validity is the degree to which an operation is similar to (converges on) other operations that it theoretically should also be similar to. For instance, to show the convergent validity of a test of mathematics skills, the scores on the test can be correlated with scores , discriminant validity Discriminant validity describes the degree to which the operationalization is not similar to (diverges from) other operationalizations that it theoretically should not be similar to. and predictive validity In psychometrics, predictive validity is the extent to which a scale predicts scores on some criterion measure.
For example, the validity of a cognitive test for job performance is the correlation between test scores and, for example, supervisor performance ratings. (Kerlinger and lee 2000) should be reported to support the claim that the questionnaire is measuring what it claims to measure.
It is now common practice in supply chain management research to report point estimates of reliability and validity coefficients. It is important to also report confidence intervals for the study population values of the reliability and validity coefficients. When researchers develop new scales, they often perform only the most rudimentary psychometric analyses and do not provide adequate evidence of what Messick (1995) refers to as consequential con·se·quen·tial
1. Following as an effect, result, or conclusion; consequent.
2. Having important consequences; significant: validity. Evidence of consequential validity provides information to help understand the metric of the response variable. In the absence of adequate consequential validity, standardized measures of effect size are usually preferred to understandardized measures.
Do not use sophisticated statistical methods when more simple and easy-to-understand methods yield the same basic conclusions. When using a new "state-of-the-art" statistical method, be sure to clearly describe the assumptions of the method and the effects of violating the assumptions. Most sophisticated statistical methods are based on assumptions that are difficult to justify and do not perform properly when their assumptions have been violated. The problem with assumption violations is exacerbated by the fact that standard diagnostic methods often are unable to detect the types of assumption violations that would have serious negative effects on the performance of the statistical procedure.
Consistent with Bonett and Wright's (2007) findings for general management journals, only a small percentage of empirical articles in leading supply chain and operations journals currently report confidence intervals. Over 20 years ago, Nelder (1985, p. 238) noted that "The grotesque emphasis on significance tests in statistics courses of all kinds...is taught to people, who if they come away with no other notion, will remember that statistics is tests about significant differences". This "grotesque emphasis" still exists today. Meaningful change in the reporting of statistical results will not occur until journal gatekeepers provide more tangible guidelines and support for authors seeking to implement this change. One suggestion involves journals using "moral suasion Moral Suasion
A persuasion tactic used by an authority (i.e. Federal Reserve Board) to influence and pressure, but not force, banks into adhering to policy. Tactics used are closed-door meetings with bank directors, increased severity of inspections, appeals to community spirit, or " tactics and actively encouraging that all accepted manuscripts report confidence intervals whenever appropriate.
We propose that leading production, supply chain and operations management journals, such as the Journal of Supply Chain Management, actively recruit editorial board members who are able to work with authors of conditionally accepted papers to insure that the appropriate confidence intervals have been reported and correctly interpreted. Furthermore, and consistent with Bonett and Wright's (2007) recommendations for general management research, we encourage that the publication criteria focus not on the size of the p-value, but on more important criteria such as the quality of the sampling design, the quality of the research design, the reliability and validity of the variables under investigation and the confidence interval widths. It is our hope that these suggestions will stimulate further dialogue and the acceptance of our recommendations regarding the use of hypothesis tests and confidence interval methods in supply chain management journals.
Bentler, P.M. and D.G. Bonett. "Significance Tests and Goodness of Fit Goodness of fit means how well a statistical model fits a set of observations. Measures of goodness of fit typically summarize the discrepancy between observed values and the values expected under the model in question. Such measures can be used in statistical hypothesis testing, e. in the Analysis of Covariance Structures", Psychological Bulletin, (88), 1980, pp. 588-606.
Bonett, D.G. "Confidence Intervals for Standardized Linear Contrasts of Means", Psychological Methods, (13), 2008, pp. 99-109.
Bonett, D.G. and T.A. Wright. "Comments and Recommendations Regarding the Hypothesis Testing Controversy", Journal of Organizational Behavior, (28), 2007, pp. 647-659.
Boring, E.G. "Mathematical Versus Statistical Importance, Psychological Bulletin, (16), 1919, pp. 335-338.
Carter, C.R., N.R. Sanders and Y. Dong "Paradigms, Revolutions, and Tipping Points: The Need for Using Multiple Methodologies within the Field of Supply Chain Management," Journal of Operations Management, (26:6), 2008, pp. 693-696.
Casella, G. and R.L. Berger. Statistical Inference, 2nd ed., Duxbury, Pacific Grove Pacific Grove, residential and resort city (1990 pop. 16,117), Monterey co., W central Calif., on a point where Monterey Bay meets the Pacific Ocean; inc. 1889. , CA, 2002.
Edwards, J.R. To Prosper, Organizational Psychology Should...Overcome Methodological Barriers to Progress", Journal of Organizational Behavior, (29), 2008, pp. 469-491.
Gardner, MJ. and D.G. Altman. "Confidence Intervals Rather Than P Values: Estimation Rather Than Hypothesis Testing", British Medical Journal The British Medical Journal, or BMJ, is one of the most popular and widely-read peer-reviewed general medical journals in the world. It is published by the BMJ Publishing Group Ltd (owned by the British Medical Association), whose other , (292), 1986, pp. 746-750.
Graybill, F.A. and H.K. Iyer. Regression Analysis: Concepts and Applications, Duxbury, Belmont, CA, 1994.
Harlow, L.L., S.A. Mulaik and J.H. Steiger (Eds.). What If There Were No Significance Tests?, Erlbaum, Mahwah, NJ, 1997.
Hunter, J.E. "Needed: A Ban on the Significance Test", Psychological Science, (8), 1997, pp. 3-7.
Kaiser, H.F. "Directional Statistical Decisions," Psychological Review, (67), 1960, pp. 160-167.
Kerlinger, F.N. and H.B. Lee. Foundations of Behavioral Research, 4th ed., Wadsworth, Belmont, CA, 2000.
Kline, R.B. Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research, APA (All Points Addressable) Refers to an array (bitmapped screen, matrix, etc.) in which all bits or cells can be individually manipulated.
APA - Application Portability Architecture , Washington, DC, 2004.
Messick, S. "Validity of Psychological Assessment: Validation of Inferences form Person's Responses and Performances as Scientific Inquiry into Score Meaning", American Psychologist, (50), 1995, pp. 741-749.
Morrison, D.E. and R.E. Henkel (Eds.). The Significance Test Controversy, Aldine, Chicago, 1970.
Nelder, J.A. "Discussion on Papers by chatfield," Journal of the Royal Statistical Society The Journal of the Royal Statistical Society is a series of three peer-reviewed statistics journals published by Blackwell Publishing for the London-based Royal Statistical Society. , Series A, (148), 1985, p. 238.
Stevens, G.C."Integrating the Supply Chain," International Journal of Physical Distribution and Logistics Management Logistics Management is that part of Supply Chain Management that plans, implements, and controls the efficient, effective, forward, and reverse flow and storage of goods, services, and related information between the point of origin and the point of consumption in order to meet , (19:8), 1989, pp. 3-8.
Tukey, J.W. "Conclusions Vs. Decisions," Technometrics, (2), 1960, pp. 423-433.
Wilkinson, L. and the Task Force on Statistical Inferences. "Statistical Methods in Psychology Journals: Guidelines and Explanations" American Psychologist, (54), 1999, pp. 594-604.
Douglas G. Bonett (Ph.D., UCLA UCLA University of California at Los Angeles
UCLA University Center for Learning Assistance (Illinois State University)
UCLA University of Carrollton, TX and Lower Addison, TX ) is a proferssor of statistics and psychology at Iowa State University Academics
ISU is best known for its degree programs in science, engineering, and agriculture. ISU is also home of the world's first electronic digital computing device, the Atanasoff–Berry Computer. in Ames, IA. Dr. Bonett's current research interests focus on the development of interval estimation methods and sample size requirement methods.
Thomas A. Wright (Ph.D., University of California, Berkeley The University of California, Berkeley is a public research university located in Berkeley, California, United States. Commonly referred to as UC Berkeley, Berkeley and Cal ) is the Jon Wefald Jon Wefald (born November 24, 1937, in Minneapolis, Minnesota) is an American educator and the twelfth President of Kansas State University, holding that position since 1986.
Wefald was born in Minneapolis and moved, at age six, with his family to Minot, North Dakota. Leadership Chair and a professor of organizational behavior at Kansas State University Kansas State University, main campus at Manhattan; coeducational; land-grant and state supported; chartered and opened 1863. There is an additional campus at Salina. Among the university's research facilities are the J. R. in Manhattan, KS. In recognition of his career accomplishments, Dr. Wright was awarded Fellow status in the Association for Psychological Science (APS) in 2007. He has published his work in such outlets as the Academy of Management Review, Journal of Applied Psychology Journal of Applied Psychology is a publication of the APA. It has a high impact factor for its field. It typically publishes high quality empirical papers.
www.apa. , Psychometrika, Academy of Management Executive, Organizational Dynamics Journal of Organizational Behavior, Journal of Occupational Health Psychology, Journal of Management and the Journal of Management Inquiry.
* Like all invited papers and invited notes, the original version of this manuscript underwent a double-blind review process.
DOUGLAS G. BONETT
Iowa State University
THOMAS A. WRIGHT
Kansas State University