Statistical interpretation of key comparison reference value and degrees of equivalence.

Key comparisons carried out by the Consultative Committees (CCs) of the International Committee of Weights and Measures (CIPM) or the Bureau International des Poids et Mesures (BIPM) are referred to as CIPM key comparisons. The outputs of a statistical analysis of the data from a CIPM key comparison are the key comparison reference value, the degrees of equivalence, and their associated uncertainties. The BIPM publications do not discuss statistical interpretation of these outputs. We discuss their interpretation under the following three statistical models: nonexistent laboratory-effects model, random laboratory-effects model, and systematic laboratory-effects model.

Keywords: interlaboratory evaluation; measurement uncertainty; variance components.

**********

1. Introduction

Key comparisons are interlaboratory comparisons that serve as technical bases for Mutual Recognition Arrangements (MRA) between national metrology institutes (NMIs) [1]. Key comparisons carried out by the Consultative Committees (CCs) of the International Committee of Weights and Measures (CIPM) or the Bureau International des Poids et Mesures (BIPM) are referred to as CIPM key comparisons. Key comparisons carried out by regional metrology organizations (RMO) are referred to as RMO key comparisons. The guidelines for carrying out CIPM key comparisons are given in reference [2].

The objectives of a CIPM key comparison are described in reference [1]. We consider two interpretations of these objectives. A common interpretation is summarized by Nielsen [3] as follows: "The purpose of measurement intercomparisons between NMIs is to test, whether measurements performed in the participating countries are consistent taking into account the uncertainties assigned to the measurements. If an inconsistency is detected, the participating countries should take the corrective actions needed to obtain consistency. Otherwise, measurement results exchanged across borders cannot be considered equivalent without adding a 'between countries uncertainty,' which would be in disharmony with the concept of the SI system of units."

This paper is based on a second interpretation of the objectives of a CIPM key comparison: Generally, the participants of a CIPM key comparison are NMIs that are members of the appropriate Consultative Committee; at least some of these NMIs provide realizations of the SI values to establish the traceability of measurements made in their countries. The purpose of a CIPM key comparison is to establish the key comparison reference value (1), the degrees of equivalence (2), and their associated uncertainties on the basis of the data provided by the participants.

This paper is limited to a simple CIPM key comparison where the common measurand is a physical quantity of stable value during the comparison. Many CIPM key comparisons are not simple because it is often impractical or impossible to realize exactly the same measurand for or by all participants. We use the symbol Y for the stable value of the measurand. The data provided by the participants of a simple CIPM key comparison are paired results and standard uncertainties [[x.sub.1], u([x.sub.1])],..., [[x.sub.n], u([x.sub.n])], where the results [x.sub.1],..., [x.sub.n] are measurements of Y. The outputs of a statistical analysis of these data are the key comparison reference value [x.sub.R], the degree of equivalence [d.sub.i] = [x.sub.i] - [x.sub.R] of the result [x.sub.i], the degree of equivalence [d.sub.i,j] = [d.sub.i] - [d.sub.j] = [x.sub.i] - [x.sub.j] of the results [x.sub.i] and [x.sub.j], and their associated standard uncertainties u([x.sub.R]), u([d.sub.i]), and u([d.sub.i,j]), respectively, for i,j = 1, 2,..., n and i [not equal to] j [1]. The key comparison reference value [x.sub.R] is an estimate for Y. An estimate for Y is a combined result of measurement determined from the data [[x.sub.1], u([x.sub.1])],..., [[x.sub.n], u([x.sub.n])].

An understanding of the difference between sampling probability distributions, used in classical (frequentist) statistics, and state-of-knowledge probability distributions, used in Bayesian statistics, is necessary for proper analysis and interpretation of the data from a key comparison. Briefly, they are defined as follows. In classical statistics, the value of the measurand is assumed to be an unknown constant, often called the true value, and each result of measurement is regarded as a realization of a random variable with a sampling distribution. A sampling distribution is a probability distribution that describes the relative frequencies of occurrence for all possible results of measurement when the conditions of measurement are hypothesized to be fixed at the intended levels [4]. The metrologist relates the expected values of the sampling distributions for the results of measurement to the value of the measurand. A classical (frequentist) statistical interpretation is a statement that relates the realized measurements to what one might expect if the key comparison could be repeated infinitely many times and throughout these repetitions the hypothesized sampling distributions continued to apply.

In Bayesian statistics, the measurement data are given constants and the value of the measurand is a random variable. A probability distribution for the value of the measurand is a state-of-knowledge distribution that describes the degrees of belief for all possible values that could be attributed to the measurand [4]. The belief is based on all available information including current results of measurement and scientific judgment based on prior and other data. Similar state-of-knowledge distributions apply to the other parameters involved in assessing the value of the measurand. A Bayesian interpretation is a statement that represents the state-of-knowledge about the value of the measurand based on state-of-knowledge distributions before measurements are made and a likelihood function conditional on the current measurements [4]. The ISO Guide [5] is consistent with a Bayesian interpretation of measurements but not with a classical (frequentist) interpretation [4].

We refer to the results [x.sub.1],..., [x.sub.n] as laboratory results. The laboratory results [x.sub.1],..., [x.sub.n] are regarded as realizations of random variables [x.sub.1],..., [x.sub.n] with sampling distributions (3). We use the symbols [X.sub.1],..., [X.sub.n] for the expected values E([x.sub.1]),..., E([x.sub.n]) of the sampling distributions of [x.sub.1],..., [x.sub.n], respectively. We refer to the expected values [X.sub.1],..., [X.sub.n] as the laboratory expected values. We use the symbols [[sigma].sub.1],..., [[sigma].sub.n] for the standard deviations S ([x.sub.1]),..., S ([x.sub.n]) of the sampling distributions of [x.sub.1],..., [x.sub.n], respectively. Here S([x.sub.i]) is the square root of the variance V([x.sub.i]) = E[[x.sub.i] - E([x.sub.i])][.sup.2] of the sampling distribution of [x.sub.i] for i = 1, 2,..., n. The uncertainties u([x.sub.1]),..., u([x.sub.n]) are statistical estimates of [[sigma].sub.1],..., [[sigma].sub.n], respectively.

References [1] and [2] do not discuss statistical interpretations of the pairs [[x.sub.R], u([x.sub.R])], [[d.sub.i], u([d.sub.i])], and [[d.sub.i,j], u([d.sub.i,j])]. A statistical analysis of the data from a key comparison and interpretation of its outputs requires assumptions and models about the relationship between the data [[x.sub.1], u([x.sub.1])],..., [[x.sub.n], u([x.sub.n])] and the value Y of the measurand. In See. 2, we discuss two assumptions, labeled as Assumption I and Assumption II, about the relationship between the laboratory expected values [X.sub.1],..., [X.sub.n] and Y. Then we discuss two classical statistics models, a nonexistent laboratory-effects model and a random laboratory-effects model, based on Assumption I. Next, we propose a systematic laboratory-effects model based on Assumption II. We describe the key comparison reference value, the degrees of equivalence, and their associated uncertainties determined by each of the three statistical models. In Sec. 3 and 4, we discuss statistical interpretations of the pairs [[x.sub.R], u([x.sub.R])], [[d.sub.i], u([d.sub.i])], and [[d.sub.i,j], u([d.sub.i,j])] under the three statistical models. Our conclusion is given in Sec. 5.

2. Statistical Assumptions and Models for the Relationship Between the Data and the Value of the Measurand

In this section, we discuss statistical assumptions and models for analyzing the data from a simple CIPM key comparison to determine the key comparison reference value, the degrees of equivalence, and their associated uncertainties.

2.1 Assumptions About the Relationship Between the Laboratory Expected Values and the Value of the Measurand

One may either assume that the laboratory expected values [X.sub.1],..., [X.sub.n] are all equal or allow for the possibility that [X.sub.1],..., [X.sub.n] may not be equal.

Assumption I: The expected values [X.sub.1],..., [X.sub.n] are all equal. The Assumption I defined so far does not specify the relationship between the results [x.sub.1],..., [x.sub.n] and Y. Therefore, in concert with Assumption I, it is generally assumed that the common expected value is equal to Y, i.e., [X.sub.1] = ... = [X.sub.n] = Y. Under Assumption I, the results [x.sub.1],..., [x.sub.n] are subject to intralaboratory variations only.

Assumption II: The expected values [X.sub.1],..., [X.sub.n] may not be equal, i.e., [X.sub.i] [not equal to] [X.sub.j] for some i,j = 1, 2,..., n and i [not equal to] j. Therefore, not all of [X.sub.1],..., [X.sub.n] may equal the value Y of the measurand. The Assumption II defined so far does not specify the relationship between the results [x.sub.1],..., [x.sub.n] and Y. Therefore, in concert with Assumption II, it is generally assumed that Y is either somewhere in the range of results [x.sub.1],..., [x.sub.n] or in the vicinity of this range (4) [6]. Under Assumption II, the results [x.sub.1],..., [x.sub.n] are subject to both the intralaboratory variations represented by the uncertainties u ([x.sub.1]),..., u ([x.sub.n]) and the interlaboratory variation arising from the dispersion of [X.sub.1],..., [X.sub.n] about Y. The differences ([X.sub.1] - Y),..., ([X.sub.n] - Y) are laboratory-effects (biases) due to unrecognized sources of error, denoted by [b.sub.1],..., [b.sub.n], in the results [x.sub.1],..., [x.sub.n]. The biases are common to all measurements in a particular laboratory but may be different for different laboratories.

2.2 Assumption About the Uncertainties Submitted by the Participants

The standard uncertainties u([x.sub.1]),..., u([x.sub.n]) submitted by the participants of a key comparison are estimates obtained by combining various estimated components of uncertainty in determining the value Y of the measurand. A combined standard uncertainty u([x.sub.i]) may be unreliable for various reasons. For example, a classical (frequentist) Type A component of u([x.sub.i]) calculated from a small number of independent measurements is unreliable (5) [5]. A Type A component of u([x.sub.i]) based on unjustified statistical assumptions may be unreliable. A Type B component of u([x.sub.i]) based on unreasonable state-of-knowledge distributions may be unreliable. A combined uncertainty u([x.sub.i]) determined from an incomplete measurement equation may be an underestimate. The unreliability of estimated uncertainties u([x.sub.1]),..., u([x.sub.n]) is a component of uncertainty in determining the key comparison reference value [x.sub.R], the degrees of equivalence [d.sub.i] and [d.sub.i,j], and their associated standard uncertainties. In this paper, we do not discuss the additional uncertainty that arises from the unreliability of u([x.sub.1]),..., u([x.sub.n]).

Classical (frequentist) statistical analyses and interpretations discussed in this paper are based on the assumption that the estimated uncertainties u([x.sub.1]),..., u([x.sub.n]) are equal to the true standard deviations [[sigma].sub.1],..., [[sigma].sub.n] of the sampling distributions of [x.sub.1],..., [x.sub.n], respectively. Most metrologists make this assumption. For example, the expression u([x.sub.W]) = 1/[square root of ([[[SIGMA].sub.i] [w.sub.i]])] for the standard deviation of the weighted mean [x.sub.W] = [[SIGMA].sub.i] [w.sub.i] [x.sub.i]/[[SIGMA].sub.i] [w.sub.i], where [w.sub.i] = 1/[u.sup.2]([x.sub.i]) for i = 1, 2,..., n, requires this assumption.

Statistical analyses based on the ISO Guide regard a laboratory expected value [X.sub.i] as a variable with a state-of-knowledge distribution having expected value [x.sub.i] and standard deviation u([x.sub.i]). Such analyses require the assumption that the estimated uncertainties u([x.sub.1]),..., u([x.sub.n]) are sufficiently reliable.

2.3 Classical (Frequentist) Statistics Models Based on Assumption I

The weighted mean [x.sub.W] = [[SIGMA].sub.i] [w.sub.i] [x.sub.i] / [[SIGMA].sub.i] [w.sub.i] and the expression u([x.sub.W]) = 1/[square root of ([[[SIGMA].sub.i] [w.sub.i]])], where [w.sub.i] = 1/[u.sup.2]([x.sub.i]) for i = 1, 2,..., n, are often used as the key comparison reference value [x.sub.R] and its associated standard uncertainty u([x.sub.R]), respectively. The use of [x.sub.W] as [x.sub.R] and u([x.sub.W]) as u([x.sub.R]) is based on the following classical (frequentist) statistics model.

2.3.1 Nonexistent Laboratory-Effects Model

The results are regarded as realizations of the random variables [x.sub.1],..., [x.sub.n], where

[x.sub.i] = Y + [e.sub.i], (1)

and [e.sub.i] = ([x.sub.i] - Y) is the error in [x.sub.i] for i = 1, 2,..., n. In this model, the parameter Y is identified with the value of the measurand and the errors [e.sub.1],..., [e.sub.n] are mutually independently distributed random variables with sampling distributions. The sampling distributions of [e.sub.1],..., [e.sub.n] are generally assumed to be normal (Gaussian). The expected values of [e.sub.1],..., [e.sub.n] are assumed to be zero and the variances of [e.sub.1],..., [e.sub.n] are assumed to be [u.sup.2]([x.sub.1]),..., [u.sup.2]([x.sub.n]), respectively. Under model (1) [represented by Eq. (1)], the expected value E([x.sub.i]) is equal to Y and the variance V([x.sub.i]) is equal to [u.sup.2]([x.sub.i]), for i = 1, 2,..., n. Since the expected values of all results are equal to Y, the model (1) is based on Assumption I. In model (1), the results [x.sub.1],..., [x.sub.n] are free of laboratory-effects (biases). Therefore, we refer to it as a nonexistent laboratory-effects model. The best least-squares estimate for the parameter Y of the nonexistent laboratory-effects model (1) is the weighted mean [x.sub.W] = [[SIGMA].sub.i] [w.sub.i] [x.sub.i] / [[SIGMA].sub.i] [w.sub.i], where [w.sub.i] = 1/[u.sup.2]([x.sub.i]) for i = 1, 2,..., n. The term best least-squares estimate (6) means that the estimate [x.sub.W] has the smallest variance among all estimates of Y that are both linear functions of the results [x.sub.1],..., [x.sub.n] and have the expected value Y. The standard deviation of the sampling distribution of [x.sub.W] is u([x.sub.W]) = 1/[square root of ([[[SIGMA].sub.i] [w.sub.i]])]. Thus the key comparison reference value [x.sub.R] based on model (1) is [x.sub.W] and u([x.sub.R]) is u([x.sub.W]). The corresponding degrees of equivalence are [d.sub.i] = [x.sub.i] - [x.sub.W] and [d.sub.i,j] = [x.sub.i] - [x.sub.j], for i, j = 1, 2,..., n and i [not equal to] j. The uncertainties u([d.sub.i]) and u([d.sub.i,j]) are determined from the sampling distributions of [x.sub.1],..., [x.sub.n] and [x.sub.R] under model (1).

Note 1: When not all uncertainties u([x.sub.1]),..., u([x.sub.n]) are sufficiently reliable estimates of the true standard deviations [[sigma].sub.1],..., [[sigma].sub.n], the true standard deviation of the sampling distribution of the weighted mean [x.sub.W] may be larger than the true standard deviation of the sampling distribution of the arithmetic mean [x.sub.A]. Thus in this case the weighted mean [x.sub.W] may be an inferior key comparison reference value to the arithmetic mean [x.sub.A].

2.3.2 Random Laboratory-Effects Model

The classical statistics model based on Assumption I for the situation where the dispersion of results [x.sub.1],..., [x.sub.n] may be more than what can reasonably be attributed to the intralaboratory variances [u.sup.2]([x.sub.1]),..., [u.sup.2]([x.sub.n]) is as follows. The results are regarded as realizations of the random variables [x.sub.1],..., [x.sub.n], where

[x.sub.i] = Y + [b.sub.i] + [e.sub.i], (2)

[b.sub.i] = ([X.sub.i] - Y) is the laboratory effect (bias) in [x.sub.i] and [e.sub.i] = ([x.sub.i] - [X.sub.i]) is the intralaboratory error in [x.sub.i] for i = 1, 2,..., n. The classical statistics assumptions to relate the results [x.sub.1],..., [x.sub.n] to Y are as follows: the laboratory biases [b.sub.1],..., [b.sub.n] are regarded as random variables having the same normal sampling distribution with expected value zero and variance [[sigma].sub.b.sup.2] [greater than or equal to] 0, called interlaboratory variance; and [b.sub.1],..., [b.sub.n] are assumed to be mutually independent and independent of the errors [e.sub.1],..., [e.sub.n]. The model (2) [represented by Eq. (2)] with these assumptions is referred to as a random laboratory-effects model [7]. Here the term random means that the biases [b.sub.1],..., [b.sub.n] are regarded as random variables with the same sampling distribution that is assumed to be normal with expected value zero and variance [[sigma].sub.b.sup.2]. Under the random laboratory-effects model (2), the expected value E([x.sub.i]) is equal to Y and the variance V([x.sub.i]) is equal to [[sigma].sub.b.sup.2] + [u.sup.2]([x.sub.i]) for i = 1, 2,..., n. The non-existent laboratory-effects model (1) is a special case of the random laboratory-effects model (2) where [[sigma].sub.b.sup.2] = 0, which means that the biases [b.sub.1],..., [b.sub.n] are all zero, i.e., [X.sub.1] = ... = [X.sub.n] = Y.

A popular estimate for the parameter Y of model (2) is the weighted mean (7) [x.sub.W] = [[SIGMA].sub.i] [w.sub.i] [x.sub.i] / [[SIGMA].sub.i] [w.sub.i], where [w.sub.i] = 1/[[s.sub.b.sup.2] + [u.sup.2]([x.sub.i])] and [s.sub.b.sup.2] is an estimate for [[sigma].sub.b.sup.2]. Reference [8] discusses various methods for determining [s.sub.b.sup.2]. The estimate [s.sub.b.sup.2] inflates each of the intralaboratory variances [u.sup.2]([x.sub.1]),..., [u.sup.2]([x.sub.n]) just enough to account for the dispersion of results [x.sub.1],..., [x.sub.n] that is not accounted for by model (1). Under the assumption that the estimated variances [s.sub.b.sup.2] + [u.sup.2]([x.sub.1]),..., [s.sub.b.sup.2] + [u.sup.2]([x.sub.n]) are regarded as the true variances of the sampling distributions of [x.sub.1],..., [x.sub.n], the best estimate of the parameter Y of model (2) is the weighted mean [x.sub.W] and the standard uncertainty associated with [x.sub.W] is u([x.sub.W]) = 1/[square root of ([[[SIGMA].sub.i] [w.sub.i]])], where [w.sub.i] = 1/[[s.sub.b.sup.2] + [u.sup.2]([x.sub.i])] for i = 1, 2,..., n [9], [8]. Thus the key comparison reference value [x.sub.R] based on model (2) is the weighted mean [x.sub.W] = [[SIGMA].sub.i] [w.sub.i] [x.sub.i] / [[SIGMA].sub.i] [w.sub.i] and uncertainty u([x.sub.R]) is u([x.sub.W]) = 1/[square root of ([[SIGMA].sub.i] [w.sub.i])], where [w.sub.i] = 1/[[s.sub.b.sup.2] + [u.sup.2]([x.sub.i])] for i = 1, 2,..., n. The corresponding degrees of equivalence are [d.sub.i] = [x.sub.i] - [x.sub.R] = [x.sub.i] - [x.sub.W] and [d.sub.i,j] = [x.sub.i] - [x.sub.j], for i,j = 1, 2,..., n and i [not equal to] j. The uncertainties associated with the degrees of equivalence are determined from the sampling distributions of [x.sub.1],..., [x.sub.n] and [x.sub.R] under model (2).

The advantage of model (2) relative to model (1) is that it allows for the possibility that the dispersion of results [x.sub.1],..., [x.sub.n] may be more than what can reasonably be attributed to the intralaboratory variances [u.sup.2]([x.sub.1]),..., [u.sup.2]([x.sub.n]). When the dispersion of [x.sub.1],..., [x.sub.n] is not more than what can reasonably be attributed to [u.sup.2]([x.sub.1]),..., [u.sup.2]([x.sub.n]), the estimate [s.sub.b.sup.2] is zero. In that case, model (2) yields the same [x.sub.R] and u([x.sub.R]) as model (1). Therefore, there is no disadvantage to using model (2) in place of model (1).

The random laboratory-effects model (2) of classical statistics is conceptually faulty for the analysis of a CIPM key comparison for the following reasons. First, the participants of a CIPM key comparison are specific NMI laboratories rather than randomly chosen from a large population of laboratories. Therefore, the biases [b.sub.1],..., [b.sub.n] may not be regarded as random variables with the same sampling distribution. Second, the assumption that the sampling distribution of the biases [b.sub.1],..., [b.sub.n] is a normal distribution with expected value zero may not be justified. The next section introduces a new model that does not assume that the biases [b.sub.1],..., [b.sub.n] are random variables with a normal sampling distribution.

2.4 A Model Based on Assumption II and the ISO Guide

A statistical analysis of the data from a simple CIPM key comparison based on Assumption II requires one to account for the uncertainty that arises from the unknown bias in a combined result of measurement that is used as an estimate for Y. Before publication of the ISO Guide, there was no generally accepted approach to account for the uncertainty that arises from an unknown bias. The approach proposed by the ISO Guide to account for the uncertainty that arises from an unknown bias is now generally accepted. So we have used the ISO Guide to develop the following systematic laboratory-effects model.

2.4.1 Systematic Laboratory-Effects Model

We start with a combined result of the form [[SIGMA].sub.i] [a.sub.i] [x.sub.i], where [[SIGMA].sub.i] [a.sub.i] = 1, that is used as an initial estimate for Y. This estimate requires the assumption that Y is within the range of results [x.sub.1],..., [x.sub.n]. We refer to the initial estimate as the uncorrected combined result (UCR) and denote it by [x.sub.UCR] = [[SIGMA].sub.i] [a.sub.i] [x.sub.i]. If [a.sub.i] = [w.sub.i]/[[SIGMA].sub.i] [w.sub.i], then [x.sub.UCR] is the weighted mean [x.sub.W] = [[SIGMA].sub.i] [w.sub.i] [x.sub.i]/[[SIGMA].sub.i] [w.sub.i], where [w.sub.i] = 1/[u.sup.2]([x.sub.i]) for i = 1, 2,..., n. If [a.sub.i] = 1/n for i = 1, 2,..., n, then [x.sub.UCR] is the arithmetic mean [x.sub.A] = [[SIGMA].sub.i] [x.sub.i]/n. Let [X.sub.UCR] = [[SIGMA].sub.i] [a.sub.i] [X.sub.i] be the expected value of the sampling distribution of [x.sub.UCR]. According to Assumption II, the result [x.sub.UCR] is subject to the bias ([X.sub.UCR] - Y). The ISO Guide recommends that the result [x.sub.UCR] should be corrected to counter its possible bias and the uncertainty associated with the correction should be included in the combined standard uncertainty associated with the corrected result. The bias ([X.sub.UCR] - Y) is an unknown constant but the correction for bias, denoted by C, is a variable with a state-of-knowledge probability distribution. If the expected value and standard deviation of a state-of-knowledge probability distribution for the correction variable C are denoted by c and u(c), respectively, then the correction applied to the result [x.sub.UCR] to counter its possible bias is c and the standard uncertainty associated with the correction is u(c).

In order to specify a state-of-knowledge probability distribution for the correction variable C, the laboratory expected values [X.sub.1],..., [X.sub.n] and the value Y of the measurand are regarded as variables with state-of-knowledge distributions and the data [x.sub.1],..., [x.sub.n] and u([x.sub.1]),..., u([x.sub.n]) are regarded as given constants. A state-of-knowledge distribution for [X.sub.i] represents the state of knowledge about the value Y of the measurand in the laboratory labeled i for i = 1, 2,..., n. The expected value E([X.sub.i]) and standard deviation S([X.sub.i]) of the variable [X.sub.i] are assumed to be [x.sub.i] and u([x.sub.i]), respectively, for i = 1, 2,..., n [5], [4]. It follows that [X.sub.UCR] = [[SIGMA].sub.i] [a.sub.i] [X.sub.i] is a variable with a state-of-knowledge probability distribution. The expected value of [X.sub.UCR] is E([X.sub.UCR]) = [[SIGMA].sub.i] [a.sub.i] E([X.sub.i]) = [[SIGMA].sub.i] [a.sub.i] [x.sub.i] = [x.sub.UCR]. In the expression (Y - [X.sub.UCR]) for the negative of bias, treated as a variable, we replace [X.sub.UCR] with its expected value [x.sub.UCR]. Then a probability distribution for C represents belief about the possible values of (Y - [x.sub.UCR]), where [x.sub.UCR] is a constant and Y is the variable. The belief about possible values of Y is based on all available information including results of measurement and scientific judgment. In reference [6], we proposed a triangular distribution for the correction variable C, with peak at [x.sub.UCR] and default limits [[x.sub.(1)] - [x.sub.UCR]] = min {[x.sub.1] - [x.sub.UCR],..., [x.sub.n] - [x.sub.UCR]} and [[x.sub.(n)] - [x.sub.UCR]] = max{[x.sub.1] - [x.sub.UCR],..., [x.sub.n] - [x.sub.UCR]}. A criticism of the proposed triangular distribution with default limits is that it is determined by the extreme results [x.sub.(1)] = min{[x.sub.1],..., [x.sub.n]} and [x.sub.(n)] = max{[x.sub.1],..., [x.sub.n]}, which are sometimes suspected to be in error.

Here, we propose a discrete-equal-probability distribution that is determined by all of the results [x.sub.1],..., [x.sub.n]. The results [x.sub.1],..., [x.sub.n] are plausible values of Y as determined by competent laboratories. (8). So the known constant differences ([x.sub.1] - [x.sub.UCR]),..., ([x.sub.n] - [x.sub.UCR]) are plausible values of (Y - [x.sub.UCR]). These differences are a statistical basis for specifying a probability distribution for C. Let [c.sub.i] = [x.sub.i] - [x.sub.UCR] for i = 1, 2,..., n. Suppose [c.sub.1],..., [c.sub.n] are assigned probabilities [p.sub.1],..., [p.sub.n]. Then the expected value of C is c = E(C) = [[SIGMA].sub.i] [p.sub.i] [c.sub.i] = ([[SIGMA].sub.i] [p.sub.i] [x.sub.i]) - [x.sub.UCR] and the standard deviation of C is u(c) = S(C) =[square root of ([[SIGMA].sub.i][p.sub.i][([c.sub.i] - c)[.sup.2])]. Frequently, the available scientific knowledge is inadequate to assign different probabilities [p.sub.1],..., [p.sub.n] to [c.sub.1],..., [c.sub.n]. Therefore, we propose the discrete-equal-probability distribution for which [p.sub.i] = 1/n for i = 1, 2,..., n. The expected value and standard deviation of C based on discrete-equal-probability distribution are c = [x.sub.A] - [x.sub.UCR] and u(c) = [square root of ([[[SIGMA].sub.i] ([x.sub.i] - [x.sub.A])[.sup.2]/n])], respectively, where [x.sub.A] = [[SIGMA].sub.i] [x.sub.i] / n is the arithmetic mean of the results [x.sub.1],..., [x.sub.n].

A measurement equation is required to incorporate correction for possible bias in a combined result of measurement for Y. The measurement equation that corresponds to the bias ([X.sub.UCR] - Y) in the uncorrected combined result [x.sub.UCR] is Y = [X.sub.UCR] + C. This measurement equation is widely applicable in metrology [10]. It suggests the following model for the value Y of the measurand:

E([X.sub.i]) = [x.sub.i], S([X.sub.i]) = u([x.sub.i]), [X.sub.UCR] = [[summation].sub.i] [a.sub.i][X.sub.i], Y = [X.sub.UCR] + C, (3)

where [a.sub.1],..., [a.sub.n] are constants such that [[SIGMA].sub.i][a.sub.i] = 1. In this model, [X.sub.1],..., [X.sub.n], [X.sub.UCR], C, and Y are variables with state-of-knowledge distributions. The expected value and standard deviation of [X.sub.i] are the given constants [x.sub.i] and u([x.sub.i]), respectively, for i = 1, 2,..., n. A state-of-knowledge distribution for the correction variable C is defined independently of the state-of-knowledge distributions for the variables [X.sub.1],..., [X.sub.n], after the latter have been specified. In particular, [X.sub.UCR] and C are independently distributed. We refer to model (3) [represented by Eq. (3)] as a systematic laboratory-effects model to distinguish it from the random laboratory-effects model (2) that regards the biases (systematic errors) [b.sub.1],..., [b.sub.n] as random variables having the same sampling distribution with expected value zero. Suppose the standard deviation of the variable [X.sub.UCR] is S([X.sub.UCR]) = u([x.sub.UCR]). Then the corrected combined result for Y determined from the systematic laboratory-effects model (3) is y = [x.sub.UCR] + c and its associated standard uncertainty is u(y) = [square root of ([[u.sup.2]([x.sub.UCR]) + [u.sup.2](c)])].

The systematic laboratory-effects model (3) allows for the possibility that not all pairs of the variables [X.sub.1],..., [X.sub.n] may be independently distributed. The variance V([X.sub.UCR]) = [u.sup.2]([x.sub.UCR]) is determined from the variances and covariances of the variables [X.sub.1],..., [X.sub.n]. When the distributions of [X.sub.1],..., [X.sub.n] are independent and [X.sub.UCR] is the weighed mean [X.sub.W] = [[SIGMA].sub.i] [w.sub.i] [X.sub.i] / [[SIGMA].sub.i] [w.sub.i], where [w.sub.i] = 1/V([X.sub.i]) = 1/[u.sup.2]([x.sub.i]) for i = 1, 2,..., n, then [u.sup.2]([x.sub.UCR]) = V([X.sub.W]) = 1/{[[SIGMA].sub.i] [w.sub.i]] = 1/[{[[SIGMA].sub.i] [1/[u.sup.2]([x.sub.i])]}. When the distributions of [X.sub.1],..., [X.sub.n] are independent and [X.sub.UCR] is the arithmetic mean [X.sub.A] = [[SIGMA].sub.i] [X.sub.i] / n, then [u.sup.2]([x.sub.UCR]) = V([X.sub.A]) = (1/[n.sup.2])[[SIGMA].sub.i] V([X.sub.i]) = (1/[n.sup.2])[[SIGMA].sub.i] [u.sup.2]([x.sub.i]) (9).

In order to specify c and u(c), one is free to use any reasonable distribution for C, based on scientific judgment. When the discrete-equal-probability distribution is used, c = [x.sub.A] - [x.sub.UCR] and u(c) = [square root of ([[[SIGMA].sub.i] ([x.sub.i] - [x.sub.A])[.sup.2]/n])]. In that case, the result of measurement for Y is y = [x.sub.UCR] + c = [x.sub.UCR] + [x.sub.A] - [x.sub.UCR] = [x.sub.A] and u(y) = [square root of ([[u.sup.2]([x.sub.UCR]) + [u.sup.2](c)])], where u(c) = [square root of ([[[SIGMA].sub.i] ([x.sub.i] - [x.sub.A])[.sup.2]/n])].

Following the ISO Guide, the result y and uncertainty u(y) determined from the systematic laboratory-effects model (3) are interpreted as the expected value and standard deviation of a state-of-knowledge distribution for the values that could reasonably be attributed to Y based on the data [x.sub.1],..., [x.sub.n] and u([x.sub.1]),..., u([x.sub.n]) [5], [4], [6]. Thus the key comparison reference value [x.sub.R] based on the systematic laboratory-effects model (3) is y and uncertainty u([x.sub.R]) is u(y). The corresponding degrees of equivalence are [d.sub.i] = [x.sub.i] - y and [d.sub.i,j] = [x.sub.i] - [x.sub.j] for i, j = 1, 2,..., n and i [not equal to] j. The uncertainties u([d.sub.i]) and u([d.sub.i,j]) are determined from state-of-knowledge distributions for the variables [X.sub.1],..., [X.sub.n] and Y.

3. Interpretation of the Key Comparison Reference Value and Its Associated Uncertainty

3.1 Classical Statistics Models Based on Assumption I

The nonexistent laboratory-effects model and the random laboratory-effects model are based on classical (frequentist) statistics. In particular, the results [x.sub.1],..., [x.sub.n] are regarded as realizations of random variables with sampling distributions and Y is an unknown constant. Therefore, the key comparison reference value [x.sub.R] is a realization of a random variable with a sampling distribution that has expected value Y and standard deviation u([x.sub.R]) = u([x.sub.W]) = 1/[square root of ([[[SIGMA].sub.i] [w.sub.i]])]. In the nonexistent laboratory-effects model [w.sub.i] is 1/[u.sup.2]([x.sub.i]) and in the random laboratory-effects model [w.sub.i] is 1/[[s.sub.b.sup.2] + [u.sup.2]([x.sub.i])] for i = 1, 2,..., n. The interval [[x.sub.R] [+ or -] 2u([x.sub.R])] determined from a classical statistics model is a confidence interval for Y computed from the data [x.sub.1],..., [x.sub.n] and u([x.sub.1]),..., u([x.sub.n]). Imagine that the CIPM key comparison could be repeated infinitely many times in exactly the same conditions using exactly the same instruments and artifacts. Now imagine that throughout these repetitions exactly the same sampling distributions continued to apply to the random variables [x.sub.1],..., [x.sub.n]. Then the confidence level is the fraction of the infinitely many hypothetical intervals, such as [[x.sub.R] [+ or -] 2u([x.sub.R])], that would include Y [4].

3.2 Systematic Laboratory-Effects Model Based on Assumption II

The key comparison reference value [x.sub.R] and uncertainty u([x.sub.R]) determined from the systematic laboratory-effects model are given constants that represent the expected value and standard deviation of a state-of-knowledge distribution for Y based on the data [x.sub.1],..., [x.sub.n] and u([x.sub.1]),..., u([x.sub.n]). The interval [[x.sub.R] [+ or -] 2u([x.sub.R])] determined from the systematic laboratory-effects model is an expanded uncertainty interval for Y. The coverage probability (level of confidence) of the interval [[x.sub.R] [+ or -] 2u([x.sub.R])] is the fraction of a state-of-knowledge distribution for Y that is encompassed by this interval [4].

4. Interpretation of the Degrees of Equivalence and Their Associated Uncertainties

4.1 Classical Statistics Models Based on Assumption I

In the random laboratory-effects model and its special case the nonexistent laboratory-effects model, the expected values of the sampling distributions of [x.sub.1],..., [x.sub.n], and [x.sub.R] are all equal to Y. Therefore, the expected values of the sampling distributions of all degrees of equivalence [d.sub.i] = [x.sub.i] - [x.sub.R] and [d.sub.i,j] = [x.sub.i] - [x.sub.j] are zero, for i, j = 1, 2,..., n and i [not equal to] j. This implies that all computed degrees of equivalence, whether small or large, are statistical estimates of zero. In particular, according to these models, all degrees of equivalence published in the key comparison database (KCDB) [11] are estimates of zero.

4.2 Systematic Laboratory-Effects Model Based on Assumption II

In the systematic laboratory-effects model, the results [x.sub.1],..., [x.sub.n] are the expected values and the uncertainties u([x.sub.1]),..., u([x.sub.n]) are the standard deviations of state-of-knowledge distributions for the laboratory expected values [X.sub.1],..., [X.sub.n], treated as variables. It follows that the degree of equivalence [d.sub.i] = [x.sub.i] - [x.sub.R] = [x.sub.i] - y is the expected value of a state-of-knowledge distribution for the laboratory effect (bias) [X.sub.i] - Y for i = 1, 2,..., n, and the degree of equivalence [d.sub.i,j] = [x.sub.i] - [x.sub.j] is the expected value of a state-of-knowledge distribution for the difference [X.sub.i] - [X.sub.j] for i, j = 1, 2,..., n and i [not equal to] j. The uncertainty u([d.sub.i]) is the standard deviation (10) of [X.sub.i] - Y and the uncertainty u([d.sub.i,j]) is the standard deviation of [X.sub.i] - [X.sub.j], for i, j = 1, 2,..., n and i [not equal to] j.

5. Conclusion

We addressed a simple CIPM key comparison where the common measurand is a physical quantity of stable value during the comparison. We discussed statistical interpretation of the key comparison reference value, the degrees of equivalence, and their associated uncertainties determined from the following three statistical models: nonexistent laboratory-effects model, random laboratory-effects model, and systematic laboratory-effects model. The first two models are based on classical (frequentist) interpretation of measurements. The systematic laboratory-effects model is based on Bayesian interpretation of measurements.

The key comparison reference value [x.sub.R] and uncertainty u([x.sub.R]) determined from the systematic laboratory-effects model represent the expected value and standard deviation of a state-of-knowledge distribution for the value Y of the measurand. Therefore their statistical interpretation agrees with the ISO Guide. According to the systematic laboratory-effects model, the degree of equivalence [d.sub.i] and uncertainty u([d.sub.i]) are, respectively, the expected value and standard deviation of a state-of-knowledge distribution for the laboratory effect (bias) [X.sub.i] - Y, for i = 1, 2,..., n, and the degree of equivalence [d.sub.i,j] and uncertainty u([d.sub.i,j]) are, respectively, the expected value and standard deviation of a state-of-knowledge distribution for the difference [X.sub.i] - [X.sub.j], for i, j = 1, 2,..., n and i [not equal to] j. Thus the degrees of equivalence determined from the systematic laboratory-effects model quantitate the agreements and disagreements of laboratory results. Therefore, the systematic laboratory-effects model is suitable for the data analysis of a simple CIPM key comparison.

Acknowledgment 

The following provided helpful comments on earlier drafts of this paper: T. V. Vorburger, Ron Boisvert, Eric Shirley, Tony Kearsley, Jim Gardener, and Nell Sedransk.

Accepted: February 17, 2004

Available online: http://www.nist.gov/jres

(1) "Key comparison reference value: the reference value accompanied by its uncertainty resulting from a CIPM key comparison [1]."

(2) "Degree of equivalence of a measurement standard: the degree to which the value of a measurement standard is consistent with the key comparison reference value. This is expressed quantitatively by the deviation from the key comparison reference value and the uncertainty of this deviation. The degree of equivalence between two measurement standards is expressed as the difference between their respective deviations from the key comparison reference value and the uncertainty of this difference [1]."

(3) We use the symbols [x.sub.1],..., [x.sub.n] for both the random variables and their realized values.

(4) If the expected value [X.sub.1] were equal to the value Y of the measurand, then according to the ISO Guide, the interval [[x.sub.1] [+ or -] 2u([x.sub.1])] would represent an approximate range of the plausible values of Y. Likewise, if [X.sub.2] were equal to Y then the interval [[x.sub.2] [+ or -] 2u([x.sub.2])] would represent an approximate range of the plausible values of Y, and so on for [X.sub.3], [X.sub.4],..., [X.sub.n]. It follows from Assumption II that any one or more of the expected values [X.sub.1],..., [X.sub.n] may be close to or equal to Y; therefore, the total interval consisting of the union of intervals [[x.sub.i] [+ or -] 2u([x.sub.i])], for i = 1, 2,..., n, represents an approximate range of the plausible values of Y. However, most metrologists assign greater belief-probability to the middle than to the ends of the total interval.

(5) The unreliability of a classical (frequentist) estimate of uncertainty arising from a small number of measurements is quantified by degrees of freedom [5].

(6) Least-squares estimation does not require that the errors [e.sub.1],..., [e.sub.n] and hence [x.sub.1],..., [x.sub.n] have normal distributions.

(7) We did not introduce a new symbol for the weighted mean determined from model (2) because model (1) is a special case of model (2).

(8) As noted in the footnote of Sec. 2.1, the total interval consisting of the union of intervals [[x.sub.i] [+ or -] 2u([x.sub.i])], for i = 1, 2,..., n, represents an approximate range of the plausible values of Y.

(9) Since the harmonic mean of positive numbers is less than or equal to their arithmetic mean, V([X.sub.W]) [less than or equal to] V([X.sub.A]). When [u.sup.2]([x.sub.1]),..., [u.sup.2]([x.sub.n]) are equal, V([X.sub.W]) = V([X.sub.A]).

(10) The standard deviation of [X.sub.i] - Y depends on the covariance between [X.sub.i] and Y for i = 1, 2,..., n. Since Y = [X.sub.UCR] + C = [[SIGMA].sub.i] [a.sub.i][X.sub.i] + C and the variable C is distributed independently of the variables [X.sub.1],..., [X.sub.n], the covariances C([X.sub.i], Y), for i = 1, 2,..., n, can be determined from the variances and covariances of [X.sub.1],..., [X.sub.n]. Then u([d.sub.i]) = [square root of ([V([X.sub.i] - Y)])], where the variance V([X.sub.i] - Y) is equal to V([X.sub.i]) + V(Y) - 2XC([X.sub.i], Y).

6. References

[1] Mutual recognition of national measurement standards and of calibration and measurement certificates issued by national metrology institutes, International Committee of Weights and Measures (CIPM), 14 October 1999, (http://www1.bipm.org/utils/en/pdf/mra_2003.pdf).

[2] Guidelines for CIPM key comparisons, International Committee of Weights and Measures (CIPM), 1 March 1999, (http://www1.bipm.org/utils/en/pdf/guidelines.pdf).

[3] L. Nielsen, Evaluation of measurement intercomparisons by the method of least squares, Report DFM-99-R39, 3208 LN, Danish Institute of Fundamental Metrology, Lyngby, Denmark (2000).

[4] R. N. Kacker and A. T. Jones, On use of Baycsian statistics to make the Guide to the Expression of Uncertainty in Measurement consistent, Metrologia 40, 235-248 (2003).

[5] Guide to the Expression of Uncertainty in Measurement, 2nd Ed., Geneva, International Organization for Standardization, ISBN 92-67-10188-9 (1995).

[6] R. N. Kacker, R. U. Datla, and A. C. Parr, Combined result and associated uncertainty from interlaboratory evaluations based on the ISO Guide, Metrologia 39, 279-293 (2002).

[7] S. R. Searle, Linear Models, 1971, John Wiley & Sons Inc., New York.

[8] R. N. Kacker, Combining information from interlaboratory evaluations using a random effects model, Metrologia 41, 132-136 (2004).

[9] R. C. Paule and J. Mandel, Consensus values and weighting factors, J. Res. Natl. Bur. Stand. (U.S.) 87, 377-385 (1982).

[10] I. Lira and W. Woger, Bayesian evaluation of the standard uncertainty and coverage probability in a simple measurement model, Meas. Sci. Technol. 12, 1172-1179 (2001).

[11] BIPM key comparison data base, (http://kcdb.bipm.org).

R. N. Kacker, R. U. Datla, and A. C. Parr

National Institute of Standards and Technology, Gaithersburg, MD 20899-0001, USA

[e-mail] 

[e-mail]

[e-mail]

About the authors: Dr. R. N. Kacker is a mathematical statistician in the Mathematical and Computational Sciences Division of the NIST Information Technology Laboratory. Dr. R. U. Datla and Dr. A. C. Parr are physicists in the Optical Technology Division of the NIST Physics Laboratory. Dr. Parr is the division chief and Dr. Datla is a group leader. The National Institute of Standards and Technology is an agency of the Technology Administration, U.S. Department of Commerce.