An index to measure and monitor a system-of-systems' performance risk.

This article extends an earlier published methodology (Garvey & Cho, 2003) for measuring the technical performance risk of a system to that of a system-of-systems (SOS). The earlier work established an approach for combining an individual system's Technical Performance Measures (TPMs) into an overall measure of performance risk, defined as the Technical Risk Index (TRI). This article extends this approach so a similar index can be developed to assess a system that is composed of many interdependent or connected systems that come together as a whole to provide an SoS capability.

**********

Technical Performance Measures (TPMs) are traditionaily defined and evaluated to assess how well a system (or a system-of-systems [SOS]) is achieving its performance requirements. Typically, dozens of TPMs are defined. Although they generate useful information and data about performance, little is available in the system engineering and program management communities on how to integrate these measures into a meaningful measure of overall performance risk. This article presents how individual TPMs may be combined to measure and monitor the overall performance risk of a system. The approach consists of integrating individual TPMs in a way that produces an overall risk index. The computed index shows the degree of performance risk presently in the system. It identifies risk-driving TPMs, enables monitoring time-history trends, and reveals where management should target strategies to lessen or eliminate the performance risks of the system.

As a system evolves through its acquisition and deployment phases, management defines and derives measures that indicate how well the system is achieving performance requirements. These measures are known as Technical Performance Measures (TPMs) (Defense Acquisition University, 2002; Blanchard & Fabrycky, 1990). Measures such as Weight, Mean-Time-Between-Failure, and Detection Accuracy are among the types of TPMs often defined. The TPMs can be taken from a variety of sources. This includes data from testing, simulations, and experimentation. Depending on the source basis for these data, and the development phase, performance data may be derived from a mix of actual or forecasted values.

As mentioned previously, the system engineering and program management communities have little in the way of methodology for quantifying performance risk as a function of a system's individual TPMs. The approach presented herein consists of computing a risk index derived from these individual performance measurements. The index shows the degree of performance risk presently in the system (or SOS), supports identifying risk-driving TPMs, and reveals where management should focus on improving technical performance and, thereby, lessen risk. When the index is continuously updated, management can monitor the time-history trend of its value. This enables management to assess the effectiveness of risk reduction actions over time.

In general, TPMs are measures that, when evaluated over time, must either decrease to meet performance requirements or increase to meet performance requirements. Thus, each TPM can be assigned to one of two categories. For this paper, Category A is defined as the collection of TPMs whose values must decrease to achieve threshold performance requirements. Category B is defined as the collection of TPMs whose values must increase to achieve threshold performance requirements.

It is assumed that TPMs are defined judiciously; that is, only those TPMs truly needed to properly measure overall technical performance are defined, measured, and monitored. Given this, acceptable performance risk can be defined as the condition when all TPMs reach, or extend beyond, their individual threshold performance values. Conversely, unacceptable performance risk can be defined as the condition when one or more TPMs have not reached their individual threshold performance values.

A GENERALIZED PERFORMANCE RISK INDEX MEASURE

The following presents a generalized index designed to measure the performance risk of a system or SoS. The index can be applied in both contexts. It provides a numerical indicator that measures how well a developing system is progressing toward its threshold performance requirements. It serves as a yardstick that enables management to measure the "distance" the system is from its minimum performance thresholds and to monitor trends over time.

To develop the generalized risk index, it is necessary to first normalize the TPM "raw" values into a common and dimensionless scale. This scale transformation is done for each TPM in each category. This allows management to compare the progress of each performance measure in a common and dimensionless scale. From these normalized scales, an overall measure of the extent to which the performance of the system meets its threshold requirements can be determined. The following general formulas illustrate how to derive this measure. They are followed by a computation example to illustrate the application context.

As mentioned previously, let Category A be the set of TPMs that need to be reduced to their threshold values. Let [v.sub.ti, Aj] be the value at time ti for the jth TPM in Category A and [v.sub.thres, Aj] be the threshold value to which the jth TPM is driven. Define [v.sub.ti, Aj] to be a normalized TPM value against its threshold as follows (assuming both [V.sub.ti, Aj] and [v.sub.thres, Aj] are greater than 0):

[v.sub.ti, Aj] = max{[v.sub.ti, Aj], [v.sub.thres, Aj]} / [v.sub.thres, Aj] (i.e., threshold met if [v.sub.ti, Aj] [less than or equal to] [v.sub.thres, Aj])

= max {[v.sub.ti, Aj], [v.sub.thres, Aj], 1}

= max {[v.sub.thres, Aj] - [v.sub.thres, Aj] + [v.sub.ti, Aj])/ [v.sub.thres, Aj], 1}

= max {1 + [v.sub.ti, Aj], - [v.sub.thres, Aj] / [v.sub.thres, Aj], 1} ([greater than or equal to] 1) Eqt 1

Equation 1 is the formula for [v.sub.ti, Aj] which brings out the overage above 1. Similarly, let Category B be the set of TPMs that need to be increased to their threshold values. Let [v.sub.ti, Bk] be the value at time ti for the kth TPM in Category B and [v.sub.thres, Bk] be the threshold value to which the kth TPM is driven. Define [v.sub.ti, Bk] to be a normalized TPM value against its threshold as follows (assuming both [v.sub.ti, Bk] and [v.sub.thres, Bk] are greater than 0):

[v.sub.ti, Bk] = min{[v.sub.ti, Bk], [v.sub.thres, Bk]} / [v.sub.thres, Bk] (i.e., threshold met if [v.sub.ti, Bk] [greater than or equal to] [v.sub.thres, Bk])

= min{[v.sub.ti, Bk] / [v.sub.thres, Bk], 1}

= min{([v.sub.thres, Bk] - [v.sub.thres, Bk] + [v.sub.ti, Bk]) / [v.sub.thres, Bk], 1}

= min{1 - ([v.sub.thres, Bk] - [v.sub.ti, Bk]) / [v.sub.thres, Bk], 1} [less than or equal to] 1) Eqt 2

Equation 2 is the formula for [v.sub.ti, Bk] which brings out the underage below 1. From the normalized values, we now calculate their average difference (or distance) from 1 for each category and use it as the category's TPM Risk Index (TRI). Assuming j = 1, 2, ..., m for Category A (m-elements) and k = 1, 2, ..., n for Category B (n-elements), then

[TRI.sub.ti, A] = [([v.sub.ti, A1] - 1) + ([v.sub.ti, A2] - 1) + ... + ([v.sub.ti, Am] - 1)] / m

= ([v.sub.ti, A1] + [v.sub.ti, A2] + ... + [v.sub.ti, Am] / m] -1 Eqt 3

[TRI.sub.ti, B] = [(1 -[v.sub.ti, B1]) + 1 -[v.sub.ti, B2]) + . + (1 -[v.sub.ti, Bn])] / n

= 1 -[v.sub.ti, B1] + [v.sub.ti, B2] + ... + [v.sub.ti, Bn] / n Eqt 4

These two indices show the average overage or underage for TPMs in Category A or Category B when their individual threshold values are rescaled to 1. To combine all normalized values into an overall risk index, we first convert the TPMs in Category A into equivalent ones in Category B. This is because the normalized values for Category A can differ in orders of magnitude from those for Category B (e.g., 1000 vs. 0.5). An overall index, based on the normalized values as calculated, will be unduly influenced by large values. The result, though correct, can be difficult to interpret.

To make such a conversion, observe that for the jth TPM in Category A with value [v.sub.ti, Aj] and threshold [v.sub.thres, Aj], an equivalent TPM in Category B can be constructed with value [U.sub.ti, A1] = 1/[v.sub.ti, Aj] and threshold [U.sub.thres, Aj] = 1/[v.sub.thres, Aj], Typically, the reciprocal of a TPM is just as practical. For example, a failure rate or a processing delay that is to be reduced can be taken in its reciprocal respectively as a mean time between failure or a completion rate that is to be increased.

The probability of a certain undesirable event (e.g., misclassification or an error exceeding the tolerance) or unavailability of a certain desirable state (e.g., system working or parts in hand) is more subtle. But their reciprocals can be viewed as the expected number of events that will contain one such undesirable event or the expected length of time that will contain one unit time of such a desirable state being unavailable. Although their complements (as opposed to reciprocals) can also be used as Category B TPMs, it is not recommended as the complements are usually close to 1 and their further improvements toward 1 do not show much difference when normalized.

By definition, the normalized value for a Category A TPM converted into a Category B TPM is:

[u.sub.ti, Aj] = min{[U.sub.ti, Aj], [U.sub.thres, Aj]} / [U.sub.thres, Aj]

= min{1/[v.sub.ti, Aj], 1/[v.sub.thres, Aj]} / (1/[v.sub.thres, Aj])

= [1 / max{[v.sub.ti, Aj], [v.sub.thres, Aj]}] / (1/[v.sub.thres, Aj])

= 1 / [max{[v.sub.ti, Aj], [v.sub.thres, Aj]} / [v.sub.thres, Aj]

= 1 / [v.sub.ti, Aj] ([greater than or equal to]1) Eqt 5

We can now treat all TPMs as being in Category B and then derive an overall risk index.

Let [TRI*.sub.ti, A] = 1 - [([u.sub.ti, A1] + [u.sub.ti, A2] + ... + [u.sub.ti, Am]) / m

[TRI.sub.ti, A] = 1 - [([v.sub.ti, B1] + [vt.sub.ti, B2] + ... + [v.sub.ti, Bn]) / n] as before

then [TRI.sub.ti, All] = 1 - [([u.sub.ti, A1] + [u.sub.ti, A2] + ... + [u.sub.ti, Am] + [v.sub.ti, B1] + [v.sub.ti, B2] + ... + [v.sub.ti, Bn]) / (m + n)

= 1 - [m(1 - [TRI*.sub.ti, A]) + n(1 - [TRI.sub.ti, B])) / (m + n)]

= [m([TRI*.sub.ti, A]) + n([TRI.sub.ti, B])] / (m + n) Eqt 8

where [TRI.sub.ti, All] is the overall TPM Risk Index for the system computed across all of the system's TPMs. Finally, a non-negative weight [w.sub.Aj] could be assigned to (1 - [u.sub.ti, Aj]) for the jth TPM in Category A and [W.sub.Bk] to (1 - [v.sub.ti, Bk]) for the kth TPM in Category B (as opposed to all having an equal weight, as assumed in the discussion above). In that case, it can also be shown that

and [TRI*.sub.ti, A] = 1 - [([w.sub.A1] [u.sub.ti, A1] + [w.sub.A2] [u.sub.ti, A2] + . + [w.sub.Am] [u.sub.ti, Am]) / [W.sub.A]] Eqt 9

where [W.sub.A] = [W.sub.A1] + [W.sub.A2] + . + [W.sub.Am]

[TRI*.sub.ti, B] = 1 - [([w.sub.B1] [u.sub.ti, B1] + [w.sub.B2] [u.sub.ti, B2] + . + [w.sub.Bn] [u.sub.ti, Bn]) / [W.sub.B]] Eqt 10

where [W.sub.B] = [W.sub.B1] + [W.sub.B2] + . + [W.sub.Bn]

and [TRI*.sub.ti, A11] = [[W.sub.A][TRI*.sub.ti, A] + [W.sub.B][TRI*.sub.ti, B]] / W Eqt 11

where W = [W.sub.A] + [W.sub.B]

This, Equation 11 is the most general form of the overall TPM Risk Index.

From the above, note that [TRI*.sub.ti, A], [TRI.sub.ti, B] and [TRI.sub.ti, A11] equally or unequally weighted, are all bounded by 0 and 1. A value of 0 for the risk indices means there are no unacceptable risks in the included TPMs, each achieving (or extending beyond) its threshold value. The risk indices can be asymptotically near 1 and that implies that each TPM value in Category A is very large when compared to its threshold and/or that each TPM value in Category B is very small when compared to its threshold, i.e., they are all far away from their thresholds. When the TPMs are moving toward their thresholds, the risk indices are moving toward 0.

COMPUTATION EXAMPLE & TIME HISTORY GRAPH

Suppose Table 1 represents a set of Category A and Category B TPMs, along with their hypothetical threshold and raw values for six measurement dates. From these data, what is the overall technical performance risk index? How is it changing over time?

From the data in Table 1 and Equations 9, 10, and 11, we can derive, for each measurement date, the TPM risk indices for the Category A and Category B TPMs, as well as for the overall TPM Risk Index. The results from these derivations are summarized in Table 2.

Note that TRI is a cardinal measure. This means its value is a measure of the "strength" or "distance" that the contributing TPMs are from their individual threshold performance values. A TRI equal to 0.5 is truly twice as "bad" as one equal to 0.25.

Figure 1 presents a time history trend of the TPM risk indices for the data in Tables 1 and 2. Here, the trend is good. All three TRIs are heading toward 0. This means all TPMs defined for the system are converging toward their individual threshold performance values. In practice, management should regularly produce a graphic summary such as this to monitor the extent that each risk index changes over time.

[FIGURE 1 OMITTED]

GENERAL EQUATION SUMMARY

This paper provides an approach and formalism for developing an overall set of quantitative indices that measure a performance risk, as a function of a system's (or system-of-systems') TPMs. Below are the general equations of the three principal risk indices.

Category A: [TRI*.sub.ti, A] = 1 - [([w.sub.A1] [u.sub.ti, A1] + [w.sub.ti, A2] + ... + [w.sub.Am][u.sub.ti, Am]) / [W.sub.A]]

where [W.sub.A] = [w.sub.A1] + [w.sub.A2] + ... + [w.sub.Am]

Category B: [TRI.sub.ti, B] = 1 - [([w.sub.B1]][v.sub.ti, B1] + [w.sub.B2]][v.sub.ti, B2] + ... + [w.sub.Bn]][v.sub.ti, Bn]) / [W.sub.B]

where [W.sub.B] = [w.sub.B1] + [w.sub.B2] + ... + [w.sub.Bn]

Overall Risk Index:

[TRI.sub.ti, All] = [[W.sub.A][TRI*.sub.ti, A] + [W.sub.B][TRI.sub.ti, B] / W

where W = [W.sub.A] + [W.sub.B]

EXTENSIONS TO SYSTEM Off SYSTEMS

This section extends the general formulation of TRI to a system that is composed of many individual systems that, when connected, provide an overall SoS capability. In this article, we use the following definition of an SoS.

DEFINITION

A system of systems is a set or arrangement of interdependent systems that are related or connected to provide a given capability, as illustrated by Figure 2. The loss of any part of the system will degrade the performance or capabilities of the whole. An example of an SoS could be interdependent information systems. While individual systems within the SoS may be developed to satisfy the peculiar needs of a given user group (like a specific Service or Agency), the information they share is so important that the loss of a single system may deprive other systems of the data needed to achieve even minimal capabilities (Chairman of the Joint Chiefs, 2003).

[FIGURE 2 OMITTED]

SYSTEM-OF-SYSTEMS TREE HIERARCHY

In Figure 2, the SoS is decomposed into its individual systems. Next, these individual systems can be further decomposed into their individual subsystems. Each element in the tree is referred to as a "node." A parent node is a node that has lower level nodes below it as its children. The top-most node represents the SoS level. The bottom leaf nodes are defined as nodes that have no children below them. For instance, in Figure 2, system 2 is a leaf node. System 1 is a non-leaf node. System 1 is a "parent node" composed of M-leaf nodes as its children. They are subsystem 11 through subsystem 1M. A parent node can also have lower-level parent nodes as its children, such as the top-most node in Figure 2. Generally, an SoS tree hierarchy should be decomposed to the level at which the contributions of individual TPMs can be directly evaluated and a TRI for that leaf node, at that level of the tree, can be computed.

COMPUTING TRI FOR SYSTEM OF SYSTEMS

Computing TRI--The TRI of the SoS is computed as a logical combination of the TRIs across the leaf nodes of the tree. Specifically, a [TRI.sub.ti, All] is computed for each leaf node x, in the same way presented in equation 11. Denote the value as [TRI.sub.ti, x], where the subscript x is to represent the set of all TPMs that are applicable to the leaf node x. Next, the [TRI.sub.ti, x] at all leaf nodes are combined to derive the [TRI.sub.ti, SoS] at the SoS level of the tree. To describe this process below, we further generalize the notation [TRI.sub.ti, x] to denote the TRI value for any node x, leaf or parent, in the SoS tree hierarchy and the subscript x now represents all the TPMs that are applicable to the node x, directly (as for a leaf node) or indirectly (as for a parent node).

Combining TRI for a parent node from its children (leaf or lower-level parent nodes) should be done according to the following rule. The overall TRI for a parent node k with M children (nodes k1, ..., kM) at time ti can be written as:

[TRI.sub.ti, k] = ([w.sub.k0][TRI.sub.ti, k0] + [w.sub.k1][TRI.sub.ti, k1] + ... + [w.sub.kM][TRI.sub.ti, kM]) / ([w.sub.k0] + [w.sub.k1] + ... + [w.sub.kM]) Eqt 12

where node k0 is an added child to the parent node k to represent the set of TPMs that are applicable across multiple or all original children of parent node k. Starting at the lowest level of an SoS tree hierarchy, Equation 12 can be used to compute the TRI for all parent nodes, as appropriate to the structure of a given SoS decomposition. Thus, the overall TRI for an SoS tree hierarchy composed of N systems (i.e., with nodes 1, ..., N as children to the top-most node of the tree) is:

[TRI.sub.ti, SoS] = ([w.sub.0][TRI.sub.ti, 0] + [w.sub.1][TRI.sub.ti, 1] + ... + [w.sub.N][TRI.sub.ti, N]) / ([w.sub.0] + [w.sub.1] + ... + [w.sub.N]) Eqt 13

where system 0 is an added child to the top SoS node to represent the set of TPMs that are applicable across multiple or all systems listed as children under the top node.

Suppose the system 1 parent node (k = 1) has just M = 3 subsystems (subsystems 11, 12, and 13) as its children. Besides the TPMs that are to be measured at each of the subsystems, we assume there is also a set of TPMs that are applicable across multiple or all subsystems (e.g., subsystem-to-subsystem integration or system level integration). For notational convenience, we use subsystem 10 to denote the collection of such TPMs and use [TRI.sub.ti, 10] to denote the TRI value computed on those TPMs. Then, the overall TRI of system 1 at time ti is as follows:

[TRI.sub.ti, 1] = ([w.sub.10][TRI.sub.ti, 10] + [w.sub.11][TRI.sub.ti, 11] + [w.sub.12][TRI.sub.ti, 12]) + [w.sub.13][TRI.sub.ti, 13]) / ([w.sub.10] + [w.sub.11] + [w.sub.12] + [w.sub.13]) Eqt 14

Clearly, if the system 1 parent node's TRI is defined solely by its children's TRI values then Equation 14 can be simplified with [w.sub.10] set equal to 0.

Other Rollup Rules--Equations 12, 13, and 14 apply a weighted average rollup rule for determining the TRI values in the SoS tree hierarchy. The rule is appropriate for a parent node when its children's performance levels are considered additive in measuring the parent node's performance level. This implies, with their assigned weights, all children's risk levels directly add to the parent node's risk level. This is probably the most common rule to use in the rollup of TRI values. Other rules may also be defined and applied accordingly. For example, referring to Figure 2 with M = 3, Equation 12 could be rewritten according to the relationship that it is considered to have among the children of the parent node system 1, as follows:

(a) If subsystems 12 and 13's performance levels are considered to be competing with each other as alternative to be selected in measuring the parent node' s performance level (i.e., the lowest risk level between the two will be selected to represent their singular risk level), then the min rollup rule applies:

[TRI.sub.ti, 1] = ([w.sub.10][TRI.sub.ti, 10] + [w.sub.11][TRI.sub.ti, 11] + [w.sub.12or13]Min{[TRI.sub.ti, 12], [TRI.sub.ti, 13]}) / ([w.sub.10] + [w.sub.11] + [w.sub.12or13] Eqt 14a

where [w.sub.12or13] is the weight assigned to the selected result between subsystems 12 and 13.

(b) If subsystems 12 and 13's performance levels are considered limiting to each other in contributing to the parent node's performance level (i.e., the highest risk level between the two will be selected to represent their singular risk level), then the max rollup rule applies:

[TRI.sub.ti, 1] = ([w.sub.10][TRI.sub.ti, 10] + [w.sub.11][TRI.sub.ti, 11] + [w.sub.12or13]Max{[TRI.sub.ti, 12], [TRI.sub.13]}) / ([[w.sub.10] + [w.sub.11] + [w.sub.12or13]) Eqt 14b

where [w.sub.12or13] is the weight assigned to the selected result between subsystems 12 and 13.

(c) If subsystems 12 and 13's performance levels are considered in parallel redundancy in contributing to the parent node's performance level (i.e., the net risk level of the two will be the product of their risk levels), then the multiplication rollup rule applies:

[TRI.sub.ti, 1] = ([w.sub.10][TRI.sub.ti, l0] + [w.sub.11][TRI.sub.ti, 11] + [w.sub.12x13][TRI.sub.ti, 12] * [TRI.sub.ti, 13]) / ([w.sub.10] + [w.sub.11] + [w.sub.12x13]) Eqt 14c

where [w.sub.12x13] is the weight assigned to the product of subsystems 12 and 13's risk levels.

(d) If subsystems 12 and 13's performance levels are considered in serial dependency in measuring the parent node's performance level (i.e., their risk levels will aggravate each other to produce a combined risk level of the two), then the complementary multiplication rollup rule applies:

[TRI.sub.ti, 1] = ([w.sub.10][TRI.sub.ti, l0] + [w.sub.11][TRI.sub.ti, 11] + [w.sub.12x13][1 - (1-[TRI.sub.ti, 12]) * (1 - [TRI.sub.ti, 13])]) / ([w.sub.10] + [w.sub.11] + [w.sub.12x13]) Eqt 14d

where [w.sub.12x13] is the weight assigned to the complementary product of subsystems 12 and 13's risk levels.

Additional rollup rules could be defined to meet other specific measuring needs. Conceptually, all these rollup rules can be expressed for any general node in an SoS tree hierarchy. But since a different combination of rules could apply to different nodes, such a general expression becomes difficult.

SUMMARY

To conclude, key features of the approach presented in the article are summarized as follows:

* Provides Integrated Measures of Technical Performance: This approach provides management with a way to transform the typically dozen or more TPMs into common measurement scales. From this, all TPMs may then be integrated and combined in a way that provides management with meaningful and comparative measures of the overall performance risk of the system (or SOS), at any measurement time.

* Measures Technical Performance as a Function of the Physical Parameters of the TPMs: This approach operates on actual or predicted values from engineering measurements, tests, experiments, or prototypes. As such, the physical parameters that characterize the TPMs provide the basis for deriving the TPM risk indices.

* Measures the Degree of Risk and Monitors Change over Time: The computed TPM risk indices show the degree of performance risk that presently exists in the system (or SOS), supports the identification and ranking of risk-driving TPMs, and can reveal where management should focus on improving technical performance and, thereby, lessen risk. If the indices are continuously updated, then management can monitor the time-history trends of their values to assess the effectiveness of risk reduction actions being targeted or achieved over time.

Lastly, the TRI calculations in this article assume the TPMs' threshold values are the goals that technical performance is driven to reach. The resulting index value measures the distance between the achieved technical performance levels and those considered minimally acceptable. Conceptually, one can use the TPMs' objective values, the desirable but more demanding technical performance levels, to replace the threshold values in the TRI calculation. The result will be an index to measure the distance between the achieved levels and those considered desirable.

ACKNOWLEDGEMENT

The authors would like to acknowledge and thank Mr. Stephen Myers, Principal Professional Staff, at the Johns Hopkins University Applied Physics Laboratory (JHU/APL) for his contributions to and review of this article.

REFERENCES

Blanchard, B. S., & Fabrycky, W J. (1990). Systems engineering and analysis (2nd ed.). Englewood Cliffs, New Jersey: Prentice-Hall.

Chairman of the Joint Chiefs of Staff. (2003, June). Operation of the joint capabilities integration and development system (CJCSM 3170.01). Washington, D.C.: GPO.

Defense Acquisition University (DAU). (2002). Risk management guide for DoD acquisition (5th ed.). Fort Belvoir, VA: DAU Press.

Garvey, P. R., & Cho, C-C. (2003, Spring). An index to measure a system's performance risk. The Acquisition Review Quarterly, (10)2, 188-199.

Paul R. Garvey is Chief Scientist, and a Director, for the Center for Acquisition and Systems Analysis at The MITRE Corporation. He is internationally recognized and widely published in the application of decision analytic methods to problems in systems engineering risk management. Garvey completed his undergraduate and graduate degrees in mathematics and applied mathematics at Boston College and Northeastern University, where he was a member of the adjunct faculty in the Department of Mathematics.

E-mail address: [e-mail]

Chien-Ching Cho is Principal Staff in the Economic and Decision Analysis Center, a department within the Center for Acquisition and Systems Analysis, at The MITRE Corporation. He has significant experience in applying operations research methods and statistical analysis techniques to a wide variety of systems engineering and analysis problems. Cho received his Ph.D. from the University of Wisconsin (Madison) in Operations Research with a minor in Statistics.

E-mail address: [e-mail]

TABLE 1. HYPOTHETICAL CATEGORY A & CATEGORY B TPM DATA SET

Category A TPM                                        Raw Value
                                           Vthres,A     V(ti,A)

Measurement Date t1
Average Processing Delay (msecs)              1.000       3.000
Mean Time to Repair (mins)                   10.000      50.000
Payload Weight (lbs)                        950.000    2112.000
Time for Engagement Coordination (sec)        0.010       0.100

                           TRI*(t1,A)         0.729      Eqt 9

Measurement Date t2
Average Processing Delay (msecs)              1.000       2.860
Mean Time to Repair (mins)                   10.000      43.000
Payload Weight (lbs)                        950.000    1764.000
Time for Engagement Coordination (sec)        0.010       0.040

                           TRI*(t2,A)         0.657      Eqt 9

Measurement Date t3
Average Processing Delay (msecs)              1.000       1.180
Mean Time to Repair (mins)                   10.000      43.000
Payload Weight (lbs)                        950.000    1328.000
Time for Engagement Coordination (sec)        0.010       0.032

                           TRI*(t3,A)         0.473      Eqt 9

Measurement Date t4
Average Processing Delay (msecs)              1.000       1.090
Mean Time to Repair (mins)                   10.000      27.000
Payload Weight (lbs)                        950.000    1189.000
Time for Engagement Coordination (sec)        0.010       0.020

                           TRI*(t4,A)         0.353      Eqt 9

Measurement Date t5
Average Processing Delay (msecs)              1.000       1.030
Mean Time to Repair (mins)                   10.000      12.000
Payload Weight (lbs)                        950.000    1008.000
Time for Engagement Coordination (sec)        0.010       0.010

                           TRI*(t5,A)         0.063      Eqt 9

Measurement Date t6
Average Processing Delay (msecs)              1.000       0.980
Mean Time to Repair (mins)                   10.000       9.000
Payload Weight (lbs)                        950.000     948.000
Time for Engagement Coordination (sec)        0.010       0.010

                           TrI*(t6,A)             0      Eqt 9

Category B TPM                                        Raw Value
                                           Vthres,B    V(ti,B)

Measurement Date t1
Interceptors Available (no. of units)       150.000      67.000
Mean Time Between Failure (hours)           500.000     100.000
Single Shot Success Probability (%)           0.950       0.870
Damage Assessment Accuracy (%)                0.995       0.600
Software Coding (no. of modules coded)      763.000     578.000

                           TRI(t2,B)          0.586     Eqt 10

Measurement Date t2
Interceptors Available (no. of units)       150.000     128.000
Mean Time Between Failure (hours)           500.000     189.000
Single Shot Success Probability (%)           0.950       0.890
Damage Assessment Accuracy (%)                0.995       0.878
Software Coding (no. of modules coded)      763.000     643.000

                           TRI(t2,B)          0.399     Eqt 10

Measurement Date t3
Interceptors Available (no. of units)       150.000     134.000
Mean Time Between Failure (hours)           500.000     223.000
Single Shot Success Probability (%)           0.950       0.910
Damage Assessment Accuracy (%)                0.995       0.940
Software Coding (no. of modules coded)      763.000     687.000

                           TRI(t3,B)          0.342     Eqt 10

Measurement Date t4
Interceptors Available (no. of units)       150.000     139.000
Mean Time Between Failure (hours)           500.000     348.000
Single Shot Success Probability (%)           0.950       0.934
Damage Assessment Accuracy (%)                0.995       0.945
Software Coding (no. of modules coded)      763.000     698.000

                           TRI(t4,B)          0.194     Eqt 10

Measurement Date t5
Interceptors Available (no. of units)       150.000     142.000
Mean Time Between Failure (hours)           500.000     379.000
Single Shot Success Probability (%)           0.950       0.940
Damage Assessment Accuracy (%)                0.995       0.999
Software Coding (no. of modules coded)      763.000     723.000

                           TRI(t5,B)          0.147     Eqt 10

Measurement Date t6
Interceptors Available (no. of units)       150.000     159.000
Mean Time Between Failure (hours)           500.000     521.000
Single Shot Success Probability (%)           0.950       0.990
Damage Assessment Accuracy (%)                0.995       1.000
Software Coding (no. of modules coded)      763.000     763.000

                           TRI(t6,B)              0     Eqt 10

Category A TPM                               Eqt 1       Eqt 5
                                           v(ti,A)     u(ti,A)      wt

Measurement Date t1
Average Processing Delay (msecs)              3.000       0.333   1.000
Mean Time to Repair (mins)                    5.000       0.200   1.000
Payload Weight (lbs)                          2.223       0.450   1.000
Time for Engagement Coordination (sec)       10.000       0.100   1.000

                           TRI*(t1,A)

Measurement Date t2
Average Processing Delay (msecs)              2.860       0.350   1.000
Mean Time to Repair (mins)                    4.300       0.233   1.000
Payload Weight (lbs)                          1.857       0.539   1.000
Time for Engagement Coordination (sec)        4.000       0.250   1.000

                           TRI*(t2,A)

Measurement Date t3
Average Processing Delay (msecs)              1.180       0.847   1.000
Mean Time to Repair (mins)                    4.300       0.233   1.000
Payload Weight (lbs)                          1.398       0.715   1.000
Time for Engagement Coordination (sec)        3.200       0.313   1.000

                           TRI*(t3,A)

Measurement Date t4
Average Processing Delay (msecs)              1.090       0.917   1.000
Mean Time to Repair (mins)                    2.700       0.370   1.000
Payload Weight (lbs)                          1.252       0.799   1.000
Time for Engagement Coordination (sec)        2.000       0.500   1.000

                           TRI*(t4,A)

Measurement Date t5
Average Processing Delay (msecs)              1.030       0.971   1.000
Mean Time to Repair (mins)                    1.200       0.833   1.000
Payload Weight (lbs)                          1.061       0.942   1.000
Time for Engagement Coordination (sec)        1.000       1.000   1.000

                           TRI*(t5,A)

Measurement Date t6
Average Processing Delay (msecs)              1.000       1.000   1.000
Mean Time to Repair (mins)                    1.000       1.000   1.000
Payload Weight (lbs)                          1.000       1.000   1.000
Time for Engagement Coordination (sec)        1.000       1.000   1.000

                           TrI*(t6,A)

Category B TPM                               Eqt 2
                                           v(ti,B)          wt

Measurement Date t1
Interceptors Available (no. of units)         0.447       1.000
Mean Time Between Failure (hours)             0.200       5.000
Single Shot Success Probability (%)           0.916       1.000
Damage Assessment Accuracy (%)                0.603       1.000
Software Coding (no. of modules coded)        0.758       1.000

                           TRI(t2,B)

Measurement Date t2
Interceptors Available (no. of units)         0.853       1.000
Mean Time Between Failure (hours)             0.378       5.000
Single Shot Success Probability (%)           0.937       1.000
Damage Assessment Accuracy (%)                0.882       1.000
Software Coding (no. of modules coded)        0.843       1.000

                           TRI(t2,B)

Measurement Date t3
Interceptors Available (no. of units)         0.893       1.000
Mean Time Between Failure (hours)             0.446       5.000
Single Shot Success Probability (%)           0.958       1.000
Damage Assessment Accuracy (%)                0.945       1.000
Software Coding (no. of modules coded)        0.900       1.000

                           TRI(t3,B)

Measurement Date t4
Interceptors Available (no. of units)         0.927       1.000
Mean Time Between Failure (hours)             0.696       5.000
Single Shot Success Probability (%)           0.983       1.000
Damage Assessment Accuracy (%)                0.950       1.000
Software Coding (no. of modules coded)        0.915       1.000

                           TRI(t4,B)

Measurement Date t5
Interceptors Available (no. of units)         0.947       1.000
Mean Time Between Failure (hours)             0.758       5.000
Single Shot Success Probability (%)           0.989       1.000
Damage Assessment Accuracy (%)                1.000       1.000
Software Coding (no. of modules coded)        0.948       1.000

                           TRI(t5,B)

Measurement Date t6
Interceptors Available (no. of units)         1.000       1.000
Mean Time Between Failure (hours)             1.000       5.000
Single Shot Success Probability (%)           1.000       1.000
Damage Assessment Accuracy (%)                1.000       1.000
Software Coding (no. of modules coded)        1.000       1.000

                           TRI(t6,B)

TABLE 2. TPM RISK INDEX SUMMARIES

Measurement    TPM Risk Index     TPM Risk Index    Overall TPM Risk
   Date        for Category A     for Category B         Index
                    TPMs               TPMs
              [TRI*.sub.ti, A]   [TRI*.sub.ti, B]   [TRI*.sub.ti, AI]
                   Eqt 9              Eqt 10             Eqt 11

    t1             0.729              0.586               0.63
    t2             0.657              0.399              0.478
    t3             0.473              0.342              0.382
    t4             0.353              0.194              0.243
    t5             0.063              0.147              0.121
    t6               0                  0                  0