Is your green light really red? Get the real story from your mission indicators. (Inside logistics: exploring the heart of logistics).
A fighter squadron I worked with a few years ago reported monthly mission-capable (MC) rates on a bar chart. For 4 consecutive months, its rates measured better than the command standard. Then in August, the MC rate busted (failed to meet) the standard. Answers were sought as to what happened in August to cause the problem. But when a different kind of chart, a control chart, was used to analyze the same data, it revealed the cause of the problem did not occur in August but was a culmination of several factors in the way aircraft maintenance had been accomplished and reported for many months. The inevitable failure would have been detectable and correctable months earlier had control charts been used. Instead, each time the rate met the standard, everyone assumed all was well. They gave the MC rate a green light for each good month, based on the recently popular traffic light chart. That was because they did not hear what the numbers were trying to tell them.
If you base decisions regarding mission indicators on charts that use the traffic light method or the ubiquitous bar or line chart, it is likely you are not getting all the information you need. When we look at charts that measure the results of the same process at fixed intervals, like MC rates by month, we glean huge amounts of information from the movement of data points, but most of us do not know how to extract that information. So we do the easy stuff, comparing the most recent month against a command standard, comparing this month against last month, and trying to make sense of trends and spikes. So how do we fix this? The good news is there is an easy-to-learn method for reading deeper into the chart. The bad news is it involves the word statistical. Statistical process control (SPC) is a cheap and easy way to use data already collected to improve command and control. Its use can literally allow us to predict performance months in advance and can provide decisionmaking guidelines for correcting or improving performance. This article looks at the deficiencies inherent in current methodologies, describes the use of control charts, and highlights how an Air National Guard (ANG) fighter wing used SPC to improve F-16 MC rates.
What Is the Problem?
Why did I say bar charts or traffic light charts do not give all the information needed? First, let me qualify this statement. There are situations where these charts are appropriate. You can use traffic-light charts for actual go or no go information. For example, availability of personnel (you need ten, and you have ten) or whether something is broken or not would be the correct use of traffic lights. Likewise, an appropriate use of bar charts would be to compare report cards of different areas in the same period (each bar represents a different base, organization, or weapon system). But sometimes, we try to force readiness information not truly go or no go in nature to fit our charts. The MC rate is one such animal. These rates are highly changeable; they are measured on a periodic basis (daily, weekly, or monthly) and compared to a standard. Traffic-light and bar charts are not a good fit for these kinds of measurements. These charts use a series of static snapshots to describe a continually changing number. They only allow us to compare two numbers or guess at patterns that seem to appear when the numbers move up and down. For example, we know when the number for this period is higher or lower than a command standard or some other number on the chart, like last month' s result. We are either better or worse, right? We either meet the standard or do not. Is that not all we need to know? If we are meeting the standard this month, we must be okay. Maybe, maybe not. While there is no arguing the truth of whether one number is higher or lower than another, that information only tells a portion of the tale. In my story about the fighter squadron, it had four consecutive green light months. But we found out later the process was not healthy. It was just that none of the symptoms were showing when the circles were colored green. How can the Air Force give green lights to sick processes? The answer is variation.
Every repeatable process contains some difference in the measured result from period to period--variation. Vital information is contained in the movement patterns of numbers on the charts, which we typically ignore or attempt to interpret using inaccurate guesswork. If MC rates are used on a monthly basis, you are likely to see the number fluctuate each month. Some months, there may be only a 1- or 2-percent shift, while other months, the rate jumps by 5 or 6 percent. But some variation in a repeatedly measured result is inevitable. If we do not know the scope of that variation because it was not measured, we would never know if we had a system, like the fighter squadron mentioned, that meets the standard today but is still guaranteed to bust the standard at some point during the year.
We already know, on some level, that every repeatable process has variation. Think of how suspicious you would be if the MC rate never fluctuated and measured 86.3 percent for 24 consecutive months. If we accept there will be some change each period, the question becomes, how much change does there have to be before we think there is something out of the ordinary happening? How steep does the trend have to be? How high (or low) does the spike have to be? Ask a dozen people, and you will get a dozen answers. There is no standard for what constitutes a true trend or spike. We have to go by gut feeling about whether the ups and downs are significant. Sometimes, we even guess right. The problem with trying to use guesswork to analyze the patterns or compare two numbers, as discussed earlier, is all these numbers are subject to some system variation and virtually guaranteed to be different. So their mere difference tells you precisely nothing. The information so crucial to command and control lies both in the numerical value and movement of numbers. The use of control charts gives a way to tell the difference between a significant change in the pattern and a random change that does not indicate a shift (for better or worse) in the system. SPC gives not only practical rules that can be quickly applied to tell signals (real departures from the norm) from noise (expected, inevitable system variation) but also guidelines on what to do about it and predicts future performance levels. Now that is useful information.
What is the Solution?
Unlike many suggestions for change in the Air Force, this one is cheap and fast. If you have monthly (or any other periodic) data and a spreadsheet program, you can begin after you finish reading this article. The finished project should look something like Figure 1. You will notice a few unfamiliar lines on an otherwise simple line chart. The dotted lines represent the limits of the window of inevitable variation. These are called control limits. This means the system will inevitably generate numbers between these lines (assuming the system is stable). The wider the lines, the larger the change in numbers expected from month to month. Knowing this has already put you ahead of the game. If you have a command standard somewhere between these limit lines, you already know the system is guaranteed to generate numbers on the wrong side of that standard. The other line in Figure 1, not normally found on the charts, is the center line. This is the average performance level. This line gives the context for other ways to interpret the data (again, assuming the system is stable). It will allow us to tell whether the changes in the numbers are part of the inevitable system variation (noise) or caused by specific, significant, out-of-the ordinary events (signals).
[FIGURE 1 OMITTED]
Stable is neither good nor bad. All stable means is that the system has proven it operates predictably and stably around some average level. How do we know if we have a stable process? See Figure 2. If, after we calculate the center line and control limits, the last of the three conditions in Figure 2 is present (a point is outside one of the control limits), the system is not stable. We need to get the process under control before we can proceed. If the system is stable, this still does not mean good. Remember that stable means predictable, and the stable process may simply be predictably bad. This could mean the performance level is not where you want it. But it could also mean there is too much variation in the system, and performance levels will fluctuate too much.
[FIGURE 2 OMITTED]
So how do we go about creating a control chart? Warning! A small amount of potentially mind-numbing statistical jargon follows. There are several different types of control charts, the discussion of which is beyond the scope of this article. The one we want to use is called the individual and moving range chart (x/mR). (1) As the name implies, it is actually two charts in one. But we will only be looking at the performance (individual) portion (Figure 1). We still need the moving range information for our calculations.
First, gather at least 5 consecutive data points (20 is better if you have historical data to draw from, but 5 is the minimum for a useful chart) for your calculation. Take the average and plot that line horizontally on a line chart.
Second, find the average moving range. For this, you need a list of moving ranges, which you get from the differences between the data points used in the first step. For example, if you used 82, 80, 79, 83, and 81 (five consecutive performance results from your system), the first moving range would be the difference between the first 2 points. In this case, that would be 2 (difference between 82 and 80). The next number would be 1 (difference between 80 and 79) and so on until you have all the moving ranges available from baseline data. Then, just take an average of these moving ranges. At this point, you should have an average performance of 81 and an average moving range of 2.25 (mean of 2, 1, 4, and 2). We are almost there.
Third, calculate the control limits. The formula for the upper control limit is the performance average + (2.66 times the average moving range). The figure 2.66 is a constant used for this kind of chart. In this case, that would be 81 + (2.66x2.25) = 87. We use the same numbers for the formula for the lower control limit, except that we subtract. So the lower control limit would be 81--(2.66x2.25)=75. Now, the chart looks something like Figure 3. This is a stable process. Note that the lines were extended into the future. We are going to plot all subsequent performance on this chart and use these lines to interpret the performance. Do not recalculate unless there is evidence of a shift in the system (as indicated by the first and second signal below).
[FIGURE 3 OMITTED]
Now all you have to do is apply the rules in Figure 2 to interpret the health of the system. The figure shows types of results that signal something out of the ordinary:
* Seven points in a row remain on one side of the average line.
* Five points continuously increase or decrease (trend).
* A single point falls outside the control limits.
If you see a signal like this, it means something has changed. That change might even be positive, evidence improvement efforts are paying off. These signals also can serve to get your attention when something goes wrong so you can eliminate the problem. I mentioned earlier that SPC can give us guidance as to actions to take. Here are those guidelines:
* If signals are present (significant change in system):
* Action = focus (perhaps corrective action) on when the signal occurred. Take corrective action if necessary.
* If signals are not present but the result is negative (a red condition), the problem does not lie in the month the red condition occurred but is a symptom of a system that allows either too much variation or operates too close to the standard.
In either case:
* Action = correct the system. Analyze and correct the multiple inputs and sources of variation throughout the system. The goal is to improve the average performance, tighten the window of inevitable fluctuation, or both.
* If signals are not present and there are no negative results but a standard value is between the control limits.
* Action = correct the system as above. It is inevitable that you will bust the standard at some point; we can predict future poor performance.
The hard part for most of us is what to do if none of these signals are present and we still have poor performance. For example, on the new control chart (Figure 3), there is a command standard of 80 (up is good). We met it for the first 2 months (green lights), busted it in month 3 (red light), and met it again the final 2 months. What would most of us have done when month 3 went red? We would have looked for reasons in month 3. This is the wrong approach. As stated above, the problem lives in the design of the system, the multiple inputs that factor into it daily. If we tried to focus on month 3, we would likely end up changing something to try to correct the symptom rather than the actual problem. This approach will fail to solve the problem and could make things worse. We probably would have taken no action in months 1 and 2 because we were green. But we had evidence of impending failure. The standard of 80 is between the control limits of 75 and 87. In a situation like this, we were bound to fail to meet the standard and knew this even when we were green. This knowledge allows us to take action prior to the actual manifestation of the problem.
This is why and how to put SPC to use in your unit. Now let us take a look at one example of SPC in action.
140th Fighter Wing Puts SPC to Work
Colorado ANG's 140th Fighter Wing at Buckley AFB began using SPC to analyze its fleet MC rates in October 1998. The logistics commander, Colonel George Clark, spearheaded the effort to derive as much information as possible from the system and focus efforts on improving the ailing health of the F-16 fleet. When they started, the average fully mission-capable (FMC) rate was 50.78 percent. They could not get a grip on the numbers, which fluctuated randomly from a low of 36 percent to a high of 70 percent. Colonel Clark's initial goal was to use control charts to "let the process measure itself," rather than relying on guesswork and conjecture from multiple parties. Up to that point, they had been using bar charts, with 2 years overlapped into one chart, which makes it virtually impossible to derive all the useful information from the system. Once everyone could see the large amount of variation in the system and that the problems were system-wide, maintenance and supply personnel began to search for ways to tighten up the system, paying attention to all the inputs and processes used to derive the FMC rate (Figure 4). We can see this system is stable, since none of the signals from Figure 2 are present. This means the system was guaranteed to give them results at an average of 50.78 percent (far below the standard of 62 percent), with inevitable and random fluctuations between the control limits of 17 and 85 percent. They continued to plot subsequent FMC rates into this chart. Early in FY00, their first signal appeared. A run of 7 points stayed above the mean. This was evidence the process had changed for the better. When the process stabilized, they recalculated new control limits and a new average line. By February 2002, their chart looked like Figure 5. The new mean jumped from 50.78 to 67.48, a 33-percent increase, pushing the average performance well above the standard of 62 percent. Equally (perhaps more) important was the decrease in variability, yielding a much more tightly controlled system.
[FIGURES 4-5 OMITTED]
The FMC rate remains at the same level of performance and variation as in FY03. Colonel Clark and his logistics team continue to use SPC in 2003 to analyze and improve mission capability.
Commanders and supervisors at all levels need to know most of the charts currently used do not give all the information needed and are frequently misleading. An understanding of variation and use of control charts can give immediate and penetrating insight into the systems. This insight gives the power to more accurately diagnose actual system health, change the focus of efforts from putting out fires to preventing them, and accurately predict future performance levels.
Much has been written about SPC, but I highly recommend a book by Donald J. Wheeler Understanding Variation: the Key to Managing Chaos. More indepth mechanics of SPC are spelled out in this book, which is entertaining and a relatively short read but is not a huge tome on statistical theory.
(1.) Donald J. Wheeler, Understanding Variation: The Key to Managing Chaos, SPC Press Inc, Knoxville, Tennessee, 1994.
Major Theriot is Chief, Logistics Readiness Inspections, Air Combat Command Inspector General Team, Langley AFB, Virginia.
|Printer friendly Cite/link Email Feedback|
|Author:||Theriot, Kenneth R.|
|Publication:||Air Force Journal of Logistics|
|Date:||Mar 22, 2003|
|Previous Article:||AFIT takes students back to the basics with its Applied Maintenance Management Concepts course. (Air Force Institute of Technology).|
|Next Article:||Transforming the way the Air Force plans, deploys, and sustains robust, global communications for the warfighter.|