Several years ago, Toyota had an infamous series of car crashes due to sticking accelerators. Two years ago, a C-17 stationed at Elmendorf AFB in Alaska crashed while performing an airshow practice session. These two incidents highlight an important characteristic of the way we use metrics and statistics. They illustrate two very important questions we must ask when performing validation or accountability, when measuring our progress toward a goal.
Let me illustrate. One of my goals for the year was to do a certain amount of reading on a daily basis, so I developed a checklist to track my progress. Whenever I accomplish the task, I put a check mark in the box for that day. This is accountability. It is measuring success or failure on a daily basis. It answers the first of two questions about the validation process:
1. Am I doing it?
Compare this to Toyota’s problem with sticking accelerators in late 2009. As it turns out, Toyota had a finely tuned statistical quality control system in place. The faulty parts were actually manufactured to the most precise specifications. The variation in the process was controlled. Toyota had a process for these parts, and they were “doing it”–they were checking off the box on their checklist every day. The problem was not the lack of quality control–they were manufacturing parts perfectly to the wrong specification. This suggests that “am I doing it?” is not sufficient for a robust accountability or validation process.
It is, however, a necessary condition. It is vitally important that the answer to the question “am I doing it?” is yes–consistently, constantly. It is the first step and the foundation. But it must be complemented by the second question:
2. Am I doing it well?
In July 2010, a C-17 crash in Alaska stunned the world. The pilot had performed the practice airshow many times–the problem was not lack of practice. Each step of the airshow profile was flown according to the checklist–mechanically, he performed it just as he had each of the previous times. The result was controlled flight into terrain (CFIT), a crash that could have and should been prevented.
The cause of the accident was not a failure to accomplish the appropriate steps of the procedure–it was a failure to assess the impact of each of those steps, a failure to answer the question “am I doing it well?” One more example will help us understand.
While navigating an airplane (or a ship or a car), one is constantly asking the question “am I on course?” We can answer this question with the help of the two questions above. If we were in a car on a highway, our criteria for success might be “staying in our lane.” We would then ask “am I doing it? am I staying in my lane?” While overseeing young drivers just learning, we might expect a lot of variation, weaving back and forth, getting alternately closer to the center and edge of the road. Yes, the student driver is staying in his or her lane but not performing it well.
Answering the latter question is much more difficult. It requires judgment. It requires leadership. It requires leaders who use judgment and understand the metric and the system of validation or accountability.
It’s not enough to have a process for collecting and analyzing data. It’s not enough to have a safety management system. It’s not enough to have quality control.
Is it even possible to create a process that answers the second question, that answers “how well”? Are we predisposed to mechanical, binary evaluation–yes or no accomplishment of a task?
__________________________________________________
Where Do We Go From Here
How do we find our way then, when we are exploring the unknown, blazing a trail into uncharted territory? How do we apply elementary statistical principles to transform uncertainty into decisive action? What is to prevent us from making a preposterous application of ATOMs when we deal with very complex situations, those in which our intuition fails?
These questions are not much different from those faced by Chuck Yeager before he ever broke the sound barrier or Neil Armstrong as he took that first step on the moon. Neither of these men, nor anyone around them–with hundreds or thousands of highly educated, very scientific people on these teams–knew what to expect. Or did they…?
ATOMs is a monthly column that introduces analytical tools of mathematics and statistics and illustrates their application. To read more about ATOMs, you can read Where Do We Go From Here, or view the online workbook here.
