Much ado about nothing?

I keep thinking that I'm done writing about Boeing. Is there really more that needs to be said at this point? And aren't there other Quality topics out there? But it's like Michael Corleone says: "Just when I thought I was out, they pulled me back in."

This time the trigger was a reader who sent me a recent article about the rash of airline safety incidents that have been making the news this spring. The article, by Kelsey Piper, appeared in Vox under the title "Are there really more things going wrong on airplanes?" Piper argues that, while the reporting of airline safety incidents is way up this spring, the actual numbers are consistent with last year, and the years before that. She summarizes the last 15 years of US commercial aviation as having "a safety record of about one or two passenger fatalities per light-year traveled."

Piper doesn't quite say it in so many words, and at the end of the article she does hint briefly that there might be a story behind the scenes. But as soon as she frames her statistics in terms of "fatalities per light-year traveled," the message to the casual reader is certainly that all this focus on airline safety is much ado about nothing. 

I always look forward to conversation with my readers, but in this case I think my reply must have been a little too abbreviated or dismissive. In any event my reader doubled down, suggesting that I look up several years of statistics from the National Transportation Safety Board (NTSB) so that I could determine independently whether we are seeing more problems with new Boeing aircraft than we saw in the past, even if the increase is being drowned out by other statistical noise.

As an aside, it's nice to know my readers have such confidence in me. But I'm not going to do that, because it is the wrong question and it hides the right one. The real story is about the changes Boeing has made in their safety management system, and about the failures of their configuration and documentation systems. These stories are critical, because they are the root causes of failures that may not even have happened yet. By contrast, counting up how many planes have lost tires while landing, or how many harmless engine fires have broken out, is a distraction. And it misses the point.

The critical fact that distinguishes these two topics is that it is perfectly possible to make a safe airplane with no Quality system (or Safety system) whatsoever! Therefore a story about Boeing hobbling their Quality system is in principle a different story from one about their current safety statistics.

Let me explain. Of course it's not likely that a team could make a safe aircraft without suitable systems in place. But if it just so happened that by random chance the team did all the right things to make the airplane safe … well then, it would be safe. The odds are against it, to be sure, but it's not impossible.

More realistically, when Boeing management removed this or that inspection step, doing so did not automatically mean that every single plane built under the new regimen was henceforth—let's say—3% less safe. The employees still remembered how to do their work, and nobody ever shows up to work wanting to do a bad job. Yes, human fallibility is always a factor, but there is no formula that rigidly connects the exact number of Quality steps in a procedure with the safety rating of the output product.

You remember that last week we talked about how Boeing management decided that Quality was "non-value-adding" overhead? This is part of why they thought that! They found, empirically, that they could eliminate one Quality inspection, save a few dollars, and no planes fell out of the sky. OK, good. How about eliminating two inspections? Three? Four? Where do we stop? You can see how, in the absence of visible negative feedback (like an increased accident rate), this could get out of hand quickly.

But wait. If that's true, why do we bother with Safety and Quality systems at all? If you can build a safe and reliable airplane without such a system, why were the Boeing executives wrong to eliminate all those extra costs?

Here's the thing: Yes, you can (in principle) build a safe aircraft without a formal Safety system. But you can't know that it is safe! What you buy with all the extra expenditures on Safety and Quality is certainty. And, of course, in order to get that certainty you implement a lot of inspections which then find problems … so the problems can be fixed before the plane is put into service. This improves the plane's safety even farther, which is all to the good.

So when Boeing pulled back their Quality system, what they did was to make their planes less certain, not less safe. This or that specific aircraft might be perfectly airworthy—who knows? To bring this discussion back to Piper's article, it is tempting to answer the question by looking at statistics: "Well, the failure rate in new planes is pretty low, so I'll take my chances." The problem with this answer is that it assumes that all planes of the same model are pretty much alike, except for the normal statistical fluctuations of the manufacturing process. But how can you know that? The presumption that all planes of the same model are pretty much alike is just one more kind of certainty. With fewer inspection steps there is less certainty, so you can't even know that the new planes are the same as the old ones. And therefore, like the ads for investment products always say, "Past performance is no guarantee of future results."

We can all be grateful that the accident rate for commercial aircraft is so very, very low. But to have confidence … or certainty … that it will stay low, we need aircraft manufacturers to rely on robust Quality and Safety systems. That's why the story that matters is about those systems, and the statistics are a tempting distraction.