Training systems and artificial intelligence

A few days ago, I got a message from a man named Momchil Penev, asking what I think about training systems and their vulnerabilities? It was an interesting question, and I had to tell him that I haven't thought about the subject in depth. Then he had a few more questions, and we started a conversation that I'll explain in some detail below. But in the meantime I started to wonder ... why haven't I spent more time thinking about training systems?

How important is training, really?

We all know that training is important to Quality: if people don't know how to do a job, they can't be expected to do it well. It even shows up in ISO 9001—clause 7.2 (c) says explicitly that if your people don't already know how to do the work, you have to make sure they learn. But as an auditor, I've never found that checking training records contributes a lot to the overall outcome of the audit, other than by checking the box for clause 7.2. If I'm going to find problems in the organization, it seems like they are always somewhere else.

But why? I think there are three reasons:

• Training records usually consist of a list of classes (for office staff) or a list of operations (for manufacturing and logistics personnel), each one signed off by a supervisor or instructor. Often there is no indication what the signature actually means, so all I can check is whether the paperwork is complete. (Remember this point, because we'll come back to it.)
• Most of the organizations I've audited are fairly small, or else they are small units inside large companies. It's hard for incompetence to hide in small companies, because people largely know what their neighbors are working on. I've been more likely to find that an employee's training record shows no signature for a skill that he knows very well, than to find a true gap in some mission-critical competency.
• On the whole, when people fail to follow established procedures, I haven't found that ignorance is the root cause. There's always something else going on. At least in my experience, the employees who take shortcuts that end in disaster know what they are supposed to do, but they choose not to do it.   

Sometimes training is not the problem!

This last point is important. Remember Alaska Airlines flight 1282, that lost a door plug shortly after takeoff on January 5, 2024? The best explanation I've ever read of the root causes behind this accident comes from an anonymous account posted to the Internet by a current Boeing employee. You can find it here. 

• This account makes the point that the door plug blew out because four bolts to hold it in place were missing.
• The bolts were missing because they had been removed in order to fix a different problem with the door plug.
• Then after that problem was fixed the bolts were never replaced because the workers forgot about them. 
• But if they had followed the established repair procedures they could not have forgotten. The only way they could possibly forget was by taking shortcuts.
• Why did they take a shortcut? Not because they were ignorant of the correct procedure, but because the procedure was too cumbersome and time-consuming. And they were already under pressure from management to hurry up.

Or consider a story that I tell in this post from a few years ago. In a factory that I represented, an external auditor wrote up one of our manufacturing personnel because he was running a plating bath and the liquid was at the wrong temperature. The procedure told him that if anything was wrong he should shut down the line and call the responsible manufacturing engineer (who was on vacation at the time). But in fact this employee had decades of experience with plating. He knew that he could compensate for the wrong temperature by making minor changes to the chemical mix in the bath. The end product was indistinguishable from one made according to the instructions, and this way we could ship it to the customer on time rather than delaying the order two weeks until the engineer returned. We got an audit finding, but from the customer's perspective his solution was perfect.* In this case, the reason the employee failed to follow the written procedure was that he was confronting an unforeseen situation, and had the deep knowledge to improvise a solution.

This is why I say that in my experience, ignorance of a procedure hasn't generally been the root cause of failure to comply. By the same token, when I am coaching a team through an 8D report to resolve some failure, I don't accept "Retrain employees" as a corrective action. There has got to be something more substantive behind the problem.  

But training systems can still be more robust

All that having been said, training systems are still necessary. And like anything else, they can always be improved. This brings me back to my conversation with Momchil Penev, who wants to use artificial intelligence technology to improve them.

Penev is the founder of skillia.AI, and his idea is an interesting one. It's a bit of a right-angle turn in our discussion, but let me take a minute to explain what he is doing.

As I say, it's an interesting idea. In general I'm something of an AI-skeptic; and as noted above, training by itself won't solve all Quality problems. But it still has to be done, and the records still have to be up to date! If Penev can find a problem and then solve it, that's how Progress happens. I wish him well. 

POSTSCRIPT: During our conversation, Penev told me that he is eager to get feedback from people in the Quality business about the performance and limitations of his tool. This means you! So while I have no financial stake in Penev's success or failure, I told him I would post his Contact page. You can reach him at https://skillia.ai/contact. Tell him what you think of his ideas, or ask how you can test the tool for free. 

__________

* For further discussion of this finding, see the referenced post.     

 

      

Boeing again, and the power of systems thinking

It seems like Boeing just can’t stay out of the news these days. 

• 
  On Saturday, June 22, Korean Air 189—a Boeing plane—developed a leak half an hour after takeoff. Within ten minutes the cabin depressurized, triggering the emergency oxygen masks. The flight was bound from Incheon Airport in Seoul, South Korea, for Taichung International Airport in Taiwan. But when the cabin lost pressure the pilots started an emergency descent to bring the plane down to a safe altitude, dropping nearly 26,900 feet in fifteen minutes. While nobody was seriously injured, nineteen passengers were hospitalized for ear pain and nosebleeds.
• 
  A couple weeks before that, on June 5, two NASA test pilots blasted off for the International Space Station aboard Boeing’s Starliner capsule. They were supposed to spend a week. But the capsule had last-minute thruster trouble after it was already launched, which almost prevented it from docking with the ISS. (It also developed several small, unrelated helium leaks.) At this writing, the two pilots have been at the ISS for three weeks (not one) … and counting. Their return has been scheduled for three different dates—and canceled each time, as NASA and Boeing continue to trouble-shoot the equipment. But a Boeing spokesman offered reassurance that that “the astronauts are not stranded.”
• Last month, Justice Department prosecutors recommended that Boeing face criminal charges for violating a 2021 settlement related to crashes of the 737 MAX in 2018 and 2019. 
• And just last week, the National Transportation Safety Board sanctioned Boeing for making inappropriate comments to the media regarding the ongoing investigation of Alaska Airlines 1282. 

What’s an airline to do?

Asking the wrong question

I wrote, “What’s an airline to do?” but it’s not clear if that’s even the right question. Let’s think about it for a minute.

Of course, it’s difficult to turn around a whole organization, especially one as large as Boeing. Growing or healing a Quality culture is a big task, and it requires coordinating a lot of elements—even something as trivial as a box of doughnuts can help make a difference. If this were the first time that Boeing had had problems, we might expect the turnaround to take a while.

But it’s not the first time. Boeing’s Quality and safety cultures have gotten worldwide attention at any rate since the crashes of Lion Air 610 in October, 2018, and of Ethiopian Airlines 302 in March, 2019. And internal discussions had already begun years before those crashes. Boeing whistleblower John Barnett testified that he began raising Quality complaints internally back in 2011.

So if the deficiencies in Boeing’s Quality and safety cultures have been well-known for over a decade, why has the company seemingly made so little progress?

A couple of months ago in this blog, we looked at how the Mayo Clinic built their Quality culture, one that continues to be healthy and effective. One of the critical elements that made the Quality initiative succeed was solid, long-term management support. Barnett and others (including other whistleblowers) have said—to put it gently—that Boeing’s management did not offer the same kind of support when Quality concerns were raised. Their focus was profitability, pure and simple.  

The funny part is, you would think that a focus on profitability ought to be enough, if only the management took a wider perspective. When customers suffer from a bad product, they sue; their damages come out of the company’s profits. When courts convict companies of criminal activities, they assess fines; those fines are paid out of the company’s profits. Besides, as we discussed with respect to the Mayo Clinic, a reputation for high Quality can itself be a key to increased profitability. Surely if only Boeing’s management were sufficiently enlightened, they would do the right things out of self-interest. Wouldn’t they? Isn’t that how this is supposed to work?

Empirically, it doesn’t seem to.     

So maybe the real question is, What’s a judge to do? Boeing has already been assessed $2.5 billion in criminal penalties; they have been subject to litigation more or less continually since the Lion Air crash; and there is no clear evidence that anything meaningful has changed. If the DOJ prosecutors have their way and Boeing is assessed another $24 billion, do we really think that will finally make the difference, when other fines and damages have failed? Or will Boeing just write it off as a cost of doing business? 

One of the recurring lessons in the Quality profession is the need to study the objective consequences of the measures we take—in other words, to assess the effectiveness of our corrective actions. If we keep doing the same thing to fix a problem and it keeps not working, the odds are pretty good that doubling down one more time isn’t going to help. Assessing fines on corporate malefactors—not just Boeing, but others as well—doesn’t seem to make much difference. Maybe it’s time to try something else.

The systems perspective

We’re often encouraged to take a “systems perspective” to solve Quality problems, though it’s not always obvious what that means in practice. But I recently ran across an example in an unexpected place. In my spare time, I’ve been reading The Dictator’s Handbook: Why Bad Behavior Is Almost Always Good Politics, by Bruce Bueno de Mesquita and Alastair Smith.* One topic they discuss is government corruption (e.g., bribes); and they point out that enacting strict laws against corruption is, by itself, not nearly enough to curtail it.** (To be very clear, I am NOT accusing Boeing executives or the FAA of corruption! But this example is meant to show how a systems perspective can illuminate questions about enforcement in general.)

Why are those laws not enough? Well, they have to be enforced by someone in the government administration. So in case the administration is also corrupt (and in some countries it is), they will not enforce laws against their friends or allies. In such countries, strict anti-corruption laws are no more than a tool that the administration can use to weed out the disloyal by engaging in very selective prosecution for something that in reality everyone does.

What Bueno de Mesquita and Smith explain is that the best way to root out corruption is to increase the number of voters that the government depends on to keep power. In an autocracy or an oligarchy, where the administration depends on a tiny number of backers, they can afford to reward those people through corruption in exchange for support. But if more people are allowed to vote, then the number of supporters needed to keep a government in power grows much larger. It then becomes pragmatically impossible to pay so many for their votes. And at that point, the anti-corruption laws can finally do their job and the prevalence of corruption starts to recede because the system no longer supports it.

This is a systems argument! And while (once again) I’m certainly not accusing anyone at Boeing of corruption, I think the example highlights how we need to approach the problem of bringing our largest and most powerful corporations to heel. Working inside the existing system by assessing fines doesn’t appear to offer much traction. The alternative is to apply some other kind of pressure to any of the very few points where there is a chance for some leverage.  

An alternative to fines

The proposal that follows is only an example, but it relies on the same kinds of systems thinking that Bueno de Mesquita and Smith use. I owe this example to the blogger and public intellectual John Michael Greer, and I will quote his description extensively in what follows.***

Greer starts by pointing out that in certain parts of Europe during the early Middle Ages, most legal punishments took the form of fines (weregild)—the same way that our courts assess fines on corporations today. Someone who injured or murdered someone else could make restitution to the victim’s family by paying the appropriate amount of money, with no additional penalty. Unsurprisingly, this arrangement meant that the very rich could commit murder uninhibited, as long as they could afford the fine. Over time, the laws changed and weregild was replaced by imprisonment or execution. Greer’s proposal is that—since a corporation is presumed to be a “legal person”—we could make similar changes in corporate law today. He describes it like this:

Imagine that a corporation—we’ll call it the Shyster Company—has just been caught deliberately selling worthless securities to widows and orphans. The district attorney files charges of felony fraud and theft in state court. The trial date arrives, the lawyers bicker, the jury finds the defendant guilty as charged, and the judge sentences the corporation to ten years in the slammer. In practice, what happens is that the judge appoints a trustee, who takes control of Shyster and all its assets. For the next ten years, Shyster is a wholly owned subsidiary of the state government. Its stock pays no dividends and has no voting rights, its directors have to find something else to do with their time, and if the trustee decides that the CEO and other overpaid office fauna get to find new jobs, they get to find new jobs—assuming that they’re not doing time themselves, as they very well could be. All profits earned by Shyster during its period of imprisonment go to the state government, subject to set-asides that pay restitution to the victims of the crime.

Meanwhile another conglomerate—we’ll call this one Dirty Rotten Scoundrel Inc.—has been caught knowingly selling food products tainted with deadly bacteria, and a dozen people have died. This time the district attorney files charges of aggravated first degree murder. The trial date arrives, the media has a field day, the lawyers bicker, the jury returns a verdict of guilty as charged, the judge sentences DRSI to death and the appeals court upholds the sentence. In practice, what happens is that on the scheduled date of execution, DRSI ceases to exist. Its stock becomes worthless, its assets are sold off in an auction in which no former shareholder is allowed to bid, its name and trademarks can never again be used by anybody under penalty of law, and its creditors get whatever scraps are left once the victims’ families receive their settlements.

OK, I guess it’s an entertaining story, but where is the systems thinking?

Fair question: Greer explains that next. 

It’s crucial that the stockholders in both cases, and the creditors in the latter case, suffer for the behavior of the corporation…. Thus under this system, if word gets out that a corporation is pushing the limits of legality, the stockholders have a very strong incentive to sell, driving down the value of the stock. Equally, if lenders become aware that a corporation is engaging in really egregious behavior, they have a very strong incentive to charge higher interest rates or even to stop loaning money to the corporation. Neither has any such incentive under the current system, which is one reason why corporations act as though their quarterly profit statements are the only things that matter. To their stockholders and creditors, this is essentially the case; this proposal would change that.

In other words, this proposal changes the incentives that a corporation faces by applying leverage to the corporation’s stock price and its access to liquidity. This kind of leverage might make a difference.****

Would this help?

Of course I can’t know for sure whether laws like this would encourage Boeing to handle its Quality culture more conscientiously. In the short run, the chain of causality is probably too long to have an immediate effect. 

On the other hand, if one of these penalties ever were to be invoked, it would likely mean a wholesale change of the company’s top management. And if the management understood that risk today, they might be more willing to take the Quality culture seriously. As Samuel Johnson once remarked, “When a man knows he is to be hanged in a fortnight, it concentrates his mind wonderfully.”

Meanwhile if you have other proposals that take adequate account of the systems incentives which drive Boeing’s management, I’d love to hear them. Please leave a comment.        

_____

* (New York: PublicAffairs, Hachette Book Group, 2011, 2022). 

** See especially Chapter 6.

*** Quotations are all from John Michael Greer, The Wealth of Nature: Economics as if Survival Mattered, 2nd ed. (Founders House Publishing LLC, 2021), pp. 213-215. An earlier draft of the same proposal can be found in an archived blogpost here, dating from January, 2010.

**** Greer says more about the legal and political aspects of his proposal in the sources cited above.

 

Much ado about nothing?

I keep thinking that I'm done writing about Boeing. Is there really more that needs to be said at this point? And aren't there other Quality topics out there? But it's like Michael Corleone says: "Just when I thought I was out, they pulled me back in."

This time the trigger was a reader who sent me a recent article about the rash of airline safety incidents that have been making the news this spring. The article, by Kelsey Piper, appeared in Vox under the title "Are there really more things going wrong on airplanes?" Piper argues that, while the reporting of airline safety incidents is way up this spring, the actual numbers are consistent with last year, and the years before that. She summarizes the last 15 years of US commercial aviation as having "a safety record of about one or two passenger fatalities per light-year traveled."

Piper doesn't quite say it in so many words, and at the end of the article she does hint briefly that there might be a story behind the scenes. But as soon as she frames her statistics in terms of "fatalities per light-year traveled," the message to the casual reader is certainly that all this focus on airline safety is much ado about nothing. 

I always look forward to conversation with my readers, but in this case I think my reply must have been a little too abbreviated or dismissive. In any event my reader doubled down, suggesting that I look up several years of statistics from the National Transportation Safety Board (NTSB) so that I could determine independently whether we are seeing more problems with new Boeing aircraft than we saw in the past, even if the increase is being drowned out by other statistical noise.

As an aside, it's nice to know my readers have such confidence in me. But I'm not going to do that, because it is the wrong question and it hides the right one. The real story is about the changes Boeing has made in their safety management system, and about the failures of their configuration and documentation systems. These stories are critical, because they are the root causes of failures that may not even have happened yet. By contrast, counting up how many planes have lost tires while landing, or how many harmless engine fires have broken out, is a distraction. And it misses the point.

The critical fact that distinguishes these two topics is that it is perfectly possible to make a safe airplane with no Quality system (or Safety system) whatsoever! Therefore a story about Boeing hobbling their Quality system is in principle a different story from one about their current safety statistics.

Let me explain. Of course it's not likely that a team could make a safe aircraft without suitable systems in place. But if it just so happened that by random chance the team did all the right things to make the airplane safe … well then, it would be safe. The odds are against it, to be sure, but it's not impossible.

More realistically, when Boeing management removed this or that inspection step, doing so did not automatically mean that every single plane built under the new regimen was henceforth—let's say—3% less safe. The employees still remembered how to do their work, and nobody ever shows up to work wanting to do a bad job. Yes, human fallibility is always a factor, but there is no formula that rigidly connects the exact number of Quality steps in a procedure with the safety rating of the output product.

You remember that last week we talked about how Boeing management decided that Quality was "non-value-adding" overhead? This is part of why they thought that! They found, empirically, that they could eliminate one Quality inspection, save a few dollars, and no planes fell out of the sky. OK, good. How about eliminating two inspections? Three? Four? Where do we stop? You can see how, in the absence of visible negative feedback (like an increased accident rate), this could get out of hand quickly.

But wait. If that's true, why do we bother with Safety and Quality systems at all? If you can build a safe and reliable airplane without such a system, why were the Boeing executives wrong to eliminate all those extra costs?

Here's the thing: Yes, you can (in principle) build a safe aircraft without a formal Safety system. But you can't know that it is safe! What you buy with all the extra expenditures on Safety and Quality is certainty. And, of course, in order to get that certainty you implement a lot of inspections which then find problems … so the problems can be fixed before the plane is put into service. This improves the plane's safety even farther, which is all to the good.

So when Boeing pulled back their Quality system, what they did was to make their planes less certain, not less safe. This or that specific aircraft might be perfectly airworthy—who knows? To bring this discussion back to Piper's article, it is tempting to answer the question by looking at statistics: "Well, the failure rate in new planes is pretty low, so I'll take my chances." The problem with this answer is that it assumes that all planes of the same model are pretty much alike, except for the normal statistical fluctuations of the manufacturing process. But how can you know that? The presumption that all planes of the same model are pretty much alike is just one more kind of certainty. With fewer inspection steps there is less certainty, so you can't even know that the new planes are the same as the old ones. And therefore, like the ads for investment products always say, "Past performance is no guarantee of future results."

We can all be grateful that the accident rate for commercial aircraft is so very, very low. But to have confidence … or certainty … that it will stay low, we need aircraft manufacturers to rely on robust Quality and Safety systems. That's why the story that matters is about those systems, and the statistics are a tempting distraction. 

               

Is Quality a "value-added" activity?

Does Quality add value?

When I was researching my posts about Boeing this spring, I ran across several sources who said Boeing had been cutting back Quality activity for years, on the grounds that Quality work was merely "overhead" and not "value-added."* So even though I've touched on this topic once or twice before,** maybe it's useful to review the question again.

It should be no surprise that I think Boeing was wrong to say that Quality doesn't add value, but in a sense they were on to something. There are two fundamental ways in which Quality differs from components like wheels or doors:

1. Quality is not tangible or material. Quality isn't a What, but a How.
2. Quality depends on the user. Quality means getting what you want, and different people want different things. So it's easy to think that Quality isn't objective.

The first point means that there's no container in inventory labelled "Quality." You can't reach in and pull out half a kilogram of Quality to install in one of the engines. Whether you build an airplane with or without Quality, mostly you use the same parts and the same tools. The difference is in how you use them. Do you really need to pay Quality personnel for that? Isn't Quality free? 

Well no, it's not. We've discussed this before. People make mistakes. The way to prevent those mistakes is to put systems in place. The systems will save you money in the long run (because you won't be paying for warranty repair or liability lawsuits), but they still cost you in the salaries of the people who run them. It's just cheaper to pay your Quality personnel a predictable sum now, than it is to pay angry customers and victorious plaintiffs incalculably more at some unexpected time in the future.

The second point is easier to explain with an example. Suppose one of Boeing's airplanes is still around far in the future, and is discovered by a band of scavengers crossing a post-apocalyptic hellscape. They won't care about the precision machining that went into the parts, nor about the multiple fail-safe systems that keep the plane in the air. All that will matter to them is that the airplane can be torn apart for scrap metal. So that precision machining will add no extra level of Quality from the perspective of the scavengers. They won't find Quality anywhere as they rip the plane apart. Doesn't that mean that Quality is subjective? 

Of course not. The answer is that Boeing's actual customers aren't scavengers in a post-apocalyptic hellscape. Boeing's customers are airlines, all of whom want the same thing—namely, to satisfy their own customers. For their part, the airline customers want to get where they are going safe and sound, and more or less on time. In cases like this, where everyone wants the same thing, Quality is absolutely objective. Anything that makes an airplane easier to fly and safer in the air is part of Quality. Anything that makes it more difficult and more dangerous is Wrong, and has to be avoided! 

All the same, I can see how these two points could mislead the Boeing management. When Harry Stonecipher took over Boeing, he avowedly set out to shift the company's focus from engineering to business. But that means that management had to focus on what was tangible and objectively quantifiable: we've all heard the admonition, "You can't manage what you can't measure." And so, ineluctably, the business focus on strict measurables with a visible impact on the bottom line meant that management had no alternative but to pay less attention to Quality.

Boeing is going through a lot of very public troubles right now, so maybe we shouldn't focus on them too relentlessly. Let's look elsewhere. Can we find other areas where Quality—an intangible that relates to customer experience and customer preferences—adds a value that people are willing to pay for?

Yes. Everywhere.

Sometimes it's not quantifiable, but it's still real. There is an old saying in sales, "Don't Sell The Steak, Sell the Sizzle." The point is that—mostly—nobody cares nearly as much about the composition of a product as they care about their experience of it. You can't eat the sizzle, but that's what people pay for. More generally, people pay for experiences that make them happy; the only time that they pay for specific physical components are when they believe that those specific components are necessary to achieve their happiness.*** But their experience, their happiness, is not tangible; and in principle it can change from one customer to another. In other words, customers pay for Quality, and not for things.

It also happens that sometimes people pay for Quality in a way that is very quantifiable! I once had an employee who used to work for a company that made medical implants. And he told me that on his very first day, his boss sat him down to say:

We sell plastic devices that cost us $5. We sell them for $125. The extra $120 pays for Quality! So don't mess it up.

Yes, Quality adds value. Sometimes you can measure it in dollars. Even when you can't, it is absolutely real.  

__________

* See for example "The last days of the Boeing whistleblower" from Fortune, March 16, 2024 (especially the next-to-last paragraph), or the On Point Podcast from NPR titled "Whistleblowers, an executive shakeup, and the future of Boeing" (especially from about 11:20 to 11:40). 

** See for example the series "Do audits really add value?" in 2021 (parts 1, 2, and 3), and the series on "Parasitic certifications?" in 2022 (parts 1, 2, 3, and 4). Or just search the blog for the phrase "value add."  

*** And sometimes this is obvious. If I drive over a nail that punctures my tire, the only thing that is going to make me happy is a new tire with no holes in it.        

                

A podcast on Boeing!

This week I had another chance to sit down with Kyle Chambers of Texas Quality Assurance, this time to talk about Boeing. Like me, Kyle has had a series of episodes dealing with Boeing's troubles in the last year, and he always brings a refreshing and very practical energy to all Quality topics. I start off talking about the FAA report that I discussed here last week; but in the course of the discussion we also cover why Boeing should hire TQA to revamp their training programs, and how to make safety classes matter to people who don't want to be there.

Please join us!

You can find the podcast version here: #QualityMatters episode 175.

Or there's a version on YouTube that also includes video, which you can find here:

Leave me a comment to let me know your thoughts! 

          

What did the FAA find?

It's all very well to sit snugly behind a keyboard and criticize Boeing's safety culture (as I have done in a number of posts this spring, for example here and here). But how much of this is just talk, and how much is based on hard data? Has anyone done the hard work to sit down with Boeing and study their culture in detail? Maybe an exercise like that could tell us something useful.

In fact, a special Expert Panel completed just such a study last month. These experts were appointed by the Federal Aviation Administration (FAA) and began to meet a year ago, at the beginning of March, 2023. They wrapped up their investigation in February 2024 after spending a full year on it. The team reviewed 7 surveys and more than 100 policies and procedures, comprising over 4000 pages. They interviewed more than 250 people across 6 locations. In the end they issued 27 findings and 53 recommendations. You can find the full report online here, and the New York Times has an article about it here. 

The report is devastating. 

More exactly, it's written in the bland bureaucratic language that is mandatory for reports like this. There are no bold headlines screaming "J'Accuse!" But I have been auditing since 1996, and I cannot remember ever reading—much less writing!—a report about a fully functioning organization* that painted in such broad strokes a picture of a management system floating so loose from its moorings.

Background and summary

The Expert Panel was formed in accordance with the provisions of the 2020 Aircraft Certification, Safety, and Accountability Act (ACSAA), Pub. L. 116-260, Div. V, § 103, which requires review of organizations that hold an Organization Designation Authorization (ODA) from the FAA. An ODA is the arrangement by which the FAA delegates certain Boeing employees to inspect Boeing's own work, on behalf of the FAA, so that the FAA does not have to assign their own people. The idea seems to be at least in part that there are a lot of inspections which are mandated by airworthiness regulations, and if all of them had to be carried out by FAA personnel then the FAA's staff and budget would have to be significantly increased. 

If you think it sounds crazy to ask a company to inspect its own work when there are serious safety risks at stake, … well, you can look up the text of the 2005 rule (70 FR 59932) establishing ODAs in the Federal Register; the "Background" section of that document explains how the idea grew incrementally over time as a way to cut down the long delays caused by airworthiness inspections. But the FAA still retains oversight of the whole process—naturally, right?—which is why the 2020 law referenced above requires all ODA holders explicitly "to adopt safety management systems (SMS) consistent with international standards and practices," and also directs the FAA "to review The Boeing Company’s ODA, safety culture, and capability to perform FAA-delegated functions." (Reference.)    

When the Expert Panel issued their report, they summarized their findings under four general headings:

• Boeing's safety culture, where they found a "disconnect" between what they heard from senior management and what they heard from the rank and file;
• Boeing's SMS, which was structured to reflect all the applicable standards perfectly but which appeared to have been glued on top of the organization with library paste;
• Boeing's ODA management structure, which the Panel conceded had been recently reorganized to make it harder for the company to retaliate against an employee finding violations while acting in the name of the FAA (but "harder" still doesn't mean "impossible");
• Other topics.

In the remainder of this post I will highlight and discuss some of the specific findings and other observations. (Sometimes I will indent my comments in blue, when I think it helps to distinguish my remarks from those of the Panel.)

Boeing's safety culture

The basic observation here is that Boeing has defined and rolled out a formal, written safety culture, but most employees don't really understand it. (Sec. 3.3) Concretely:

• Many employees, when interviewed, didn't know about "Boeing's enterprise-wide safety culture efforts, nor its purpose and procedures." (Sec. 4.1, #1)
• Even employees who knew the terminology of the safety culture couldn't use it in a sentence. (Sec. 4.1, #2)
• Some Boeing sites have good, "confidential, non-punitive reporting systems" in place—but not all of them. (Sec. 4.1, #3)
• Managers can investigate reports in their own reporting chain, which means they risk not being impartial. (Sec. 4.1, #4)
• Employees don't know which reporting system to use for safety problems. Employees don't really trust any of the reporting systems, and prefer to report safety problems to their bosses. Employees especially don't trust the anonymity of the "preferred system." Employees do not (reliably) get informed of the outcome when they do report through these systems. (Sec. 4.1, #5)
• My comment: When you first hear it, "reporting safety problems to your manager" doesn't sound like a bad idea. (Although naturally people who report problems should still hear back how they were dispositioned, or they'll start to think that reporting is a waste of time.) The reason that "reporting safety problems to your manager" can become a problem is that ….
• When employees report safety problems to their managers, it's often done verbally. So there is no way to know if any particular problem ever made it into the reporting system. And if a problem didn't get into the system, there's no way to track whether it was ever analyzed or fixed. (Sec. 4.1, #6) 

Boeing's SMS

Grigory Potemkin,
architect of the system?

The Panel makes a number of high-level observations about Boeing's SMS, before diving into the details. Among these observations are the following:

• All the SMS documents are new, and there is no traceability to the changes from what came before. (Sec. 3.4, para. 4)
• Most of the SMS documents cover general conduct and do not translate to the concrete working level. (Sec. 3.4, para. 5)
• Many employees don't really understand the elements of the SMS, or else they think it is a management fad that won't stick around. (Sec. 3.4, para. 10) 
• Many employees point out that Boeing already had a detailed safety system before the SMS was implemented—so why do we need this new one now? (In fact the old system is still referenced in many procedure documents.) (Sec. 3.4, para. 11)
• Boeing requires employees to take safety training classes, but doesn't test whether they learned anything. (Sec. 3.4, para. 13)

In other words, the Panel says that Boeing's shiny new SMS—which complies perfectly with all the relevant requirements and standards—is a Potemkin system. 

After those general observations, the specific findings might be an anticlimax, but here are a few of them:

• The complexity of the SMS documentation, and "the constant state of document changes," make it hard for employees to understand it. (Sec. 4.2, #10)
• Boeing uses an SMS dashboard to track safety goals, but employees (and some managers) don't understand what it is or how to use it. (Sec. 4.2, #12)
• There are different tracking systems for the SMS and for the legacy safety systems, and many people are confused by them. (Sec. 4.2, #12, cont'd.)
• Since Boeing has kept all the legacy safety systems in place, employees across the company don't trust that the new SMS will last long. (Sec. 4.2, #13)
• Boeing has procedures on how to evaluate safety-relevant decisions, but there's nothing to explain how to tell which business decisions count as safety-relevant. (Sec. 4.2, #14)

In other words, employees don't understand the SMS and they have no motivation to learn it.

Boeing's ODA management structure

The Panel's general observation about the ODA program is that it is getting harder to fill, because participating inspectors (called Unit Members, or UMs) are retiring faster than new ones are being brought onboard. (Sec. 3.5, paras. 4-6; sec. 4.3, #18)

But the detailed findings have to do mostly with the risk that UMs could fear retaliation for speaking out about problems:

• Boeing has not eliminated the possibility of retaliation when UMs raise safety concerns, and some UMs have experienced what looks like retaliation. Other UMs are not willing to help or step in, and their help is rejected as interference. (Sec. 4.3, #16)
• Boeing says they took steps to make sure the ODA program is working correctly, but cannot provide proof. (Sec. 4.3, #17)
• In which case, did they really do anything?
• Supposedly Boeing has changed the ODA organizational structure, but nobody knows how. Employees still report to their old managers. Procedures are still written around the old structure. (Sec. 4.3, #19) 

There are some other smaller findings as well.

Other topics

Of the findings classified as "Other matters," the two that concern me the most state (in different ways) that input from pilots is treated inconsistently: if it comes into Executive A, it is treated seriously and addressed; but if it comes into Executive B, it might get lost or forgotten. (Sec. 4.4, #23 and #24) Less alarming are some technical points about how to handle the relationship between Boeing and the FAA in the future.

But a couple of the other general observations are worth noting.

Right at the beginning, Boeing welcomed the Panel and made sure to say that they looked forward to open collaboration. But the Panel says that in fact, Boeing answered questions rather as if the evaluation were an audit or a deposition, and asked for no input of any kind. (Sec. 2.6, paras. 12-13; sec. 3.2, para. 1)

So I have to ask, Did Boeing expect to learn anything from this evaluation? Or was the intent simply to get through it as fast as possible, with as few findings as possible? Because clearly, if you approach the whole exercise in a defensive frame of mind, you leave open fewer chances to learn and improve from the experience. 

Also interesting: the Board of Directors emphasized that they use safety-related performance metrics "when determining both Annual Incentive Pay and Long-Term Incentives." These metrics include, for example, "the requirement for executives to complete Boeing's Safety Management System training." This statement was intended to demonstrate Boeing's commitment to safety. (Sec. 3.7, paras. 5, 9, and 12)

The problem is, I think it demonstrates the reverse. Safety metrics in the bonus program? No! On the contrary, safety should be more important than any bonus program! Ironically, when you pay people for something, you cheapen it. At that point people start weighing one part of the bonus against another: Let's see, if I'm willing to give up a few dollars on safety, we can sell a lot more planes and by the end of the year the difference will more than make up for what I lost. Dollarizing the safety program is irresponsible if not worse. Safety should be non-negotiable, and paying people for it makes it negotiable. (I discuss this point in more detail in this post here.)

On the other hand, I understand why the Board of Directors would take this approach. To the man with a hammer, every problem looks like a nail. And it does seem like, in the last couple of decades, money is the hammer that Boeing's management has learned how to use. 

It's a long report. But I think it explains why Boeing has gotten into its present straits. From my point of view, the fundamental problems are all around system implementation. Boeing tried to create a new system, but went for the quick-n-easy approach rather than making sure the new system was fully implemented and integrated at all levels in the organization. As a result, people don't know what to do! Even people who want to do the right thing—and I firmly believe that this includes nearly everyone, nearly all the time—don't know how to do the right thing so that errors get caught, followed up, and fixed … and so that they themselves don't get in trouble for finding those errors in the first place.

Too much system can be as much a problem as not enough system. There's a balance and it always has to be pragmatic. I may have said this once or twice before now. 

__________

* I have participated audits that were meant as gap analyses, for organizations that wanted ISO 9001 certification and knew they weren't ready yet; and the results of those were often far worse than this one. But it was no surprise because the organizations knew in advance they had a lot of work to do.    

                

The news just keeps coming!

I thought I was done writing about Boeing's current Quality problems, but the news just keeps coming and coming. Some of the stories simply confirm what we've already said about Boeing's current Quality culture; other stories talk about legal issues, and have less to do with Quality strictly understood. But one way or another, there continue to be a lot of them.

Here's a quick sampling of recent stories that I've found around the Internet:

• About sudden leadership changes at Boeing:
• Head of Boeing 737 MAX programme out amid safety concerns at planemaker | Aviation News | Al Jazeera
• About the Boeing 737 MAX a week ago that had a gear collapse after the plane landed in Houston:
• United Boeing 737 MAX Suffers Gear Collapse After Landing - One Mile at a Time
• About criminal investigations into the Alaska Airlines flight 1282 failure:
• Report: DOJ opens criminal investigation into Alaska Airlines blowout | AP News
• US opens criminal probe into Boeing 737 MAX mid-flight blowout | transport News | Al Jazeera
• About Boeing's treatment of their suppliers through a program that demanded more production for less money:
• Boeing strikes conciliatory tone with suppliers amid 737 MAX crisis | Reuters
• Boeing Rolling Back Controversial “Partnering For Success” Deal? - Mentour Pilot
• About recent interviews with Ed Pierson, formerly Boeing 737 Program Sr. Manager, who explains why he refuses to fly on a 737 MAX and what risks he still sees in Boeing manufacturing:
• ‘I’m Not Trying to Cause a Scene. I Just Want to Get Off This Plane.’ - POLITICO
• Former Boeing 737 Program Sr. Manager Says Boeing Company Is WORSE NOW Than Before Two MAX Crashes! (youtube.com)
• About the sudden death last Saturday of Boeing whistleblower John Barnett, with what the Charleston County coroner described as "a 'self-inflicted' wound." 
• Boeing whistleblower found dead in US (bbc.com)
• (If I can judge from a totally unscientific review of comments on Twitter replying to this news, the number of people who believe Barnett's wound to have been truly self-inflicted seems to be pretty small just now. Naturally I make no accusations.)

It's an exciting time.

Ziad Ojakli, Boeing EVP

But the story I want to write about is a different one. In some ways it is smaller and quieter than the ones I just listed, but it sheds a helpful light on one of the least glamorous—but most critical!—of all the Quality disciplines. Yes, I'm talking about records control, and about how Boeing's records control system seems to have failed them at the worst possible moment.

The basic story is told by the Seattle Times here, and Associated Press chimes in here for corroboration. Briefly, it all started with the investigation into Alaska Airlines flight 1282, when a door plug blew out while the plane was in the air. The investigation revealed that four bolts were missing which were supposed to hold the door plug in place.

Why were the four bolts missing?

They had been removed to facilitate earlier rework.

Why was there rework?

There was damage to five rivets which had to be repaired. The procedure to repair those rivets required that the door plug be removed temporarily. Then after the repair the door plug was replaced.

Why weren't the four bolts replaced when the door plug was replaced?

… good question. Here the trail runs cold. The logical thing would be to ask the person who did the repair, but we don't know who that was.

Wait, … what?? How can we not know who did the repair? Surely that information was captured as part of the repair documentation!

You would think so. But up till now Boeing has been unable to provide that documentation. And last Friday, Ziad Ojakli, Boeing executive vice president and the company’s chief government lobbyist, sent a letter to Sen. Maria Cantwell of the Senate Commerce Committee, saying, "We have looked extensively and have not found any such documentation." He added as a "working hypothesis: that the documents required by our processes were not created when the door plug was opened."

Let me repeat that, just to be clear:

• Boeing's procedures require complete documentation of any rework, whenever rework is done. (So far, so good.)
• But now they can't find the documentation for rework that was done two months ago.
• The company's executive management is willing to tell a Senate Committee that he thinks maybe the documentation was never generated.  

This is terrifying.

To be more exact, there are several possible explanations for this turn of events, and every single one of them is terrifying!

One possibility is that the documentation really wasn't generated for this particular rework. 

But in that case, what else are they doing that hasn't been documented? How could you ever know? (Hint: you couldn't.) And if you don't know what work has been done on an airplane, why would you ever be willing to fly on one again?

Another possibility is that the documentation was generated, but Boeing can't find it.

This raises the same fears. If you can't find your documentation, it might as well not exist. At that point you are totally unable to use the documentation: for example, to monitor trends, or to connect the dots between one failure and another. You can't do anything proactive, and you can't even do much that's reactive. All you can do is wait for the next plane to fall out of the sky.

And of course a third possibility is that Boeing is brazenly lying to a Senate Committee.

In some ways, I almost hope this last one is the answer. I would rather that a company like Boeing be competent, even while doing something villainous, than that they succumb to floundering ineptitude. At the very least, a competent villain is more likely to build planes that keep flying.

But if you make the conscious decision to lie to the Senate, it's because you are hiding something really bad. Nobody does that on a whim. And so, once again, I start to worry about "What else don't we know?"

Yes, of course there are other possibilities, but mostly I think they add filigree details to the ones I have already sketched out. Maybe the documentation was created, but then the guy who did the work snuck into the records system and destroyed it afterwards so he wouldn't get in trouble when flight 1282 lost its door plug in such a dramatic way. Or maybe his friend did it on his behalf. And naturally it's easy to understand why this guy would be afraid of being in the spotlight nationwide. What's not easy to understand—what is, in fact, flatly inexcusable—is why any company as big as Boeing would tolerate a document control system that could be subverted so easily by a single bad actor.

You keep documentation for a reason. And even when the documentation embarrasses you, it's better to provide it (and own up in public to your mistakes) than to hide it (and leave everyone wondering whether things are even worse than they really are).

When I first started writing about Boeing's troubles (back in January) I tried to put those troubles in the best possible light by pointing out how few failures there have been (as a fraction of the total number of flights in a year) and by explaining that the whole point of a Quality Management System is to help you handle failures gracefully.

But document and records control is the single most basic element of any QMS. If Boeing never generated (or cannot find) rework documentation for a recent job, then their QMS fundamentally isn't working.

There is no way to tell this particular story so that it sounds good.

Photo from the National Transportation Safety Board

          

The myth of the silver bullet

For the last few weeks we've been talking about corporate culture: in particular, about whether you can build a company's culture deliberately, and about how far that culture is implicated when things go well or badly. So it was through a delightful synchronicity that I recently ran across two very different sources which spoke to this topic in rather different ways.

The Patagonia case study

Building a culture ...

The first was a talk given by Carlos Conejo, LSSMBB, under the auspices of ASQ, about "The Patagonia Ethos." Conejo reviewed the outdoor clothing company Patagonia, and explained how they built a corporate culture deliberately and systematically. Back in the old days, when Yvon Chouinard (the founder) first started to make climbing equipment, he told customers they shouldn't expect quick responses during climbing or skiing seasons. Then, as the company grew, they introduced:

• Flexible work arrangements
• Casual dress code
• Flat organization
• No private offices
• Health food in the offices
• On-site daycare
• Transparent communications to employees
• Classes for employees on how to get involved in local, grassroots environmental causes 

100% of the electricity used by the company is from renewable resources. 

98% of the raw materials used by the company are recycled. 

If you have old gear from Patagonia, you can send it in and they will repair it. 

These principles make Patagonia's gear more expensive than that from their competitors, but customers gladly pay the higher prices because they support the company's mission.

Then in 2022, Patagonia transferred all its paying (but nonvoting) stock to the Holdfast Collective, "a nonprofit dedicated to fighting the environmental crisis and defending nature." The voting (but non-paying) stock went to the Patagonia Purpose Trust, "created to protect the company’s values." Chouinard described these transfers by saying, "Earth is now our only shareholder." (Interestingly, Robert Bosch GmbH has a very similar ownership structure.)

All of these steps have contributed to a clear and embedded corporate culture.

... but not a silver bullet

But it's not all roses. Conejo explained that one of the consequences of the company's pervasive informality was that for many years they were very weak when it came to formal planning, budgeting, and performance management. Then when it finally became clear that these activities were needed, they created a home-grown solution that lurched too far in the other direction. For a while, the business planning process took three whole months to plan each year. Partly this is because—in the name of transparency—it engaged all employees at all levels clear across the organization. But many of these employees had no previous experience in (or even exposure to) business planning or the rudiments of project management. So the value of their input was compromised, or else they had to take the time out to learn the subjects they were contributing to. 

Ultimately, Patagonia grew past these problems. They scaled back the planning process while continuing to emphasize openness and the development of their employees. But two overall messages were inescapable. 

First, culture is important but it is not a silver bullet. You need systems too. 

Second, every culture has its own failure mode. There is no "perfect culture"; each one has some strengths and some weaknesses. Which ones predominate is partly a matter of which circumstances the company faces.     

Boeing, again

All of which brings us back to Boeing.

In recent posts* I've suggested that Harry Stonecipher (Boeing President 1997-2001 and 2003-2005) deserves a measure of criticism for deliberately dragging the Boeing culture away from a focus on solid engineering and toward a focus on the economic bottom line. But the second source that I ran across a few days ago was a blog post that provided important insight into that transition. (See "The Myth Of Old Boeing," by Bill Sweetman.) 

What Sweetman makes clear is that Boeing, back in the days before Stonecipher took over, may well have had a solid culture; and the engineers were surely very smart. But their configuration-control system dated from World War Two! By any normal standards, Boeing should have been totally incapable of building airplanes for multiple customers** in the modern day. The only thing that saved them—for a while—is that they had low-ranking employees on the production floor who understood the archaic configuration system backwards and forwards, and who worked around it with heroic effort in order to get the planes built. But these were individual human beings. One by one they got old and retired. And we all know that any system which relies on heroes to get the job done will fail sooner or later.

This was the challenge that Stonecipher faced when he took over the company. Yes, he insisted that Boeing start thinking about the economics of profit and loss. And yes, in the end, it's possible that he went too far. But part of his motivation at the time was to drag Boeing—kicking and screaming—away from a configuration-control system that made factory production pointlessly expensive and mind-numbingly inefficient.

In other words: if it hadn't been Stonecipher, it would have been someone else. The only other alternative would have been for Boeing to collapse under the weight of its own inefficiency.

To repeat the two points above:

1. Culture is important but it is not a silver bullet. You need systems too.
2. Every culture has its own failure mode, and there is no "perfect culture."

For those of us in the Quality business, none of this should be controversial. In a sense, culture is about making sure that all your people are approaching their work in the right way. But Deming taught us years ago that "A bad system will beat a good person every time." That's why you need both. 

__________

* See specifically here and here.    

** A configuration-control system manages how changes or alternatives are introduced into a design. If you sell a single basic product to several customers, each of whom insists on their own unique package of options, you need a sophisticated configuration-control system to keep track of all the variations so that (for example) United gets airplanes tailored for United and not for American. By the early 1990's, Boeing's system for handling these variations was woefully out of date.