Managing critical issues

We've talked before about problem-solving (see for example this post here), but what do you do when everything comes unglued at once—when a problem hits the fan, and suddenly you are fielding calls from reporters and attorneys when you haven't even gotten all the facts yet? Yes, you need a robust problem-solving protocol, but at the same time you need so much more than that! You need to manage the news cycle, because a careless or unguarded comment can turn public opinion implacably against you or ruin your chances in court, even though you haven't finished your investigation yet so you really don't know what happened!  

A few weeks ago, I attended a very thorough presentation on exactly this topic. The speakers were Shubhada Sahasrabudhe and Shalabh Tandon of QuRIuS Consulting LLC, the talk was sponsored by the ASQ Phoenix Section (704), and you can find a link to the YouTube video at the bottom of this post. The overall message is based on their recent book, Don't Panic, Pivot: Managing Critical Issues to Prevent Crises, and it is well worth your time and attention.

The authors start by explaining that a critical issue is not the same as a crisis, though it may cause a crisis if it is not addressed in a timely way. Major disasters can be caused by a succession of seemingly-trivial errors that were ignored, rather than set right; and the authors look at specific examples in some detail, including the Deepwater Horizon explosion, and the collapse of the Francis Scott Key Bridge. In each case, there were small signs in advance that something wasn't quite right. Based on this insight, they define a critical issue as follows: 

• An issue is any unresolved event that disrupts or hampers the normal operation, and that fails to meet the published or agreed-upon specifications.
• An issue is critical if it has time-sensitive impact (financial, safety, functional, etc.).
• And if not addressed urgently, a critical issue can become a crisis.  

Why does it matter? Well of course nobody wants to cause a catastrophic oil spill, or to destroy a major bridge. But the authors are careful to point out that the consequences of such a crisis affect multiple dimensions. Even if you momentarily set aside humanitarian concerns to take a cold-blooded look at the organization's own interests, a disaster hurts the organization in at least three areas: it damages operational efficiency, it can be ruinously expensive, and it soils the brand. (Regarding this last point, I have friends who check their travel plans to make sure they are not flying on Boeing planes, ever since the door blew out of Alaska Airlines 1282.*) So it is clearly in the organization's interest—even its narrowest self-interest—to watch for critical issues and correct them.

How do you do it? The authors sketch a method that looks, in broad outline, a lot like the protocol for handling an 8D. But there are a couple of critical additions.** 

1. Right at the beginning, in D2 (State the problem), evaluate whether this issue is critical. Make sure you understand the severity of the consequences, in case the issue blows up.
2. Then, during the impact assessment (typically also part of D2), make sure you address all relevant impacts: to operations, to finances, to stakeholders, and to your brand.
3. During D3 (Contain the problem), it is not enough to prevent the problem from spreading. You also have to contain the brand damage by making whatever public statements are necessary and appropriate, consistent with what you know at this point. Never speculate in public! Stick to the known facts, and promise you will come back with the full story when you have it. 
4. As you proceed through D4 (Find the root cause) and D5 (Define corrective action), validate your corrections before implementing them. Make sure that they address all affected stakeholders, and that they do not inadvertently cause further harm!

Perhaps the biggest change is that the authors break out Communication as a separate step between D6 (Implement corrective actions) and D7 (Assess risks and learn lessons). This addition confirms the point that harm to the brand can be as damaging as harm to customers or other stakeholders. And the authors give clear advice for all communication: make it timely, make it clear and unambiguous, and make it consistent. All of these points are important, but it is easy to overlook the last one. Or rather, it's easy, but don't overlook it! If you say one thing today and then backtrack tomorrow, people will think you are hiding something and assume the worst.

Finally, the authors break D7 (Assess risks and learn lessons) into two parts, because there is so much important work that has to be done there. On the one hand, they insist on the critical importance of a detailed After Action Review. But then you also have to implement Preventive Measures so that the problem can never, ever happen again. Both steps are important, so they list them separately to make sure neither gets shortchanged.

After all this, there's still a lot in their presentation that I haven't even touched. A single blogpost can do only so much. But check out the YouTube video below; if that inspires you, check out their book. There's a lot of detail here, and a lot of process to follow. But if you ever have to deal with a "critical issue"—even once!—you will be grateful to have a defined process that helps you keep your head amid the tumult. And of course implementing a system will be cheaper than trying to wing it, possibly by orders of magnitude. It's good to be prepared, and these authors can help prepare you.  

__________

* See also this post and the ones following.

** When I worked for Bosch, we had a similar procedure for similar circumstances. But this one is published, and to my knowledge the Bosch procedure has not been published yet. So I am happy to discuss this procedure.  

           

Egg cartons

Last week I talked about preventive actions in general, and remarked that you have to make sure you are solving the right problem. But of course there are other requirements too, in order for a preventive action to be really effective. It should be (for example) simple, inexpensive, and easy to replicate at scale. So I thought it would be fun to take a minute to look at one of the most successful preventive actions of our time, an invention so ubiquitous that it can be hard to remember the problem it was designed to solve.

Consider the egg carton.

The modern egg carton was invented in 1911 by Joseph Coyle, a newspaperman in British Columbia, after overhearing an argument between a hotel owner and a deliveryman over an order of eggs that had arrived broken.

"[Coyle] said there's got to be a better way. And … well, the rest is history," Fergus Tomlin, director of the Bulkley Valley Museum, said in an interview.

The modern egg carton is an ingenious device—so ingenious that it becomes easy to think of it as an obvious tool or a fact of nature, rather than a preventive measure created by human intelligence to solve a problem. But a moment's reflection reveals some interesting questions. Among these:

Eggs have been fragile as long as people have been eating them; so why did it take until 1911 for someone to invent a protective container to carry them in?

What was it about Coyle's design that made it universally adopted?

For the first question, I think there are two answers. Over the long term, it is only comparatively recently that we have developed economic markets large enough that the fragility of eggs carried to market could be treated as one large problem rather than as hundreds of small, individual problems. And if anyone did happen to develop an egg carton back in the Middle Ages, he never wrote it down in any document that has survived—so the idea died with him. By the dawn of the twentieth century, by contrast, we had national markets and a patent system to spread innovations.

The second answer is that other people before Coyle did come up with solutions to the problem, but their solutions simply weren't as good! There were wooden crates designed to carry eggs in quantity, but they were large and bulky. At best, they were suited to large commercial transactions, but they were too cumbersome for anyone who wanted to by a simple dozen.

This brings us the the second question: What was it about Coyle's design that made it universally adopted? And I think I anticipated the answers in my introduction above.

Coyle's design was simple: The whole carton is one part. There are no interior slats to insert and align, in order to keep the eggs in place.

Coyle's design was inexpensive: His egg cartons were made of cardboard or paper pulp, not wood.

Coyle's design was easy to replicate at scale: Ironically, this is why Coyle never became a millionaire. Other people took his simple design, made this or that minor improvement, and then patented the result. In a brief Google search I found four different US patents for variant egg cartons; it would not surprise me if there are more.

US patent 1269394, to Joseph Coyle        

US patent 1543443 to Morris Koppelman

US patent 1975129 to Francis H. Sherman

US patent 3458108 to Walter H Howarth, Gerald A Snow, and Harold A Doughty

At a personal level, this outcome may be disappointing. Years later, in an interview, Coyle's daughter said, "As is so often the case with inventors, he was no match for the sharp practices of big business and their sharper lawyers. The Coyle carton made several millionaires but dad was not one of them. We lived comfortably, but not affluently."

But from a Quality perspective, in a sense this is the best possible outcome. Innovations should be spread, picked up, adapted, improved-upon, and then those improvements should be passed along in their turn. We learn from others, and then others in their turn learn from us. The Quality goal is to solve problems as well as possible. And the problem of carrying eggs without breaking them was solved—almost (if not quite) as well as possible—by Joseph Coyle in 1911. 

           

Gatekeeping

Last week I wrote a post for this blog, and—as usual—tried to publicize it in LinkedIn. But I mostly failed. My very first post about the article (in my own timeline) went through well enough; but when I tried to cross post in special-interest groups, I invariably got the message, "Oops - we were unable to complete your request. Please try again later." And "trying again later" produced the same result. It was odd.

LinkedIn Customer Support has hitherto offered a number of theories for what happened, none of which match the facts of the case. (Perhaps I explained the problem in a confusing way at first; but as we have emailed back and forth, I have tried to make the picture clearer.) But I have been able to post other notifications since then. So it is clear to me that my account has not been blocked, and I have not violated some overall limit on the number of posts. As far as I can tell, the only plausible explanation left is that my post triggered a content filter. The filter must have determined my post to be spam, not because there were too many copies of it (There weren't.) but somehow because of what I said. In other words, I must have run afoul of some gatekeeping protocol.     

Gatekeeping isn't necessarily a bad thing, though in this case it proved inconvenient for me. In essence, gatekeeping is a form of preventive action. For example, many large companies have policies that require all their suppliers to be certified to ISO 9001. This is a gatekeeping requirement, and it is intended to weed out certain problems before they start. We have discussed at length that certification to ISO 9001 doesn't necessarily mean a company does good work. But it does mean that they have a way to handle complaints, and it does mean that they will work next week more or less the same way they worked this week. Those two reassurances—just by themselves—are huge. 

By the same token, if you determine that a certain sort of message interferes with the purposes of your online community, it's only natural to take action in advance to prevent that kind of message from ever being posted. I assume that the LinkedIn filters probably thought my post was some kind of advertisement—maybe even an advertisement for some kind of artificial intelligence tool. While advertising is allowed on LinkedIn, it has to go through a certain procedure before it is posted. It would not surprise me to learn that LinkedIn has tools—maybe even AI tools, ironically—that screen individual posts to look for advertisements that have been disguised as personal notifications. It wouldn't even surprise me if someone showed me a list of the features that this tool looks for, the features that identify a probable advertisement-in-hiding, and it turned out that by a simple accident I had written some of those features into my post—so that the gatekeeping tool rejected it.

It wouldn't surprise me. But at this point that's pure speculation. I have seen no such list yet, and LinkedIn Customer Service has not confirmed that this is the problem. I am just guessing.

But let's take this line of thought one step farther. We all know that if you are going to derive preventive actions or lessons learned from a problem investigation, the actions or the lessons have to be based in some kind of causal way on the things you learned during the investigation. In the same way, if you decide to gatekeep your social media community to keep out AI-based advertising (or whatever it turns out to be), you should make sure that this is what you are really doing in practice. Look at your algorithm closely, and review it from time to time. Also, if someone's posts get caught by the algorithm and he complains, that might mean he's a human being (rather than an AI) so you might be able to use that information to refine your algorithm even more. 

Meanwhile I'll try to write notifications that don't sound like ad copy. That's a preventive action on my part, to avoid this circumstance arising again.   

           

When doctors go bad

What do you do when doctors go bad?

Of course it happens. Doctors are human beings, and they have the same faults as other human beings. They can be rude, short-tempered, and hard to work with. They can make enemies in the workplace. They can make mistakes with patient care, and sometimes those mistakes are serious. Some doctors commit sexual harrassment. Some doctors are addicted to alcohol or other drugs. All these things happen. But what can anyone—a clinic, a hospital, or the profession at large—do about it?

Is it a Quality problem?

In a sense this sounds like a Quality problem. Certainly when a doctor goes bad, his patients (and colleagues) aren't "getting what they want." But in the same breath we have to admit that it is a very unusual kind of Quality problem!

• As noted, doctors are human beings; and we don't know nearly as much about human behavior as we do about machines. So while a machine may require no more than simple adjustment to correct its behavior, humans are more complicated.
• In fact, humans have free will, and therefore cannot be simply adjusted the way a machine can. So if a doctor is behaving badly, we have to get his consent before we can expect him to behave any better.
• Why do we need his consent?
• Partly to respect his innate human dignity, that he is free and not a slave.
• And partly because if he does not consent, he or his friends may take steps to frustrate the changes we want him to make.¹ 
• There may also be barriers of an organizational kind.² 
• Complainants might fear legal or organizational retribution.
• The offending doctor might be sufficiently valuable to the organization that management takes his side in a complaint.
• The delay built into processing a complaint might give the offending doctor time to build alliances, and might contribute to a sense in the workplace that the offending behavior is considered "normal."

What has been done so far?

"Disruptive behavior" by doctors is not new. I found one article that surveyed the relevant literature, all the way back to a news item in the New York Times from June 20, 1875.³ But it seems that systemic responses are a comparatively recent development. Back in the late 1970's Dr. Kent Neff, a psychiatrist working in Portland, Oregon, chaired a committee created by the Oregon Medical Association to address concerns about alcoholism among doctors working in hospitals. In 1994, he used the lessons learned from this committee to take charge of the Professional Assessment Program at Abbott Northwestern Hospital in Minneapolis, a program to help hospitals and medical groups with troubled doctors.⁴ Neff has continued to champion this topic in the years since then, as have others of course. In 2012, he prepared a presentation⁵ for The Foundation for Medical Excellence, summarizing the research and practical steps that had been taken at that time. And over the last two decades (at least in the United States), the medical community has been making progress on this front.

Why is it happening only now? I don't know. Studies suggest that disruptive physician behavior (DPB) is not uncommon,⁶ and it is obviously detrimental both to the workplace and to patient outcomes. And yet it seems that even during a century when medicine became ever more professionalized, this topic was overlooked or brushed to the side. According to one study, part of the problem is that there are at least 207 different terms used to describe DPB, so that systematic literature searches are prohibitively difficult. It is possible that this question belongs in a history of medicine, and not in a Quality blog. In any event, the topic of how to respond to DPB is alive today.  

Can we approach this as a Quality problem?

Actually, yes! That's more or less the approach that has been taken. After what I said above about how unusual a Quality problem this is, you might be skeptical. On the other hand, remember that I have characterized the 8D process as "a way to apply the scientific method to solving your organization's problems." And in fact, it turns out to be general enough that we can apply it here too.

So, for any medical organization (hospital, clinic, or other medical provider) to innoculate itself against DPB, the steps look something like this:

Ground rule: Don't blame individuals

• Neff's research is absolutely clear that disruptive physicians are often good doctors (at least potentially) who are caught in a maladaptive system. He insists that the process must be respectful, confidential, and based on objective data. All of these provisions are critical parts of getting the consent of the affected physicians to the new culture that you plan to implement.

D1: Name a team

• Because the solution involves a system-level approach, there will have to be a team to implement it.
• More critically, the team has to include top management, because the work needs their support. They are the only ones who can break down the organizational barriers that might stand in the way. Therefore the administration must clearly understand the cost—in dollars!—that DPB drains from the business every year in operational friction, on top of the risk of huge damages in case of patient harm. And remember—when I reviewed the Quality culture of the Mayo Clinic two years ago, one of my findings was that their "whole Quality initiative had unflinching management support over the long haul."    

D2: State the problem

• Spell out clearly what exactly constitutes unacceptable behavior, and also what constitutes acceptable behavior. It sounds extreme, but there are some physicians who have never had these things articulated for them.
• As Peisah et al. state: "How can those who lack internal loci of control or appropriate social and behavioural skills (for whatever reason), behave appropriately if nobody has articulated what behaviour is expected of them, or the consequences of such behaviour?"

D3: Contain the problem

• Every organization will do what it must to keep the doors open while working this topic.

D4: Find the root cause

• A lot of research has already been done on this question, and mostly the root causes aren't different from one organization to the next. Among the common characteristics of doctors who exhibit DPB are:
• High intelligence but poor emotional or social skills.
• A history of suffering emotional neglect or abuse. 
• A maladaptive personality.⁷ (Naturally some of these features interrelate.)
• Genuine psychiatric disorders (e.g., depression).
• "Life in the fast lane."
• Medical illness.
• That having been said, it's still early days. Peisah et al. lament that "much of the abundant literature pertaining to DPB is theoretical, with little empirical investigation."
• This is a long list of causes. There does not appear to be a single root cause that can be proven responsible because turning it on and off turns the effect on and off like a light switch. But Peisah et al. argue, "We make this complicated because it is, and because ironically, the more contributors to the problem, the more there are potential targets for remediation, again justifying our position of hope."

D5: Brainstorm possible corrective actions

• Mostly not needed, because some well-defined corrective actions already exist.

D6: Implement corrective actions

• Staff education and training. Identify champions within the staff for the new culture.
• Write policies and procedures to define a consistent reporting and review process supported by audits of adherence.
• Access to support for physical and mental health, and to interventions when necessary.

D7: Assess risks and learn lessons

• Reflect on how the system attracts and fuels bad behavior.
• Institutionalize the new culture through constant repetition. Again, remember our earlier consideration of the Mayo Clinic. In fact, the literature on DPB highlights the Mayo Clinic in particular as an organization where this kind of cultural change has been successful.⁸  

D8: Close the 8D and thank the team

• It's hard to go wrong with pizza.

Of course there are subtleties that are specific to medical care. And clearly there is a whole literature addressing this topic with no explicit connection to the Quality discipline.

But I find it reassuring that the basic techniques of Quality can be made to fit this kind of problem too, unusual though it be. It's another indication that these tools really are as broadly applicable as we have always said they are. So that's nice to know. 

__________

¹ There is an extensive discussion of this risk in Markian Hawryluk, "Doctors under the influence," The Bend Bulletin, Thursday, December 9, 2010. A successful Oregon program for alcoholic doctors, one which guaranteed their confidentiality and emphasized recovery, was replaced by a program with a more punitive focus. The intent was to prioritize justice for patients who had been harmed. The unintended consequence was that doctors stopped volunteering for the program, and also stopped referring their colleagues to it. 

² Citations to professional medical literature will generally be made with a URL. (And when possible, I will direct the URL to the specific relevant section of the article.) But the first time I reference an article, I will give the formal citation as a footnote. For this point, see specifically: Peisah C, Williams B, Hockey P, Lees P, Wright D, Rosenstein A. Pragmatic Systemic Solutions to the Wicked and Persistent Problem of the Unprofessional Disruptive Physician in the Health System. Healthcare (Basel). 2023 Sep 1;11(17):2455. doi: 10.3390/healthcare11172455. PMID: 37685490; PMCID: PMC10487014. I will refer to the authors of this article in the text as "Peisah et al." As for the article's title, Wikipedia defines a "wicked problem" as "a problem that is difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often difficult to recognize." 

³ Yes, more than a hundred and fifty years ago. That date is not a misprint. The article surveying the literature is: Petrovic MA, Scholl AT. Why We Need a Single Definition of Disruptive Behavior. Cureus. 2018 Mar 18;10(3):e2339. doi: 10.7759/cureus.2339. PMID: 29796352; PMCID: PMC5959733. 

⁴ Atul Gawande, "When Good Doctors Go Bad," New Yorker, July 31, 2000. (Published in the August 7, 2000, print edition.) 

⁵ Kent Neff, "Understanding & Managing Physicians With Disruptive Behavior," The Foundation for Medical Excellence, Organizational Professionalism, October 19, 2012.

⁶ "The data presented show that disruptive behaviors are frequently observed in the daily life of health professionals, and compromise the quality of care, the safety of the patient, and can lead to adverse effects." See: Moreno-Leal P, Leal-Costa C, Díaz-Agea JL, Jiménez-Ruiz I, Ramos-Morcillo AJ, Ruzafa-Martínez M, De Souza Oliveira AC. Disruptive Behavior at Hospitals and Factors Associated to Safer Care: A Systematic Review. Healthcare (Basel). 2021 Dec 23;10(1):19. doi: 10.3390/healthcare10010019. PMID: 35052183; PMCID: PMC8775368. 

⁷ Compare, for example, this article: Bucknall V, Burwaiss S, MacDonald D, Charles K, Clement R. Mirror mirror on the ward, who's the most narcissistic of them all? Pathologic personality traits in health care. CMAJ. 2015 Dec 8;187(18):1359-1363. doi: 10.1503/cmaj.151135. PMID: 26644545; PMCID: PMC4674404. 

⁸ Mueller PS. Incorporating professionalism into medical education: the Mayo Clinic experience. Keio J Med. 2009 Sep;58(3):133-43. doi: 10.2302/kjm.58.133. PMID: 19826207.      

      

How do the bad units know where to go?

When your customer returns a failed product, there are things you regularly do to understand the problem. Depending on the product, you might check its serial number against your production logs, to see when it was manufactured. Was it part of a batch that you already know had problems? Maybe you check its composition. Does it include components that have given you trouble before? There are multiple avenues you can explore, as part of your root-cause analysis investigation. But do you remember to ask your customer, What exactly were you doing when the product stopped working?

This is not who the story is about!
But I found the illustration on the
Internet and couldn't resist. 

Last week I heard the craziest story. Some customer had an accident that left him unable to walk. But his job required him to move around a lot, so he got a motorized wheelchair. After a month, the engine burned out, and he had it replaced. A month later, the new engine burned out, so he had that one replaced too. And the month after that ... well, you get the idea.

It seems that the wheelchair manufacturer and the customer's insurance company let this cycle go on for some time before they finally began to investigate. What exactly was this guy's job, anyway? It turns out he was a high school football coach. And his idea of how to do the job involved racing back and forth on the sidelines during games, so he could get a close look at what was going on. On the grass. In the mud. All through football season. If he got stuck in a mud patch or a gopher hole, he just jammed the wheelchair into overdrive until he got free. And somehow the engines in his wheelchair kept burning out.

Gosh, who would have guessed?

It reminded me of a story I heard years before, in a problem-solving class. A large manufacturer of high-end cookware kept getting pans returned where the ceramic finish had melted in a way that made the pans unusable.* They studied their manufacturing process, spent a lot of time and money on improvements, and it made no difference to the return rate for this particular failure.

Finally, after a lot of frustration, they hired a problem-solving expert who looked at the overall return data. This company shipped product throughout North America, but all the returns with this problem came from Toronto. Right away he told the company to stop wasting time and money reengineering their manufacturing process.

"Why?" they asked. "How can you be sure that's not the problem?"

"If the problem was caused by manufacturing, it would be evenly distributed across all your customers. But look at this map. How do all the bad pans know to go to Toronto?" 

"Then what's the cause?"

"Put me on the next flight to Toronto, and I'll tell you."

In the end, he discovered there was a popular cooking program on a local TV station which recommended that viewers clean their pans in a certain way. That cleaning method just so happened to destroy the ceramic finish on this company's pans. I don't remember how they finally solved the problem. But until they asked how their customers were using the products, their investigations were fruitless.

So remember to ask. If you see failure data that doesn't fit the patterns you expect, what other pattern does it fit instead? There has to be one. And when you have all the facts, the picture is going to make sense. As long as the failures make no sense, you don't have all the facts.

Just remember: How do all the bad pans know to go to Toronto?

It's a powerful question.

__________

* I heard this story years ago, so I may have some parts of the story slightly garbled. But the point should be clear.      

Normalization of deviance

A year and a half ago, I wrote about disasters—and about how hard it can be to see them coming. I made the point that when we analyze a disaster retrospectively, we are likely to be led astray because we know how it's all going to turn out. Because of Hindsight Bias, in particular, we think it should have been obvious to everyone that a disaster was imminent, when in reality it might not have been clear at all. It is important to remember this bias when we try to understand a disaster, so we can look at events with the eyes of those who participated in them, to derive working lessons for the future.

But not all disasters are like this. Sometimes the risk of a disaster really is obvious to the people involved at the time, according to data they already have—they see the data, they understand the risk, and then somebody decides just to go ahead and do it anyway.

The Challenger disaster

"On January 28, 1986, Space Shuttle Challenger broke apart 73 seconds into its flight, killing all seven crew members aboard. The spacecraft disintegrated 46,000 feet (14 km) above the Atlantic Ocean, off the coast of Cape Canaveral, Florida, at 16:39:13 UTC (11:39:13 a.m. EST, local time at the launch site). It was the first fatal accident involving an American spacecraft while in flight."*

What happened? It was a cold day, and the rubber O-rings which sealed a joint in the right Space Shuttle Solid Rocket Booster were stiff. So they didn't seal the joint adequately. Shortly after liftoff, gases from within the rocket booster leaked out and started to burn through the larger structure. Now, the space shuttle and all its launch framework were made out of steel—of course. But the rocket boosters burned at 5600℉; and at that temperature, steel boils!** (Not melts—boils.) So naturally the whole assembly burst apart.

But how much of this could we have predicted ahead of time? It turns out the answer is, All of it. Recently I ran across a lecture on YouTube that breaks it down.*** (This lecture is saved in four parts, of which the first two discuss the Challenger disaster from a Quality perspective. The other two parts give valuable advice for managing your career and your life, but I won't focus on them here. You can scroll to the bottom of this post to find links to the lecture itself.)

In summary, the speaker (Mike Mullane) explains the sequence of events. 

• During the initial design reviews, the O-rings were designated at "Criticality 1," meaning that a failure could entail the destruction of the vehicle and the loss of life. "Criticality 1" also meant that any damage to the O-rings constituted adequate cause to abort the mission and redesign the shuttle. 
• Sure enough, after the shuttle's second flight (years before Challenger), the team recovered the parts and detected damage on the O-rings. 
• But for this and that reason the team decided to go ahead with a third launch, and the third flight was fine.
• In future flights, sometimes the O-rings were damaged and sometimes they weren't.
• After-action reports regularly called out the risk posed by damage to the O-rings. Multiple memos, over a period of two years or more, described the O-ring issue as "urgent." 
• But each flight was successful. So the project got the idea that the O-ring problem wasn't that big a deal. Every time the issue was raised, it was granted a standing waiver.
• Until, of course, one day it was a big deal after all ....  

What is the normalization of deviance?

Mullane explains that the "normalization of deviance" stems from nothing more than the natural human tendency to take shortcuts under pressure. We know what the "right" way to do a job is, and when we are relaxed we are happy to follow it. But then time runs short, or money runs short, or something else happens—it could be anything, really—and we get under pressure. So we take a shortcut, to make the job easier.

And most of the time, after we take that shortcut ... nothing happens! The job gets finished with no problem. So the next time we are under pressure, we remember that shortcut and do it again. And then again. Pretty soon, the "shortcut" has become the normal way of working. The "deviance" (a deviation from the defined and approved method) has become "normalized."

We've seen this before. Last year, when I was writing about Boeing, I explained how their cost-cutting drive led them to gut what used to be a robust safety management system. One of the factors at work was exactly this dynamic. I wrote:

They [Boeing management] found, empirically, that they could eliminate one Quality inspection, save a few dollars, and no planes fell out of the sky. OK, good. How about eliminating two inspections? Three? Four? Where do we stop? You can see how, in the absence of visible negative feedback (like an increased accident rate), this could get out of hand quickly.

That's what happened with Challenger. Word for word.

How do you protect against it?

Fine, how do we avoid this?

The short answer is almost too simple: Don't do that! But that sounds obvious, and yet this dynamic continues to afflict people every single day. So really, what do we do?

Mullane lists four points that he thinks are critical:

1. Recognize your vulnerability: Everybody thinks, It won't happen to me. I know all about this problem, so that makes me immune. I watched a video on YouTube. I read a blog post in Pragmatic Quality. I know better than to fall into this trap. Nice try. But the other people, those ones who did fall into this trap? They were plenty smart too. All of them "knew better." But when they felt pressured, their brains reacted automatically. It can happen to you too, exactly the same way. So watch for it.
2. Execute to meet standards: This is the core of it. Plan the work, and then work the plan. Mullane explains the Air Force has a saying, "The flight manual is written in blood." In other words, every instruction in the flight manual was put there because one day somebody did something different and it turned out badly. Don't let the next one be you. If the manual says, "Abort the mission when the red light flashes," and then the red light flashes, ... abort the mission. Simple as that. 
3. Trust your instincts: Mullane makes a big point of saying that we often know more than we understand consciously, and that our instincts are there to keep us alive. So if something just feels ... off, somehow ... wrong, but you can't put your finger on quite why ... trust that feeling. Probably the thing really is wrong, and at some level you even know why. It just hasn't percolated up into your consciousness yet, but it will.
4. Archive and review near-misses and disasters: Learn from other people's experience, so you don't have to go through the same thing. Look at the disasters—or the near-misses, where things came out fine but almost didn't—that your own team has experienced. But then try to find out about other teams as well. Look for the big disasters (or near-misses) in your industry, the ones that make the news. Read everything you can, and then flow down to your team what you have learned.

And then, if we do those four things, are we home free?

I'm pretty sure nobody can promise that. But if you do these things you'll be miles ahead. And you will have reduced the odds of normalizing deviance as far as you can.

If you want more details, Mullane's lecture is a good one.

Mike Mullane's lecture, part 1/4: What is normalization of deviance?

Mike Mullane's lecture, part 2/4: How do you protect against normalization of deviance?

Mike Mullane's lecture, part 3/4: Responsibility: https://www.youtube.com/watch?v=Wuk_DoX-rz8

Mike Mullane's lecture, part 4/4: Courageous self-leadership: https://www.youtube.com/watch?v=DABsxJtNcYg

__________

* Quoted from Wikipedia, "Space Shuttle Challenger disaster." I have used this article for basic information about the disaster.  

** For specifics see this flyer from Northrop-Grumman on the Five-Segment Booster, especially the "Booster facts" on page 1. 

*** The lecture was posted to YouTube about ten years ago, but I don't know when it was given. The speaker is Mike Mullane (website, Wikipedia), an engineer, weapon systems officer, retired USAF officer, and former astronaut. He was talking to the International Association of Firefighters (IAFF) about the "Normalization of deviance."      

        

Another take on 5-Whys

A couple months ago, I saw a fun cartoon on LinkedIn. It was attached to a perfectly sound article which you can find here. But the cartoon is what I wanted to share.

And of course this is right. Sometimes analysis sputters out. You might not have the data you need to get a first-class root cause. Other factors might interfere as well. This work can be hard.

It's still valuable to do, of course.

                

Have you used the Five Hows?

The August 2024 edition of Quality Progress contains a brief article about the Five Hows, and I'd love to know if you have used them yet. Leave me a note to tell me your story!

Most of us have already heard about the Five Whys: it's the simplest tool for root-cause analysis. I describe it in multiple earlier posts: for example, I explain how the concept works in this post here. And recently I applied it (purely as a demonstration) to a current question in American public policy in this analysis here. (I discuss it in other places too, including here and here.) 

In principle the Five Hows are exactly the same thing, except that instead of asking "Why did that happen?" you ask "How will we do this?" So they look forward rather than backward. They are for planning rather than diagnosis.

So far, so good. But the documentation I have found up till now has been awfully thin on examples. The article in Quality Progress is only one page long and it's written entirely at the level of theory. I have found a few other articles, but the information they provide about how to use Five Hows are not consistent with each other.

For example, this article that Dennis Henry posted in LinkedIn (from November 2023) treats the Five Hows strictly as a supplement to Five Whys: the idea is still to find the root causes of a problem, but Henry says that by asking "How ... How ... How ...," we can uncover detailed technical root causes instead of managerial ones.

This article from Quality One (undated, no author listed) still restricts the Five Hows to problem-solving, but applies them to brainstorming corrective actions. In other words, once you have found the root cause of a problem, you next ask "How ... How ... How ..." in order to fix it.  

Back in my first article on Five Whys, I worked an example that started with "My car won't start," and the root cause turned out to be "I didn't maintain the car according to the schedule in the manual." (This was a fictional example.) So in that case, the Quality One article would propose that I next ask, "How can I make sure to maintain my car according to the schedule in the manual?" The answer would probably involve adding reminders to my calendar, or something similar.    

Then there's this article in Medium, that George Spasov wrote back in February 2017. Spasov tells us that he actually invented the Five Hows method independently. And he also gives one detailed example of how to use it outside of traditional problem-solving.

His remarks on inventing the method are as follows:

This technique was developed by… well … me.

I was trying to figure out a way to achieve a very specific goal for my work. By being a heavy user of the 5 Whys I knew that there must be a way to reverse the process. To be honest, I don’t know whether somebody else had drawn the same conclusion as me. If you know a similar concept and the person behind it, I would love to hear about it and collaborate on the matter.

His example goes like this:

• How can I improve my brand positioning? By getting more quality exposure to my target audience.
• How can I get more quality exposure to my target audience? By communicating the right messages to the right people.
• How can I communicate the right messages to the right people? By first understanding what these messages are and who is my target audience.
• How can I understand what these messages are and who is my target audience? By first making an analysis of my existing clients and their interests.
• How can I make an analysis of my existing clients and their interests? By making a drill-down of this data from the analytics.

Someone might say that "improving your brand positioning" is a kind of "problem" that Spasov is solving with this analysis, but it's certainly not traditional problem-solving! This is somehow more like corporate strategy than it is like traditional corrective action, and I'm excited to think that Quality tools can be as useful in the boardroom as they are in the laboratory or on the production floor.

I only wish I had found more examples of this kind of usage.

So tell me your stories! Have you started using Five Hows? And if so, what kinds of problems have you used them to solve? Do you find them most useful to supplement traditional root-cause analysis? Or have you been able to deploy them—as Spasov did—to build corporate strategy as well?

Drop me a note. I'd love to hear from you!

   

Embrace your failure!

Earlier this month, I published two posts that were directly connected to current political issues in the United States, first here and then (as a follow-up) here. The political issue in question was illegal immigration. My focus in the articles was: first, to review the results of the Administration's recent "root cause" policy on immigration; and second, to suggest how the Administration's root cause analysis could have been improved—possibly to support a more effective policy, but at any rate to be completer than it was. And my overriding imperative was to handle the subject in a non-political way.

After I published the first article, a good friend argued that if I really thought I could keep the discussion non-political, then I was fooling myself. More exactly, her argument was as follows:

• Since Vice President Harris was put in charge of the "root cause" policy, any evaluation of the policy will necessarily be construed by others as an evaluation of her. If I say that the policy failed, readers will construe me to mean that she personally failed.
• Any personal evaluation of the success or failure of a candidate for the Presidency—if expressed during the middle of a campaign—will necessarily be construed by others as a partisan statement, either an endorsement or an attack. If I say that a candidate failed at something, readers will construe me to have attacked the candidate.
• Therefore I shouldn't publish such an article unless I am prepared for my writing to be used as artillery in the campaign by one side or the other.

We talked about it for a while, and I told her I disagree for at least two reasons. In the first place, I simply don't have that wide a reach. Mostly my posts are read by a small pool of people interested in aspects of the Quality business. And it's highly unlikely that anything I say about the technicalities of root cause analysis will drive you to change your vote. I assume you already know how you are going to vote, and nothing I say will make the slightest difference.

More importantly, the whole point of Quality analysis is that we can learn from our mistakes by analyzing them, so that we do better next time. In this sense, it is critical to embrace your failures, and not hide them! Because when we fail at something, it's never on purpose. Always we think we are doing the right thing; we have a plan of some kind, and we are following it with the intention of reaching a goal. This is what deliberate behavior looks like. But the consequence is that the very next time we have another goal, we'll do exactly the same thing—unless something stops us. Unless we intervene with ourselves to make a change.

If our plan is right and we succeed, of course that's fine. But if we fail—that's when it gets important to pay attention. If we fail, that means the plan wasn't right, which in turn means that there was something we missed when we analyzed the situation. If we aren't careful, we'll miss the same thing again next time and fail in the same way. Conversely, if we don't want to fail the same way next time, we have to figure out what we missed last time and take care not to miss it again.

This is why we have to own our failures and understand them.

As for the Administration's "root cause" policy on immigration, when I say that it failed that's no more than a historical fact. The plan was that by taking these actions, the Administration intended to bring about those outcomes. But in fact the actual outcomes were very nearly the opposite of what the Administration had in mind. (I explain the details in my article, and give links to external documentation that substantiates all the historical data.) 

OK, fine. This happens all the time, to administrations from both parties. There's nothing partisan in recognizing the facts. The next step should be to look at the analysis and find the flaw, so that the next time around will be better.

Of course, there are a lot of moving parts in any political situation. And right now we are facing an election. It is unlikely that today's Administration will launch a major new initiative in the last months of this year. Maybe something will be done next year, but at this point we don't know who will be responsible then. So it's hard to know what to expect.

But understanding your failures is still a really valuable thing to do.    

                

Root causes on the back of an envelope

Two weeks ago, I posted an article in this space about the "Root causes of illegal immigration." I started from a Congressional Research Service report entitled "Central American Migration: Root Causes and U.S. Policy"; and I argued that the Administration's root cause analysis went astray by failing to distinguish between bounded cases (where the number of causes is strictly limited) and unbounded cases (where solving some causes just makes room for others). Then I asked readers whether they would like to see me work out my own root cause analysis of the same problem.

At this writing, seven readers have said "Yes." So I'm going to give it a shot—partly to respond to their interest. 

But the exercise drove home for me another important fact about problem-solving. Finding the root cause is only half the battle! After your analysis has generated a list of potential causes (and for many complex problems there are more than one), you then have to decide which causes you can address. And from that smaller list, you may have to filter again to find which causes you are willing to address! Admittedly if your widget-stamping machine is out of alignment, this last part is easy. Of course you're willing to re-align the widget stamping machine. But in other kinds of complex problems—especially in system problems, of which political problems are prime examples—every action taken over here will have some unintended side effect over there. So before you charge in to fix stuff, you have to weigh what it is worth fixing.

This is important, and I ran smack-dab into it while working through this example.

With that said, let me make a couple of apologies and disclaimers.

Nothing here is brilliant. My earlier post sparked a discussion in the myASQ community, and Jason Wesolowski of TX-RX Systems made all the same points I'm going to make here, with breathtaking elegance, in a couple of paragraphs. My version is longer so I can show my work; but my hat is off to Jason for seeing the issues right away, and for expressing them with admirable concision. 

I am no expert on illegal immigration. There will be points that I miss. This article is a back-of-the-envelope calculation, nothing more. All I really know is the logic of root cause analysis. So if you understand illegal immigration as a subject-matter expert, please take the systematic method that I show you here and flesh it out with better real-life information.

I said this before, but it bears repeating: This is not a post about immigration, nor about the election! In the first place, as noted, I am no expert on these topics. I have no idea what the best immigration policy should be. In the second place, there is lots to discuss here at the level of data and logic, without letting political opinions slow us down. I will keep my opinions out of the conversation.

The discussion unfolds in three steps. First, is there a problem? Second, what are the potential causes? Third, let's evaluate those causes to see which we want to address.  

Is there a problem?

First we have to check whether we have a problem at all? Don't go to a lot of effort to find the root cause for something that's not a problem. So, ... Is illegal immigration a problem?

I think the answer has to be Yes, for at least two reasons.*

From a procedural point of view, it is always bad when an organization's written procedures say one thing, and people in their daily work do something else. (And in civil society, the laws are our "procedures.") When the written procedures and the daily work disagree, it's at best confusing. Nobody knows what the real rules are. Also there is a risk that you'll get punished today for doing what got you praised yesterday. It's maddening.

The solution, when there is a gap between the written procedures and the daily work, is always—always!—to bring them back together. Maybe you enforce the procedures as written. Maybe you rewrite them to match the familiar habits. Maybe you meet in the middle. But tolerating a situation where the organization is chronically out of compliance with its own procedures degrades performance, degrades trust, and degrades morale. It is poisonous. (I talk more about the role of procedures in this post, among other places.)  

From a humanitarian point of view, when someone enters the country without the protection of the law, unscrupulous people can abuse or exploit him. His landlord or his employer can mistreat him, and he has no recourse. If he complains, the authorities discover that he is here illegally, and deport him. So the landlord and the employer are free to do a lot of bad things.

Temporary foreign workers clear
a field west of Montreal. (CBC)

Abuses can turn up even when a migrant enters the country legally. Just a few days ago, the United Nations issued a special report on Canada's Temporary Foreign Worker (TFW) program, comparing it in some instances to a "contemporary form of slavery."** However the case may turn out with Canada's TFW program once the dust settles, it is at any rate a legal public program. How much greater are the opportunities for abuse when the migrant worker has entered the country illegally and can claim no public protection!    

What are the root causes?

This post is already too long, so I have pushed the details of the analysis to another page. YOU CAN SEE THE ANALYSIS BY CLICKING HERE!

Also, when I started work on this I began trying to arrange the root causes on an Ishikawa diagram. I didn't get very far, but you can see the results at the bottom of this page.

But here is the list of potential root causes coming out of that analysis:

• Things are bad in a migrant's home country.
• The US is a big country with a large population.
• The US has very long land borders.
• The US has very long coastlines.
• The US is a good place to live.
• Lots of people find jobs in the US.
• Sometimes the filters at the border might fail to detect an illegal entrant.
• Sometimes the procedures at the border might fail to detain an illegal entrant.
• There is no reliable way to track someone once he is out of official sight.
• The legal entry procedure is very difficult and takes a long time.

If anything here doesn't make sense, the analysis goes through the reasoning in detail.

Now we get to the hard part.

Evaluating root causes, deciding where to act

Some of these points are not actionable. Of the rest, most are facts we probably don't want to change. There is one point that holds promise, but even it is tricky. Let's look at them.

Facts we can't change

The first four points on the list are simple facts that are out of our power.

Things are bad in a migrant's home country.

My last post explained that this fact is the one the Administration tried to change in their "Root Cause" initiative. And they succeeded in improving conditions in three countries. But bad conditions in other countries meant that the next year saw more migrants anyway. Pragmatically, the world is too large and there are too many countries where people are suffering for this to be a useful line of approach. 

That doesn't mean we should never help other countries! I take no position in this article either for or against foreign aid. But we can't expect foreign aid to reduce the immigration pressure.

The US is a big country with a large population.

Yes it is.

The US has very long land borders.

If you want barriers along our borders, that might have a deterrent effect in some places. But there are a lot of miles to cover.

The US has very long coastlines. 

You can't build walls in the water, and there's a lot of coastline.

Facts we (probably) don't want to change

Now we get to points that are more actionable. But we might want to think twice before acting.

The US is a good place to live. 

If we deliberately made the US a post-apocalyptic wasteland, no one would want to come here. But that's crazy. Let's not do that, OK?

Lots of people find jobs in the US.

Norway is a nice place to live, but there are very few illegal immigrants because it is almost impossible to find a job if you don't have legal status. An article from 2017 explains that "'Norway must be one of the most transparent societies in Europe.' It’s also highly regulated, where residents’ addresses are registered with local authorities and banks can’t open accounts for people lacking official identification numbers." Or consider New Zealand: an online discussion thread on exactly this topic explains that New Zealand prosecutes employers who hire illegal immigrants, and the fines are devastating. 

In principle we could do the same in the US, but do we want to? That depends. Most of my professional work has been in high-tech, and whenever I have taken a new job I have been required to show proof that I can legally work in the US.*** But I'm pretty sure there are industries or professions where the rules are not so strict, and I think there is a limit beyond which more rules would feel burdensome. If you ask the teenager down the street to babysit for a couple of hours, do you really have to check that her papers are in order first? (And then keep records to prove that you checked!) Or where is the line? There may be a little room on this front, but I expect not a lot.

Sometimes the filters at the border might fail to detect an illegal entrant.

I don't know if this ever happens, but logically it has to be evaluated. The problem is that every detection system has a failure rate. If we increase the sensitivity of the detection system used at the border, there will be more false positives. In other words, a more sensitive system is more likely to deny entry to normal citizens returning from vacation! Such an "improvement" won't be popular for long.

Sometimes the procedures at the border might fail to detain an illegal entrant.

Again, I don't know if this ever happens, but logically it has to be evaluated. But supposing it were true, what then? Are we asking for sturdier holding facilities at each port of entry? Are we asking for officials to deploy tougher methods? I don't know, but none of that is likely to look good on the evening news. So again, such an "improvement" won't be popular for long.

There is no reliable way to track someone once he is out of official sight.

Here's another point that is in principle easy to fix. Just make the US a police state, where no one can go anywhere without official papers and permission. Every time you arrive in a new town, check in with the Secret Police before you even find a room for the night. Then if anyone does slip past the border illegally, it will be child's play to track him down.

But who wants to live in a country like that?? I don't. You don't. Nobody does. So again, let's not do that! 

This leaves one last cause to consider.

The last root cause

The legal entry procedure is very difficult and takes a long time.

Honestly I think this one looks the most promising, if only because the others look so bad. In principle, "all it takes" to fix this is for Congress and the President to agree on something simpler than what we have now. (Quit snickering! 😀) And it's a common­place of process management that you get people to follow your processes by arranging things so that compliance is easy and noncompliance is hard. Make following the procedure the easiest way to do something, and everyone will follow the procedure. It's the same here. If it is easier to obey the law than to break it, everyone will obey the law. That's what we want.

Or maybe not. Maybe getting that agreement is hard. Maybe we as a country haven't come to a consensus on what we want here. That's not a problem that root-cause analysis can solve. The best I can do is to expose the problem, so that at least we know what we have to answer. But actually to answer it requires political work, and not Quality work.

Don't forget that a complete solution might involve several initiatives at once. Notwithstanding everything I said about the other root causes, they might be involved in partial measures that overlap each other to keep the problem from getting worse while a full solution is implemented.

But, speaking purely from the perspective of this Quality analysis, I think that tackling the last root cause—i.e., reforming immigration law—comes the closest to a winning play.

That doesn't mean it will be easy.

__________

* If you think there are more reasons than these two, that's always possible. I don't claim this is a complete list. 

** You can find the UN report here. For news reports, see here and here. The Canadian government responded on August 16 with a blistering news release accusing that the UN "special rapporteur" "spoke to no employers, did not visit a single farm, and spoke only with 100 workers." Up till now I have been able to find this news release only on LinkedIn, here.  

*** Since I am a citizen, that has not been hard to do.