Value-added auditing

Have you ever been at work when someone announced a new procedure that you knew right away was guaranteed to fail? What do you do about it? Of course if it's in your authority you can intervene immediately, but what if it's not? Is there any tool you can use to make your point?

Yes there is. Audits.

I used to work for a company who shipped products all over the world. Once we introduced a new product that had to be certified in each country where we shipped it, because one of the components was regulated as a communications device. This meant in each new market we had to supply a sample of the device, fill out a bunch of paperwork, and supply a fee; in exchange we got the right to import the product to that country for a certain period of time before we had to do it all again. But of course the expiration dates differed from country to country. So we had to keep track of the expiration dates for each country we shipped to, in order to make sure that we didn't inadvertently sell a product into a country where the certification had expired last week.

The design engineers were all in one office. The warehouse and the order-entry personnel were in a different facility, two time zones away. And when we recognized the problem posed by the need to certify this one component in country after country, someone proposed the following solution:

Let the Engineering Manager track the dates, because he's the one that has the specialized design knowledge to fill out all those government forms. He'll keep a big matrix that lists all the countries we ship to, and the current expiration date for each country. Then whenever he updates that list, he'll e-mail it to the Order-Entry Desk. That way they will always have a current list; and whenever they input an order for this product, they'll check the list to make sure we are really allowed to sell into that country. Foolproof!

Is it just me, or is it obvious to you, too, that this plan was guaranteed to fail? The most basic reason is that everything about this procedure was an exception. None of it was part of anyone's normal work routine.

The Engineering Manager had to remember to update the file in a timely way, even though that had almost nothing to do with the rest of his job. When he did update the file, he did it by hand.

The order-entry personnel had to check their emails in a timely way, and then fish out this file whenever it was updated. Then they had to put it somewhere, and remove the old outdated copies (to avoid confusion).  They did all of these steps by hand, with no system to guarantee even that the new files got stored in the same place where the old ones had been. (And of course if they accidentally stored different versions in different places, what would guarantee that they'd always find the latest one when they had to look for it?)

Finally, the Order Desk handled orders for thousands of different products. Among all those thousands of products, only this one had a rule that they had to stop everything and check a hand-built matrix to look up country and date before placing the order.

As I say, everything about the procedure was an exception. It was guaranteed to fail. But it was outside my area of responsibility, and it hadn't failed yet. So I couldn't really say anything.

What I did was to make notes to prepare for the next round of internal audits, which had already been scheduled for a few months later. When the time came, I asked the Engineering Manager for the latest copy of his matrix, and I scheduled an audit of the Order Entry Desk.

The people who operated the Order Entry Desk were really sharp, and we finished the routine part of the audit easily with no findings and lots of time to spare. So then I asked them for help. I reminded them of this procedure that was supposed to be in place—"Oh yes, we remember"—and then explained that I was afraid it was going to break down. Could they please help me? They opened the Order System in read-only mode, and searched for all orders for the product in question, starting with the latest one. For each order, they read me the country that it was shipped to and I checked it against the list. 

Within the first half dozen orders they read me, we found TWO that had shipped to a country where the certification had expired. Amazingly enough, that country's customs officials hadn't caught the shipments to reject them. But technically it was still a violation of a legal requirement. So I thanked the Order Entry auditees for their help, and wrote it up as a Major Nonconformity. I made a point to emphasize in the Closing Meeting that this finding was found while auditing the Order Desk, but it wasn't their fault: it was a system issue, and had to be addressed as such.

Soon after the company implemented a system of automated flags in the order tool itself, completely eliminating the manual exception-handling. It was a much better approach.

               

Digging Deeper with Root Causes

This morning, Quality Magazine published my article, "Digging Deeper with Root Causes." It's their article now so I won't post the text of it here, but you can find it by following the link. I hope you find it useful!

Finding Root Causes: How to Identify and Understand the Core Issues

This morning, Quality Magazine published my article, "Finding Root Causes: How to Identify and Understand the Core Issues." It's their article now so I won't post the text of it here, but you can find it by following the link. I hope you find it useful!

Do things right, or Do the right thing?

Two weeks ago, when I started writing about the Mayo Clinic, most of the feedback I got was favorable. It seemed that people were glad to read good news for a change. But Ian Hendra, of Clearline Services in New Zealand, offered a comment on LinkedIn that was very critical, not only of the Mayo Clinic but of the whole health-care industry. The gist of his argument was that medical care as practiced these days focuses (outside of surgery) on the use of pharmaceuticals, even for conditions that can be more thoroughly and reliably treated with diet and nutrition. While his comment rapidly plunged into detailed arguments about diet and insulin resistance, he touched on a wider issue. If I were to rewrite his argument to make it at once more general and more focused on the underlying Quality topics, it would sound something like this:

Any hospital whose Quality Policy says they prioritize patient outcomes should do everything they can to help patients get well. But this means they should use methods that have (1) a high rate of success, while involving (2) minimal intervention in the body's natural operation, in order to cause (3) the fewest unwanted side effects. Many of the treatment methods common in modern medicine, by contrast, involve significant intervention in the body's natural operation, and sometimes trigger serious side effects. Therefore a hospital who takes the Quality Policy seriously should step back from many of the most common treatment methods in modern medicine, and replace them with others.  

Now, I'm certainly not a doctor, and I have no expertise to discuss this critique at the medical or scientific level. What is more—to be very clear—I don't know the research that Mr. Hendra is relying on, so I have no idea whether he's got his medical or scientific facts straight. But what interests me is the Quality issue. If some patients do better when they are treated according to nonstandard methods, does Mayo's Quality commitment require its doctors to adopt those nonstandard methods? Is that what a commitment to the Quality of patient outcomes really means?

In one sense, of course it would be nice. From a patient's point of view, getting well is typically the only measure of the Quality of a course of treatment. At the same time, there are a number of pragmatic considerations that every hospital or medical center has to keep in mind.

They have to have specialists who understand any method of treatment that they offer. The more types of treatment they offer, the more specialists they need in different fields. This point should be uncontroversial, but we can't forget it.

The treatment methods they offer have to be known to work. This point is trickier than it sounds. Human health is phenomenally complex, and even the most routine courses of treatment regularly fail on somebody. It's also true—and documented—that some very unusual courses of treatment have nonetheless worked very well on some patients.* It can be hard to know where to draw the line, but you have to draw it somewhere; otherwise, with no regular approaches, a doctor really doesn't know where to begin.

Finally, the treatment methods often require not only consent but active participation (or at least compliance) on the part of the patient. This is where diet and nutrition encounter their most frequent difficulties. Most patients are famously very bad at following nutritional advice. Lots of people "know they should eat better than they do," but that doesn't change their food choices in real life. If a doctor says, "Take this medication once a day for two weeks," patient compliance is usually pretty high. If the same doctor tells the same patient, "Cut out high-cholesterol foods and refined carbohydrates every day for the next thirty years," compliance drops off fast. You don't have to look for conspiracies by the pharmaceutical industry to guess that there are practical reasons doctors overwhelmingly choose the former approach over the latter.

In light of these considerations, what is a hospital to do? At a pragmatic level, as I suggested a minute ago, they have to draw the line somewhere. Implicitly or explicitly (and of course it is better to do  it explicitly) they have to say, "These are the kinds of services we provide. We will provide them as well as we can. But if you want some other kind of service, you have to go elsewhere." This statement defines the Scope of their Quality System. Inside that Scope, it is possible to optimize their performance. But there is no meaningful way to optimize performance of tasks that are outside of Scope, because a hospital (or any organization) shouldn't take on those tasks to begin with. And if you try to widen your Scope until it covers everything, you can never optimize because you can never get your arms around the task. So you can never get started. 

From  Matt Groening's School is Hell.

To capture this in a snappy formula, an organization's Quality Commitment is focused on ensuring that the organization does things right, and not that it does the right thing. There must, of course, always be room somewhere to discuss whether the organization really is doing the right thing, or whether the Scope should be amended to include new and different kinds of activities. But these discussions take place necessarily outside the Quality System (though they might incidentally involve some Quality personnel). In a hospital, these discussions would include whether to offer or adopt fundamentally new or alternative forms of treatment; in a business, they would involve whether to pivot to a new line of business or a new form of operation. These discussions are important but they are fundamentally strategic, while most routine Quality work is tactical: Given a certain set of objectives, how can we best achieve them?

No Quality System will ever make everything perfect, any more than it can prevent all accidents ever. And it is not really the job of the Quality System to break the paradigms in which the organization operates. But inside those paradigms, of course, it is a powerful tool for making things as good as possible.

__________

* In addition to the normal kinds of care made available by conventional medicine, there have been documented cases where some patients have responded well to such treatments as: 

• diet and nutrition (Mr. Hendra's preferred approach)
• placebos
• acupuncture
• homeopathy
• Christian Science practice
• laying on of hands
• prayer  

Obviously it would be tough to ask a hospital to provide all of these.

  

Quality and cost

Last week I wrote about the Mayo Clinic, and how their most recent Quality Initiative pushed them to the very first rank among the world's hospitals. When I looked for what principles undergirded or enabled this achievement, one of the ones I identified was this:

Mayo chose to focus on Quality first, not cost; and their metrics for Quality were tied directly to patient outcomes. This is the correct way to define metrics: start by defining what results you want to achieve, and then build metrics to support those results. In modern management, metrics drive so much other activity that it is critical to get this part right up front.

Now that's great, but it's fair to ask, How can Mayo afford to focus on Quality before cost? Don't they have bills to pay? For most companies—maybe all—neglecting cost is a quick way to go broke. What's special about Mayo's situation that apparently allowed them to defy this basic law of corporate gravity?  

Of course the answer is that Mayo didn't literally neglect cost. In fact, the article that I summarized last week (ASQ's "Journey To Perfect: Mayo Clinic And The Path To Quality") makes clear that an earlier quality initiative in the 1990's was in fact canceled for reasons of cost. What happened since that time (apparently) was that Mayo reëvaluated their approach, and began to address costs intentionally, and consistently with their mission. This intentionality shows up in several ways.

For example, Mayo has pushed to standardize practices across its multiple facilities. As my post last week pointed out, the criterion for selecting which practice to choose (among various alternative ways to do the same thing) was generally based on data about patient outcomes. But the mere fact of standardizing at all lowered costs. And cost is one factor tracked (among many others) in the massive data collection that Mayo hosts about all its surgical activities.

Mayo's approach to paying physicians is equally intentional. Physicians are compensated with salary only—no bonuses or special treats. The goal is "to remove financial incentives to do more than is necessary or less than desired for the patient." Of course the salaries are structured so that physicians with more experience earn commensurately more, and the various specialties are compensated competitively with peer institutions. And an additional benefit of this structured, salary-only compensation plan is that it makes it easier for Mayo to find and correct inequities, in order to avoid pay discrepancies based on non-relevant factors like race or sex. But the core motivation is to reduce the risk of corrupting the patient experience.

The key, though, seems to have been when Mayo realized "that patient-centered care is a win for financial outcomes. Quality was not simply continuous improvement; it was the vision and mission of the organization." In other words, while an awareness of costs matters internally (in order to keep the books balanced), cost is not a market-differentiator externally, at least not for the Mayo Clinic. Patients don't come to Mayo to get their x-rays done cheaper. They come to Mayo to get their care done better. Patients seek out the Mayo Clinic because they want its Quality.

Can other businesses do the same thing? Yes and no. To some extent, Mayo benefits from the economics of health care. The proliferation of so many different insurance models for health care means that almost nobody pays the "list price" for any given medical procedure, and so those prices mostly are not posted or even visible. Often enough, a patient won't even know the price for any but the most routine care until after the care has been provided. And health care is simply a different kind of good from candy bars or movie tickets. If movie tickets cost too much, people may try to stream the movies at home for less or simply forgo seeing them at all. But someone who needs an important medical procedure is much less likely to forgo it because of the cost. For all these reasons, cost is a weak market differentiator in the health care industry.

All the same, companies with a reputation for Quality can thrive in any industry even if their products cost more. I used to work for Bosch, which is a manufacturing company. And manufacturing is an industry where even minor differences in cost can make a huge difference. Bosch's products are never the cheapest ones available; but people buy them eagerly because they know the products are going to work. Every time.

A few months ago, we discussed that—Philip Crosby notwithstanding—Quality isn't really free. Building any kind of Quality System takes time, effort, and money. But in the long run, such a system pays for itself by reducing defects. And if you are good enough, it draws customers from around the world who want to work with you because you are the best.

          

Building a healthy Quality culture

For much of this spring, I've written about system and culture failures. I've written the most about Boeing, but I've addressed other companies too (even ones like Toyota and Patagonia, who are mostly pretty good). After a while it can get grim. One reader wrote me to say, "I am too depressed about the Boeing issue."

What if we look at a success story, for once? A company that built on a history of doing good work, and lifted themselves to world-class performance? It could be a refreshing change, if nothing else.

A couple weeks ago Dale Weeks, another reader, called my attention to an ASQ Case Study about the Mayo Clinic. The study, titled "Journey To Perfect: Mayo Clinic And The Path To Quality," was written in July 2012. That means it is more than a decade old, but it is hardly out of date. The Mayo Clinic is still among the best hospitals in the world. This Newsweek website rates them, in fact, #1 in the world. Mayo's own website states, more modestly, that "Mayo Clinic Hospital — Rochester is top-ranked in more specialties than any other hospital and has been recognized as an Honor Roll member by U.S. News & World Report's 2023–2024 'Best Hospitals' rankings." Whatever they are doing, they are doing it right and it seems to be working.

What are they doing? The article describes a history of Quality initiatives at Mayo, so the first thing is that Quality has been a sustained focus over many years. But the heart of the article is about the Quality initiative in the early 2000's, and this had several elements. We have discussed some of them before. 

• Mayo implemented a "Fair and Just Culture," in which "every member of the medical team is encouraged to report anything that does not seem quite right, without fear of reprisal." Even better, the article gives examples which suggest that Mayo really means it. (By contrast, for example, Boeing's culture sounded good on paper, but did not encourage transparent communication in practice.)
• Mayo asked all departments to display their performance data, so that any room for improvement would be visible to everyone. (Back in the first year of this blog we discussed the value of making your performance data visible to everyone.)
• Mayo used this performance data to look for best practices, and then spread those best practices across all their multiple hospitals. Sometimes the data related to patient outcomes: when one hospital had a much-lower patient failure rate than all the others in administering warfarin, Mayo studied that hospital's procedure and made it standard. Other times, the data was frankly economic: when they found that one orthopedic surgeon could do hip replacements more profitably than any of the others (while all of them had the same rates of patient outcomes), his method became the one to use.
• While of course Mayo had to track their financial metrics the same way any business does, those were never the center of attention. Mayo's key metrics focus on "patient and quality outcomes, patient safety, and the patient experience." We've talked before about why it is more valuable to measure problem-resolution than operational activity, and also about why it is essential to see how the problem looks to your customer. These are lessons Mayo has clearly taken to heart.
• Mayo made training available to all employees on Quality methods, including topics like Six Sigma, lean, and reengineering; and encouraged everyone to take them. They even issued pins of different colors that employees could wear, to show how far they had progressed through the curriculum.
• Finally, Mayo implemented an extensive knowledge management infrastructure.

All of these measures are excellent, of course, and it is no surprise that the outcomes have been so good. But I found myself wondering, Why doesn't everybody do this? If the Mayo Clinic can focus on Quality this relentlessly and achieve such glowing results, then it is clearly possible. And if it's possible, why doesn't everyone else do the same thing? Why—not to put too fine a point on it—isn't Boeing following this very same game plan right now?

The article isn't structured around cause-and-effect. It's a case study, and not a root-cause analysis. But as I combed through it, I think I found three fundamental points that made all the others possible.

• First, Mayo had a history of selecting executives from among its practicing physicians. I have sometimes argued that management roles should not be handed out as rewards for the very best physicians or surgeons, because it's better that those experts continue to save lives in the examining room and the surgical theater. At the same time, it clearly makes a difference when the executives understand the work at a deep level. Once upon a time, Boeing's executives were engineers, too.
• Second, Mayo chose to focus on Quality first, not cost; and their metrics for Quality were tied directly to patient outcomes. This is the correct way to define metrics: start by defining what results you want to achieve, and then build metrics to support those results. In modern management, metrics drive so much other activity that it is critical to get this part right up front.
• Third, it is clear that the whole Quality initiative had unflinching management support over the long haul. Without a clear vision of what to do, and an unwavering commitment to doing it, none of this could have been accomplished.

These three points aren't universal. Not every company selects its executives from its working experts; not every company prioritizes Quality over cost; and not every management team is committed to implementing Quality in every aspect of their business. But they could do. If Mayo can keep finding ways to improve, so can we all. 

                

Does my test car have to be safe?

Last week I got a call from a friend and former colleague, who wanted to talk about stage gates. (You remember I've discussed them before here and here.) Specifically, he wanted my perspective on the question, "When do all the formal requirements have to be incorporated into a product? Can you wait till you release it, or do they have to be there before you start testing?"

Phrased like that the question sounds pretty innocuous, and also simple: It depends. Naturally all the requirements should be in place before you release the product, because otherwise you're not really done. And you'd like everything to be ready before you start testing, so that you can test it all together to make sure it works. If one feature isn't ready when you start testing, you can't test that feature; then when you finally get to it, there's always a risk you'll find a problem which impacts something else you thought was already buttoned-up. At the same time there are plenty of projects where this or that feature isn't quite ready when testing begins, and the project makes a conscious decision to proceed at risk. As I say, it depends.

Then as we talked some more, I began to understand that my friend had a real-life example in mind, one which made the discussion a little more pointed.

Last October, a GM Cruise self-driving taxi struck and dragged a pedestrian 20 feet.* This accident was only the latest of several last year, some of which got significant attention in the news.** After the October accident, the National Highway Traffic Safety Administration began an investigation; the State of California suspended Cruise's permission to operate autonomous vehicles; and Cruise recalled its entire fleet for reprogramming. Cruise's interaction with regulators appears to have been remarkably clumsy; while Cruise says they had no intention to mislead either the regulators or the public about what happened, apparently their first communications did exactly that. Since then Cruise has fired nine executives and let go a quarter of its staff. The CEO and one co-founder both resigned, and Cruise published a blog post blaming its corporate culture.*** 

So my friend's real question was something like, "What precautions do you have to take before you let an autonomous vehicle out on the public roads? Let's say you are still in the test phase; can you legitimately skip some of the safety features because your car is 'not released yet'? And if so, what stops you from ending up like Cruise?"

It's a good question, because it calls attention to the difference between formal requirements and pragmatic requirements. Often we think of formal requirements as more numerous than pragmatic ones: pragmatically, a hammer should drive nails; formally, there's an entire ISO standard of technical specifications for steel hammer heads. (That's ISO 15601:2000, in case you wondered.) But in a pre-release situation, sometimes the distinction reverses. Depending on how your Quality Management System is written, many product requirements might not be formally mandatory before release. But you may need them for pragmatic reasons, all the same. 

To be clear, Cruise's vehicles had been fully released, because Cruise was operating a business and collecting fees. But there are plenty of other companies making autonomous vehicles, or wanting to make them. So my friend wasn't asking "What did Cruise do wrong?"—though it would be fascinating to dive into that question, if we had the data—but rather, "If Whizzbang Motors decides to build an autonomous car tomorrow, what's to stop one of their test drives from ending just as badly as this well-publicized accident?"

So I started at the beginning. Quality means getting what you want. And obviously you don't want to hurt people. So Quality means making sure that you don't. For a released product, especially in the automotive market, that means complying with a long list of federal regulations; and that, in turn, means proving that you comply with them by providing objective evidence like documented test results. For an unreleased product, you may not have all of that documentation yet. But you still have to avoid hurting people, and that means you have to figure out what it is going to take.

The earliest tests, when you are first developing a new car, can probably be carried out in a laboratory. After that, you move to a test track, or proving grounds. These are environments that simulate real driving conditions, but without the innocent bystanders. The only people on the test track (at any rate during a test) should be people attached to the development project, who know what the vehicle is capable of and where the errors or problems are likely to be. Even so, sometimes test drivers die in accidents. The only consolation is that nobody takes a job as test driver without knowing the risks ahead of time.

And when the day comes that you finally have to test on the public roads? Doubtless there will be public, legal requirements that you have to meet before you can put your car on the roads, even if it's not released yet. But beyond those legal requirements, you have pragmatic and moral requirements to have done everything in your power ahead of time to make sure the car is safe. Some people will insist that "Nothing can be made 100% safe," and in a literal sense they are right. But that is never an excuse for doing any less than you possibly can.

When it comes to the safety of any pre-release product, your first thought should never be "What do the rules make me do?" Ask yourself instead, "How do I avoid hurting people?" 

__________

* See, for example, either of the following articles: 

• GM Cruise robotaxi unit faces US probes over dragging incident, vows reforms | Reuters
• GM’s Cruise reveals dual US probes into grisly collision and company’s response | Self-driving cars | The Guardian.  

** See for example this accident in April 2023, or this one in August. In all of 2023, Cruise vehicles were involved in 23 crashes, out of a fleet which numbered at least 950 at the time of the October accident. 

*** Wow, where have we heard that before? 😀