Quality methods for the win!

I've often argued that Quality methods are just as useful in real life as they are in the workplace, but this month's issue of Quality Progress is running a story that makes the point better than I ever could. Jared Watson, a senior quality engineering specialist at L3Harris Technologies, published an article about how he used systematic quality improvement methodologies to win big on the game show "Jeopardy!" 

You can read the full article online. But to summarize, Watson won his first rounds (in June 2023) with only "a month of preparation and a lifetime of enthusiastic—if amateur—curiosity." But then he was invited to compete in the Tournament of Champions, and he realized that his preparation would have to reach a whole new level. So he analyzed his gaps as clearly as possible, and made a plan to address them.

The first step was honesty. In my original run, I’d lost thousands of dollars on geography clues, including two Daily Doubles, and geography is the most frequently tested “Jeopardy!” category. If I didn’t confront that weakness, it would beat me again.

Next came help. Improvement accelerates when we invite others into the process. In my case, I had a sleeping trivia giant in my house ...: my wife, Jasmin. What began as a simple request for a few flash cards turned into more than 10,000 handcrafted study prompts .... Her support became the backbone of my preparation.

But tools and support only matter if you put in the work. Early on, I tried to master new material in a single sitting by drilling the same cards until I was frustrated. It didn’t work. What did work was steady, sustainable repetition: reviewing each card no more than twice a day, trusting that consistency—not perfection—would lock in the knowledge over time. And it did....

This lesson is one our field knows well: Quality improves through honest assessment, smart use of resources and disciplined, incremental effort. Whether on the “Jeopardy!” stage or in our organizations, consistent practice beats the illusion of overnight mastery every time. 

With this solid preparation, Watson posted a perfect performance during the quarterfinals, answering 21 for 21 and all three Daily Doubles. Later he was eliminated in the semifinals, but the power of the method to get him that far was absolutely clear. As Watson himself concluded, "That’s what quality looks like in practice: A commitment to the process, not just the final score."

Congratulations!

           

Sampling bias

A friend forwarded this cartoon to me a couple of weeks ago. It seems to come from the website Sketchplanations, drawn by Jono Hey. You can find the original here.  

Of course statistics are central to the Quality discipline, so we need to understand how sampling works. And I think the cartoon makes a serious point in that regard, rather deftly.

It's also funny, and high time too!

            

Feedback on Santa's audit!

Wow! I got a lot of comments on last week's post about how far Santa Claus complies with ISO 9001. Thanks especially to Pia Hamrin, Dawn Ringrose, Petro Shoturma, and Jeremy Panitz for their detailed feedback!

What interested me was the remarkable level of unanimity among the responses. Of course each commenter had a unique perspective. But since Santa Claus is fictional,* any commonality among the responses must be traceable to the second half of the discussion—namely, to the expectations that grow naturally out of working with ISO 9001. Those of us who work with the standard regularly have come to know that some clauses need a lot more attention than others, because the impact they have on organizations is so consequential.

Even if you aren't worried about Santa Claus, ask yourself whether these same clauses aren't important topics in your organization.

Common concerns

It was no surprise to me that the two topics which attracted the most attention were Internal controls and Complaint handling. Businesses differ a lot: organizational contexts can be very different, as can the quality of customer requirements. For some companies those topics might require a lot of attention, while for others the same topics might manage themselves. But internal controls always matter—to everyone. And without complaint handling, you have no feedback on your operations to make sure your customers are satisfied. So nearly everyone agreed to focus on those areas.

Naughty or nice?

Everyone knows that Santa Claus checks whether each child is "naughty or nice," but how does this happen? Dawn's reply assured me that Santa uses a proprietary algorithm. But Pia asked how that algorithm is validated, and whether the sources are documented? Jeremy suggested that Santa consults with children's deities around the world to make his determinations.

Complaint handling

Petro and Pia both raised the issue that there is no publicized mechanism for complaint handling. Jeremy suggested a resolution to this topic, though, by arguing that "If the customer isn't satisfied with the gift they got from Santa Claus, the gift did not come from Santa Claus."**

Less worried

By contrast, nobody was very worried about document retention. Dawn suggested that the children's letters must be fully digitized, and nobody else took up the subject. I think this lack of concern reflects our common experience that yes, of course document retention is important; but on the other hand documentation findings are generally the weakest kind of finding in any audit.

Similarly, Dawn and Jeremy both assured me that internal audits were done by impartial elves, where "no one audits their own toy line." And I think we have all experienced that even in organizations where the Quality Management System is informal or not very mature, it is generally possible to find people who can treat the process objectively, and who can therefore do reliable audits.

Unique observations

Then there were the unique observations, that pushed the discussion in an unexpected direction.

Jeremy argued that Santa Claus is thought of as a toymaker, but that the real focus of his design expertise is in logistics. In commenting on clause 8.3.2 (Design and development planning), he explains, "Santa has only 24 hours to deliver toys and other gifts all across the world. That means he's got to know the weather patterns. So his design and planning is less on toys and gifts and more on safety and delivery of the goods." Jeremy doesn't use this example, but I assume he would compare Santa (in this respect) to McDonald's, who didn't redesign the hamburger but revolutionized how it was delivered to the customer.

For her part, Dawn answered the question about children changing their minds by enunciating the rule, "Last Wish Wins." But that rule means that the audit plan has to spend a lot more time checking scrap and rework under clause 8.7 (Control of nonconforming outputs). As long as the only topic was the elves' workmanship, we could assume that the elves are magical and don't make mistakes; so overall scrap should be minimal. But if children can change their minds late in the game, there is sure to be scrap as earlier gifts are replaced with later ones. Thus does any answer in one part of the audit open up new trails in another part—and this, too, is a familiar experience to all of us.

Naturally I don't mean to suggest that any clause is unimportant. In the right (or wrong) circum­stances, any clause can trip you up. But I found it interesting and reassuring that the feedback on this topic aligned so closely with my own sense about where the critical risks are in any ISO 9001 implementation. Again, even if the North Pole doesn't interest you much, your own organization does. And it is worth looking over where your own risks are.

__________

* Or at any rate I'll assume so for the purposes of this post! 

** It's an interesting idea, to which I note only that, however this principle might work for Santa, it's not available for the rest of us.     

         

Does Santa Claus comply to ISO 9001?

Years ago, when my kids were little, they would spend an hour or so every Christmas Eve watching the NORAD Santa Tracker. You've probably heard the story behind how NORAD tracks Santa's progress through the skies every year. And indeed every year there are press releases updating the public on the current status of the Santa-tracking program.

These days I'm out of touch with the latest developments in aviation, but I'm correspondingly more interested in international standards. So today I find myself wondering, How far does Santa Claus comply with ISO 9001?

It's not an idle question. Last year an investigative reporter went to the North Pole—at least, that's what the article said—to check the operations of Claus Enterprises against OSHA's Top 10 workplace-cited standards for 2024. The results weren't pretty, though Santa himself put the best face on what he called "“an unfortunate misunderstanding," and pledged a prompt corrective action plan.

But even if safety issues have been addressed, what about Quality? Millions of children rely on Santa for presents each year. Can he really be counted on to come down all the right chimneys? Does he stand behind his work, or does he just fly away "like the down on a thistle"?

Up till now I have been unable to find an ISO 9001 certificate* for any part of Santa's operations, neither his workshop nor his delivery service. Partly the problem may be geographic: I'm not aware of any formal Certifying Body with jurisdiction over the North Pole. All the same, surely it's a good idea to do an advance check—like a pre-audit—to find which clauses need the most attention. So Santa, if you are reading this, treat it as a little free consulting, to help you get positioned.**  

These, then, are some clauses of the standard where I would expect discussion in an audit:

4.1 Understanding the organization and its context

Ever since the climate change amendment last year, this clause has included the requirement, "The organization shall determine whether climate change is a relevant issue." Well if the Polar ice cap is about to melt, that sounds relevant to me! Does Santa have plans to relocate? And where?

4.2 Understanding the needs and expectations of interested parties

How often are these needs and expectations reviewed or updated? I assume that the desires of little children have been broadly similar for centuries. But what about other players in the same market? Does Santa face competition from online retailers, or has he worked out some kind of deal?

5.2.1 Establishing the quality policy

What exactly is Santa's Quality Policy? Does he strive for 100% satisfaction? We know that every year some children are disappointed; so if the policy calls for 100%, there might be a problem. And what if a gift is defective, or has to be returned? Does Santa replace it himself, or does he require the parents to do the work? (See also clause 8.4 Control of externally provided processes, products and services.) 

7.5.3 Control of documented information

Children write letters to Santa from all over the world. Are those kept for future reference, in case there is some dispute over what a child really requested? And in what form? (For example, are the original letters kept on paper, or are they digitized?) If the letters are kept on paper, how are they preserved in the harsh Arctic climate?

A second topic is the retention policy: how many years is a letter kept? Santa is always said to be wise and to have a long memory; but the storage requirements for each new year's batch of letters must be enormous. Are they kept for a year? For a decade? For centuries? How are they indexed? How easily can they be retrieved?

8.2.3 Review of the requirements for products and services

There's another kind of control needed for the children's letters: regular review. Some children might request toys that are dangerous, or that are illegal in their place of residence. How is this review conducted, and how are conflicts resolved?

In addition to all the other reviews, subclause 8.2.3.1(c) requires a review of "requirements specified by the organization." Presumably this includes comparing all requests against the List of Who's Naughty and Nice. But how is that list managed and kept up to date? And how often is a naughty child really given a lump of coal instead of some other present? (I can think of children over the years that I thought had earned a lump of coal who never seemed to get one!) 

8.2.4 Changes to requirements for products and services

What about children who change their minds about what they want, and who send multiple letters? Is there a tracking system to allow the second letter to be filed with the first one, or to replace it completely?

8.3.2 Design and development planning

What is the ratio of customized gifts to off-the-shelf toys? The publicity photos of Santa's Workshop look like they are mass-producing toys, but there must be a lot of specialized requests as well. How is this handled? (See also clause 8.5.2 Identification and traceability.)

8.4 Control of externally provided processes, products and services

How much of Santa's work is subcontracted to local merchants, or to parents? And on what terms? Does Santa still get all the credit, in all cases?

8.5.2 Identification and traceability

How far is lot-traceability a relevant issue? (Note that if every job were custom—see also clause 8.3.2 Design and development planning—then lot-traceability would not be a requirement.)

8.5.4 Preservation

The North Pole has a cold, harsh climate. The same is true for much of the lower atmosphere. How does Santa preserve presents in good condition, and keep them from being damaged by the harsh environment?

8.6 Release of products and services

Does Santa keep records of every gift, and are those records filed with (or cross-indexed to) the corresponding request letters? This raises some of the same questions we discussed above under clause 7.5.3 Control of documented information. 

9.1.2 Customer satisfaction

What is Santa's current level of customer satisfaction? Has it been trending up or down? In case of any downward trend, has Santa chosen to implement corrective actions, and what do those look like?

9.2 Internal audit

Who does internal audits for Santa's enterprises? Are the auditors selected in a way that "ensure[s] objectivity and the impartiality of the audit process"? Is the scope of the internal audits the same as the scope of the enterprises, or are there some areas that are off-limits to the auditors? Can I see the results of the latest full audit?

9.3 Management review

Can I see the records of the latest management review? Has Santa ever introduced a major change to the operations as a result of findings in management review?

• If yes, what was it?
• If no, then is the organization really using the management review, or is it just a dog-and-pony show?

10.3 Continual improvement

Tell me about the latest continual improvement initiatives.

Having said all this, I have to admit that it would be the chance of a lifetime to audit Santa's Workshop at the North Pole. An operation with such vast scope, that has to achieve such delicate precision in its outputs, would be amazing to review.

So Santa—again—if you are reading this and want someone to do a test-audit before you call a regular Certifying Body, leave a comment on this post with your contact information and I'm sure we can work something out.

And to everyone else, have a jolly day, a delightful holiday season, and a very happy New Year! 

__________

* I did find one article here that claims Santa Claus is certified to ISO 9001 and a host of other standards as well, but it offers no links to objective evidence.  

** And with a little luck, giving away free consulting might be enough to move me from Naughty to Nice!     

 

On sabotage

It's been a while since I posted anything funny here. Let me know if this counts as funny.

Some years ago, the CIA declassified a document published by the Office of Strategic Services (OSS) back during World War Two. The document was called Simple Sabotage Field Manual (Strategic Services Field Manual No.3), and it was written to support our allies trapped behind enemy lines as they tried to disrupt Axis war-fighting capability and Axis society generally. It discusses motivation, timing, and tools, and then gives a lot of specific recommendations. You can download a copy of the document from here (or here).

What could be funny about this, you ask? Well, the book got the attention of management consultants and other business excellence experts, because there is a section on how to sabotage "organizations and production." And the advice in that section just sounds so ... normal! 

Here are just a few highlights:

• Organizations and Conferences
• Insist on doing everything through "channels." Never permit short-cuts to be taken in order to expedite decisions.
• When possible, refer all matters to committees, for "further study and consideration." Attempt to make the committees as large as possible- never less than five.
• Haggle over precise wordings of communications, minutes, resolutions.
• Managers and Supervisors
• Don't order new working materials until your current stocks have been virtually exhausted, so that .the slightest delay in filling your order will mean a shutdown.
• Hold conferences when there is more critical work to be done.  
• Multiply paper work in plausible ways. Start duplicate files.
• Apply all regulations to the last letter.
• Office Workers
• Misfile essential documents.
• Tell important callers the boss is busy or talking on another telephone.
• Spread disturbing rumors that sound like inside dope.
• Never pass on your skill and experience to a new or less skillful worker.

The list goes on and on. And, as I say, the actions mostly sound like normal human clumsiness, short-sightedness, officiousness, and inefficiency. I've seen some of these measures with my own eyes, and I bet everyone I know can say the same thing—not always and not everywhere, but at some jobs, from some people, at some times.

The thing to remember is that while you or I might call this inefficiency, the OSS called it sabotage! When our organizations stumble through the day allowing these things to go on, we are in fact sabotaging ourselves. Now, in spite of all that, most of us get through the day intact anyway. Most of our organizations live to do business again tomorrow. We don't shut ourselves down. 

But think how amazing our performance would look if we didn't sabotage ourselves!  

If you have time for a longer-form discussion, Kyle Chambers and Caleb Adcock from Texas Quality Assurance chatted about this very topic in two back-to-back podcasts back in early March. The first is half an hour, and the second is just about 45 minutes. As always, they discuss the issues with their trademark humor and pragmatism. 

  
      

Why are you an auditor?—REDUX!

It's the New Year,* and I hope 2025 will be good for us all! In case any readers are still coping with a surfeit of champagne, let's start with something light.

A few months ago, I published a post asking "Why are you an auditor?" I told the story of Amir, who took a job as an auditor in order to learn how to manage a business. His plan was that, once he had learned how to run a business, he was going to quit auditing for entrepreneurship.

After writing this article, I posted notices pointing to it in a number of venues. One of these venues was the myASQ community, where Anthony DeMarinis of AJD Quality Solutions responded with the answers that he gives his students to the exact same question. I've wanted to share them with you ever since I read them, and this looks like a fine time to do so.

Top 10 Reasons to Become an Auditor

10  Get to see the big picture with exposure to Top Management

 9  Benchmark other areas and promote Out-Of-Box Thinking

 8  Good way to increase your personal knowledge and keep your job

 7  Paid time off and a diversion from your regular job

 6  Acquire transferable skills to prepare for the next layoff

 5  Experience with conflict resolution (which may come in handy at home)

 4  Free dinner at fancy restaurants with the Auditee

 3  Opportunity to network and look for another job

 2  You're not the Auditee and get to hassle someone else for a change

 1  Unlimited POWER!!!

Once again, I wish you all a very Happy New Year! Let's do good things in 2025.

  

__________

* OK, technically that was yesterday. But it's only been 2025 for a scant 32 hours so far.     

Another take on 5-Whys

A couple months ago, I saw a fun cartoon on LinkedIn. It was attached to a perfectly sound article which you can find here. But the cartoon is what I wanted to share.

And of course this is right. Sometimes analysis sputters out. You might not have the data you need to get a first-class root cause. Other factors might interfere as well. This work can be hard.

It's still valuable to do, of course.

                

Quality and Despair

It's Hallowe'en. Once again, it's time for something spooky.

This year I'm going to quote a commercial product. It's a Demotivational® poster from the folks over at Despair Inc. It starts with one of the basic commonplaces of the Quality business, and spells out the awful consequences of taking it literally.

The race for quality has no finish line;

so technically, it's more like a death march.   

Click here to jump to their site, where you can order these words of wisdom on a poster or a mug.

Happy Hallowe'en, everyone!

    

What's "proportionate"?

When the ISO 9001 standard requires you to take action to address your risks and opportunities, it includes this admonition in clause 6.1.2: Actions taken to address risks and opportunities shall be proportionate to the potential impact on the conformity of products and services.

OK, I guess that's fine, but what does it mean? What does it take for your actions to be "proportionate" to the risks they address?

The word is never defined—or at any rate, neither ISO 9001 nor ISO 9000 define it. But intuitively I think we all have a sense for what it means, don't we? The basic idea is ancient: Nothing in excess.* If you face a risk that might, at worst, cost you $100, then it is foolish to spend $1000 to prevent it. That cost, or that level of effort, is disproportionate to the $100 downside that you face from the unmitigated risk. Most of the time, we probably don't need a definition more exact than that.

But "sometimes the clearest way to explain what a rangatang [sic] is, is to 'tell what it ain't.'"** A couple of months ago, I stumbled upon a blog post from 2016 that explains the concept of disproportionality with crystal clarity. The author of the post—Quinn Dunki of Blondihacks—just wanted to set up an automated cat feeder, so that her cat would be fed on time and she didn't have to watch the clock every day. Simple, right?

Turns out her cat had a different idea. Her cat's idea was, "How do I get this machine to give me more food than Quinn wants me to have?" So Quinn had to make some adaptations to her automated cat feeder, to protect it from the prying paws of her cat. As she says at the opening of her blog post, "The trick is to be smarter than the animal with a brain the size of a walnut."

But of course, Quinn worked on this problem part-time, and her cat worked on it full-time. 

You can read the results here.

In the end, Quinn won. But I'm pretty sure nobody would say that the effort she expended was proportional either to the benefits she gained or to the risks she was avoiding.

Verbum sapienti sat. 

__________

* "Μηδὲν ἄγαν" was one of the three proverbs said to have been inscribed at the entrance to the Greek temple at Delphi. See here for more information.  

** Owen Ulph, The Fiddleback: Lore of the Line Camp (San Francisco: Browntrout Publishers, 1995), p. 23.      

Five laws of administration

It's the last week of the year, so let's end on a light note. Here are five general principles that I've picked up from working in many organizations over the years.

Hard, but easy

1. The more sophisticated a task, the easier it is to get authorization to proceed.

2. The more far-reaching the implications of a proposed policy, the easier it is to get approval.

Easy, but hard

3. The easier a task is to describe, the more time it takes to complete.

4. Procedures that are easy to write, are hard to implement.

Law of purchase requests

5. No Purchase Request can be approved until the cost* of the discussions justifying the decision exceeds the cost of the item to be purchased.

Do you have others? Send me yours.

And to all my readers, have a very Happy New Year!

__________

* (calculated as the sum across all participants of the hours spent in discussion times each participant's hourly rate of pay) 

          

ISO haunted house

Next Tuesday is Hallowe'en. 

So in honor of the holiday, here's a quick glimpse into the ISO/ANSI office Hallowe'en party. If you dare ....

All thanks and credit go to Randall Munroe, the creator of xkcd. You can find the original of this comic online here.

           

Quality process meme

Once again trying my hand at memes. Let me know if you think it works.

This one follows on from the discussion in this post a couple of years ago.

   

Quality Auditors: "What my friends think I do"

It has been more than two months since I posted anything light-hearted here. Time for something funny.

Here is a meme that I created many years ago, back when "What my friends think I do" was actually current across the Internet. I hope you like it.

      

Quality policy meme

You may remember I wrote about Quality Policies in a post last year. Then a couple of weeks ago, Kyle Chambers and I discussed them on his #QualityMatters podcast. Here I take one more pass, this time in meme format.

Feel free to let me know if you think I should stick to essays, and leave memes for people who are funnier and more creative ....  😀

Lizard People and the Context of your Organization (My first podcast!)

Last week I sat down with Kyle Chambers of Texas Quality Assurance to record my first-ever podcast. We had a wide-ranging discussion: the original idea was that we would talk about Context of the Organization and how your COTO analysis can benefit you. But while we were at it, we also talked about how to design your QMS, why policy matters, how to avoid common pitfalls in auditing ... and how the world is secretly run by intergalactic space lizards!

Check it out.

You can find the podcast version here: #QualityMatters episode 147.

Or there's a version that also includes video, which you can find here: 

Leave me a comment to let me know your thoughts!

Asking about goals

After a discussion like the one that stretched across the last four posts, maybe we can afford to take a minute for something more light-hearted.

The following conversation really happened, in an audit where I was the company's internal Quality representative, accompanying an external auditor on his interviews.