Doesn't anybody know how to play this game?

Quality professionals have to know at least something about a lot of different fields. 

• We don't have to be logicians, but we have to know enough logic to work through a 5-Why analysis or to build a cause-and-effect diagram. 
• We don't have to be statisticians, but we have to know enough statistics to manage processes with a control sheet, or to sort failures with a Pareto chart. 
• We don't have to be experimental scientists, but it helps if we know something about the design of experiments.

We also have to understand organizational behavior and management. Most of the improvements we identify can't be implemented without the help of other people, generally people who don't report to us. So we have to know when and how to introduce changes so they can slide into place with the least disruption. It helps to develop a feel for the organization we work in, so that our colleagues in other departments are disposed to support us rather than to block us. 

That's why I was so startled at the very casual way I learned that ASQ had recently updated the procedure to elect leadership committees (or "Boards") for local geographical sections.

• I first heard an announcement (with no special urgency) that there would be a training class on the updated procedures on July 15. I was going to be out of town that day, so I made a point to listen to the recording as soon as I got back.
• Sure enough, I listened to the recording on July 23. It outlined a complex new procedure but specified no deadlines.
• That evening there was a general leadership meeting, and someone asked what the deadlines were for the new procedure. No one in the meeting knew, so the organizer took an action to find out and let us know.
• Only the next day, on July 24, did the meeting organizer send us a screenshot that the first deadline for the new procedure had been on July 8—more than two weeks earlier!

The discovery that we were already late caused a little consternation among our regional leadership. Our contact in leadership found and supplied two documents to clear up the confusion. Unfortunately they disagreed. One said to hold elections for two weeks, starting in October, with a slate of candidates compiled in August. The other said to hold elections for four weeks, starting in September, with a slate of candidates compiled in July.

Some time later I tracked down a copy of the procedure document itself. Turns out the new version had been released back in February, and the first deadline for the election process is actually July 1. But the first time anybody was told about it was in a training class two weeks later that didn't specify deadlines. As I say, the notification was very ... relaxed.

OK, you might say, so it was a simple bureaucratic mix-up. No big surprise. These things happen all the time. Why should it matter to you so much?

And my real answer is an emotional one, not a reasoned one. But we're ASQ! We're supposed to know better than that. We're supposed to be better than that!

But are we? On reflection it's not so obvious, for at least two reasons.

First, ASQ is an organization like any other. It has to hire staff, like any other. ASQ's members are Quality profes­sionals, but there is probably no require­ment that the Society's profes­sional staff all spend a certain number of years in the Quality trenches before coming onboard. Such a require­ment would be, at the very least, highly unusual; it would also force ASQ to pay well above market rates for office staff. 

Second, even seasoned Quality professionals sometimes make bone-headed mistakes. I know I have.* What we hope for, when working with Quality professionals, is not that they be perfect, but that they be ready to examine their own mistakes openly and dispas­sionately in order to learn from them. And naturally we hope also that they bring with them the memory of past lessons learned, perhaps in the form of a checklist or procedure, so that they don't have to learn from any particular failure more than once. But to ask that they never fail is simply to ask too much.

Maybe ASQ can carry out a root-cause analysis on the mistakes in this roll-out, and then assign long-term corrective and preventive actions. In other words, maybe ASQ can use the current muddle to improve the process for handling updated procedures. That kind of outcome could turn today's jumble and disorder to advantage. Let us hope.

__________

* Tip of the hat to the memory of the great Graham Chapman, in "The Mouse Problem."     

           

Can a Checklist Replace Your Audit Program?

A couple days ago, Quality Magazine published my article, "Can a Checklist Replace Your Audit Program?" It's their article now so I won't post the text of it here, but you can find it by following the link. I hope you find it useful! 

 

Quality in hard times

In last week's post, I remarked on the risk that mismanagement can teach employees not to care about the quality of their work: specifically, if employees are penalized for doing good work, they will quickly learn not to repeat the experience. A reader commented that this outcome doesn't even require active mismanagement; the simple failure to reward good work can do damage by itself. She wrote that the absence of any "consistent opportunity for reward can create a 'well, why bother then?' atmosphere." And of course she's right. I suspect many of us have experienced contemporary examples of exactly this dynamic at one point or other. In further discussion, she gave me a concrete historical example from the history of the British Royal Navy, whose officer corps had become so crowded during the nineteenth century that no further promotions were possible. Nor were there a lot of wars for the Navy to fight, once Napoleon was defeated in 1815. So with no chance for fighting and no hope of promotion, many officers stopped caring about their work and the corps began to rot.  

'What comes next? They become careless of themselves; most of them take to drink, more or less, for that is a habit almost impossible to wholly abstain from, while leading that kind of monotonous life; from that they become despised by their seniors in the service'.... The diary of a seaman in the Leander in the Pacific in 1863-5 notes seven officers court-martialed for drinking, not counting the chief engineer who died of it.*

Not officers, but you get the idea.

But this observation raises a further question. What does reward mean? Are rewards always money? If yes, we have a problem, because no company can count on consistent profitability, year in and year out. One year, somehow, times are going to be tough. One year, somehow, there won't be any extra cash on hand to distribute rewards for even the best service. And it would be painful to think that hard times might destroy the incentive for quality work.**

I have worked for companies that hit hard times, and my experience is that we continued to do good work. But everyone's case is a little different. There were a couple of people who were just there for the paycheck, or who had financial obligations that couldn't wait: they left promptly for greener pastures. And there were a couple who were laid off as the company retrenched. But many of us stuck it out, and continued to do the best we knew how.

If I ask myself Why? I see a number of factors. One is that we had already built up an ésprit de corps from working together in better times, so that none of us wanted to let down our colleagues. We weren't supporting The Company (which is more or less an abtraction), but we were supporting Larry in the next office, and Frank down the hall—people with whom we might bicker about little stuff, but whose work we still respected.

A second factor is that management communicated with us openly, and—so far as we could tell—treated us fairly. They rolled out information in All-Hands Meetings, so that everyone got the same message at the same time. And they let us know, "Here's what we can do for you today; here's what we want to do for you but can't right now; here's how we plan to get there." Knowing that we were all in the same boat made a difference. Knowing that we weren't being played for suckers made us willing to do our part.  

We discussed similar points in this blog last year. Mostly people like working together, so they are willing to cut the company some slack when things go wrong. But people can think for themselves, so they want you to tell them the truth. (And they can usually tell if you are lying!) People need you to give them honesty, respect, and simple justice every single day—which is far more often more than they ask for a raise or a bonus. So if you (as manage­ment) are willing to fulfill your obligations to give your people honesty, respect, and justice—even when you can't back those up with more money—they are generally willing to return the favor by continuing to fulfill their obligations to you. 

Thus can you ask for quality work even in hard times. But only thus!

__________

* From N.A.M. Rodger, The Price of Victory (New York: W.W. Norton & Co., 2024), p. 75. I thank Terri Williamson for raising the point out of her study of the Royal Navy, and for providing the citation. 

** The point would become even more acute in case the economy as a whole were ever to enter a contraction spiral, because then employees couldn't just jump to a profitable competitor. For explorations of this possibility, see e.g. the research of J.M. Greer, for example here.     

           

Where do you start? — take two

Last week, I wrote about employee engagement—and espcially about the research that suggests a strong correlation between employee engagement and business performance. But while I was writing, my mind started playing with the idea to see where it would lead.

In the first place, while the article that I reviewed was careful to assert no more than a correlation between engagement and performance, I reflected: If there were a causal connection, which direction would it point? Is it more likely that good business performance would cause employees to feel like they understand their jobs and have the tools they need? Or is it more likely that employees who do understand their jobs and do have the tools they need will perform well in the marketplace? Pretty clearly the second one is more likely.

In the second place, the article makes it clear that the measures for employee engagement are actionable. If your employees feel neglected and unengaged, there are concrete things you can do to improve the situation: make sure they know what is expected of them, make sure they have the tools that they need, praise them when they do well, and so on.

Then suddenly I thought of an article that I posted here four years ago. At the time, I asked: if you have just taken over responsibility for the Quality system in an organization that needs reform, where do you start? At a practical level I stand by what I said then, that internal system audits are the only way to get objective information about what's missing. But if you have several parts of the organization that need attention, where should you focus first?

Based on the Gallup results, I think the answer may be: Management.

After all, IF it is at least plausible that improved engagement drives higher quality work and improved results, and IF management can take concrete steps to improve engagement, THEN doesn't it seem likely that trying to improve performance without those concrete steps by management will be an uphill battle all the way? We have already seen that treating your people with justice and respect can be an important form of preventive maintenance to make sure the organization functions the way it should. So it stands to reason that if you want to reform or reinvigorate an organization, those are the points you should check first.

Now I want to draw a distinction. If you have to address "Quality issues" throughout an organization, that can mean two different things.

• Case 1 is a case like the Riverville plant that I discussed last month. The people there wanted to do a good job, but they had a different definition of how to get there than the managers of Octopus Enterprises who had recently acquired the place.
• Case 2 is more severe. It's the kind of place where people don't care about doing a good job, and just want to get through the day.  

In Case 1 people still have the right attitude, so I think the core issue is one of communicating how the new system will help them achieve the same good results they are already trying to achieve with the old system. Ideally, the new system will make things easier, though as we saw in the case of Riverville, the transition can be tough.

But Case 2 is brutal. And there I have to ask, How did things ever get this bad in the first place? My assumption—and I admit it is just an assumption—is that most people start off wanting to do a good job. If that's no longer the case in a particular workplace, my first guess is that the employees have been taught not to care about doing good work by management practices that penalize them when they do. The other possibility is that management inadvertently hired one or two bad apples, and then failed to remove them before they succeeded in corrupting the others. Either way, it is unlikely that the rest of the organization will reform until things are improved inside management.

And really, ... how would you reform an organization's Quality focus without insisting on Quality in management? I think it would have to be like the old Internet meme: The beatings will continue until morale improves.

CAVEAT: I have never had sole responsibility for turning around the Quality performance of an entire organization. And I know that ideas which make sense at my desk are often wrong in the real world. 😀 So I might be wrong about this too. If you have experience that leans in a different direction, please leave a comment to tell me about it!  

           

How much does engagement matter?

We’ve talked about “engagement of people” as a fundamental Quality principle, and at a superficial level it sounds obvious. Disgruntled employees are distracted by being unhappy, so they are likely to turn out bad work; happy employees are not distracted, so they can focus on what they are doing and turn out good work. On the other hand, the world is full of simple formulas that sound right but don’t stand up to empirical investigation. What about this one? Is it true? Or is it just an agreeable fairy tale?

In fact, the correlation between employee engagement and business outcomes has been the subject of fairly extensive research over the years. Last month I ran across a recent study by the Gallup organization, called “The Relationship Between Engagement at Work and Organizational Outcomes: Q^12® Meta-Analysis: 11th Edition” This report dates from 2024, and it is part of an ongoing project by Gallup. There are references in the notes to how the results have shifted since the previous edition. (Answer: not much.)

Briefly, what does the report say?

• The authors define a way to measure employee engagement, and then they identify eleven different ways to measure business performance. 
• They review a whole lot of companies, covering over three million employees. 
• And they find a consistent correlation: the companies whose employees feel the most engaged also have the best performance.

Why do you care?

• I’m glad to see the correlation with performance. Yes, I expected it. But it’s nice to see that it’s true.
• I like the way they measure employee engagement, because the metrics are actionable. I’ll explain what I mean in a minute.

Tell me about the method

Sure. The research pulled together 736 studies across 347 organizations in 53 industries, located in 90 countries; in total they reviewed 183,806 working groups, containing 3,354,784 employees. [See p.4 of the report.] This paper itself is a “meta-analysis,” a statistical method to combine studies of different sizes in order to filter out any idiosyncrasies and make the results comparable. The study establishes correlations but does not directly address causality. [See p.17.] In some cases, the authors deliberately omitted data from 2020, when effects related to COVID-19 clearly overwhelmed the effects that they were studying. [See p.18.] Overall, the findings “show high generalizability across organizations…. [M]ost of the variability in correlations across organizations was the result of sampling error, measurement error or range restriction in individual studies.” [See p.28.]

Fine, what were the numbers?

The study rated organizations based on their employee engagement scores. Then they calculated their business performance on each of eleven measures. Then they compared the business performance of those companies with high engagement against the business performance of those companies with low engagement. And they calculated the percent differences in the two sets of rankings. Here is the key table of results: [See p.5.]

Median percent differences between top-quartile and bottom-quartile units were:

• 10% in customer loyalty/engagement
• 23% in profitability
• 18% in productivity (sales)
• 14% in productivity (production records and evaluations)
• 21% in turnover for high-turnover organizations (those with more than 40% annualized turnover)
• 51% in turnover for low-turnover organizations (those with 40% or lower annualized turnover)
• 63% in safety incidents (accidents)
• 78% in absenteeism
• 28% in shrinkage (theft)
• 58% in patient safety incidents (mortality and falls)
• 32% in quality (defects)
• 70% in wellbeing (thriving employees)
• 22% in organizational citizenship (participation)

The paper spends many pages explaining the method and the calculations, but this table is the payoff. Companies with high engagement have 78% less absenteeism than companies with low engagement? Wow. Where do I sign up?

Wait—how do they measure employee engagement?

I thought you’d never ask.

The metric is something developed by the Gallup organization over years. It’s called Q¹², and it is based on a list of thirteen questions which the employee rates from 1 to 5. Twelve of these questions are very specific and concrete; one of them is a general evaluation. Here is the list: [See p.12.]

• Q00.  (Overall Satisfaction) On a 5-point scale, where 5 means extremely satisfied and 1 means extremely dissatisfied, how satisfied are you with (your company) as a place to work?
• Q01. I know what is expected of me at work.
• Q02. I have the materials and equipment I need to do my work right.
• Q03. At work, I have the opportunity to do what I do best every day.
• Q04. In the last seven days, I have received recognition or praise for doing good work.
• Q05. My supervisor, or someone at work, seems to care about me as a person.
• Q06. There is someone at work who encourages my development.
• Q07. At work, my opinions seem to count.
• Q08. The mission or purpose of my company makes me feel my job is important.
• Q09. My associates or fellow employees are committed to doing quality work.
• Q10. I have a best friend at work.
• Q11. In the last six months, someone at work has talked to me about my progress.
• Q12. This last year, I have had opportunities at work to learn and grow.

I said above that I like this way to measure engagement, because nearly all of these points are actionable. The Gallup organization deliberately chose to ask about issues that individual managers can correct! So if someone carries out a Q¹² survey at your place of work and your department scores low in this or that area, mostly you (if you are a manager) can do something about it. Make sure your people know what is expected of them; make sure they have the tools that they need; make sure they are doing work they are suited for; praise them when they do well; and all the rest. These are part of your job as a manager anyway. The use of this metric means that if you carry out all these tasks conscientiously, you can overcome any bad score related to engagement. You can improve. And we have already seen that quality in management is a critical benefit to the whole organization. 

And in other news ….

Meanwhile, for those who thought the whole point was obvious from the beginning, take heart. In other news, researchers have carried out an expensive study to prove that the best bait for mice is cheese.*      

__________

* “During a six-month period in 1973, The New York Times reported the following scientific findings: A major research institute spent more than $50,000 [$377,390 in 2026 dollars!] to discover that the best bait for mice is cheese….” From Jerry Mander, Four Arguments for the Elimination of Television (New York: Quill, William Morrow and Company, 1978), p. 53.

           

Auditing a system that uses AI

We’ve talked before about some of the Quality issues related to the use of artificial intelligence (AI) technology, but mostly in an incidental kind of way.* But the adoption of AI by many companies seems not at all incidental. So it is fair to ask how the introduction of AI is going to affect core Quality functions. How will AI change—say—auditing?

It's a good question, and there have been a couple of attempts to answer it in the past year. Last October, Elisabeth Thaller and Jorge Bravo Carreño wrote an article "When the QMS Thinks for Itself" for Quality Progress. Then just last month the ISO 9001 Auditing Practices Group published a formal guidance paper on "Auditing a Quality Management System that uses Artificial Intelligence (AI) systems." It is perhaps not surprising that the article and the guidance paper repeat the same basic points,** and they are pretty much the same points you would make if someone asked you about the same topic. Nothing here is unexpected.  

What makes this consistency possible is that ISO 9001 is structured to apply to any kind of work. So the basic questions are always the same. What are your processes? What are your risks? What are your training requirements? And are you getting the results that you want?

The article in Quality Progress identifies five main thoughts to keep in mind when planning an audit in an organization that uses AI, and the sugges­tions in the APG guidelines can almost be mapped directly onto the same list. The five main thoughts are these:

1. Identify where the organization is actually using AI. ("AI may be hiding in plain sight.")
2. Use the process approach to identify inputs and outputs in the normal way. ("The process approach still applies—even when the process thinks for itself.")
3. Identify risks. ("Risk-based thinking isn’t optional.")
4. Identify training needs, responsibilities, and authorities. ("Competence and oversight are essential.")
5. Identify results and look for objective evidence. ("Focus on evidence, not the shine.")

And while I'm certainly not going to quote all ten pages of the guidance document in a blog post—you can download the whole thing for free by using the link above—let me quote just a couple of items that fit under each of these five headers, to show you the overlap.

Identify where the organization is actually using AI.

• When preparing the audit, the auditor should:
• Identify whether the organization uses any AI system type for any of its processes within the scope of the QMS.
• Determine the specific function(s) of these processes within the QMS, and
• Recognize assigned responsibilities in relation to these processes. [§2, p.4]

Use the process approach.

• Where AI system(s) are used within operational processes, has the organization ensured that their use supports the achievement of intended results?
• Are such processes monitored, and are changes related to the use of AI system(s) controlled?
• How does the organization ensure the consistency, correctness, and reliability of AI outputs? [§3, cl. 8, p.7]

Identify risks.

• Has the organization considered the implications of the use of AI system(s) when determining its internal and external issues?
• Has the organization identified statutory and regulatory requirements in relation to the use of AI systems, such as those related to data privacy and information security? [§3, cl. 4, p.4-5]
• Has the organization determined and addressed risks and opportunities related to the use of AI system(s)? [§3, cl. 6, p.6]

Identify training needs, responsibilities, and authorities.

• Are adequate resources available to maintain and update AI system(s), including access to technical expertise, data quality management, and cybersecurity support?
• Is the organization able to demonstrate that personnel who use, manage, or oversee AI system(s) meet the determined competence requirements? [§3, cl. 7, p.6]
• Does top management take accountability for ensuring that the AI system(s) within the QMS support its effectiveness and achievement of intended results? [§3, cl. 5, p.5]

Identify results and look for objective evidence.

• Are customer satisfaction trends evaluated for potential impacts arising from AI-enabled processes or interactions?
• Is internal auditing addressing the effectiveness of AI-influenced processes?
• Is information related to the performance and effectiveness of AI systems an input into management review? [§3, cl. 9, p.8]
• Are nonconformities or complaints involving AI system outputs recorded, analyzed, and addressed? [§3, cl. 10, p.8]

There shouldn't be anything shocking in this list. One of the strengths of ISO 9001 is precisely its flexibility. And that flexibility comes from seeing all kinds of work through a lens that highlights certain common features. AI is just one more topic viewed through the very same lens. This is why I said above that you would have come up with the same points if you had taken the time to work through the standard in detail.

But most of us are too busy with our day jobs to do that, so it's convenient that the Auditing Practices Group has done it for us. The next time you have to audit an organization that uses AI in its management system, check out the guidelines document to see if any of the suggestions help you. 

__________

* In this post, I asked what the word “Quality” really means in the context of AI. In this one, I discussed how AI tools might be able to enhance training records. 

** Partly this outcome is unsurprising because the same person—Elisabeth Thaller—was the lead writer for the article and the chair of the committee that wrote the guidance document. Chris Paris published an article taking the APG committee to task because most of them have no special background in AI. But it appears from her LinkedIn profile that Thaller does have at least some background.        

           

Sampling bias

A friend forwarded this cartoon to me a couple of weeks ago. It seems to come from the website Sketchplanations, drawn by Jono Hey. You can find the original here.  

Of course statistics are central to the Quality discipline, so we need to understand how sampling works. And I think the cartoon makes a serious point in that regard, rather deftly.

It's also funny, and high time too!