Can a Checklist Replace Your Audit Program?

A couple days ago, Quality Magazine published my article, "Can a Checklist Replace Your Audit Program?" It's their article now so I won't post the text of it here, but you can find it by following the link. I hope you find it useful! 

 

Auditing a system that uses AI

We’ve talked before about some of the Quality issues related to the use of artificial intelligence (AI) technology, but mostly in an incidental kind of way.* But the adoption of AI by many companies seems not at all incidental. So it is fair to ask how the introduction of AI is going to affect core Quality functions. How will AI change—say—auditing?

It's a good question, and there have been a couple of attempts to answer it in the past year. Last October, Elisabeth Thaller and Jorge Bravo Carreño wrote an article "When the QMS Thinks for Itself" for Quality Progress. Then just last month the ISO 9001 Auditing Practices Group published a formal guidance paper on "Auditing a Quality Management System that uses Artificial Intelligence (AI) systems." It is perhaps not surprising that the article and the guidance paper repeat the same basic points,** and they are pretty much the same points you would make if someone asked you about the same topic. Nothing here is unexpected.  

What makes this consistency possible is that ISO 9001 is structured to apply to any kind of work. So the basic questions are always the same. What are your processes? What are your risks? What are your training requirements? And are you getting the results that you want?

The article in Quality Progress identifies five main thoughts to keep in mind when planning an audit in an organization that uses AI, and the sugges­tions in the APG guidelines can almost be mapped directly onto the same list. The five main thoughts are these:

1. Identify where the organization is actually using AI. ("AI may be hiding in plain sight.")
2. Use the process approach to identify inputs and outputs in the normal way. ("The process approach still applies—even when the process thinks for itself.")
3. Identify risks. ("Risk-based thinking isn’t optional.")
4. Identify training needs, responsibilities, and authorities. ("Competence and oversight are essential.")
5. Identify results and look for objective evidence. ("Focus on evidence, not the shine.")

And while I'm certainly not going to quote all ten pages of the guidance document in a blog post—you can download the whole thing for free by using the link above—let me quote just a couple of items that fit under each of these five headers, to show you the overlap.

Identify where the organization is actually using AI.

• When preparing the audit, the auditor should:
• Identify whether the organization uses any AI system type for any of its processes within the scope of the QMS.
• Determine the specific function(s) of these processes within the QMS, and
• Recognize assigned responsibilities in relation to these processes. [§2, p.4]

Use the process approach.

• Where AI system(s) are used within operational processes, has the organization ensured that their use supports the achievement of intended results?
• Are such processes monitored, and are changes related to the use of AI system(s) controlled?
• How does the organization ensure the consistency, correctness, and reliability of AI outputs? [§3, cl. 8, p.7]

Identify risks.

• Has the organization considered the implications of the use of AI system(s) when determining its internal and external issues?
• Has the organization identified statutory and regulatory requirements in relation to the use of AI systems, such as those related to data privacy and information security? [§3, cl. 4, p.4-5]
• Has the organization determined and addressed risks and opportunities related to the use of AI system(s)? [§3, cl. 6, p.6]

Identify training needs, responsibilities, and authorities.

• Are adequate resources available to maintain and update AI system(s), including access to technical expertise, data quality management, and cybersecurity support?
• Is the organization able to demonstrate that personnel who use, manage, or oversee AI system(s) meet the determined competence requirements? [§3, cl. 7, p.6]
• Does top management take accountability for ensuring that the AI system(s) within the QMS support its effectiveness and achievement of intended results? [§3, cl. 5, p.5]

Identify results and look for objective evidence.

• Are customer satisfaction trends evaluated for potential impacts arising from AI-enabled processes or interactions?
• Is internal auditing addressing the effectiveness of AI-influenced processes?
• Is information related to the performance and effectiveness of AI systems an input into management review? [§3, cl. 9, p.8]
• Are nonconformities or complaints involving AI system outputs recorded, analyzed, and addressed? [§3, cl. 10, p.8]

There shouldn't be anything shocking in this list. One of the strengths of ISO 9001 is precisely its flexibility. And that flexibility comes from seeing all kinds of work through a lens that highlights certain common features. AI is just one more topic viewed through the very same lens. This is why I said above that you would have come up with the same points if you had taken the time to work through the standard in detail.

But most of us are too busy with our day jobs to do that, so it's convenient that the Auditing Practices Group has done it for us. The next time you have to audit an organization that uses AI in its management system, check out the guidelines document to see if any of the suggestions help you. 

__________

* In this post, I asked what the word “Quality” really means in the context of AI. In this one, I discussed how AI tools might be able to enhance training records. 

** Partly this outcome is unsurprising because the same person—Elisabeth Thaller—was the lead writer for the article and the chair of the committee that wrote the guidance document. Chris Paris published an article taking the APG committee to task because most of them have no special background in AI. But it appears from her LinkedIn profile that Thaller does have at least some background.        

           

Make your matrix certificate work

Last week I introduced the concept of a matrix certificate, and I thought it could be useful if I explained the approach in a little more detail. Mostly the idea is self-explanatory, but it is worthwhile to keep one or two points in mind so they don't trip you up.

When ISO first introduced its management system standards, each certificate was tied to a specific location where specific work was done. So the certificate was printed with an address and a scope statement, as in the example to the left.* 

But many organizations have more than one location, and often the processes are identical from one site to the next. There is not a lot of variety, for example, between this McDonald's and that one. So organizations began asking their Certifying Bodies** if there were any way to group multiple locations under a single certificate. Since the CB's, for their part, had to manage a lot of redundant information for all of these sites, they were keen to agree. And so CB's began to introduce the "matrix certificate."

A matrix certificate works just like a standalone certificate, except that the address and scope are replaced by a matrix: Site 1 does this kind of work, Site 2 does that kind of work, and so on. There is no requirement for a literal matrix on the certificate itself, so long as the information is there. A matrix is a convenient format when there is a large number of sites to consider. When there are only two or three, the CB might simply use paragraphs, as in the example to the right immediately below.***  

The other place you see a genuine matrix is in the internal paperwork that supports the certification: at the CB, and at the organization itself. These internal matrices look something like the one that I show at the bottom of this post, and they correlate the infor­mation needed to manage the audit program. This starts by listing the sites and their scopes or func­tions, just like on the certif­icate. But then the internal matrix also tracks how many people work at each site. This is because there is a formula used by all the CB's to determine how many audit-days**** they have to schedule at each location. The formula uses both number of employees and type of work as inputs to the calculation: a big plant needs more audit-hours than a small plant, and manufacturing requires more attention than HR or sales. Finally, the matrix should track how long the audit cycle is for each site. Normally with a multi-site certificate, the company headquarters is audited every year; also, manu­fac­turing is likely to be audited every year. The example I give below has one site for each of those functions. But then it also shows three sites that mostly do engineering or product design, that follow exactly the same procedures, and that are therefore more or less interchangeable. In order to reduce the overall audit costs (and to simplify scheduling), the CB has put each of these on a three-year cycle, so that they rotate. In any given year the CB plans audits in three locations; but over the three years before recertification, they visit all five sites.

On the whole, using a matrix certificate is a good way to control your certification expenses if you are in a situation where it works for you. But of course there are risks. The main risk is that one of the critical entries in your matrix will change: for example, you might have heavy layoffs at one site, or (conversely) hire a whole new department. Either way, you have to make sure to report the change to your CB in a timely way, so that they can recalculate your audit days. One way or another, this might change your plans for the year.

Or you might move functions from one site to another. Since each function makes its own contribution to the calculation of audit days, this move might make a difference to the result even if the number of employees involved is not high. A specialized technical function, for example, might employ only a few people; but it might require a lot of Quality equipment to make sure it operates correctly, and in some cases all that infrastructure might have to be audited.

Finally, there is a risk that two different sites are supposed to be doing the same work according to the same procedures, but one site decides to implement local improvements without telling the other. I have seen this happen, where an organization had two Distribution Centers. One was more or less a glorified shipping dock at the far end of a building that did a lot of other work; the other was a pure Distribution Center with no other functions. Back when the organization first codified their written procedures, someone wrote a single set of logistics documents and told both organizations to use them. But in the intervening years, the Site Manager of the standalone Distribution Center brought in a lot of equipment to modernize his operations. It was exciting to visit, because every time I arrived they had improved something new. But somehow the word never got back to the first operation. I had workers from that one tell me with a straight face, "We have exactly the same procedures they have there," when there was no longer anything in common. And I had to explain, "No you don't. And at the very least your documentation has to be updated to reflect that reality."

Note that when changes like this take place, it can undermine the logic behind a matrix certification. The justification for auditing Site 3 in place of Site 4 and Site 5 is that they follow the same procedures in all three sites so you'll see the same evidence anyway. Once that is no longer true, it is harder to keep down the number of audits.  

It is also true that breakthrough improvements on such a scale are mostly the exception rather than the rule. And if there is a chance for your organization to make breakthrough improvements, having to replan your audit program is a small price to pay!

__________

* I found this illustration online by searching for "sample iso 9001 certificate." I have no professional, personal, or financial connection to the organization in question. 

** A Certifying Body (CB) is the registrar that send out your external auditor, manages the audit paperwork, and issues your certificate. There are lots of CB's in the world. If you look at the certificates that I posted above as examples, the first is from TÜV SÜD, and the second is from DEKRA. These are both well-known CB's. Just by coincidence, I have never worked with either one.  

*** Again, I found this illustration by searching online and have no connection with the company. 

**** One audit-day means one full day of auditing by one auditor. So four audit-days might mean one auditor for four days, or two auditors for two days, or one auditor for three days plus another who joins him for only one of those days, or any other combination that adds up to four.     

           

How do you certify a big corporation?

Last summer, in the middle of a podcast about something else, Kyle Chambers raised a question: how is it possible to certify a really big company to a standard like ISO 9001? His point was that really big companies have so many parts that they can't all play together. (I mention the podcast in this post here.) Kyle might have meant the question rhetorically, but of course he is right. If you have scores of locations spread across multiple continents—engaged in dozens of lines of work—how can you possibly coordinate them all into a single management system? How can you possibly certify anything so complex?

You can't. So you break it in pieces.

More exactly, you divide your really big company into sub-units of a useful size, and then make each sub-unit responsible for its own certification. Then in your sales and marketing literature you speak of the company as a unified whole, announce that "Con­glom­erated Enterprises has been proudly certified to ISO 9001 since ...," and follow with the date of the first certificate that came through.

How big is "a useful size"? It really depends. I've seen it done several ways.

In general, there are two competing pressures at play in determining the right size for an entity to serve as the scope of certification.

• On the one hand, you don't want the scope too narrow, or you'll have to pay for too many audits. In other words, if three offices that are all in the same city all do exactly the same kind of work, you can save some money by letting them share a certificate. You pay for less overhead at the Certifying Body, and you pay for fewer audits (because all three offices are doing the exact same thing). This all looks good on a balance sheet.
• On the other hand, you don't want the scope too broad, because there's always the risk that someone goes crazy one day. If you have a hundred locations all sharing the same certificate, and someone in a tiny office way out in Far Foodle starts violating an important policy, the next auditor might rate it as a Major Nonconformity. Then that Major might put the certificate at risk for all hundred locations!

Where you draw the line while balancing these two imperatives depends a lot on your organizational culture. I've worked for one multi-site operation whose corporate ethos was entrepreneurial, and who had created a number of sites by acquisition. So there was a distinct chance that two sites might not be on the same page. For that company, each site was strictly responsible for its own certifications. The company had a blanket contract with a CB, because they got a volume discount. But inside that blanket contract, we were on our own. I made the arrangements for my location, but not for the others; I worked with the auditor and tracked the findings for my location, but not for the others. We never had a crisis where one location risked losing certification, but the company's management took no chances.

But then later I worked for another company that bundled eight sites across the United States all into one certificate. We supported the same product line, and we were all in the same geographic region. So the company decided that was enough commonality that it made sense for us to share a certificate, and we could sink or swim together. I made the arrangements for eight locations, tracked findings for eight locations, and flew around the country a fair bit to support audits when they happened. 

This meant I carried out eight internal audits a year, but not nearly so many external audits. We had a "matrix certificate," which meant that the CB did a sampling every year: they always audited headquarters, and they always audited our one factory, but then they would pick one or two other sites randomly and leave it at that. Over the years, of course, they ended up visiting every site; but every spring I had a long discussion with their scheduler to agree where they should go. 

"You visited A last year, and you visited B the year before, but you've never audited C. How about seeing them this year?"

"Wait, we've never been to C? What do you even do at that site?"

"It's basically the same work as B, so I don't expect any significant findings. But you haven't been there yet, so it might be good to visit."

"Sure, I guess. How many people work there ...?"       

As long as the processes and systems really are common across locations, this can be a useful way to proceed. Next week I'll say a little more about how matrix certificates work.

       

Feedback on Santa's audit!

Wow! I got a lot of comments on last week's post about how far Santa Claus complies with ISO 9001. Thanks especially to Pia Hamrin, Dawn Ringrose, Petro Shoturma, and Jeremy Panitz for their detailed feedback!

What interested me was the remarkable level of unanimity among the responses. Of course each commenter had a unique perspective. But since Santa Claus is fictional,* any commonality among the responses must be traceable to the second half of the discussion—namely, to the expectations that grow naturally out of working with ISO 9001. Those of us who work with the standard regularly have come to know that some clauses need a lot more attention than others, because the impact they have on organizations is so consequential.

Even if you aren't worried about Santa Claus, ask yourself whether these same clauses aren't important topics in your organization.

Common concerns

It was no surprise to me that the two topics which attracted the most attention were Internal controls and Complaint handling. Businesses differ a lot: organizational contexts can be very different, as can the quality of customer requirements. For some companies those topics might require a lot of attention, while for others the same topics might manage themselves. But internal controls always matter—to everyone. And without complaint handling, you have no feedback on your operations to make sure your customers are satisfied. So nearly everyone agreed to focus on those areas.

Naughty or nice?

Everyone knows that Santa Claus checks whether each child is "naughty or nice," but how does this happen? Dawn's reply assured me that Santa uses a proprietary algorithm. But Pia asked how that algorithm is validated, and whether the sources are documented? Jeremy suggested that Santa consults with children's deities around the world to make his determinations.

Complaint handling

Petro and Pia both raised the issue that there is no publicized mechanism for complaint handling. Jeremy suggested a resolution to this topic, though, by arguing that "If the customer isn't satisfied with the gift they got from Santa Claus, the gift did not come from Santa Claus."**

Less worried

By contrast, nobody was very worried about document retention. Dawn suggested that the children's letters must be fully digitized, and nobody else took up the subject. I think this lack of concern reflects our common experience that yes, of course document retention is important; but on the other hand documentation findings are generally the weakest kind of finding in any audit.

Similarly, Dawn and Jeremy both assured me that internal audits were done by impartial elves, where "no one audits their own toy line." And I think we have all experienced that even in organizations where the Quality Management System is informal or not very mature, it is generally possible to find people who can treat the process objectively, and who can therefore do reliable audits.

Unique observations

Then there were the unique observations, that pushed the discussion in an unexpected direction.

Jeremy argued that Santa Claus is thought of as a toymaker, but that the real focus of his design expertise is in logistics. In commenting on clause 8.3.2 (Design and development planning), he explains, "Santa has only 24 hours to deliver toys and other gifts all across the world. That means he's got to know the weather patterns. So his design and planning is less on toys and gifts and more on safety and delivery of the goods." Jeremy doesn't use this example, but I assume he would compare Santa (in this respect) to McDonald's, who didn't redesign the hamburger but revolutionized how it was delivered to the customer.

For her part, Dawn answered the question about children changing their minds by enunciating the rule, "Last Wish Wins." But that rule means that the audit plan has to spend a lot more time checking scrap and rework under clause 8.7 (Control of nonconforming outputs). As long as the only topic was the elves' workmanship, we could assume that the elves are magical and don't make mistakes; so overall scrap should be minimal. But if children can change their minds late in the game, there is sure to be scrap as earlier gifts are replaced with later ones. Thus does any answer in one part of the audit open up new trails in another part—and this, too, is a familiar experience to all of us.

Naturally I don't mean to suggest that any clause is unimportant. In the right (or wrong) circum­stances, any clause can trip you up. But I found it interesting and reassuring that the feedback on this topic aligned so closely with my own sense about where the critical risks are in any ISO 9001 implementation. Again, even if the North Pole doesn't interest you much, your own organization does. And it is worth looking over where your own risks are.

__________

* Or at any rate I'll assume so for the purposes of this post! 

** It's an interesting idea, to which I note only that, however this principle might work for Santa, it's not available for the rest of us.     

         

Does Santa Claus comply to ISO 9001?

Years ago, when my kids were little, they would spend an hour or so every Christmas Eve watching the NORAD Santa Tracker. You've probably heard the story behind how NORAD tracks Santa's progress through the skies every year. And indeed every year there are press releases updating the public on the current status of the Santa-tracking program.

These days I'm out of touch with the latest developments in aviation, but I'm correspondingly more interested in international standards. So today I find myself wondering, How far does Santa Claus comply with ISO 9001?

It's not an idle question. Last year an investigative reporter went to the North Pole—at least, that's what the article said—to check the operations of Claus Enterprises against OSHA's Top 10 workplace-cited standards for 2024. The results weren't pretty, though Santa himself put the best face on what he called "“an unfortunate misunderstanding," and pledged a prompt corrective action plan.

But even if safety issues have been addressed, what about Quality? Millions of children rely on Santa for presents each year. Can he really be counted on to come down all the right chimneys? Does he stand behind his work, or does he just fly away "like the down on a thistle"?

Up till now I have been unable to find an ISO 9001 certificate* for any part of Santa's operations, neither his workshop nor his delivery service. Partly the problem may be geographic: I'm not aware of any formal Certifying Body with jurisdiction over the North Pole. All the same, surely it's a good idea to do an advance check—like a pre-audit—to find which clauses need the most attention. So Santa, if you are reading this, treat it as a little free consulting, to help you get positioned.**  

These, then, are some clauses of the standard where I would expect discussion in an audit:

4.1 Understanding the organization and its context

Ever since the climate change amendment last year, this clause has included the requirement, "The organization shall determine whether climate change is a relevant issue." Well if the Polar ice cap is about to melt, that sounds relevant to me! Does Santa have plans to relocate? And where?

4.2 Understanding the needs and expectations of interested parties

How often are these needs and expectations reviewed or updated? I assume that the desires of little children have been broadly similar for centuries. But what about other players in the same market? Does Santa face competition from online retailers, or has he worked out some kind of deal?

5.2.1 Establishing the quality policy

What exactly is Santa's Quality Policy? Does he strive for 100% satisfaction? We know that every year some children are disappointed; so if the policy calls for 100%, there might be a problem. And what if a gift is defective, or has to be returned? Does Santa replace it himself, or does he require the parents to do the work? (See also clause 8.4 Control of externally provided processes, products and services.) 

7.5.3 Control of documented information

Children write letters to Santa from all over the world. Are those kept for future reference, in case there is some dispute over what a child really requested? And in what form? (For example, are the original letters kept on paper, or are they digitized?) If the letters are kept on paper, how are they preserved in the harsh Arctic climate?

A second topic is the retention policy: how many years is a letter kept? Santa is always said to be wise and to have a long memory; but the storage requirements for each new year's batch of letters must be enormous. Are they kept for a year? For a decade? For centuries? How are they indexed? How easily can they be retrieved?

8.2.3 Review of the requirements for products and services

There's another kind of control needed for the children's letters: regular review. Some children might request toys that are dangerous, or that are illegal in their place of residence. How is this review conducted, and how are conflicts resolved?

In addition to all the other reviews, subclause 8.2.3.1(c) requires a review of "requirements specified by the organization." Presumably this includes comparing all requests against the List of Who's Naughty and Nice. But how is that list managed and kept up to date? And how often is a naughty child really given a lump of coal instead of some other present? (I can think of children over the years that I thought had earned a lump of coal who never seemed to get one!) 

8.2.4 Changes to requirements for products and services

What about children who change their minds about what they want, and who send multiple letters? Is there a tracking system to allow the second letter to be filed with the first one, or to replace it completely?

8.3.2 Design and development planning

What is the ratio of customized gifts to off-the-shelf toys? The publicity photos of Santa's Workshop look like they are mass-producing toys, but there must be a lot of specialized requests as well. How is this handled? (See also clause 8.5.2 Identification and traceability.)

8.4 Control of externally provided processes, products and services

How much of Santa's work is subcontracted to local merchants, or to parents? And on what terms? Does Santa still get all the credit, in all cases?

8.5.2 Identification and traceability

How far is lot-traceability a relevant issue? (Note that if every job were custom—see also clause 8.3.2 Design and development planning—then lot-traceability would not be a requirement.)

8.5.4 Preservation

The North Pole has a cold, harsh climate. The same is true for much of the lower atmosphere. How does Santa preserve presents in good condition, and keep them from being damaged by the harsh environment?

8.6 Release of products and services

Does Santa keep records of every gift, and are those records filed with (or cross-indexed to) the corresponding request letters? This raises some of the same questions we discussed above under clause 7.5.3 Control of documented information. 

9.1.2 Customer satisfaction

What is Santa's current level of customer satisfaction? Has it been trending up or down? In case of any downward trend, has Santa chosen to implement corrective actions, and what do those look like?

9.2 Internal audit

Who does internal audits for Santa's enterprises? Are the auditors selected in a way that "ensure[s] objectivity and the impartiality of the audit process"? Is the scope of the internal audits the same as the scope of the enterprises, or are there some areas that are off-limits to the auditors? Can I see the results of the latest full audit?

9.3 Management review

Can I see the records of the latest management review? Has Santa ever introduced a major change to the operations as a result of findings in management review?

• If yes, what was it?
• If no, then is the organization really using the management review, or is it just a dog-and-pony show?

10.3 Continual improvement

Tell me about the latest continual improvement initiatives.

Having said all this, I have to admit that it would be the chance of a lifetime to audit Santa's Workshop at the North Pole. An operation with such vast scope, that has to achieve such delicate precision in its outputs, would be amazing to review.

So Santa—again—if you are reading this and want someone to do a test-audit before you call a regular Certifying Body, leave a comment on this post with your contact information and I'm sure we can work something out.

And to everyone else, have a jolly day, a delightful holiday season, and a very happy New Year! 

__________

* I did find one article here that claims Santa Claus is certified to ISO 9001 and a host of other standards as well, but it offers no links to objective evidence.  

** And with a little luck, giving away free consulting might be enough to move me from Naughty to Nice!     

 

Priorities in repair

There are costs when you develop a product. One of the easiest to forget is the cost to repair it later. I don't mean when the customer breaks something by mis-using it; you can charge for repairs like that. I'm thinking of the ordinary cost of warranty repair. What happens when the product breaks down through normal wear and tear, inside the warranty period? Or in a product powered by software, what happens when the customer has an unexpected use-case, and discovers a brand-new bug?

You fix it, of course. All reputable companies do. But that costs time and money. In the high-tech case, it may require the attention of very talented engineers.

Long ago—and yes, it seems like it was in a galaxy far, far away—I worked for a startup that had a problem. We were developing a NewProduct™ that was late. It relied on a very specialized technology. The whole work rested on just one or two of our engineers.

To make things worse, we had recently released an OldProduct™ that relied on similar technology. When problems were reported, they were routed to the same engineers to fix. But every hour those engineers spent fixing a bug in the OldProduct was one more hour they weren't working on the NewProduct. So the NewProduct continued to get later.

One day, our President had had enough. In a moment of frustration, he fired off an email that said, "Effective immediately we will stop fixing bugs on OldProduct until NewProduct is released!" Clearly he hoped that this message would set the right priorities for the company, so that we could finally make some progress.

By coincidence, our annual ISO 9001 surveillance audit was scheduled for just a month later. And you can probably guess what happened. At one point the auditor asked me, "How do you respond to customer complaints?" In the course of the discussion, I mentioned the recent email about OldProduct.

Our auditor took a very dim view of this story. Had we really stopped fixing all bugs on OldProduct? He reminded me that clause 8.5.2 of ISO 9001:2000* specifically requires the organization to review nonconformities, including customer complaints. If we really had stopped fixing all reported bugs, he would have to write that up as a Major process nonconformity in the audit. 

I hadn't worked closely with this product, so I called the Customer Service Department. They explained that no, we hadn't stopped fixing bugs. When he sent that email, our President was just blowing off steam. What we had done in reality was to set up a filtering system to prioritize bug-fixing.

• If OldProduct genuinely didn't work, we fixed the problem.
• But if OldProduct actually did work, but just not the way the customer wanted it to work—this was by far the majority of the cases!—we took the customer's improvement suggestions and put them all in a big pool to be addressed when we had time. This meant we would look at them after NewProduct finally got released.

The Customer Service folks brought objective evidence to prove that yes, one or two bugs really had been addressed since President's email, so the filtering system was working. Our auditor downgraded his finding to a suggestion that it would be helpful if everyone in the organization had the same understanding about this policy, and the rest of the audit was uneventful.

This all took place long ago, but I remember it because the distinction is important. 

• On the one hand, there are things—like repairing genuine flaws under warranty—that you simply have to do: all responsible companies do them, and they are required by external standards like ISO 9001. 
• But on the other hand, you often have broad latitude to prioritize your work in a way that makes sense based on current conditions.

This flexibility can make all the difference. 

__________

* Yes, we were still working to the 2000 edition of the standard. Remember this was many years ago!   

    

Climate change and bad audits

A while ago—six weeks, now that I check—I wrote about how to audit the recent Climate Amendment to ISO 9001. I based my advice on a guidance document issued by the ISO 9001 Auditing Practices Group, entitled (unimaginatively enough) ISO 9001 Auditing Practices Group Guidance on: Auditing Climate Change issues in ISO 9001. At the time, my basic conclusion was that the requirements were not heavy or onerous.

A couple of weeks later, Kyle Chambers and Caleb Adcock of Texas Quality Assurance released a podcast on the exact same topic. (See the YouTube link below.) I usually appreciate Kyle's pragmatic approach to the ISO standard, so I assumed he would see this guidance the same way I did; but it was a while before I had time to listen to his podcast. It turns out, though, that his take is almost the opposite of mine! He summarizes the APG document as "a whole guidance document of gotchas!"

How is this possible? After listening to the rest of Kyle's podcast, I think I see what happened. Like so much in the ISO world, it all comes down to context and interpretation. Also, it seems like some of Kyle's clients have been saddled with really poor auditors. I'll explain what I mean as we go on.

Agreements

Let me start with the parts where we agree. Kyle's basic advice to his clients (around 21:00) is that you shouldn't need to do anything you aren't already doing to manage your business. But you may have to document what you are doing, and wrap it up in the language of risk management. Caleb makes the same point in different words (around 22:10) by spelling out that if you have determined that climate change has a significant impact on your business, then naturally you are going to implement steps to manage that impact so that you can keep your doors open. You'll do this with or without ISO. So just document it.

If you believe that climate change has no impact to your business at all, Kyle suggests (24:48) that you should still keep an eye on the legal and regulatory aspects affecting you, in case they change.

And Caleb makes the critical point (13:53) that "The auditor is not the risk police." In other words, when a business makes this or that determination about the relevance of climate change to their operations, it is not up to the auditor to overrule them.

I agree with all of this. 

Bad audits

Where Kyle and I part company is over what to expect in your audits. When I read the guidance document, I approach it as an auditor and I think about what I would do. Practically speaking, an auditor is constrained by the clock: you've got to get through the whole organization, and you've got only a few hours to cover it all. So mostly you don't have time to add any new topics, because the old topics will already keep you busy enough. (I discuss some of the auditor's mindset in this post here.) Therefore, if I were the auditor, I would scan through the ten-page document to look for two or three extra questions I could ask. Then, depending on the answers, I'd see where the trails led from there. Obviously this guidance document includes a lot more than two or three questions—it's ten pages long, after all!—but there are lots of different companies out there to audit. Questions that work for one won't work for another. So the document includes a lot of examples to choose from. 

But some of Kyle's clients have had auditors who took a very different approach. He describes (at around 8:00 and following) having long arguments with auditors who insisted that the client organizations had to answer every single question in the document, or at least a couple of questions from every section—regard­less whether the questions made any sense for the business! Kyle kept saying, "The company doesn't do that," and the auditor kept insisting, "My Certification Body gave me a form to fill out and I have to put a piece of evidence in every blank. So I don't care if it's irrelevant. Give me an answer anyway!"

This argument went on for a long time.

In a sense I feel bad for the auditor. If the CB really did give him a form like that, and if they really did insist that he had to fill out every blank, then they set him up for failure. But whoever is ultimately responsible, this is bad auditing! How much time did the auditor waste arguing over these points? Whatever it was, he didn't get it back later. The more time he spent barking up the wrong tree on climate change, the less time he had to check calibration, operational controls, handling of noncon­forming material, documents and records, internal audits, or management review. In fact, if a company were unethical, they could use arguments over climate change evidence to run out the clock, so they never had to confront hard questions in other parts of the operation. And any auditor who lets himself be played like this isn't good at his job.

I wish I could say this will never happen, but obviously I'd be wrong because it has. All I can say is that no experienced auditor should let himself get tangled up like this, and I'm sorry that Kyle's clients got stuck with someone who did. 

Other topics

Podcasts are conversations, and conversations meander. In the course of this discussion, Kyle and Caleb raise a number of interesting or tantalizing side topics. 

• At about 15:30, Caleb asks if there's a difference between addressing risks from climate change and addressing risks from natural disasters. (Answer: It depends.) 
• At 18:50, Kyle asks how it is possible for a really big company to get certified to ISO 9001, because it has so many parts that they can't all play together. (Answer: I've worked for global companies that were committed to ISO 9001 certification, so it's possible. But you're right that it ain't easy.) 
• And a little later, at 19:28, they ask, "What about the company who says 'We have only 100 employees, so how can we save the polar bears?'" (Answer: I've talked to small companies too, and yes, that's a concern. But some of the answers are interesting.) 

Each of these deserves a longer answer, along with plenty of other questions that they raise but don't have time to pursue. I won't take the time here. But if you'd like me to address one or another of them, leave a comment to let me know.

Meanwhile here is a link to their podcast on YouTube.

      

"Did you bribe the auditor?"

I've talked in earlier posts about how formally analyzing the Context of the Organization can support you in assessing the business risks you face, or the scope of your management system. But sometimes your context can have a huge impact even in a far less formal way.

Years ago, I worked for a small company here in Santa Barbara, California, that was acquired by a Big Company with headquarters in Europe. A couple of years after the acquisition, Big Company informed us that we had to certify to ISO 14001, the environmental management system standard.

So we set to work, building on our ISO 9001 quality system to create an integrated management system addressing both standards. We worked with an expert from another of Big Company's American offices, trained everyone at our location about the new system, and scheduled an audit. We passed the audit, and got our certificate. So far, so good.

The management from Big Company's home office back in Europe congratulated us, of course, but at the same time they expressed considerable incredulity. Our office had had a lot of trouble getting our initial certification to ISO 9001 some years before, so our European colleagues tended to think of us as chronically a day late and a dollar short. At the same time, when it came to ISO 14001 the European home office hadn't achieved certification until their third external audit! How did we—of all people—manage to score on our first? One vice-president even asked me, "Did you bribe the auditor?" I think he was joking—he made it sound like a joke—but clearly he was also rattled.

Just to be clear, no of course we didn't bribe the auditor. (Long time readers will remember this post from two years ago, where I explain exactly why bribery is a bad thing, as if that weren't already obvious!) I don't remember what I told our VP—the question was so unexpected that I probably fumbled it. But months later, in a very prolonged case of l'esprit d'escalier, I figured out the true answer to explain the seeming incongruity that was troubling him.

In the first place, certification to ISO 14001 generally involves implementing two kinds of measures in your organization: environmental measures and system measures.

• Environmental measures are the programs you put in place in order to address your environmental impacts. These can be simple things like recycling your waste paper and your printer toner cartridges, or they can be large programs to reduce pollution or toxic waste resulting from your manufacturing activities. It all depends on what your current environmental impacts happen to be.
• System measures are (in essence) a series of paperwork exercises, that integrate these environmental activities into an overall management system. These measures include assessing your current environmental impacts, planning which ones to address, planning the programs to address them, measuring the results (i.e., the effectiveness) of those programs, and then using the output of those measurements to replan for next year.  

Certification to ISO 14001 requires that both kinds of measures are in place and working smoothly. And the reason our European home office had so much trouble getting their certificate was that they had to implement measures like a recycling program (and others, of course) from scratch. But it took a while for their employees to develop new habits. 

So an auditor would come to the site and hear that the company had implemented a recycling program. Then he would see that all the recycling bins were empty, and that people discarded copier paper into the regular trash. Naturally he would write a nonconformity that the recycling program was ineffective. Multiply this one example by enough other instances of the same basic problem, and you can see why it took them three tries to get their certificate.

But in our office the project was a lot easier. Remember that we were located in Santa Barbara, California. Santa Barbara was the site of a major 1969 oil spill; public outrage at the spill kickstarted local environmental activism, resulting in the 1970 Environmental Rights Day, and (a few months later) the very first Earth Day. Consequently, state and local ordinances governing the environmental behavior of businesses are already very strict, and have been for many years. Any company operating here has to comply to keep their doors open, and employees take environ­mental measures for granted.

So when we began our project to achieve ISO 14001 certification, half the job was already done. All we had to implement were the system measures. We did the assessments, determined that our current environmental measures were appropriate, and set up metrics to track their effectiveness.

We didn't have to bribe the auditor. We didn't have to be smarter than our European colleagues, and we didn't have to work harder. We just benefitted from our location, and from all the extra requirements which that location implied. In that sense, our context did half the work for us.  

__________

         

Auditing the climate amendment

In February of last year, the ISO formally adopted an Amendment to ISO 9001, relating to climate change. (In a stroke of ironic timing, I published an article with Quality Digest just days before the decision, on the subject, "What would it mean if this amendment were adopted?") And right away, voices began calling out, "How do we audit the new requirement?"

Strictly speaking the Amendment added one new requirement to ISO 9001:2015, and one note. The requirement is added to clause 4.1, which discusses Context of the Organization (including the internal and external issues facing the organization), and it provides:

The organization shall determine whether climate change is a relevant issue.

The note is added to clause 4.2, which discusses the needs and expectations of the organization's interested parties, and it reminds the reader:

Relevant interested parties can have requirements related to climate change.

That's it.

Now, the note is literally just a reminder. It is not auditable, which means that there is no need for the organization to show an auditor any objective evidence related to it. But whenever the ISO standard uses the word "shall," that word creates a requirement. The organization has to do it, and has to be able to prove it to an auditor.

Only, ... what questions can the auditor ask to check whether the organization made the determination they are required to make? Auditors started asking for guidance.

And the ISO 9001 Auditing Practices Group rode to the rescue! I don't remember if I've mentioned this group before, but they write papers on how to audit to ISO 9001. They have a website here, which hosts a large library of these papers available for free download. If you ever want official advice on how to audit some part of the ISO 9001 standard, check their library first.

Sure enough, just a month after the Amendment went into effect, the APG published ISO 9001 Auditing Practices Group Guidance on: Auditing Climate Change issues in ISO 9001. You can find it online for free, located here.

As a side note ... I remember when this document first came out, somebody on LinkedIn remarked that the paper is ten pages long. He argued that for it to be so long "proves" that the new requirement "must be" heavy and onerous. I no longer remember if he went so far as to accuse the Lizard People of using this requirement to crush small businesses, and I have lost the link so I can no longer re-read his post to check. But I almost expected him to.

So as a public service, let me reassure you: There is no sign that any Lizard People wrote this paper, and the requirements are not heavy or onerous.

Fine, what does it say?

The basic advice is simple:

• Stay objective.
• Ask if the organization has decided climate change is relevant to them?
• If yes, what steps are they taking?

Let's look at each of these briefly.

Stay objective

When this amendment was first debated, I heard one person say:

"It is impossible for anyone to pretend they aren't affected by climate change. Therefore the answer to the question 'Is it relevant?' has to be Yes, and if someone says No you can write him up!"

The APG rejects this line of thinking. They point out that each organization has a right to make this decision for themselves. The APG paper goes on to say:

In some parts of the world climate change and its causes are controversial topics. Consistent with ISO 19011:2018 Guide­lines for auditing management systems, auditors are to maintain objectivity and neutrality when auditing climate change issues. They should not express personal beliefs relating to climate change. The role of the auditors is to assess whether the organization determined if climate change issues are relevant or not in relation to their QMS and its intended results and, if that is the case, then how it is addressed within the QMS.

This amendment does not require an organization to have climate change initiatives unless it has been identified as a relevant issue to achieve the intended results of the QMS.

Is it relevant?

So the first thing to ask is whether the organization thinks climate change is relevant to them? This means: Do they think that there is a credible risk that climate change can affect their ability to supply their goods or services to customers?

While you are asking that, the paper suggests that you might also check whether the organization has any statutory or regulatory requirements that could affect the answer? And likewise, do they have any contractual requirements that could affect the answer? Obviously you (as an auditor) want to see that the organization's decision on this point is aligned with their legal or contractual requirements.

And then the paper suggests other directions you might explore:

• Is the organization starting to use renewable materials?
• Have they been asked to move to carbon neutral products and services?
• Have energy issues pushed them to reorganize their operations or infrastructure?
• Are they seeing a greater frequency of storms, floods, fires, or drought? And if so, do those developments affect their employees, their supply chain, or their distribution?

And so on. The paper lists many more questions that you might consider.

What steps are they taking?

If the organization determines that climate change is out of scope for their Quality Management System, the discussion ends there. But if they decide that it is a relevant issue, then the basic logic of ISO 9001 requires that they address it one way or another. Naturally the details will be different for every organization. But you can follow the climate theme as a thread throughout the rest of the audit:

• When you audit the scope of the QMS in clause 4.3, you can check whether they still use the same industry codes as before. Are they still making the same products, or in the same line of work?
• When you audit risks and opportunities in clause 6.1, what risks and opportunities have they explicitly identified that relate to the climate? How have they evaluated them? Have they assigned actions to address the risks or capitalize on the opportunities?
• When you audit changes in clause 6.3, consider all the other changes listed here. Has the organization handled them in a planned and controlled way?
• When you audit the work environment in clause 7.1.4, has it been affected by the climate in any way? Has the organization taken any steps?
• When you audit operations in clause 8, how have their operations been affected? (The APG paper offers a full page of suggested questions for clause 8 alone, but of course the details will always depend on the individual organization.)
• When you audit performance evaluation in clause 9, what aspects of climate impact or climate mitigation are the organization monitoring and measuring? Are they monitoring and measuring data that are related to the impact or the risks they have identified? Are the tools and methods that they use for this monitoring and measuring actually fit for purpose? And how is the organization checking to ensure that the changes they have implemented really work?

Again, the paper gives many more suggestions besides these. And of course you would never use every single question. It all depends on what fits, based on the details of the organization.

None of it is heavy or onerous. And it all flows logically from the actual requirements. But I am glad to know that the suggestions are available, as a support.

If you're not familiar with the Auditing Practices Group, be sure to check out their library.     

    

What is Evidence-based decision making?

Last week I quoted the Michael Toy on the unpredictability of major improvements. In the same blog post that I referenced, he also remarks, "Reality Is a Good Thing." His point, of course, is that you have to base your decisions on actual facts, however unpleasant those facts might look. If you base your decisions on wishful thinking instead, that doesn't make the facts go away! All it means is that you will be unprepared when you run smack into them.

This sounds obvious. Why do we even need to mention it?

The problem is that if we are not careful, it is easy for us to confuse ourselves about what the facts really say. It is always tempting to believe what we hear from our friends, or what we think sounds obvious, or what we read in the papers, or what we get from a well-advertised numerical model.

What we need is evidence.

Sherlock Holmes looking for evidence.

ISO 9000:2015 recommends a number of "possible actions" that an organization might choose to take, to ensure that decisions are based on objective facts—that is, on evidence. Among these actions are such measures as:

• determine, measure and monitor key indicators to demonstrate the organization’s performance;
• make all data needed available to the relevant people;
• ensure that data and information are sufficiently accurate, reliable and secure;
• ensure people are competent to analyse and evaluate data as needed. (2.3.6.4)     

These measures should look familiar, because they are all fundamental requirements of the corresponding management system standard ISO 9001:2015. It is only a small exaggeration, in fact, to argue that the whole point of the ISO 9001 schema for quality management systems is to ensure that organizations have the data they need so that they can make evidence-based decisions about how to improve. Look at that list again:

• The organization has to determine key indicators of its own performance: but these are just the quality objectives that ISO 9001 requires in clause 6.2, along with the other performance monitoring required in clause 9.1.
• The data has to be made available to all the relevant people (as I discussed in this post from 2021 about process management).
• The data has to be accurate, reliable, and secure: but this is why the organization has to control its own documentation (as provided in ISO 9001, clause 7.5).
• People have to be competent to understand and analyze the organization's data: but this is why the organization has to determine its competency requirements and then take steps to make sure they are met. (All these requirements can be found in ISO 9001, clause 7.2.)

And so on.

For that matter, what is an internal audit (ISO 9001, clause 9.2) but a tool for collecting evidence—"objective evidence" is the technical phrase—of how well the organization is fulfilling its compliance and quality obligations? Then this evidence is discussed in Management Review (Ibid., clause 9.3) to feed decisions about the strategic direction and tactical implementation of the quality system. Again, the whole structure of the standard's requirements is designed to provide factual evidence to management to serve as the basis for decisions. We discussed a couple of years ago why this information has to be factual, and why (in consequence) you should never lie to your auditor!

So yes, in a sense the principle is obvious. But it's still important to state it explicitly. Fortunately, if you work with a standard like ISO 9001, its requirements do much of the work for you.

Of course, the importance of weighing evidence and accepting reality stretches far beyond the cloistered world of quality management system standards. One of the best stories I ever heard about facing reality head-on comes from cattle ranching, which is a far cry from most applications of ISO 9000 or ISO 9001. It goes like this:

A story is told of a Tonopah preacher who was delivering a funeral sermon over the casket of an old range veteran long and widely known in Nye County. The garrulous clergyman was waxing unctuously eloquent. "Old Dan is not dead," he declared. "He has just taken the highest trail ...." The sardonic voice of another old puncher sounded from a rear pew of the church, "I got a hundred sez he's dead!"*

Reality is a Good Thing.

__________

* Owen Ulph, The Fiddleback: Lore of the Line Camp (San Francisco: Browntrout Publishers, 1995), pp. 69-70.