"Is there anything you want me to write up?"

Over the years I've gotten to work with a lot of other auditors, and I've learned something from each of them. Sometimes they've just had a really interesting perspective on the work of Quality: I remember an external auditor who explained over lunch that a few years before he had started his own business (something unrelated to Quality) and it failed. When he analyzed the failure, he concluded that the root cause was that he didn't know how to run a business. So he trained to become a Quality auditor, which would allow him to look at many other companies and study how they were run. His plan was that when he finally felt he had learned how to run a business, he would quit auditing and try again.

But sometimes I've learned techniques. One of the most surprising was when I was working with a colleague on an internal audit, and as we were about to wind up one interview he asked, "Just one more question: Is there anything you would like us to write up? That it would help you for us to write up?"

Wait, … what? In my experience most auditees look at an audit like some kind of oral exam: the last thing they usually want is to volunteer something to be written up.

But my friend was completely serious. He pointed out that as internal auditors we are there to help the organization improve. And all we are ever able to see is a sampling. So maybe there's something that isn't working the way it should, but that we missed. And if it would help our auditee for us to look at it, why not ask for it?

In fact the auditee said yes there was, or at least maybe. She wasn't sure, but what did we make of these project requests she had gotten yesterday? We looked at them, and they were requests for her to set up and track projects where half the estimated durations were blank and the costs were listed as "Don't know." We agreed that this didn't look like enough for her to work with, but for various reasons there wasn't an easy way for her to push back. (That sounds unlikely the way I'm describing it, but I'm leaving out a lot of details.) Most of the project requests that she got were just fine, so there didn't appear to be a systemic problem. But we did write a minor nonconformity that she was being asked to move forward without the planning data she needed to do her job.

In the big picture, that was probably the most meaningful finding we wrote during that entire audit, the one most likely to help the organization improve. And we would never have gotten it if my friend hadn't asked for it.

I've used that question ever since. Often the answer is No, but even then I think it helps the auditee see the audit differently. It helps make the point that this really is a collaborative effort — that we really are on the same team.

Of course there are risks to watch for, when you ask a question like this. Once in a while an auditee will take this as an invitation to air some personal grudge against a coworker, or to try to score political points in a fight between departments. Obviously you have to watch for those and can't write them up.

But it's a good question. And I'm grateful to my friend for having taught it to me.

     

Auditing and consulting

My last couple of posts (see here and here) have suggested a kind of relationship between an auditor and the audited organization that has real risks, so let me talk about them briefly.

There is a fundamental principle that auditors must not do consulting. The difference is that an auditor tells you what's wrong (how your organization is deviating from its requirements) and a consultant tells you how to fix it. The reason to keep these roles separate is that combining them poses a temptation for the auditor to abuse his authority: first, he writes up a list of nonconformities; second, he comes back charging $500 an hour to tell the organization what they have to do to clear the nonconformities; third, he comes back next year to see if they did exactly what he said — and if not, he writes more nonconformities, ad infinitum. Permanent employment for the auditor, but really bad for the client. If you separate auditing from consulting, you prevent this cycle. So the general rule is, "I can tell you that you don't conform to your requirements, but I can't tell you how to correct the problem."

The basic principle is a good one, especially in the case of an external (or third-party) auditor who gets paid every time he shows up on the premises. But for internal auditors the distinction between auditing and consulting is often not so practical. In the first place, unless the organization is large enough, whoever does the internal audits is very likely the same person who will be assigned to lead or coach the corrective action team because there is literally no-one else available and qualified. In the second place, even during the audit itself it's not unusual to hear the question, "Why is it wrong to do what I'm doing? I don't understand what that paragraph of the standard even means. What should I do differently so that I'm not violating the requirement?" When someone asks you a question like that, the line between explaining the finding and consulting on how to fix it becomes so thin it almost disappears.

In my last couple of posts, I say that sometimes you might talk to the organization's management before rating a finding, or you might take into account topics like the organization's overall level of maturity. This advice is most appropriate in internal audits, where the distinction between auditing and consulting is already compromised for the reasons I described above. What about the risk that the auditor might abuse his authority? In the internal case that risk is minimized because if the auditor starts asking for something crazy, the department can easily escalate over his head to his manager to ask for intervention. And when you are all on the same team — when you are all paid out of the same payroll — there is no advantage to the auditor in demanding things that don't make the company healthy and prosperous.

It is important to understand that there is a difference between auditing and consulting, and also why the line between them is drawn so sharply. But then when it comes to working in the real world, like with everything else, what you do depends on risk and judgement: what are the concrete risks here and now, and how do you judge that you can meet them most effectively?

      

Minors and Majors

Last week I talked about how to distinguish Minor Nonconformities from Opportunities for Improvement. Now I'll review the difference between Minor Nonconformities and Major Nonconformities.

The difference to the organization is that Majors get a lot more attention and typically require a lot more work to close. If the auditor from your registrar raises a Major in an external audit, it can block your certification or recertification. Depending on the finding and the contract with your registrar, you may have to pay for a re-audit within a specified time frame (far sooner than you were planning for!) to prove that the nonconformity has been corrected and permanently prevented. Because the consequences for external Majors are so significant, organizations frequently define heavy procedures to handle internal Majors — so that they get immediate and sustained management attention, to make sure they have been resolved before the external audit.

In short, Minors can be comparatively innocuous but Majors are The Scary Ones. But what is the real difference? When do you write a Major?

The definition can be found in ISO 17021:2015, and it relates to the idea of a Quality Management System (QMS). Briefly, if the failure is a system failure, it's a Major; if not, it's a Minor. More exactly, according to definition 3.12 a major nonconformity is a:

nonconformity that affects the capability of the management system to achieve the intended results. 

In the same way, definition 3.13 tells us that a minor nonconformity is a:

nonconformity that does not affect the capability of the management system to achieve the intended results.

But what are "the intended results"? In a broad sense this probably has something to do with healthy operation and customer satisfaction; but in a narrow sense, surely every single procedure in the organization has as one of its intended results that everyone in its scope should comply with it. And if you take the term that broadly, then any failure would count as a Major. Clearly that can't be the right way to see the question.

In casual conversation, the difference is usually described in terms of extremes: a Major is "a total breakdown of the system," while a Minor is "an isolated one-off error." Of course this leaves most nonconformities somewhere in the middle, with the auditor having to decide whether a finding is more like the first or more like the second. 

For example, suppose the organization has defined a specific template for all their internal documents; but when you examine these documents during the audit you find that nobody uses this template except the Quality department. Is that a Major or a Minor?

On the one hand, it's clearly not "an isolated one-off error," since you see the very same error almost everywhere. It certainly looks like the error is "systemic," or at any rate like there is no functioning system for introducing document templates and making sure everyone uses them.

On the other hand, will the organization take it seriously if you write a Major for document templates? 

• Some will, especially if they have contractual requirements to other interested parties related to the use of those templates: but those organizations won't have this finding in the first place. 
• An organization where this finding turns up is an organization that doesn't see any reason to care about internal document templates — and is therefore an organization that will never take such a Major seriously.
• But can you as an auditor, in good professional conscience, justify calling it a Minor? Can you honorably say that it meets the definition of a Minor? 
• It depends. If it's an internal audit, consider talking to them. Ask top management — the people who will receive your report when you are done — whether having a uniform format for internal documents is one of their "intended results." If not, then this finding does not affect the achievement of "intended results," and could count as a Minor.
• That doesn't mean that any time the organization doesn't care about something it's a Minor, of course. Some "intended results" (like customer satisfaction or legal compliance) are so serious that the organization has to care. So be reasonable.
• Note also that as the organization matures, so will their list of "intended results." Maybe in a few years they will have reached a place where they take document formatting more seriously. By the time that happens, though, you probably won't see this particular finding any more.       

My very favorite explanation of the difference, though, came from a class discussion back when I took my first Lead Auditor training class. The instructor had just made the point that an audit is a sampling exercise. You can never see everything that goes on inside an organization. And one of the students had a concern.

Student: If an audit is a sampling exercise, doesn't that mean there's a big risk that when we audit an organization they might have huge, serious problems and we don't see them?

Instructor: That will never happen.

Student: But you just said an audit is a sampling. What if the big problems are all over here and we happen to be looking over there? What if we just miss them?

Instructor: That happens with Minors all the time. In fact, I guarantee that any time you do an audit, there will be minor nonconformities going on in the organization that you will miss. But if the organization has Majors, you will know it by the time you reach the Receptionist's desk! You will smell them! You will know they are there. The point is that if the organization suffers from major nonconformities, their attitude will come through in so many little things that it will be impossible for them to hide it, or for you to miss it. And then — since you already know the Majors are there waiting to be found — all you have to do is find them.  

That's obviously a very informal criterion, but it makes the point beautifully.

Is that really a Nonconformity?

When you carry out a Management System audit, one of the tasks is to decide how to rate your findings. You stumble over something that's just not right, but is it really a Nonconformity, or is it something milder — say, an Opportunity for Improvement? If it's a Nonconformity, is it a Major or a Minor?

You'd think it would be easy to decide these things, and of course sometimes it is. Out on the production floor you find someone working to revision A of a certain procedure, but back in the office you already confirmed the current revision is B: obviously this is some kind of nonconformity against Document Control. What's interesting is that sometimes it's not so obvious.

This week I will talk about distinguishing Minors from Opportunities. Next week I will talk about distinguishing Minors from Majors.

The audit from Hell

Once upon a time, I was co-auditor on an internal audit of a manufacturing plant; the Lead Auditor (whom I'll call "Jackie," which was not her name!) was a colleague brought over from Europe. Jackie and I got on very well at a personal level, but when we compared notes at the end of the first day I was stunned: I had audited these departments over here and came back with something like four or five nonconformities, while she had audited those departments over there and came back with over a dozen. The next day, it was the same story. In the end (it was a four-day audit) we combined some of the findings to make the list shorter, but still we reported something like twenty-five Minors and one Major. 

Was the plant really in such bad shape? No, not really. They were sloppy about their paperwork and some other administrative topics. I would have been happier if they had taken those things more seriously. But none of this casual attitude found its way into the products. When you read the long list of findings in our report, none of them jeopardized customer satisfaction in any direct way. 

So why the ghastly report?

1. The plant had recently been acquired by a new owner, who had a whole set of their own procedures. (The new owner was headquartered in Europe, which is why Jackie was flown out to do the audit.)
2. Jackie was the Lead Auditor, which gave her the final say over what went into the report and how we should rate it.
3. And Jackie's perspective — every time she found anything out of alignment with the way the new owner did things — was to insist, "But the procedure clearly says …."

Sometimes that's the right way to audit, but this wasn't one of those times. Nobody in that plant learned anything useful by being forced to slog through a root-cause analysis for why the corporate document templates hadn't been rolled out yet, or why the plant hadn't implemented the standard corporate labeling conventions in the warehouse, or why the supplier audit results weren't posted on a website so they'd be easy to find. All they learned was to hate internal auditors, which didn't make my work any easier in the years to come.

And it was all so unnecessary. Of course the errors had to be written up. But there has to be room to apply a measure of judgement or discretion in the writing. The trick is to find rationally defensible criteria for that judgement or discretion, so that you don't just assign ratings based on how you feel that day. In the months following that audit, I chewed over this problem a lot. And I think I have an answer.

Minor Nonconformity or Opportunity for Improvement?

The key question is this: How mature is the organization you are auditing? The purpose of the audit is to drive continual improvement. That means the organization has to learn something from your findings, something they can use to get better. A truly mature organization can learn from even the most trivial findings. So if you are auditing a mature organization you can write up any mismatch between the procedures and what you observe in reality, confident that your finding will lead to improvement. If you are auditing an immature organization, though, you have to pick your battles. Here's how.  

Let's say you are auditing an organization that still has some maturing to do. During the audit, you identify several places where the procedures say one thing, and the auditees or operators do something different. Interrogate each situation as follows:

1. Check the procedure that your auditee failed to follow. Remember that nobody writes a procedure for no reason: every step in a procedure was written there in order to keep something from going wrong. Focus on the steps that your auditee failed to follow, and ask yourself: What is the risk that these steps were written to avoid?
2. Now check back with the auditee. Explain that the procedure was written to keep this or that bad thing from happening. Then ask: Do you have some other way to prevent these risks? Can you show me that there's no chance of these bad things happening?
3. If the answer is Yes, then from the perspective of the end result the auditee actually complies with the procedure because he has succeeded in eliminating the risk that the procedure was written to eliminate. 
• In this case, write an Opportunity for Improvement as follows: "Procedure 123 says to do … in order to avoid the risk of …. In fact the operator avoids that risk in another way, by doing …. Consider reviewing the procedure to determine whether it can be rewritten to match how the operator really works, or whether the operator should change his work to bring it in line with the procedure."
4. On the other hand if the answer is No, you have to write a Nonconformity. But at that point the discussion isn't about "compliance with the procedure"; it's about the live, unaddressed risk that could jeopardize operations. 
• In this case your message to the organization should be, "You folks are playing with matches, and it's a lucky thing you haven't burnt your fingers yet. Fix this before your luck runs out." You don't even have to mention "compliance with the procedure"; and the organization will be grateful that you identified a risk, instead of resentful that they have to fill out a bunch of audit paperwork.

For the immature or semi-mature organization, redirecting the focus like this is a big help.