Do audits really add value? Part 3 of 3

In the last couple of weeks, I've discussed the question whether external, third-party audit results are reliable. On the one hand, I've given reasons it is fair to be suspicious of them; on the other, I've had experiences where they have proven uncannily perceptive. What's the middle ground, the synthesis of these two conflicting positions? Last week I tipped my hand by saying that "yes, we can trust our audit results, provided that we understand clearly what the job of an external audit really is and don't expect it to do something else instead." In what follows, let me try to spell out what that job is.

What an external audit is not

In the first place, an external audit is not a complete health-check. After all, it is a commonplace that auditing is a sampling operation. Third-party auditors routinely remind clients of this during their Opening or Closing Meetings. And in an earlier post I mentioned an audit instructor who said clearly, Any time you do an audit, there will be minor nonconformities that you will miss. Even if your organization passes, that doesn't guarantee that everything is perfect. Notice that for this reason, it is not necessary for an auditor to make the experience painful for the client, because he's not even trying to catch everything. So if someone (like the fellow I quoted two weeks ago) says that auditors are taking a "kinder and gentler" approach than they did back in the early 1990's, that doesn't have to be a problem.

That's what an external audit does not do. Now what does it do?

Enforcement

The first thing an audit does is to support the enforcement of the organization's Quality Management System. Every QMS involves imposing a set of rules on the organization; and no matter how engaged the employees are, there will always be someone who thinks that this particular rule shouldn't apply to him. And maybe for a while he gets away with it: management's attention is somewhere else, and his colleagues don't feel like leaning on him. But sooner or later somebody schedules an audit. And then the message — from management and colleagues alike — suddenly becomes the same: Dude, even if you think the rule is dumb you have got to comply with it or else the auditor will write us up.  And that message is often convincing even when no other message has worked.

Fresh eyes

Sometimes there is something wrong in your system which you know is wrong, but you walk past it every single day and after a while you stop seeing it. I had something like this happen to me. The local organization I worked in had a procedure for processing 8D problem reports. It was based on a global procedure that covered our whole division worldwide, but there were local adaptations for one reason and another. Anyway, the global procedure changed, which meant that we (that means I) had to change the local one to match it. The adjustment was straightforward; I knew exactly what I had to do. So I put it on my to-do list. This was six months before our next surveillance audit.

You know what happened next. One thing and another interrupted me before I could work on it immediately, and then it slid far enough down the list that I didn't see it often. Occasionally I would notice it and remember, Oh right, I still have to fix that. But about that time another problem would cross my desk and I'd forget again.

And then our auditor showed up. As we reviewed the corrective action system, he asked to see our 8D procedure. I gave it to him, and he read to about page 2 where he suddenly asked, It says here you process 8Ds like this. Is that true? And then I remembered, Oops! Not any more we don't. I was going to fix that, wasn't I? Of course he wrote a nonconformity, and to answer it I finally updated the document correctly. It doesn't seem like a big issue in the grand scheme of things, but if he hadn't written that finding I might never have remembered to do it. And as I discussed last summer, it actually does matter that your procedure documents be accurate. 

System integrity

There is at least one more job that an external audit does reliably. It guarantees the overall integrity of the system. To explain what I mean, let me tell another story.

Years ago, I worked in a place that was struggling to implement a disciplined QMS. We had gotten ISO 9001 certification, but keeping things at a sustainable level was a challenge. It seemed like every year I was writing Major Nonconformities in our internal audits.

So after one of our external surveillance audits, the General Manager took a few minutes out of his next staff meeting to complain that the audit process was useless.

Me: What do you mean "useless"?

General Manager: Well that guy spent a few days here, he seemed to talk to everyone, but then he gave us a clean bill of health! What's wrong with him? Didn't he see that we had seven Major Nonconformities in our internal audits? How could he say that our system is working OK?

I wasn't sure how to answer, so after the meeting I forwarded that question to the auditor (whose contact information I had kept). And he answered:

Auditor: Yes, I saw those seven Majors. But you found them, didn't you? They were all clearly stated in the internal audit report; and when we checked the action plans, the root-cause analyses looked reasonable and the corrective measures were on-schedule. The system was working exactly the way it's supposed to work.  

Then he went on.

Auditor: Look, if you want me to come out there, photocopy your internal audit results, sign my name to them, and then spend the rest of the week in a bar — and get paid for it — I can do that. But that's not going to give you a lot of value. So it's more important to me to make sure that your overall system is hanging together and functioning the way it should. Of course you're going to have problems or hit bumps in the road. That's normal. The important part is how you react to those problems, and right now you guys are doing fine.

And that's what I mean by guaranteeing the overall integrity of the system. This is why an external auditor doesn't have to find every little thing the organization is doing wrong: because if the system is working correctly, the organization will find those problems themselves. Therefore the one critical thing that the external auditor has to ensure is that the system itself is working.

This point relates also to our earlier discussion of the difference between Minors and Majors. Two weeks ago, when I listed reasons to be suspicious of audit results, most of those reasons applied to Minors. Didn't they?

• If auditors used to strain at gnats and no longer do, that has to mean that they used to write a lot of Minors and no longer do, because writing Majors has always been the exception. 
• More to the point, think about the external audit that started this whole train of thought, where the auditor asked a few simple questions and then wrapped up the audit. What made that possible was that the overall system was functioning just fine — and in that office, by that time, it was. Yes, if he had been more focused he could have found a few Minors for us to chase after. But fundamentally that's not what we needed from him. We had internal audits for that — and customer complaints, and nonconforming material reports, and the whole armamentarium of Quality Management tools. What we needed from him was assurance that the system was intact, and it was. 
• And while experts certainly disagree, I would argue that they are a lot more likely to disagree over Minors than over Majors because Minors are one-off failures. They are almost incidental. And therefore there is a lot more room for personal, subjective judgement to come into play. Majors, on the contrary, are by definition failures that endanger the system. My old instructor might have been exaggerating when he said that "if the organization has Majors, you will know it by the time you reach the Receptionist's desk!" But it is pretty hard to mistake a system breakdown for a one-off failure, or vice versa.   

From this point of view, the most important job of the external auditor is to find and report Majors, if there are any. Minors are lagniappe. If the external auditor happens to find them, of course he reports them; but if he doesn't, somebody else will. On the other hand if the system has broken down, that "somebody else" might never come along. So the external auditor has to report on Majors. 

And for that reason, as long as we remember the difference between what external audits must do and what they cannot pretend to do, we can continue our audit programs with a good conscience.