Auditing is a fundamental part of any management system. In the four-step cycle of "Plan-Do-Check-Act," auditing is the core of step three, "Check." Without audits we would never know if our systems were working.
But there are internal audits and external audits. And is the information we get from external audits actually reliable? Is it useful? Or are there institutional or organizational factors that skew or corrupt the auditing process?
I think the real picture is not a simple one, so I'll break up my answer into three parts, all based on my own personal experience. In the first post — this one — I'll give reasons to suspect that maybe external audits don't really add value. In the second post, I'll backtrack and give reasons that maybe they do add value after all. And in the third post I'll try to find a middle ground that does justice to both opinions.
You may be able to guess where I'll end up before I get there, or you may have opinions of your own. Either way, feedback is always welcome. Please feel free to comment.
My thinking along these lines all started after a singularly-unimpressive third-party audit. The auditor walked around, asked a couple of aimless questions to which we gave answers that he apparently found acceptable, and wound up. All done.
Afterwards I sat around talking with the site management, and we tried to understand what had happened. Partly it was just that this particular guy was a little goofy, and we've all met auditors for whom we could say the same. (Christopher Paris even jokes about it in his auditor cartoons.) But the General Manager remarked that our experience that day seemed, in his mind, about par for the course. He said that all the auditors he could remember seemed to have been more-or-less colorful individuals, but he couldn't think of any findings that made much of a difference to the organization. So what's the point, he asked. Is there any real value that we get out of interrupting operations for a day or two while we host these guys? Do their reports actually help us build a better organization? Or is this all just a song-and-dance we have to go through as part of the price for getting our certificate? Because we really do need the certificate. But is that all we're getting?
In reply, I explained something that another third-party auditor had described for me several years ago. When ISO 9001 was first introduced, so this man had told me, auditors were very strict. Getting certification was a big deal. But over time there were more and more registrars available, and they were in competition with each other. This meant that if a company was refused certification by, say, BVC or BSI, they could always call up DNV or DQS or Perry-Johnson and try again. The risk of losing repeat business pushed the registrars to grade more and more leniently, and to market themselves as “partners” in improving your management systems. They wrote fewer nonconformities and more “opportunities for improvement” … suggestions for things you might want to consider.
The General Manager listened to all of this politely. Then, bless his heart, he jumped in with exactly the right question: You know who else behaved like that? The bond-rating agencies, back in 2008. And see how well that turned out for everybody!
Even when auditors aren't deliberately throwing slow softball pitches, though, there's another risk to the reliability of any audit report. Experts disagree. Often they disagree a lot.* If you have ever performed an audit and then discussed it with another trained auditor, you know that no two auditors will choose to include exactly the same data points; and even if they can agree on a particular finding, it's not infrequent that they'll rate it at different levels of severity. We always ask for "objective evidence" to justify any finding, and rightly so. But what we then do with that objective evidence depends more than we like to admit on subjective assessments and personal expertise.
So if audit reports are, in the end, based (at least partly) on the auditor's personal, subjective opinions — and if auditors are under institutional pressure not to be terribly hard on their clients — this brings us back to the original question: Do audits really add any value? And if you look at it from one point of view, you wouldn't be crazy if you suspected that the answer might be No.
But don't touch that dial. I'll come back next week to look at the question from another point of view.
__________
* You can find this topic discussed at great length in, for example, the medical field, where it would seem to be a matter of life-or-death to get the answers right. See for example this article from 2008, this one and this one from 2010 (both based on the same book), or this one from 2020. This extreme variability is one of the reasons that medicine relies more and more often on objective checklists instead of personal expertise, as described here.
No comments:
Post a Comment