The dark side of process metrics

In my last post I talked at some length about how to define Key Performance Indicators (KPIs) or metrics to measure your processes. But there is a risk here. Too many companies decide that the best way to ensure process-adherence and continual improvement is to give people an incentive, so they tie employee bonus payments to the achievement of specific process metric goals. 

Superficially this idea sounds reasonable. Don’t people work harder when they have incentives? Shouldn’t you pay people more when they do what you want? But in practice it can drive perverse consequences.

Once upon a time I worked in a branch office for a company, where I was responsible for implementing the Quality Management System. It was important to my boss that all of the corporate processes be fully in place and fully adopted, so he gave me a goal – one that counted in my annual bonus – that there should be no Major Nonconformities in the internal audits at that branch. In the abstract, this kind of made sense: a Major Nonconformity would mean that some entire process had never been implemented or had completely broken down, and therefore would suggest strongly that I hadn’t been doing my job. The only problem was that I also did most of the internal audits. So when I encountered a potential Major, what was I supposed to do: overlook it, or torpedo my bonus?

In fact I always wrote them up, and just accepted losing that fraction of the bonus. And I was so inexperienced back then that it didn’t occur to me to explain to my boss that he was (unintentionally) incentivizing me to fake my audits. But when I look back in retrospect I just shake my head in dismay. How could neither of us have seen what a bad idea that was?

More generally, when you incentivize a specific outcome, be prepared for people to increase the frequency of that outcome, even if it becomes disconnected from the overall Quality goal. Once again Scott Adams captured this effect perfectly, in a cartoon strip where the company offers its software engineers a ten-dollar bonus for every bug they find and fix. (Like last time, I am providing only a link and not a copy of the strip in order to stay on the right side of Fair Use.) Nor can you avoid the problem by defining the metric more cleverly, because:

There is no metric in the world that cannot be gamed.

So if bonuses are paid only when the process metrics are green, be prepared for the metrics to turn green regardless how well the processes are doing in real life. This doesn’t have to mean that anyone is actually lying: I assume most people care too much for their integrity to sell it so cheaply. But people are also remarkably creative; so if there is a way to do more of the incentivized activity by disconnecting it from the broader goal, people will figure it out and even think that they are doing the right thing. It’s what you asked for, isn’t it?

You need accurate process metrics to know if the processes are working, so tie your bonuses (if any) to something else.   

     

Updated housekeeping details: let's move to once a week

Back when I started this blog I posted a list of housekeeping details. For the most part, everything I said there still stands.

But I've come to think that posting once every two weeks isn't often enough. So I'm going to shift to once a week -- still on Thursday mornings, at 8:00 am, Pacific Time.

And from time to time there may be unplanned, off-cycle posts like this one, if I want to comment on something right now that isn't a normal part of talking about how QMSes work. Or at any rate I don't want to rule out the possibility.

Once again, comments or feedback are welcome. Thanks!

      

Processes 3: Measuring them

When I talk about "measuring" a process, I really mean monitoring or measuring the results. The point is to find out whether your process is succeeding or failing. In other words, Are you getting what you want?

In order to distinguish success from failure, you have to do three things: set up some objective criterion of what constitutes success at the beginning of a cycle (a year, a quarter, a project, or whatever); check at the end of the cycle to see how you did; and then publish the results to anyone who needs to know. I'll talk about each of these.

Define your metric 

Typically you measure each process by one or more performance metrics or Key Performance Indicators (KPIs). First you decide what to measure, and then you set a goal that you want to achieve. 

Maybe you decide to measure the effectiveness of the product development process by setting a target that 80% of your development projects will complete on time and on budget. 

Maybe you decide to measure the effectiveness of the shipping process by setting a target of 98% on-time-delivery of orders. 

In each case the idea is that if the process is working correctly, you will hit the target; and if you fail to hit the target, something might be wrong with the process. In principle you can choose almost anything as long as it makes sense – but let me qualify that remark. Some metric definitions are better than others.

There is an extensive literature about how to design process metrics, and I don't propose to summarize it all. But one of the most common bits of advice recommends defining "SMART metrics," where "SMART" is an acronym defined as follows:

• Specific: Say exactly what has to be done.
• Measurable: Make the goal quantitative, so you can measure whether you got there or how much progress you made.
• Achievable: Don't ask for the impossible, or nobody will bother to try.
• Relevant: Make sure the goal is connected to the work in some natural way.
• Time-bound: Define a goal to be completed by a specific point in the future, so everyone knows what target to aim at.

These are perfectly good criteria, and if you can define your metrics this way by all means do so. But some topics don't lend themselves to the SMART criteria; for example, in some cases the most meaningful measure of success might be qualitative or binary rather than quantitative. So if you find you need a shorter and more general list of criteria for your process metrics, I think the bare minimum list of features for any meaningful metric is this one instead:

• Achievable: Success has to be possible, or no-one will try.
• Non-trivial: Failure has to be possible, or no-one will care.
• Unambiguous: You have to be able to tell the difference between success and failure or your metric does you no good – because the whole point of a metric is to tell you whether you succeeded or failed.

Again, if you are able to define SMART goals, do it. But if you can't, at least make sure your goals are Achievable, Non-trivial, and Unambiguous. Those are – and have to be – the basics.

Measure your performance

This should be straightforward, shouldn't it? You defined your metric so that it's measurable. Go measure it.   😀

Publish your results

Now that you've found out how you are doing, tell the world. 

Not literally the whole world, of course. Don't tell your competitors. Don't post your results on the Internet, or on the front page of the daily newspaper. But inside your organization, transparency is a virtue. 

And really, even if you try to restrict the results to those who have a "need to know," that list is still pretty long. 

• All the functional managers responsible for the process have to be informed, because they can't manage effectively without feedback. 
• All the functional workers who implement the process have to be informed, because you cannot hold people responsible for meeting a goal without telling them what the goal is and whether they met it. 
• Top management have to be informed, because they are steering the whole ship and have to know if they are too close to an iceberg. 

So at that point, why not make the information available to anyone who wants to look it up? Post it on a bulletin board outside your office, and keep it updated as a normal routine. When you are transparent with your colleagues, it says that you trust them; and it makes them more likely to trust you. This is always a good thing. Or think of it this way: 

• If the numbers are good, you have no motive to hide them.
• If the numbers are bad because of reasons outside your control, you have no need to hide them.
• And if the numbers are bad because of reasons inside your control that you haven't addressed yet, you shouldn't be able to hide them. 

Never manage your work in a way that you are ashamed of, and you never have to be ashamed of the results. Even when you fail.

Is that all? Not quite. There are ways that KPIs can go badly wrong, if you aren't careful. But that's a different topic from the one in this post. What I have explained here are the formal criteria for any metrics that are useful and functional. On the other hand, if a KPI meets all the right formal criteria but you are measuring the wrong things, your results still won't be any help. That's the topic of my next post. 

    

Off-cycle post: Can "lying to the public" be part of your process?

This post falls entirely outside my regular, planned sequence (the next of which is going to be Part 3 on process management -- Part 2 is here), and also outside the two-week cadence that I defined back in the beginning. (Is two weeks too long to wait between posts? Leave me a comment either way.) But I want to comment in a timely way on Christopher Paris's recent blogpost as follows:

Caspian Pipeline, Responsible for Falsifying Reports Related to Oil Spill, Holds ISO 14001 Environmental Certifications - Oxebridge Quality Resources

By all means read the whole story for the juicy details, as well as the story from the Moscow Times on which it is based. But the gist is in this one-sentence summary:

The Caspian Pipeline company responsible for falsifying the size of a recent Black Sea oil spill "guaranteed" its environmental safety capabilities by pointing to ISO 14001 certificates issued by Bureau Veritas Group and accredited by UKAS.

Of course the environmental damage is appalling, and the firm's mendacity is morally reprehensible. But I have to admit that when I first read the article, a small contrarian voice in the back of my head asked, "But what if they were following their internal processes? What if one of their processes says, 'In case of an environmental disaster, lie to the press.'? Could that make them still compliant to ISO 14001 after all?"

So I had to go check the standard. Turns out the answer is "No, your process can't tell you to lie to the public," but it's not as obvious as you would hope. The ISO 14001:2015 standard contains 25 instances of the word "communicate" or "communication" in the normative clauses 4-10. Of those 25 instances, three say that communication has to take into account the organization's compliance obligations (7.4.1, 7.4.3, and 9.1.1), and only one (in 7.4.1) requires the organization to "ensure that environmental information communicated is consistent with information generated within the environmental management system, and is reliable." But yes, that should be enough to prohibit lying to the public. Also, if the organization's compliance obligations happen to include truthful reporting then those three references apply as well.

What about other standards? ISO 9001:2015 references "communicate" or "communication" 17 times, but nowhere does it explicitly say those communications have to be truthful. Clause 7.4 gives the most guidance, but all it says is:

The organization shall determine the internal and external communications relevant to the quality management system, including: 
a) on what it will communicate; 
b) when to communicate; 
c) with whom to communicate; 
d) how to communicate 
e) who communicates.

ISO 45001:2018 takes the same approach as ISO 14001, requiring that:

When establishing its communication process(es), the organization shall:
-- take into account its legal requirements and other requirements;
-- ensure that OH&S information to be communicated is consistent with information generated within the OH&S management system, and is reliable.

In other words, standards that regulate topics where people might get hurt and around which there are probably legal regulations say something about keeping your communications "reliable."

Does this incident prove the system of certification and accreditation to be corrupt? Not by itself. In the absence of other information, it's always possible that someone went crazy five minutes before his press conference began, and therefore went completely off-script. No process can prevent isolated instances of non-compliance. What matters is how the organization responds to the situation after the fact, and whether Bureau Veritas follows up this paper trail during their next audit to ensure that the organization's response was meaningful and appropriate. Inquiring minds want to know....

Maybe Christopher Paris will be able to follow up on this news later. If you haven't seen his Oxebridge blog yet, by all means check it out.

    

Processes 2: Owning them

 Along with defining a process, you have to name one person as its Process Owner. To be clear, I don't mean there is only one Process Owner in the whole company: that might happen somewhere, but it's not the normal case. What I mean is that each process has one Owner. Depending how things work at your place, it's possible that one person might end up owning a couple different processes, or possibly not.

The Process Owner is the go-to person for all issues, problems, or discussions related to that process. In more formal terms, the Process Owner is the (one) person who has authority to define the process or to change it, and who also has the responsibility to ensure that it works. Let's unpack this.

First, I say the Process Owner has the authority to define the process (or change it).I never say that he has to have the knowledge or expertise all by himself. 

Of course he might. I once worked at a place where the Warehouse Manager knew the status of every order, every planned shipment, and every piece of inventory; he also knew more about the ins-and-outs of storage and shipment than anyone else in the company. He was awesome.

But it's not always true. I've also seen cases where one process (in this case, the product development process) was so fantastically complicated that there was a whole department devoted to keeping it current. The Process Owner was the overall VP of Engineering, but he had to rely on his experts to discuss the details.

It might happen that one process touches many departments, so that the Process Owner has to get multiple people to buy in before he can finalize his process and publish it. He probably has to align with the owners of the other processes that provide inputs into his, or of those to whom he provides his outputs in turn. All of this is possible.

But at the end of the day, there has to be one person who is responsible. If there isn't one Owner responsible, then in practice nobody is responsible. And in that case, if the process ever fails – or breaks down, or proves dysfunctional (and one day they all do) – nobody is really responsible to fix it. Yes, the organization will find ways to ignore the process or work around it. Work will still get done. But the official goals will all be missed and the formal indicators will all be red. And by then the process will have become just an obstacle, instead of a tool.

Second, I say that the Process Owner has the responsibility to ensure that the process works. This responsibility entails a couple of different things.

On the one hand, when the Process Owner defines the process, he had better insert steps to make sure associates are really following the process and the daily work is on track. These steps might be as simple as filling out a checklist or signing off a traveler, but they could also be a lot more complex if necessary.

On the other hand, the Process Owner also has to check the process performance, to make sure the company is getting the expected results. Then once he has checked the process performance, he has to report to Top Management. If the process is giving the results it is supposed to give, life is roses. If not, he had better find out why not. It might be just a routine blip in performance, a normal statistical deviation that doesn't worry anyone. Or it might be a sign that something is wrong, that the process no longer meets the company's needs and has to be adjusted. Of course you don't want to change your basic processes for light and transient causes; so in the first case it's probably best to sit tight and see if next quarter is better. But no process is perfect, and no process lasts forever. Sooner or later, they all have to be adjusted, reworked, or scrapped. Watch closely so you can tell when this has come to pass.

My third post in this sequence is about how to measure process performance. Reporting to Top Management takes place during Management Review, and I'll discuss that in the future (probably some months from now). 

In other words, this post covers the basic What of the Process Owner's job. In the next post we'll start to discuss the How.

      

Processes 1: Writing them

If we agree that formalizing your processes can be useful in some ways, what's next? Two things: define or document each process (so far as you need to), and then manage or control it. Managing a process, in turn, also requires at least two things: that the process have an Owner, and that it be measured in some way.

Originally I tried to fit all this information into one post, but it got long and ugly. So I'm going to break it into bite-sized pieces, to keep the topic light and easy:

1. Documenting a process
2. Process Owners
3. Measuring a process

Then I'll add a fourth post as an appendix, about some of the risks to watch for in process measurement.

Sound fun? 😀 Let's do it.

What do you say?

When you document a process, say only what you truly need to say and then stop. 

• If something is a legal requirement, spell it out.
• If something is a safety guideline, spell it out.
• If something is complicated enough that you don't want people improvising, spell it out.

But when you get to the point where it really doesn't matter how somebody does something so long as the thing gets done – in that case spell out the What and stay silent about the How. Again, say only what you need.

There are a couple of reasons for this. 

• It makes the documents shorter, so they are easier to write and easier to read.
• Therefore it makes them easier to follow.
• Therefore your documents will feel like less of a burden. 

Most people already roll their eyes when they are faced with too much documentation. If you can keep it simple, your people can focus on their real work rather than scratching their heads over subparagraph 17(a).

At a minimum, you should list the inputs into your process, and the outputs from it. And try to answer the basic question, "What are we doing here?"

It can also be helpful to point out how one process interacts with another, and what resources each one requires. (These latter points are required by the ISO 9001 standard; so if you want certification, be sure to do them. If not, do as much as you find helpful..) 

How do you say it?

Along with keeping your documents short, keep the language direct. Use the second person (like I'm doing here). Use active verbs wherever they fit. In fact, maybe it's simpler if you just get The Elements of Style, by Will Strunk Jr. and E. B. White, the pre-eminent style guide for American English. If you follow their advice in writing, you don't need mine.

Who does the writing?

As I discussed previously, the content has to come from your experts in the task at hand. But generally they'll be grateful if somebody else can help them with the wordsmithing and the typing.

Where do you store the documents when they are done?

Wherever the people who have to use them can find them later. If they all have access to a computer, then an online document repository is great. If they don't, you might need to store them on paper. Remember that if you store the documents on paper, there has to be some way for people to get copies at the same time that the master is kept safe; also, if you distribute copies on paper you have to be sure to collect them all when the document gets updated so that nobody is following out-of-date instructions. That's one reason that online repositories are so popular -- because they make updating easier. But they only work if everyone can reach them.

Depending on what you are doing, document management can get a lot more complicated. But these are the basics. So let me follow my own advice and stop here for now. If you'd like to see more detail later, please leave me a Comment to say so. 

"If only you have the right process …."

Some stories you just can't make up. You think, "No one would believe it if I told a whopper like that." Then it happens in real life.

That's how I felt when I was sitting in a meeting one day. The Big Boss from the home office was berating us for our failure to follow the finer points of the company's product development process. And then suddenly he said:

"Don't you understand why this is so important? If only you have the right process, and if you follow it carefully enough, you are guaranteed to succeed!"

Guaranteed? Really? If we follow the process closely enough, then our ideas don't have to be any good, our implementation doesn't have to be efficient, we'll never have an accident, and our competitors will go out of business?

Cool.

Also absurd. In the first place, there's no such thing as the perfect process. And to be fully clear, no company anywhere ever earned a dime by following a documented process.*

But if following your process carefully doesn't guarantee success, why do it? When we do audits we always check that people are following their processes, and we write nonconformities when they aren't. Why do we care?

First, a documented process is just a tool. Following a written procedure won't guarantee you Quality, any more than it will guarantee you success. On the other hand, remember that Quality just means getting what you want, and if you use the right tool for the job you are a lot more likely to get what you want in the end. So – what job does a documented process help you accomplish?

There are a couple that are easy and obvious, plus one that is a little harder to see until you take a step back.

The simple reasons for documenting a process are:

1. to help you remember it,
2. to make it easier to train others.

These reasons are obvious when the process itself is complicated, but sometimes I've been lulled into forgetfulness even when the job was simple. As for training, there have been any number of times I've taken over a job from someone else who "just knew" how to do it, and the first thing I always do is to write a checklist (even if only for myself).

The third way you can use a documented process comes into play when you are trying to get your arms around your business as a whole – either to make things more efficient or because something, somewhere, just isn't working. To do that you have to know what is going on – to know what everyone is really doing. And to do that, you refer to a written procedure.

Fine, those are the reasons to write a process; but why the obsession with enforcing it? It depends. 

In some cases it really makes a difference. Cleanliness and sterilization procedures in health care can mean the difference between patients getting better and patients getting seriously worse. Precision and process control in aeronautics can mean the difference between flying and falling. And in any other highly-regulated industry, strict process enforcement can mean the difference between legal compliance (on the one hand) and the government shutting your doors (on the other).

What about when the consequences of non-compliance are a lot milder? In the first place, if it truly doesn't matter how you do a task, rewrite the procedure so that it only says what to do (what your output should be) and not how to do it. In the second place, even then process enforcement still matters if only because: If you don't enforce the process, you have no way of knowing whether the things you wrote in it are true. And if you don't know whether it's true, the process is just waste paper. 

Notice that in this last case you are not ensuring the quality of the outcome or work product; in this specific case you are ensuring the quality of the written document itself, so that you can then use it for all the tasks described up above. What this means is that if you try to enforce the process and find out halfway through that the procedure as-written is wrong, you have to be ready to change it. Never force someone to do something wrong** merely to guarantee formal compliance*** with an internal document. But be ready to update the document so that what it describes is right, and enforce that. Only then is it a functional tool.

 __________

* In fairness, this particular Big Boss wasn't usually a foolish man. I think maybe he was having a bad day. Or else we had really frustrated him.

** The definition of "wrong" depends on context, but includes (for example) anything illegal, immoral, or impossible. It should also include anything impractical, with the caveat that sometimes a requirement looks impractical until you step back to see how it fits into the bigger picture.

*** I speak here of formal compliance, not legal compliance. Legal compliance is non-negotiable. A breach of legal compliance counts as "wrong," as explained immediately above.