Can you be TOO optimized?

There's an old story about a wise man that has some relevance for bloggers as well.

Once upon a time there was an old man renowned for his wisdom. Whenever a discussion got tangled and tempers rose, he always told a parable that resolved the question perfectly. Finally a little boy asked him, How could he unfailingly pull out a story each time that summarized the issue so well? "Ah," said the old man, "you don't understand my method. I don't ransack my brain looking for a parable to match each new conversation. Rather, I introduce a topic of conversation for which I already have the perfect parable."

Blogging is a little like that. We bloggers are sure we have the right answer, whatever the question. And by the time you read a post all the way to the end, we've usually made the point pretty convincingly. It almost looks like we're right about everything. (Or I hope it does. 😃) But then, ... who picked the topic for the post in the first place? We did! So of course we're going to pick topics where we think we can win the argument.

But I've been thinking recently about a plant I encountered a few times during my career, where things seemed slightly off-kilter but I could never tell why. I didn't work there, and they weren't part of my normal responsibilities. But I visited a couple of times, to support audits or for other reasons. The audits turned up findings, as audits always do; and the findings were always addressed in the usual way. Somehow, though, I never got the sense that we uncovered anything more than surface issues. It felt like there was something more basic that we were missing.

This plant was located in a town I'll call Riverville, so I can refer to the place as the Riverville plant.

Recently I've been formulating a hypothesis that might account for part of the situation there. So I want to describe the hypothesis, and ask whether you think this could pose a problem in real life? I won't go into a lot of details about how things were at Riverville, if only so that I don't identify it too plainly. But let me know if you think the circumstances I describe could cause some related mis-fires.

Then if it turns out that I'm all wrong, I can go back to questions where I (think I) already know the answer in advance. 😀  

The Riverville plant

Visiting and auditing

The Riverville plant had been in operation for many years. The people were deeply competent, and committed to their work. Also, because these things happen, the plant had undergone multiple changes of ownership throughout the decades it had been in business. Sometimes the new owners ran it as a standalone operation. But occasionally the new owners wanted it to integrate into a larger ecosystem of other businesses that they had also acquired. When I knew them, Riverville had been recently acquired by Octopus Enterprises (not their real name).

Despite the acquisition, though, the plant never felt like the other Octopus plants that I had seen. When visitors started to quote a lot of the normal Octopus manage­ment terminology, I thought I could see the Riverville old-timers smile to them­selves, as if to say, "Sure, we'll play along until you go home. But that's not how it really works." When I audited, I kept turning up forms that I'd never seen at any of the other Octopus plants. "Don't you use the standardized Octopus forms?" "Oh sure, sure, ... mostly we do. But that one's an exception because of the special way we handle [topic] here. It's authorized, though. Let me show you the procedure document!"

And there was always a procedure document to back it up. There were so many procedure documents, in fact. Sometimes I thought it would be an interesting project to collect all of them and paste them together, to try to get a picture of how the whole plant functioned. But of course I couldn't. As I said, I never worked there regularly, so in the normal course of things I would have had no business even raising the question. And in the absence of a total breakdown of their system—which never happened during the time I was familiar with the plant—I could never have made an argument to justify the time and effort. Besides, mostly the folks at Riverville were able to provide the right management deliverables back to Octopus headquarters when asked, and mostly they were able to get their regular work done in a routine way.

Fully optimized

As for those forms? Oh, the forms were awesome! Someone had put a lot of time into these. There were links to data sources on the internal network, so that when you needed to fill in fields 5-15 you just clicked a button on the form. Then you added a little more data by hand and clicked another button, and right away the form routed itself to the next person in line. None of this was managed by some overall data management tool. It was all hard-coded into the forms with URLs that pointed to archive repositories where project information was stored in suitably-structured Word and Excel files. 

Rebuilding the engine in flight.
This is not recommended.

In fact, the forms were the key to my recent hypothesis. I think the Riverville plant was too optimized to change! They had worked together for so long that they had optimized every single transaction in a large and diverse factory. But they optimized them all according to the outlines of the old Riverville management system. If a new owner let them run themselves, and just showed up at the end of the year to collect the profits, there was no harm done. But when Octopus Enter­prises tried to make them integrate with the Octopus manage­ment system, the only way they could do it was to run two systems in parallel. They couldn't stop using the old forms, because everyone relied on them. They couldn't stop using the old data­bases, because the forms pulled data from them. They couldn't stop working in the old way, because everyone in the company depended on every step of the old system. To replace the old Riverville system with the new Octopus system would have meant rebuilding the airplane engine in flight. 

I was never able to dig deeply enough to prove that the Riverville personnel were running two systems in parallel—I mean, as opposed to adapting the Octopus system by adding "just a couple" of the old Riverville forms—but it would make sense of my general observation that somehow things were a bit off. As we discussed last week, running the Octopus management system as a Potemkin system would have been a perfectly natural response, if the Riverville management felt that they were being forced to make changes that didn't help them any. 

The problem, of course, is that working in two parallel systems takes twice as much effort as working in one. Also, if you have data in two places, it is almost impossible to keep them both consistent. Whatever options Riverville faced, this one cannot be called The Easy Choice.

End of the Bronze Age

My thinking about Riverville has been supported in an interesting way by an article that appeared in LinkedIn around the end of March. The author is Marco Nutini, a risk-management expert from Brazil, and the title is "1177 BC Called. We're Not Listening." 

It's a fun article, and not long. Read it, if you have a few minutes. Nutini starts off summarizing his thesis with admirable brevity:

In the late Bronze Age, the Eastern Mediterranean was a marvel of interconnection. Egypt, the Hittites, Mycenaean Greece, Ugarit, Cyprus — linked by trade routes, diplomatic marriages, and supply chains that moved copper, tin, grain, and luxury goods across vast distances. It was, by the standards of the era, a globalized world.

Then it collapsed. Not from a single cause, but from all of them at once.

His point is that the highly interconnected world of the late Bronze Age was optimized for current circumstances. Assuming that the world stayed at peace, that the rain fell reliably, and that no major population centers were wiped out by earthquake or volcano—assuming all that, the world ran just fine. But when those assumptions failed—when drought, famine, and earthquake came, when cities revolted and the Sea Peoples were landing on every shore—the finely-calibrated economy of the late Bronze Age could not withstand the shocks. It fell apart, collapsing into the Greek Dark Ages. Optimization for normal times is not the same thing as resiliency in trying times.

Admittedly, having your factory bought by Octopus Enterprises shouldn't be as devastating as having your seacoast raided by pirates, or your island explode. But either way, I think there is a risk that over-optimization can make you fragile and unable to adapt when circumstances change.

If I'm wrong, feel free to tell me so!