Thursday, November 19, 2026

The scariest word in the English language

Last week, I wrote about an article from 1960 which predicted (somewhat tongue in cheek) that the world would end on Friday, November 13, 2026. I thought it would be a light-hearted piece with insights into the use of numerical models, and no more than that.

By the next day, readers were asking if I could tell the future. No, strictly speaking the world didn't end. But on midday Thursday, just as my post was going live, Microsoft started losing servers. Within an hour, Outlook and Copilot were unavailable in 74 countries, shutting off access to email, calendars, and a host of Microsoft 365 tools that businesses rely on for everyday work.

The day after that—Friday the 13th itself—the FAA issued a nationwide ground stop lasting four and a half hours, and preventing 3500 flights from taking off. Planes that were already in the air were allowed to land, but no new flights were allowed to leave until the problem was contained. The follow-on effects were even greater: in the end, nearly 8000 flights were canceled, and over 10,000 flights were delayed. These included regular passenger flights as well as cargo flights, and even charter flights used by many professional sports teams to get to scheduled engagements. Nobody was injured, but the effect on customer plans and schedules across the country was catastrophic.

[When I first wrote this article, those were the only outages in the news. The fact that more have followed, day after day since then, doesn't change any of the conclusions I draw here, but just makes the whole topic more acute.]   

The crowning irony is that both these devastating failures had the same proximate cause, namely, each was triggered by a recent system upgrade.* For the Microsoft outage, Fox News reports that:

A software bug introduced in July went undetected until heavier traffic and reduced server capacity triggered a chain reaction during peak U.S. business hours Nov. 12, Microsoft executive Charles Lamanna said in a post on X.... The flaw came from a fix developed specifically for Microsoft 365 gateway servers, Lamanna said. Neither the defect nor that fix applies to broader Windows deployments.

Calvin Trillin is—quite rightly!—afraid
of the word upgrade.
Meanwhile, for the FAA outage, Reuters reports that:

A massive, nearly five-hour shutdown of the US aviation system that has disrupted more than 20,000 flights since Friday may have been the fault of a single router installed by government contractor L3Harris, government officials said Saturday. 

In short, last week provided ample confirmation of Calvin Trillin's mordant observation some years ago that the scariest word in the English language today is upgrade.**

But why? Why are upgrades so bad? Why do we all assume that upgrades to our personal devices will mean worse performance, and—more alarmingly—why do organizations responsible for mission-critical systems (like Microsoft and the FAA) permit upgrades to those systems that shut them down? Don't they test this stuff before releasing it? 

Of course they test it. I assume they test it extensively.*** The problem is that, as systems become more complex, it gets more difficult to test them thoroughly. Even though the logic inside any given circuit is deterministic, the performance of specific tasks can be affected by factors such as system load, background processes, interrupt activity, and resource contention. As a result, the behavior of sufficiently complex systems tends to become non-deterministic. The same inputs do not always produce the same outputs, because it's possible that something else on the network might interfere unpredictably. So test engineers have to make their best guess about what to test, in what combinations, and for how long. If a certain behavior never surfaces until after—oh, let's say 10,000 hours of continuous uptime—and if the system was tested for only 9000 hours of continuous uptime, then the test will never detect the behavior. (Just for reference, 1 year = 8766 hours.) Microsoft's spokesman explained that their outage took them by surprise for exactly this reason:

Lamanna said the software change had operated worldwide for more than three months without problems. Microsoft’s testing missed the flaw because its labs did not reflect the mix of traffic and hardware that later triggered it. The bug was worse on newer, more powerful machines, while testing tended to focus on weaker hardware.

So does that mean that Microsoft's test engineers were wrong, wrong, wrong? In a sense I guess they must have been, because their choice led to a worldwide outage. But it might still have been a reasonable choice. I can understand wanting to prioritize tests on weaker equipment, on the assumption that the stronger equipment can look out for itself. This once, that was the wrong call to make, but it might have been the right call a dozen other times. 

Real root cause analysis is supposed to prevent the recurrence of the problem. I wish the best for the problem-solving teams at Microsoft and the FAA—and at all the other organizations that make mission-critical systems which affect our lives—but I fear that the ineradicable complexity of these systems means that we will continue to see unplanned outages at random intervals, because we will never be able to test them in ways that perfectly simulate the real world.

On the other hand, I would be glad to be proven wrong.

__________

* I specify "proximate cause" because problem analysis is still under way (at least for the FAA failure), so the real root causes aren't known yet or have not been made public. Microsoft blames their failure on a "latent defect," but that is just a description of the situation and not a cause per se. 

** Trillin first made his remark in this article for the New York Times. Since then, the concept has been picked up by Robert Parkhill writing into the Guardian, and by John Michael Greer in a recent post. 

*** Though I've never actually worked for either Microsoft or the FAA.   

           

Thursday, November 12, 2026

Will the world end TOMORROW?

Will the world end tomorrow? That sounds crazy, but ... not so fast. Let's stop and look at it.


In 1960, a team of respected scientists published an article in the journal Science, entitled "
Doomsday: Friday, 13 November, A.D. 2026." The article appeared in Volume 132, Issue 3436, dated November 4, 1960, running on pages 1291 through 1295. The authors were Heinz von Foerster, Patricia M. Mora, and Lawrence W. Amiot, and the main topic was a study of the growth of human population. The authors pointed out that any population is governed by its rate of increase (fertility) and its rate of decrease (mortality). They set up the relevant differential equations, analyzed them based on the known historical data, and finally settled on a closed-form equation to calculate N, the human population of the Earth at any given time t.

Note that the denominator of the expression includes the term (t0 – t), where the current time and date (repesented by t) is subtracted from a reference date t0. Obviously if we want to take this formula seriously, we have to know when t0 is, because on that date the denominator of the expression falls to zero and the value of N becomes undefined. What is perhaps more frightening, in the days, hours, and minutes before t0, the denominator will be very small but still positive. So at that point the value of N will be very large—and growing ever more rapidly—but still (for the moment) defined.

The authors recognized that their article required them to make a determination of the value of t0, and so they did. After careful consideration of the empirical data regarding current and historic human populations, they determined that t0 would fall late in 2026, on the 317th day of the year. (Arithmetically, 0.87 x 365 = 317.55.) The 317th day of 2026 is Friday, November 13. Tomorrow! By coincidence, that is also Heinz von Foerster's 115th birthday, or it would be if he were still alive.

So if the von Foerster formula is accurate, we should expect the human population of the Earth to balloon towards infinity today and tomorrow, and then presumably to vanish as the value becomes undefined. If the formula is accurate, tomorrow is Doomsday.*

Heinz von Foerster, 1963, from University
of Illinois publicity department
Logically the next question has to be ... exactly how accurate is that formula? Well, back in 1960 it was pretty accurate, because it was built from the observed data. And in fact for some years thereafter, the predicted population values tracked the actual ones pretty closely. Nearly four decades after publication, the formula predicted that the world's population at the beginning of 1997 would be 5,986,622,074. The actual value was 5,924,787,816, or 98.967% of the predicted value. That's remarkably close.

Then somewhere around the turn of the millenium, the growth of human population began to slow down. Current projections are that human population will stop growing in absolute numbers before the end of the 21st century, and will begin to decline. Many explanations have been offered; I don't know if there is professional agreement on a single one. But in any event, the geometric increase predicted by the von Foerster doomsday formula no longer represents empirical reality.

We can afford to exhale. The world is unlikely to end tomorrow. 

Now, you might ask: This story is fun, but what does it have to do with Quality? Simply this. We use a lot of mathematical models in Quality. Those are valuable, because if we can test products mathematically we don't have to destroy them in real life to find their breaking points. Mathematical testing is a lot less wasteful than destructive testing. But you have to make sure that the mathematical model is valid. Compare it at all the critical points against experimental data. If time is a relevant factor, then check it again after some time has elapsed. It is easy to be misled by a fancy model that no longer checks out. 

We've discussed this topic before, but the point bears repeating.       

__________

* On the other hand, the formula also predicts that the human population of Earth today—Thursday, November 12—should be 65,335,000,000,000. That's over 65 trillion people. Spoiler alert: it's not.  

           

Thursday, November 5, 2026

Quality tools in kindergarten

Last month we examined the story of an engineer who used Quality methods to win at Jeopardy!; and I have made the argument more than once (for example here) that Quality methods are just a form of organized common sense that builds on re-using known solutions to known problems. There is nothing inherently industrial or commercial about the Quality mindset. All that is required is a desire to do things successfully in an organized way.

So why not use Quality tools in ... oh, I don't know ... let's say kindergarten? Well, it turns out someone has. And I've linked a five-minute video at the bottom of this post in which a couple of subject-matter experts (or SMEs, in this case kindergarteners) explain to an interviewer how their class uses process flow diagrams and control charts.    

I don't know the origin of this video. If you do, please leave me a note in the comments. But the concept is straightforward.

In the first place, the SMEs explain their flow chart for how to begin the day. You come into the room, and then have to check whether you have a backpack. If yes, unpack it and hang it up, and then return to the regular flow; if no, just proceed with the regular flow. Then you put your name up for lunch, sign in, sit down on the rug, and focus on the calendar.

This all sounds pretty straightforward. Did the class really need a flowchart to tell them that? Well yes. The SMEs explain that before they had the chart, kids would wander into the room and not know what to do. Sometimes students with backpacks would forget to take them off or unpack them, but would just sit down on the rug still wearing their backpacks. Or students might forget to focus on the calendar, thus losing time later in the day. 

Next they explain the control chart that tracks how long it takes them to clean up at the end of the day. Each day, the clean-up time is recorded (in minutes) with a dot on the chart. There is a line showing the expected average time, and bands showing the acceptable variation. The SMEs explain very clearly that the goal is to keep their time near or below the planned average, but not to drift too far above it. The interviewer asks, "Well what if it took you a lot longer? What if one day the dot was way up here, past the limit line?" His interviewee explained very soberly, "That would mean we were out of control." And there was no doubt in his voice that that would be a bad thing.

So yes, you can apply Quality tools in kindergarten. I hope you enjoy the video. 


 

Thursday, October 29, 2026

The good ship Vasa

I have long thought that project managers should all have a framed painting of the Swedish warship Vasa to hang in their offices. It must be the most famous object lesson in history for the importance of requirements management.

The Vasa was built between 1626 and 1628, to be the flagship of Sweden's naval squadron in the Stockholm Archipelago. [This history is condensed from the article in Wikipedia.] The ship was designed by Henrik Hybertsson (shipwright at the Stockholm shipyard), relying on requirements from King Gustav II Adolf (known to history as Gustavus Adolphus the Great). Hybertsson was a master shipbuilder, and the Vasa should have been a great warship. But after the keel was laid and the fundamental architecture of the ship could no longer be changed, the King kept asking for improvements. The ship needed more sails, and taller ones. It had to carry more cannons, and heavier ones. And so on. No one had the courage to say "No," so the project drifted behind schedule. Then Hybertsson fell ill and died. A new shipwright took over the job. By this time the King was impatient, and wanted the team to finish their work so the Vasa could be launched as soon as possible.

On her maiden voyage, the Vasa left port and sailed roughly 1400 yards into Stockholm harbor. The weight of all her extra sails and cannons meant that she rested very low in the water. After a few minutes, a light breeze tipped her slightly to one side ... and even that little was too far. She began to take on water, and sank on the spot.

Gustavus Adolphus. When he asks
for more cannons, tell him "No."
Although she failed as a warship, the Vasa had a much more successful afterlife as a museum. In 1961, the ship was finally recovered from the harbor floor. It turned out that the mud had preserved her very well, and many of the artifacts on board were still in good condition. Archeologists worked for several years to restore the ship, and in 1990 the Vasa Museum opened to the public.

The Vasa is fascinating as a museum. I've been there, and I recommend it whole-heartedly. But of course it was a complete failure as a warship. And the reason it failed so utterly is precisely that the project team failed to control their requirements.

Once the keel was laid, the ship's architecture was more or less fixed. It could support sails of a certain height, and cannons of a certain weight. No more than that. And these limits derived from the laws of physics, which cannot be overruled by royal decree.

I'm pretty sure that every case of scope creep in every large project since then has the same basic elements: an initial design based on one set of assumptions, work that has progressed far enough that it is not economical to start over, and then new requirements from someone who doesn't understand their consequences and whom no one has the courage to confront. But that's why every project management office in the world should be decorated with a painting of the Vasa. It may be painful to tell your King "No," but it's better than letting your grand warship sink to the bottom of the harbor less than a mile into her maiden voyage.    

           

Thursday, October 22, 2026

Pay your suppliers

A few weeks ago, Christopher Paris published an article about companies who don't pay their suppliers—or rather, strictly speaking, companies who make their suppliers wait four, five, or six months to get paid, even when the contract plainly requires payment in 15 days net. He describes having dealt with companies just like this, and I believe it.

Gothmog rejecting an invoice.
Back when I was first starting out, I worked indirectly with a company I'll call Gothmog Enterprises. (Don't even try to guess who they were in real life; my LinkedIn profile doesn't go back that far.) Gothmog Enterprises was a very large company. And there were persistent rumors that they preferentially did business with small, local suppliers who were desperate for the orders. Then when the invoice arrived—so the rumors used to claim—Gothmog Enterprises would pay 60% and stamp it "Payment in Full." This left the suppliers with a painful choice: continue to do business with Gothmog, knowing that they would only ever get 60 cents on the dollar; or cut them off, and be cut out of the massive volume of business that Gothmog made available. Of course it was a terrible choice for the suppliers, and for Gothmog to force it on them was unambiguously evil. 

Maybe it was only a rumor and never really happened.

"ISO 9001 would never let you
mistreat your suppliers!"
Now, Chris Paris sells consulting related to management systems, for example (but not only) those certified to ISO 9001 or AS9100. So it would be tempting to think he shouldn't have to deal with this kind of problem. You might even think to yourself, Surely a company with a functioning quality system wouldn't be allowed to play games like this with their suppliers! But this would be to look at the world through rose-colored glasses. Not sure you agree? Let's check.

ISO 9001:2026 covers all the requirements relating to suppliers in clause 8.4. (Property belonging to suppliers but stored at your workplace is covered in clause 8.5.3.) There is no prefatory material in clause 8.4, but just three subclauses.

  • Clause 8.4.1 says that anything provided by a supplier and ultimately bound for your customers must conform to requirements, and must be put under a suitable control to ensure that it conforms.  
  • Clause 8.4.2 says that you have to figure out what controls are appropriate, and whether to apply them to the things you buy, or to the suppliers themselves, or both.
  • Clause 8.4.3 says you have to tell the supplier about all your requirements, and about all the controls you intend to apply.
  • There's nothing about paying your bill afterwards.

And that's it! Oh, I suppose with this latest edition of the standard you could try to build an argument around clause 5.1.1(i), which says that top management is supposed to "promot[e] quality culture and ethical behavior." Maybe you could argue that letting your suppliers wait for months before they are paid is a funny kind of "ethical behavior." It would certainly be worth making the argument. But don't expect to win.

This might look like a gap in ISO 9001, but I'm not sure I agree. I'd rather call it a limitation of scope. ISO 9001 was never intended to describe ever single thing a company should do in order to qualify as praiseworthy or good. It is specifically a quality management system standard, which means that it should be focused strictly on customer satisfaction. If I can make my customers happy while being an unethical jerk to my suppliers, ... well, morally I still shouldn't do it; but as long as my customers don't know that I am an unethical jerk they won't care. In the short term, before my picture starts showing up on social media headlining lurid exposés of my crimes and misdeeeds, my customers will still be satisfied. And in that sense—for that short term—my orc-ish behavior towards my suppliers is out of scope for ISO 9001.

Where the treatment of suppliers does re-enter the larger Quality ecosystem is in the more general consideration of the seven Quality principles (spelled out in ISO 9000). One of these principles is Relationship management, which covers relationships with all interested parties. And in case it's not already obvious, suppliers are always interested parties. The point of the discussion of these principles in ISO 9000 is that even if yes, technically a company can get away with observing the letter of the ISO 9001 requirements while still being a scoundrel in other respects, in the long run it's not a sustainable strategy. Over time, suppliers will walk away from them, blacklist them, and refuse to do business with them—exactly the way that Chris Paris describes in the article I mentioned at the beginning of this post! Any company who wants to secure the ability to do business in the long term has to do more than narrowly follow the rules. 

Among other things, this means paying suppliers.  

           

Thursday, October 15, 2026

When half a loaf is worse than none

You work in a small office that's part of a big organization, and the experts at Headquarters just issued a new regulation. Everyone has to comply. You don't have the budget to implement the whole thing, but you could do half of it with no trouble. Should you do that half, just to show good faith? Isn't half a loaf better than none? 

Not so fast. It all depends.

Of course sometimes that's the right thing to do. Implement Part One now, and make a plan to implement Part Two next year when you can afford it. Communicate with the people at Headquarters so everyone understands the situation, and make incremental progress.

But evaluate every situation on its own. Sometimes implementing half a loaf will undermine you, and you would do better to wait till you can do the whole thing at once.

I was in exactly this situation once, years ago. I was working for a small, entrepreneurial startup that had just been acquired by a global company. The global company had a lot of systems and regulations that we had to learn in a hurry.

One bundle of regulations addressed security. Every office and every plant worldwide—except ours—issued employee badges. Those badges let you into the building at your site. If there were restricted areas at your site, your badge could be fine-tuned to let you into exactly the areas you were allowed to access. If you went to work in a new location, your badge could be reprogrammed to open the doors at the new location instead. It was a complex, interconnected system.

We had none of that.

And right away, visitors started to comment. 

"What, no badges? How can you work for Big Company without badges?" 

"Gosh, the next time I need a new printer I'll just fly out here and take one from your office since your security is so lax."

"Does Headquarters know that you guys don't have badges?"

Someone in my department decided we should do something right away. She started scheduling appointments for everyone in our office to have a picture taken, and ordered a box of badge blanks. She was halfway through taking all the pictures before I had a chance to sit down with her and ask what she was thinking.

"Well everyone keeps saying we need badges. I figured I could make badges, and then our guests would be happy."

"But badges are just part of the picture," I reminded her. "What about the electronic keys and the huge IT infrastructure that links us into the rest of Big Company worldwide? Where is the budget for that going to come from, and who is going to implement it?"

"I thought we could do that later."

That was the key. She figured half a loaf is better than none, and in a different case she might have been right. But in this specific case, it didn't work, and couldn't possibly work.

I told her, "Think a minute. Suppose it is the day after you issue badges to everyone, and explain they are mandatory. How are you going to enforce them?"

"What?"

"Say that Frank forgets his badge at home. But he can still get into the building, right? Because the badge isn't an electronic key, so it isn't hooked into the security system. Did you think we were going to hire guards to stand at the door and stop people without badges?"

"No."

"That's right. There's no budget for that. Only then what happens is that everyone realizes the rule about wearing badges has no teeth. They can openly ignore it, with no consequences. And then the next time we have to implement a new rule from Big Company, it'll be twice as hard."

"So what do you want me to do? Should I run around the building every day to check that people have their badges on?"

"No. What I want is for us to wait and do nothing until we can afford to implement the whole package all at once."

That's what we did, and it took a couple of years until the electronic entry system became a high enough priority for our management to pay for it. In the meantime, Big Company understood that we were non-compliant with that specific set of security requirements, and treated the situation as a known risk to be managed. Mostly it didn't take a lot of attention, and we all went about our business just fine. When the day finally came, we did everything at once; we were fully integrated into Big Company's IT infrastructure; and, yes, we got our badges.

Every case is different. There is no substitute for thinking through all the details to understand what happens if you take this option or that one. But sometimes it is simpler just to be noncompliant—fully above-board, so that you are known to be non-compliant—than to get tangled in the complexities of trying to do something halfway.

           

Thursday, October 8, 2026

Quality methods for the win!

I've often argued that Quality methods are just as useful in real life as they are in the workplace, but this month's issue of Quality Progress is running a story that makes the point better than I ever could. Jared Watson, a senior quality engineering specialist at L3Harris Technologies, published an article about how he used systematic quality improvement methodologies to win big on the game show "Jeopardy!" 

You can read the full article online. But to summarize, Watson won his first rounds (in June 2023) with only "a month of preparation and a lifetime of enthusiastic—if amateur—curiosity." But then he was invited to compete in the Tournament of Champions, and he realized that his preparation would have to reach a whole new level. So he analyzed his gaps as clearly as possible, and made a plan to address them.

The first step was honesty. In my original run, I’d lost thousands of dollars on geography clues, including two Daily Doubles, and geography is the most frequently tested “Jeopardy!” category. If I didn’t confront that weakness, it would beat me again.

Next came help. Improvement accelerates when we invite others into the process. In my case, I had a sleeping trivia giant in my house ...: my wife, Jasmin. What began as a simple request for a few flash cards turned into more than 10,000 handcrafted study prompts .... Her support became the backbone of my preparation.

But tools and support only matter if you put in the work. Early on, I tried to master new material in a single sitting by drilling the same cards until I was frustrated. It didn’t work. What did work was steady, sustainable repetition: reviewing each card no more than twice a day, trusting that consistency—not perfection—would lock in the knowledge over time. And it did....

This lesson is one our field knows well: Quality improves through honest assessment, smart use of resources and disciplined, incremental effort. Whether on the “Jeopardy!” stage or in our organizations, consistent practice beats the illusion of overnight mastery every time. 

With this solid preparation, Watson posted a perfect performance during the quarterfinals, answering 21 for 21 and all three Daily Doubles. Later he was eliminated in the semifinals, but the power of the method to get him that far was absolutely clear. As Watson himself concluded, "That’s what quality looks like in practice: A commitment to the process, not just the final score."

Congratulations!



           

Five laws of administration

It's the last week of the year, so let's end on a light note. Here are five general principles that I've picked up from working ...