Last week, I wrote about an article from 1960 which predicted (somewhat tongue in cheek) that the world would end on Friday, November 13, 2026. I thought it would be a light-hearted piece with insights into the use of numerical models, and no more than that.
By the next day, readers were asking if I could tell the future. No, strictly speaking the world didn't end. But on midday Thursday, just as my post was going live, Microsoft started losing servers. Within an hour, Outlook and Copilot were unavailable in 74 countries, shutting off access to email, calendars, and a host of Microsoft 365 tools that businesses rely on for everyday work.
The day after that—Friday the 13th itself—the FAA issued a nationwide ground stop lasting four and a half hours, and preventing 3500 flights from taking off. Planes that were already in the air were allowed to land, but no new flights were allowed to leave until the problem was contained. The follow-on effects were even greater: in the end, nearly 8000 flights were canceled, and over 10,000 flights were delayed. These included regular passenger flights as well as cargo flights, and even charter flights used by many professional sports teams to get to scheduled engagements. Nobody was injured, but the effect on customer plans and schedules across the country was catastrophic.
[When I first wrote this article, those were the only outages in the news. The fact that more have followed, day after day since then, doesn't change any of the conclusions I draw here, but just makes the whole topic more acute.]
The crowning irony is that both these devastating failures had the same proximate cause, namely, each was triggered by a recent system upgrade.* For the Microsoft outage, Fox News reports that:
A software bug introduced in July went undetected until heavier traffic and reduced server capacity triggered a chain reaction during peak U.S. business hours Nov. 12, Microsoft executive Charles Lamanna said in a post on X.... The flaw came from a fix developed specifically for Microsoft 365 gateway servers, Lamanna said. Neither the defect nor that fix applies to broader Windows deployments.
![]() |
| Calvin Trillin is—quite rightly!—afraid of the word upgrade. |
A massive, nearly five-hour shutdown of the US aviation system that has disrupted more than 20,000 flights since Friday may have been the fault of a single router installed by government contractor L3Harris, government officials said Saturday.
In short, last week provided ample confirmation of Calvin Trillin's mordant observation some years ago that the scariest word in the English language today is upgrade.**
But why? Why are upgrades so bad? Why do we all assume that upgrades to our personal devices will mean worse performance, and—more alarmingly—why do organizations responsible for mission-critical systems (like Microsoft and the FAA) permit upgrades to those systems that shut them down? Don't they test this stuff before releasing it?
Of course they test it. I assume they test it extensively.*** The problem is that, as systems become more complex, it gets more difficult to test them thoroughly. Even though the logic inside any given circuit is deterministic, the performance of specific tasks can be affected by factors such as system load, background processes, interrupt activity, and resource contention. As a result, the behavior of sufficiently complex systems tends to become non-deterministic. The same inputs do not always produce the same outputs, because it's possible that something else on the network might interfere unpredictably. So test engineers have to make their best guess about what to test, in what combinations, and for how long. If a certain behavior never surfaces until after—oh, let's say 10,000 hours of continuous uptime—and if the system was tested for only 9000 hours of continuous uptime, then the test will never detect the behavior. (Just for reference, 1 year = 8766 hours.) Microsoft's spokesman explained that their outage took them by surprise for exactly this reason:
Lamanna said the software change had operated worldwide for more than three months without problems. Microsoft’s testing missed the flaw because its labs did not reflect the mix of traffic and hardware that later triggered it. The bug was worse on newer, more powerful machines, while testing tended to focus on weaker hardware.
So does that mean that Microsoft's test engineers were wrong, wrong, wrong? In a sense I guess they must have been, because their choice led to a worldwide outage. But it might still have been a reasonable choice. I can understand wanting to prioritize tests on weaker equipment, on the assumption that the stronger equipment can look out for itself. This once, that was the wrong call to make, but it might have been the right call a dozen other times.
Real root cause analysis is supposed to prevent the recurrence of the problem. I wish the best for the problem-solving teams at Microsoft and the FAA—and at all the other organizations that make mission-critical systems which affect our lives—but I fear that the ineradicable complexity of these systems means that we will continue to see unplanned outages at random intervals, because we will never be able to test them in ways that perfectly simulate the real world.
On the other hand, I would be glad to be proven wrong.
__________
* I specify "proximate cause" because problem analysis is still under way (at least for the FAA failure), so the real root causes aren't known yet or have not been made public. Microsoft blames their failure on a "latent defect," but that is just a description of the situation and not a cause per se.
** Trillin first made his remark in this article for the New York Times. Since then, the concept has been picked up by Robert Parkhill writing into the Guardian, and by John Michael Greer in a recent post.
*** Though I've never actually worked for either Microsoft or the FAA.










