Microsoft’s preliminary explanation for the November 12 Microsoft 365 outage points to a July software defect, shifted Teams traffic and reduced gateway capacity, according to Neowin. Automatic traffic rerouting ended up giving healthy servers the same problem. Source
Microsoft said on November 14 that services had fully recovered after more than a day of disruption, Reuters reported. The outage interrupted email, calendars and meetings for businesses and governments internationally. Source
Three problems, one very bad combination
Neowin’s summary of Microsoft executive Charles Lamanna’s explanation describes three problems colliding:
- A hidden July defect. A software fix introduced a bug in Microsoft 365’s network gateway servers that could make them stop responding under certain workloads. Source
- More Teams traffic. Several hours before the major outage, Teams traffic moved from an alternative gateway to the primary gateway that also handled Exchange Online. This was a normal failover scenario, but it increased the load and changed traffic patterns. Source
- Less available capacity. About 14% of gateway server capacity was already unavailable because of an unrelated problem from the previous day. The remaining capacity was sufficient for normal workloads, but the combination of shifted traffic and reduced capacity triggered the defect. Source
The faulty update had passed regular testing and rolled out globally over several weeks. That testing primarily used older, less powerful hardware with fewer CPU cores; the bug was more likely to occur on newer servers with more cores, Neowin reports. Source
As servers stopped responding, automatic rerouting sent their traffic to healthy machines. The extra load then triggered the same defect on those machines. In less than two hours, the failure spread across 80% of Microsoft’s network gateway fleet, according to Neowin’s account. Source
Separate capacity partitions and regions would normally contain this kind of failure, but another issue prevented those safeguards from working. Neowin’s account does not explain that issue further. Some services, including Teams, OneDrive, SharePoint and Office apps, could move away from the affected gateways; Exchange Online could not use the same failover mechanism.
Comments