Microsoft has revealed some details about what it believes caused the recent global outage of Office 365 and some of its other platforms. Users were left behind after the Office 365 outage around the world, and other services like Microsoft Teams, Office.com, Power Platform, and Dynamics365 were affected as well. According to Microsoft, the outage was caused by a failed deployment of an Azure AD service update.
Office failure
A preliminary report from the company found that the update was released too soon, as it did not follow the company's usual testing regimen. This typically involved moving through five "rings" prior to release, allowing Microsoft to test any change or update with a defined group of vetted testers. This time, however, a bug in Microsoft's Secure Deployment Process (SDP) caused the update to be deployed to all rings instead of the first proper test ring. "Azure AD is designed to be a geographically distributed service deployed in a multi-partition active-active configuration in multiple data centers around the world, built with isolation boundaries," Microsoft said in its post-preliminary report. incident. Typically, changes initially target a validation ring containing no customer data, followed by an inner ring containing only Microsoft users, and finally our production environment. These changes are rolled out in phases across five rings over multiple days. " "In this case, the SDP system failed to correctly target the validation test ring due to a latent flaw that affected the system's ability to interpret the implementation metadata. Therefore, all rings were attacked simultaneously. The implementation Incorrect error resulted in degraded service availability. Following the unexpected release, Microsoft claims to have attempted to roll back "minutes after the impact" using its automated recovery systems that would normally have limited the duration and severity of the impact. "However, the hidden flaw in our SDP system had corrupted the deployment metadata and we had to resort to manual restore processes. This significantly extended the time to mitigate the issue," the company's report said, explaining why the issue affected users around the world. Users who were already signed in to Office 365 or one of the other services are not were affected.Via BleepingComputer