OakLeaf Systems: Windows Azure Compute Availability Logs: North and South Central US Data Centers, 2/29 and 3/1/2012

Sunday, March 04, 2012

Windows Azure Compute Availability Logs: North and South Central US Data Centers, 2/29 and 3/1/2012

Following is a transcript of the logs for the Windows Azure Compute service disruption on 2/29 and 3/1/2012 for Microsoft’s North Central US (Chicago) and South Central US (San Antonio) data centers in chronological order:

[Windows Azure Compute] [North Central US] [Yellow] Windows Azure Compute Availability

Feb 29 2012 10:55AM We are experiencing an issue with Windows Azure Compute in the North Central US sub-region. Incoming traffic may not go through for a subset of hosted services in this sub-region. Deployed applications will continue to run. There is no impact to storage accounts either. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 12:30PM We are still troubleshooting this issue and capturing all the data that will allow us to resolve it. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 1:45PM We are executing restoration steps to mitigate the issue. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 2:30PM We have determined that about 6.7% of hosted services in this sub-region are impacted by this issue. The restoration steps to mitigate the issue are underway. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 3:30PM The restoration steps to mitigate the issue are underway. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 4:30PM The restoration steps to mitigate the issue are still underway. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 5:30PM The restoration steps to mitigate the issue are still underway. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 6:30PM The restoration steps to mitigate the issue are still underway. This incident impacts Access Control 2.0, Marketplace, Service Bus and the Access Control & Caching Portal in the same regions where Windows Azure Compute is impacted. As a result affected customers may experience a loss of application functionality. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 7:30PM We are actively recovering Windows Azure hosted services in this sub-region. More and more customers applications should be back up-and-running even if service management functionality is not yet restored. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 8:30PM We have recovered over half of the Windows Azure hosted services that were impacted. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 9:30PM Recovery efforts are still underway. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 10:30PM Recovery efforts are still underway. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 11:30PM Recovery efforts are still underway, we are 70% through the recovery process and we restored full service management functionality for more customers in this region. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Mar 1 2012 1:00AM Recovery efforts are still underway. The majority of Windows Azure customers and services have been restored. However, a few customers and services remain affected and we are working aggressively to restore full functionality. We restored full service management functionality for most customers in this region. We have published a recap of the incident since it started on the Windows Azure team blog (http://blogs.msdn.com/b/windowsazure/archive/2012/03/01/windows-azure-service-disruption-update.aspx). Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Mar 1 2012 3:00AM We are working on stabilizing the Windows Azure Platform as well as following-up with all customers who were impacted by this incident. The recovery process continues. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Mar 1 2012 4:30AM Recovery efforts are still underway, we are 85% through the recovery process in terms of impacted hosted services. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Mar 1 2012 7:30AM Our recovery efforts to restore compute service to impacted customers in this sub-region are complete. A small number of customers in this sub-region may face long delays during service management operations. We request any customers experiencing compute or service management issues in this sub-region to reach out to us through the support channel described on this site: (http://www.windowsazure.com/en-us/support/contact). We apologize for any inconvenience this incident causes our customers.

Mar 1 2012 10:15AM We have confirmed that full functionality is restored in the North Central US sub-region. We apologize for any inconvenience this incident caused our customers.

[Windows Azure Compute] [South Central US] [Yellow] Windows Azure Compute Availability

Feb 29 2012 10:55AM We are experiencing an issue with Windows Azure Compute in the South Central US sub-region. Incoming traffic may not go through for a subset of hosted services in this sub-region. Deployed applications will continue to run. There is no impact to storage accounts either. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 12:30PM We are still troubleshooting this issue and capturing all the data that will allow us to resolve it. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 1:30PM We are still troubleshooting this issue, and verifying the most probable cause. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 2:30PM We have determined that about 28% of hosted services in this sub-region are impacted by this issue. The restoration steps to mitigate the issue are underway. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 3:30PM The restoration steps to mitigate the issue are underway. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 4:30PM The restoration steps to mitigate the issue are still underway. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 5:30PM The restoration steps to mitigate the issue are still underway. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 6:30PM The restoration steps to mitigate the issue are still underway. This incident impacts Access Control 2.0, Marketplace, Service Bus and the Access Control & Caching Portal in the same regions where Windows Azure Compute is impacted. As a result affected customers may experience a loss of application functionality. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 7:30PM We are actively recovering Windows Azure hosted services in this sub-region. More and more customers applications should be back up-and-running even if service management functionality is not yet restored. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 8:30PM We have recovered over half of the Windows Azure hosted services that were impacted. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 9:30PM Recovery efforts are still underway. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 10:30PM Recovery efforts are still underway. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Feb 29 2012 11:30PM Recovery efforts are still underway, we are 70% through the recovery process and we restored full service management functionality for more customers in this region. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Mar 1 2012 1:00AM Recovery efforts are still underway. The majority of Windows Azure customers and services have been restored. However, a few customers and services remain affected and we are working aggressively to restore full functionality. We restored full service management functionality for all customers in this region. We have published a recap of the incident since it started on the Windows Azure team blog (http://blogs.msdn.com/b/windowsazure/archive/2012/03/01/windows-azure-service-disruption-update.aspx). Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Mar 1 2012 3:00AM We are working on stabilizing the Windows Azure Platform as well as following-up with all customers who were impacted by this incident. The recovery process continues. We are also focusing on restoring full service management functionality in this sub-region. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Mar 1 2012 4:30AM Recovery efforts are still underway, we are 85% through the recovery process in terms of impacted hosted services. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Mar 1 2012 7:30AM Our recovery efforts to restore compute service to impacted customers in this sub-region are complete. A small number of customers in this sub-region may face long delays during service management operations. We request any customers experiencing compute or service management issues in this sub-region to reach out to us through the support channel described on this site: (http://www.windowsazure.com/en-us/support/contact). We apologize for any inconvenience this incident causes our customers.

Mar 1 2012 4:57PM We have confirmed that full functionality is restored in the South Central US sub-region. We apologize for any inconvenience this incident caused our customers.

OakLeaf Systems’ live demonstration Cloud Apps are hosted in the South Central US data center and had a total of 35 minutes of downtime. See Uptime Report for my Live OakLeaf Systems Azure Table Services Sample Project: February 2012 post of 3/3/2012.