Thursday, February 01, 2007

Google's Blogger bX-vjhbsj Meltdown

Google earned a whopping US$1.06 billion (33% of revenues of US$3.21 billion) in the fourth quarter but it's obvious that the new 800-lb computing gorilla isn't devoting much of those enormous resources to its Blogger group.

As I was updating my post about the Coast Guard's search for Jim Gray at about 8:00 this morning (Thursday), I noticed the OakLeaf blog was loading very slowly and the editing feature was slowing down, too. Finally, I began receiving the following error page when attempting to review my posts:
Apparently, the Old Blogger network started falling down on Tuesday, January 23, infected New Blogger and, after a week's incubation period, New Blogger finally keeled over and died this morning. Here's the (reverse) chronology from the Blogger Status site:
Thursday, February 01, 2007 Some users see the error code bX-vjhbsj when trying to view a blog. We have identified the source of this error and will push the fix to the site shortly. In the meantime, hitting Refresh in the browser may workaround the problem. We apologize for the inconvenience. Posted by Pal at 08:20 PST.
Wednesday, January 24, 2007 New Blogger has had troubles this morning, which you may have seen with slow requests and 502 error pages.We are working on this and hope to get it resolved very soon.Update, 1PM: These issues have been resolved. We are continuing to monitor and diagnose closely. We apologize for this morning’s poor performance.You may still see 502 errors on occasion, but waiting a few seconds and reloading the page should work. This is mostly limited to a handful of blogs, however. Posted by Pete at 12:14 PST
Tuesday, January 23, 2007 Publishing on the old Blogger was failing with "Error 550" since approximately 5:30PM PST. It was fixed at approximately 8:00PM PST.New Blogger was unaffected. Sorry for the inconvenience. Posted by Jessica at 20:21 PST
Checking the Blogger Help Group, users began experiencing major-scale bX-vjhbsj errors starting at about 2:30 a.m. PST. By 9:00 a.m. there were more than 300 "How Do I?" posts and about 150 posts on the topic in the "Something Is Broken" and other categories.
Cringely's January 19, 2007 column, "When Being a Verb is Not Enough: Google wants to be YOUR Internet," says "Google intends to take over most of the functions of existing fixed networks in our lives, notably telephone and cable television. ... It's a grand plan, but can Google pull it off? Yes they can."
I don't think "they can" until Google puts its own network house in order and takes better care of its current patrons. Waiting "a few seconds and reloading the page" won't work with today's—much less tomorrow's—Internet users.

Subsequent Events: This afternoon (about 2:30 p.m. PST) I saw the following update message on the Blogger Status blog:

Update 10:50AM (PST): The issue has been fixed. Unfortunately pushing out the new build involved a few minutes outage. We apologize for trouble.

The bX-vjhbsj errors seem to be gone, but it's still taking much longer than usual (perhaps an order of magnitude) to open the OakLeaf blog on a 3-Mbps commercial DSL connection. Most of the delay is due to latency, the period between the HTTP request and the start of the HTTP response, often as much as 10 seconds. Other sites (RedmondMagazine, Microsoft, Techmeme, etc.) are behaving normally. As of 4:30 p.m. PST, I was still receiving 502 errors from the OakLeaf blog.

The Google Blogoscoped site reported more Google problems this morning in this Google News Broken Just Now post. Search Engine Land reports similar Google News problems starting at 7:03 PST and received this message from Google:
A number of users have had difficulty accessing Google News today. We know how important Google News is to our users and are working as quickly as possible to fully resolve the problem.
Cause and effect or just an interesting coincidence?
Technorati tags: , , , ,