OakLeaf Systems: Storing the Android MiniPCs Site’s Log Files in Windows Azure Blobs

Saturday, September 07, 2013

Storing the Android MiniPCs Site’s Log Files in Windows Azure Blobs

OakLeaf Systems’ Android MiniPCs and TVBoxes WordPress blog is a demonstration Windows Azure shared Web Site instance that uses a MySQL database for storage. The Windows Azure team’s Scott Guthrie announced on 9/3/2013 a new capability to store Web Site log files in Windows Azure text blobs, which you can store indefinitely and analyze with a variety of commercial tools, as well as Hadooop with Windows Azure HDInsight services.

This post describes how I made the change from the file system to Windows Azure blob storage and how I read the data with some of the commercial tools I commonly use with Windows Azure blobs. Many Windows Azure Web Site (WAAS) users are new to Windows Azure, so this tutorial assumes no prior Azure experience.

Android MiniPCs and TVBoxes is a relatively recent and specialized site, so it doesn’t receive a lot of traffic. Log blobs averaged about 1 MB per hour when this post was written. As the total size of the blobs grows, I’ll post additional articles about how I analyze them.

Setting Up Blob Storage for Web Site Diagnostics

To enable blob storage for Web Site diagnostics, do the following:

1. Click STORAGE in the navigation pane, select the storage account for your Web Site, androidminipc for this example, click the CONTAINERS menu, and click the Add button to open the dialog. Type the NAME of the blob container, androidhttplogs for this example, and accept the Private ACCESS default, unless you want to grant public access to the blob or its continer:

2. Click OK (check mark) to create the container.

3. Click the Web Sites icon, open the Web Site of interest in the Windows Azure Portal and click the CONFIGURE menu item and scroll down to the Site Diagnostic section, which defaults to FILE SYSTEM storage with a 35 MB quota:

4. Click the STORAGE button to display a MANAGE STORAGE button and click it to open a Manage Storage for Site Diagnostics dialog. Select the storage account associated with your Web site from this list, androidminipc for this example, and type the container name you specified in step 1:

5. Click the OK (check) button to close the dialog and return to the Site Diagnostics section:

6. Mark the SET RETENTION check box to open the RETENTION PERIOD text box, specify the number of days to retain data, and click the buttons for added diagnostic details, if you want:

7. Click SAVE to retain your changes:

Note: The SAVE action specified, but didn’t create, the blob container.

Reading Log Data from Blobs

I use Red Gate Software’s (formerly Cerebrata’s) Azure Management Studio for working with Windows Azure storage and diagnostics. To open the log file for viewing in Notepad or downloading to your local PC as a text file, do the following:

1. Return to the Management Portal’s STORAGE page, select the androidminipcs storage account, and click the MANAGE ACCESS KEYS button to open the Manage Access Keys dialog:

2. Click the PRMARY ACCESS KEY text box’s Copy button to copy the full key value to the clipboard.

3. Start Azure Management Studio, right-click the navigation pane’s Storage Accounts node, and choose Add Storage Account Connection to open the dialog of the same name. Type the Storage Account Name and paste the Storage Account key into the text boxes, mark the Use SSL check box, and accept the Development default or change the the Connection Group:

4. Click OK to add the storage account to the Storage Accounts list, expand the Blob Containers list, click androidhttplogs to display the container’s contents. Navigate to the end of the year, month, day and hour hierarchy to expose the current *.log blob:

5. Double-click the *.log item to open it in Notepad:

6. Click the Download button to open the Folder dialog, select or create a folder to store the downloads, and click OK to complete the download:

7. Open the *.log file in Notepad, remove the leading # and space characters (see step 5) and save the changes.

8. Open the *.log file in Excel to start the Text Import Wizard, select the Delimited option and mark the My Data Has Headers check box:

9. Click Next and clear the Tab, mark the Space delimiter check box, and accept the other defaults:

10. Click Next and, with the Date column selected, mark the Date option and select YMD format:

11. Click Finish to display the log file in Excel:

Note: Pingdom interrogates the site every five minutes to create monthly uptime reports, such as Uptime Report for My Live Windows Azure Web Site: July 2013 = 99.37%.

Stay tuned for the article describing log analytics techniques.

Full disclosure: I’m a paid contributor to Red Gate Software’s ACloudyPlace blog and Simple-Talk newsletter and have received a free license for their Azure Management Studio.