Windows Azure and Cloud Computing Posts for 8/3/2011+

A compendium of Windows Azure, SQL Azure Database, AppFabric, Windows Azure Platform Appliance and other cloud-computing articles.

• Updated 8/5/2011 8:00 AM with link to Steve Marx’ detailed Playing with the new Windows Azure Storage Analytics Features article of 8/4/2011 in the Azure Blob, Drive, Table and Queue Services section.

Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:

• Azure Blob, Drive, Table and Queue Services
• SQL Azure Database and Reporting
• Marketplace DataMarket and OData
• Windows Azure AppFabric: Apps, Access Control, WIF and Service Bus
• Windows Azure VM Role, Virtual Network, Connect, RDP and CDN
• Live Windows Azure Apps, APIs, Tools and Test Harnesses
• Visual Studio LightSwitch and Entity Framework v4+
• Windows Azure Infrastructure and DevOps
• Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds
• Cloud Security and Governance
• Cloud Computing Events
• Other Cloud Computing Platforms and Services

---

Azure Blob, Drive, Table and Queue Services

The Windows Azure Storage Team explained Windows Azure Storage Analytics in an 8/3/2011 post:

Windows Azure Storage Analytics offers you the ability to track, analyze, and debug your usage of storage(Blobs, Tables and Queues). You can use this data to analyze storage usage to improve the design of your applications and their access patterns to Windows Azure Storage. Analytics data consists of:

• Logs
  • Provide trace of executed requests for your storage accounts
• Metrics
  • Provide summary of key capacity and request statistics for Blobs, Tables and Queues

Logs

This feature provides a trace of all executed requests for your storage accounts as block blobs in a special container called $logs. Each log entry in the blob corresponds to a request made to the service and contains information like request id, request URL, http status of the request, requestor account name, owner account name, server side latency, E2E latency, source IP address for the request etc.

This data now empowers you to analyze your requests much more closely. It allows you to run the following types of analysis:

• How many anonymous requests is my application seeing from a given range of IP address?
• Which containers are being accessed the most?
• How many times is a particular SAS URL being accessed and how?
• Who issued the request to delete a container?
• For a slow request –where is the time being consumed?
• I got a network error, did the request reach the server?

Metrics

Provide summary of key statistics for Blobs, Tables and Queues for a storage account. The statistics can be categorized as:

• Request information: Provides hourly aggregates of number of requests, average server side latency, average E2E latency, average bandwidth, total successful requests and total number of failures and more. These request aggregates are provided at a service level and per API level for APIs requested in that hour. This is available for Blob, Table and Queue service.
• Capacity information: Provides daily statistics for the space consumed by the service, number of containers and number of objects that are stored in the service. Note, this is currently only provided for the Windows Azure Blob service.

All Analytics Logs and Metrics data are stored in your user account and is accessible via normal Blob and Table REST APIs. The logs and metrics can be accessed from a service running in Windows Azure or directly over the Internet from any application that can send and receive HTTP/HTTPS requests. You can opt in to store either the log data and/or metric data by invoking a REST API to turn on/off the feature at a per service level. Once the feature is turned on, the Windows Azure Storage stores analytics data in the storage account. Log data is stored as Windows Azure Blobs in a special blob container and metrics data is stored in special tables in Windows Azure Tables. To ease the management of this data, we have provided the ability to set a retention policy that will automatically clean up your analytics blob and table data.

Please see the following links for more information:

• MSDN Documentation (this link will be live later today)
• Logging: Additional information and Coding Examples
• Metrics: Additional information and Coding Examples

Nothing heard from the Windows Azure Storage Team for three months and suddenly three posts (two gigantic) in two days!

Steve Marx (@smarx) posted a link to his web app to change Windows #Azure storage analytics settings on Twitter: http://storageanalytics.cloudapp.net/. Take heed of Steve’s caution:

The XML elements’ values (white) are editable:

• Steve’s detailed Playing with the new Windows Azure Storage Analytics Features article of 8/4/2011 explains how to use an updated version of http://storageanalytics.cloudapp.net/, which uses a self-signed certificate to encrypt transmissions:

From the introduction to Steve’s post:

Since its release yesterday, I’ve been playing with the new ability to have Windows Azure storage log analytics data. In a nutshell, analytics lets you log as much or as little detail as you want about calls to your storage account (blobs, tables, and queues), including detailed logs (what API calls were made and when) and aggregate metrics (like how much storage you’re consuming and how many total requests you’ve done in the past hour).

As I’m learning about these new capabilities, I’m putting the results of my experiments online at http://storageanalytics.cloudapp.net. This app lets you change your analytics settings (so you’ll need to use a real storage account and key to log in) and also lets you graph your “Get Blob” metrics.

A couple notes about using the app:

1. When turning on various logging and analytics features, you’ll have to make sure your XML conforms with the documentation. That means, for example, then when you enable metrics, you’ll also have to add the <IncludeAPIs /> tag, and when you enable a retention policy, you’ll have to add the <Days /> tag.
2. The graph page requires a browser with <canvas> tag support. (IE9 will do, as will recent Firefox, Chrome, and probably Safari.) Even in modern browsers, I’ve gotten a couple bug reports already about the graph not showing up and other issues. Please let me know on Twitter (I’m @smarx) if you encounter issues so I can track them down. Bonus points if you can send me script errors captured by IE’s F12, Chrome’s Ctrl+Shift+I, Firebug, or the like.
3. The site uses HTTPS to better secure your account name and key, and I’ve used a self-signed certificate, so you’ll have to tell your browser that you’re okay with browsing to a site with an unverified certificate.

In the rest of this post, I’ll share some details and code for what I’ve done so far.

---

Jai Haridas, Monilee Atkinson and Brad Calder of the Windows Azure Storage Team described Windows Azure Storage Metrics: Using Metrics to Track Storage Usage on 8/3/2011:

Windows Azure Storage Metrics allows you to track your aggregated storage usage for Blobs, Tables and Queues. The details include capacity, per service request summary, and per API level aggregates. The metrics information is useful to see aggregate view of how a given storage account’s blobs, tables or queues are doing over time. It makes it very easy to see the types of errors that are occurring to help tune your system and diagnose problems and the ability to see daily trends of its usage. For example, the metrics data can be used to understand the request breakdown (by hour).

The metrics data can be categorized as:

• Capacity: Provides information regarding the storage capacity consumed for the blob service, the number of containers and total number of objects stored by the service. In the current version, this is available only for the blob service. We will provide table and queue capacity information in a future version. This data is updated daily and it provides separate capacity information for data stored by user and the data stored for $logs.
• Requests: Provides summary information of requests executed against the service. It provides total number of requests, total ingress/egress, average E2E latency and server latency, total number of failures by category, etc. at an hourly granularity. The summary is provided at service level and it also provides aggregates at an API level for the APIs that have been used for the hour. This data is available for all the three services provided – Blobs, Tables, and Queues.

Finding your metrics data

Metrics are stored in tables in the storage account the metrics are for. The capacity information is stored in a separate table from the request information. These tables are automatically created when you opt in for metrics analytics and once created the tables cannot be deleted, though their contents can be.

As mentioned above, there are two types of tables which store the metrics details:

1. Capacity information. When turned on, the system creates $MetricsCapacityBlob table to store capacity information for blobs. This includes the number of objects and capacity consumed by Blobs for the storage account. This information will be recorded in $MetricsCapacityBlob table once per day (see PartitionKey column in Table 1). We report capacity separately for the amount of data stored by the user and the amount of data stored for analytics.
2. Transaction metrics. This information is available for all the services – Blobs, Tables and Queues. Each service gets a table for itself and hence the 3 tables are:

• $MetricsTransactionsBlob: This table will contain the summary – both at a service level and for each API issued to the blob service.
• $MetricsTransactionsTable: This table will contain the summary – both at a service level and for each API issued to the table service.
• $MetricsTransactionsQueue: This table will contain the summary – both at a service level and for each API issued to the queue service.

A transaction summary row is stored for each service that depicts the total request count, error counts, etc., for all requests issued for the service at an hour granularity. Aggregated request details are stored per service and for each API issued in the service per hour.

Requests will be categorized separately for:

• Requests issued by the user (authenticated, SAS and anonymous) to access both user data and analytics data in their storage account.
• Requests issued by Windows Azure Storage to populate the analytics data.

It is important to note that the system stores a summary row for each service every hour even if no API was issued for the service in that hour, which would result in a row showing that there were 0 requests during the hour. This helps applications in issuing point queries when monitoring the data to perform analysis. For APIs level metrics the behavior is different; the system stores a summary row for individual APIs only if the API was utilized in that hour.

These tables can be read from like any other user created Windows Azure Table. The REST URI for the metrics table is http://<accountname>.table.core.windows.net/Tables(“$MetricsTransactionsBlob”) to access the Blob Transaction table and http://<accountname>.table.core.windows.net/$MetricsTransactionsBlob() to access rows stored for Blob service in the transaction table.

What does my metrics data contain?

As mentioned, there are two different types of tables for the metrics data. In this section we will review the details contained in each table.

Blob capacity metrics data

This is stored in the $MetricsCapacityBlob Table. Two records with capacity information will be stored in this table for each day. The two records are used to track the capacity consumed by the data that the user has stored separately from the logging data in $logs container.

Table 1 Capacity Metrics table

Transaction metrics data

This is stored in the $MetricsTransactions<service> tables. All three (blob, table, queue) of the services have the same table schema, outlined below. At the highest level, these tables contain two kinds of records:

• Service Summary – This record contains hourly aggregates for the entire service. A record is inserted for each hour even if the service has not been used. The aggregates include request count, success count, failure counts for certain error categories, average E2E latency, average Server latency etc.
• API Summary – This record contains hourly aggregates for the given API. The aggregates include request count, success count, failure count by category, average E2E latency, average server latency, etc. A record is stored in the table for an API only if the API has been issued to the service in that hour.

We track user and Windows Azure Storage Analytics (system) requests in different categories.

User Requests to access any data under the storage account (including analytics data) are tracked under “user” category. These requests types are:

• Authenticated Requests
• Authenticated SAS requests. All failures after a successful authentication will be tracked including authorization failures. An example of Authorization failure in SAS scenario is trying to write to a blob when only read access has been provided.
• Anonymous Requests - Only successful anonymous requests are tracked. The only exception to this are the following which are tracked:
  • Server errors – Typically seen as InternalServerError
  • Client and Server Timeout errors
  • Requests that fail with 304 i.e. Not Modified

Internal Windows Azure Storage Analytics requests are tracked under the “system” category:

• Requests to write logs under the $logs container
• Requests to write to metrics table

These are tracked as “system” so that they are separated out from the actual requests that users issue to access their data or analytics data.

Note that if a request fails before the service can deduce the type of request (i.e. the API name of the request), then the request is recorded under the API name of “Unknown”.

Before we go over the schema for the metrics transaction tables (the table columns and their definitions), we will define some terms used in this table:

• For service summary row, the column represents the aggregate data at a service level. For API summary row, the same column represents data at an API level granularity.
• Counted For Availability: If we have listed an operation result count in the below table as “Counted For Availability”, then it implies that the result category is accounted for in the availability calculation. Availability is defined as (‘billable requests)/(total requests). These requests are counted in both the numerator and denominator of that calculation. Examples: Success, ClientOtherError etc.
• Counted Against Availability: These requests are counted in the Availability denominator only (were not considered as ‘billable requests). This implies that requests are counted only in the denominator and will impact the availability (for example, server timeout errors).
• Billable: These are the requests that are billable. See this post for more information on understanding Windows Azure Storage billing. Example: ServerTimeoutError, ServerOtherError.

NOTE: If a result category is not listed as “Counted For/Against Availability”, then it implies that those requests are not considered in the availability calculations. Examples: ThrottlingError …

See original post for a five-foot long “Table 2 Schema for transaction metrics tables for Blobs, Tables and Queues.” …

Turning Metrics On

A REST API call, as shown below, is used to turn Analytics on for Metrics. In this example, logging is turned on for deletes and writes, but not for reads. The retention policy is enabled and set to ten days - so the analytics service will take care of deleting data older than ten days for you at no additional cost.

Request: 
PUT http://sally.blob.core.windows.net/?restype=service&comp=properties 
HTTP/1.1
x-ms-version: 2009-09-19
x-ms-date: Fri, 25 Mar 2011 23:13:08 GMT
Authorization: SharedKey sally:zvfPm5cm8CiVP2VA9oJGYpHAPKxQb1HD44IWmubot0A=
Host: sally.blob.core.windows.net

<?xml version="1.0" encoding="utf-8"?>
<StorageServiceProperties>

    <Logging>
        <Version>1.0</Version>
        <Delete>true </Delete>
        <Read> false</Read>
        <Write>true </Write>
        <RetentionPolicy>
        <Enabled>true</Enabled>           
        <Days>7</Days>
    </RetentionPolicy>
    </Logging>

    <Metrics>
        <Version>1.0</Version>
        <Enabled>true</Enabled>           
        <IncludeAPIs>true</IncludeAPIs>
        <RetentionPolicy>
            <Enabled>true</Enabled>           
            <Days>10</Days>
        </RetentionPolicy>
    </Metrics>

</StorageServiceProperties >

Response:
HTTP/1.1 202 
Content-Length: 0
Server: Windows-Azure-Metrics/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 4d02712a-a68a-4763-8f9b-c2526625be68
x-ms-version: 2009-09-19
Date: Fri, 25 Mar 2011 23:13:08 GMT

The logging and metrics sections allow you to configure what you want to track in your analytics logs and metrics data. The metrics configuration values are described here:

• Version - The version of Analytics Logging used to record the entry.
• Enabled (Boolean) – set to true if you want track metrics data via analytics
• IncludedAPIs (Boolean) – set to true if you want to track metrics for the individual APIs accessed
• Retention policy – this is where you set the retention policy to help you manage the size of your analytics data
  • Enabled (Boolean) – set to true if you want to enable a retention policy. We recommend that you do this.
  • Days (int) – the number of days you want to keep your analytics logging data. This can be a max of 365 days and a min of 1 day

For more information, please see the MSDN Documentation. (this link will be live later today)

How do I cleanup old Metrics data?

As described above, we highly recommend you set a retention policy for your analytics data. If set, the maximum retention policy allowed is 365 days. Once a retention policy is set, the system will delete the records in the metrics tables and log blobs from the $logs container. The retention period can be set different for logs from metrics data. For example: If a user sets the retention policy for metrics to be 100 days and 30 days for logs, then all the logs for queue, blob, and table services will be deleted after 30 days and records stored in the associated tables will be deleted if the content is > 100 days. Retention policy is enforced even if analytics is turned off but retention policy is enabled. If you do not set a retention policy you can manage your data by manually deleting entities (like you delete entities in regular tables) whenever you wish to do so.

Searching your analytics metrics data

Your metrics can be retrieved using the Query Tables and Query Entities APIs. You have the ability to query and/or delete records in the tables. Note that the tables themselves cannot be deleted. These tables can be read from like any other user created Windows Azure Table. The REST URI for the metrics table is http://<accountname>.table.core.windows.net/Tables(“$MetricsTransactionsBlob”) to access the Blob Transaction table and http://<accountname>.table.core.windows.net/$MetricsTransactionsBlob() to access rows stored in them. To filter the data you can use the $filter=<query expression> extension of the Query Entities API.

The metrics are stored on an hour granularity. The time key represents the starting hour in which the requests were executed. For example, the metrics with a key of 1:00, represent the requests that started between 1:00 and 2:00. The metrics data is optimized to get historical data based on a time range.

Scenario: I want to retrieve capacity details for blob service starting from 2011/05/20 05:00a.m until 2011/05/30 05:00a.m

GET http://sally.table.core.windows.net/$MetricsCapacityBlob()?$filter=PartitionKey ge ‘20110520T0500’and PartitionKey le ‘20110530T0500’

Scenario: I want to view request details (including API details) for table service starting from 2011/05/20 05:00a.m until 2011/05/30 05:00a.m only for requests made to user data

GET http://sally.table.core.windows.net/$MetricsTransactionsTable()?$filter= PartitionKey ge ‘20110520T0500’and PartitionKey le ‘20110530T0500’and RowKey ge ‘user;’ and RowKey lt ‘user<’

To collect trending data you can view historical information (up to the point of your retention policy) which gives you insights into usage, capacity, and other trends.

What charges occur due to metrics?

The billable operations listed below are charged at the same rates applicable to all Windows Azure Storage operations. For more information on how these transactions are billed, see Understanding Windows Azure Storage Billing - Bandwidth, Transactions, and Capacity.

The following actions performed by Windows Azure Storage are billable:

• Write requests to create table entities for metrics

Read and delete requests by the application/client to metrics data are also billable. If you have configured a data retention policy, you are not charged when Windows Azure Storage deletes old metrics data. However, if you delete analytics data, your account is charged for the delete operations.

The capacity used by $metrics tables are billable.

The following can be used to estimate the amount of capacity used for storing metrics data:

• If a service each hour utilizes every API in every service, then approximately 148KB of data may be stored every hour in the metrics transaction tables if both service and API level summary are enabled.
• If a service each hour utilizes every API in every service, then approximately 12KB of data may be stored every hour in the metrics transaction tables if just service level summary is enabled.
• The capacity table for blobs will have 2 rows each day (provided user has opted in for logs) and that implies that every day approximately up to 300 bytes may be added.

Download Metrics Data

Since listing normal tables does not list out metrics tables, existing tools will not be able to display these tables. In absence of existing tools, we wanted to provide a quick reference application with source code to make this data accessible.

The following application takes the service to download the capacity/request aggregates for, the start time and end time and a file to export to. It then exports all metric entities from the selected table to the file in a csv format. This csv format can then be consumed by say excel to study various trends on availability, errors seen by application, latency etc.

For example the following command will download all the ($MetricsTransactionsBlob) table entities for blob service between the provided time range into a file called MyMetrics.txt

DumpMetrics blob requests .\MyMetrics.txt “2011-07-26T22:00Z” “2011-07-26T23:30Z”

const string ConnectionStringKey = "ConnectionString";

static void Main(string[] args)
{
    if (args.Length < 4 || args.Length > 5)
    {
        Console.WriteLine("Usage: DumpMetrics <service to search - blob|table|queue> <capacity|requests> <file name to export to> <Start time in UTC for report> <Optional End time in UTC for report>.");
        Console.WriteLine("Example: DumpMetrics blob capacity test.txt \"2011-06-26T20:30Z\" \"2011-06-28T22:00Z\"");
        return;
    }

    string connectionString = ConfigurationManager.AppSettings[ConnectionStringKey];

    CloudStorageAccount account = CloudStorageAccount.Parse(connectionString);

    CloudTableClient tableClient = account.CreateCloudTableClient();

    DateTime startTimeOfSearch = DateTime.Parse(args[3]);
    DateTime endTimeOfSearch = DateTime.UtcNow;

    if (args.Length == 5)
    {
        endTimeOfSearch = DateTime.Parse(args[4]);
    }

    if (string.Equals(args[1], "requests", StringComparison.OrdinalIgnoreCase))
    {
        DumpTransactionsMetrics(tableClient, args[0], startTimeOfSearch.ToUniversalTime(), endTimeOfSearch.ToUniversalTime(), args[2]);
    }
    else if (string.Equals(args[1], "capacity", StringComparison.OrdinalIgnoreCase) && string.Equals(args[0], "Blob", StringComparison.OrdinalIgnoreCase))
    {
        DumpCapacityMetrics(tableClient, args[0], startTimeOfSearch, endTimeOfSearch, args[2]);
    }
    else
    {
        Console.WriteLine("Invalid metrics type. Please provide Requests or Capacity. Capacity is available only for blob service");
    }
}


/// <summary>
/// Given a service, start time, end time search for, this method retrieves the metrics rows for requests and exports it to CSV file
/// </summary>
/// <param name="tableClient"></param>
/// <param name="serviceName">The name of the service interested in</param>
/// <param name="startTimeForSearch">Start time for reporting</param>
/// <param name="endTimeForSearch">End time for reporting</param>
/// <param name="fileName">The file to write the report to</param>
static void DumpTransactionsMetrics(CloudTableClient tableClient, string serviceName, DateTime startTimeForSearch, DateTime endTimeForSearch, string fileName)
{
    string startingPK = startTimeForSearch.ToString("yyyyMMddTHH00");
    string endingPK = endTimeForSearch.ToString("yyyyMMddTHH00");

    // table names are case insensitive
    string tableName = string.Format("$MetricsTransactions{0}", serviceName);

    Console.WriteLine("Querying table '{0}' for PartitionKey >= '{1}' and PartitionKey <= '{2}'", tableName, startingPK, endingPK);

    TableServiceContext context = tableClient.GetDataServiceContext();

    // turn off merge option as we only want to query and not issue deletes etc.
    context.MergeOption = MergeOption.NoTracking;

    CloudTableQuery<MetricsTransactionsEntity> query = (from entity in context.CreateQuery<MetricsTransactionsEntity>(tableName)
                                            where entity.PartitionKey.CompareTo(startingPK) >= 0
                                            && entity.PartitionKey.CompareTo(endingPK) <= 0
                                            select entity).AsTableServiceQuery<MetricsTransactionsEntity>();

    // now we have the query set. Let us iterate over all entities and store into an output file.
    // Also overwrite the file
    Console.WriteLine("Writing to '{0}'", fileName);
    using (StreamWriter writer = new StreamWriter(fileName))
    {
        // write the header
        writer.Write("Time, Category, Request Type, Total Ingress, Total Egress, Total Requests, Total Billable Requests,");
        writer.Write("Availability, Avg E2E Latency, Avg Server Latency, % Success, % Throttling Errors, % Timeout Errors, % Other Server Errors, % Other Client Errors, % Authorization Errors, % Network Errors, Success,");
        writer.Write("Anonymous Success, SAS Success, Throttling Error, Anonymous Throttling Error, SAS ThrottlingError, Client Timeout Error, Anonymous Client Timeout Error, SAS Client Timeout Error,");
        writer.Write("Server Timeout Error, Anonymous Server Timeout Error, SAS Server Timeout Error, Client Other Error, SAS Client Other Error, Anonymous Client Other Error,");
        writer.Write("Server Other Errors, SAS Server Other Errors, Anonymous Server Other Errors, Authorization Errors, Anonymous Authorization Error, SAS Authorization Error,");
        writer.WriteLine("Network Error, Anonymous Network Error, SAS Network Error");

        foreach (MetricsTransactionsEntity entity in query)
        {
            string[] rowKeys = entity.RowKey.Split(';');
            writer.WriteLine("{0}, {1}, {2}, {3}, {4}, {5}, {6}, {7}, {8}, {9}, {10}, {11}, {12}, {13}, {14}, {15}, {16}, {17}, {18}, {19}, {20}, {21}, {22}, {23}, {24}, {25}, {26}, {27}, {28}, {29}, {30}, {31}, {32}, {33}, {34}, {35}, {36}, {37}, {38}, {39}, {40}",
                entity.PartitionKey,
                rowKeys[0], // category - user | system
                rowKeys[1], // request type is the API name (and "All" for service summary rows)
                entity.TotalIngress,
                entity.TotalEgress,
                entity.TotalRequests,
                entity.TotalBillableRequests,
                entity.Availability,
                entity.AverageE2ELatency,
                entity.AverageServerLatency,
                entity.PercentSuccess,
                entity.PercentThrottlingError,
                entity.PercentTimeoutError,
                entity.PercentServerOtherError,
                entity.PercentClientOtherError,
                entity.PercentAuthorizationError,
                entity.PercentNetworkError,
                entity.Success,
                entity.AnonymousSuccess,
                entity.SASSuccess,
                entity.ThrottlingError,
                entity.AnonymousThrottlingError,
                entity.SASThrottlingError,
                entity.ClientTimeoutError,
                entity.AnonymousClientTimeoutError,
                entity.SASClientTimeoutError,
                entity.ServerTimeoutError,
                entity.AnonymousServerTimeoutError,
                entity.SASServerTimeoutError,
                entity.ClientOtherError,
                entity.SASClientOtherError,
                entity.AnonymousClientOtherError,
                entity.ServerOtherError,
                entity.SASServerOtherError,
                entity.AnonymousServerOtherError,
                entity.AuthorizationError,
                entity.AnonymousAuthorizationError,
                entity.SASAuthorizationError,
                entity.NetworkError,
                entity.AnonymousNetworkError,
                entity.SASNetworkError);
        }
    }
}

/// <summary>
/// Given a service, start time, end time search for, this method retrieves the metrics rows for capacity and exports it to CSV file
/// </summary>
/// <param name="tableClient"></param>
/// <param name="serviceName">The name of the service interested in</param>
/// <param name="startTimeForSearch">Start time for reporting</param>
/// <param name="endTimeForSearch">End time for reporting</param>
/// <param name="fileName">The file to write the report to</param>
static void DumpCapacityMetrics(CloudTableClient tableClient, string serviceName, DateTime startTimeForSearch, DateTime endTimeForSearch, string fileName)
{
    string startingPK = startTimeForSearch.ToString("yyyyMMddT0000");
    string endingPK = endTimeForSearch.ToString("yyyyMMddT0000");

    // table names are case insensitive
    string tableName = string.Format("$MetricsCapacity{0}", serviceName);

    Console.WriteLine("Querying table '{0}' for PartitionKey >= '{1}' and PartitionKey <= '{2}'", tableName, startingPK, endingPK);

    TableServiceContext context = tableClient.GetDataServiceContext();

    // turn off merge option as we only want to query and not issue deletes etc.
    context.MergeOption = MergeOption.NoTracking;

    CloudTableQuery<MetricsCapacityEntity> query = (from entity in context.CreateQuery<MetricsCapacityEntity>(tableName)
                                            where entity.PartitionKey.CompareTo(startingPK) >= 0
                                            && entity.PartitionKey.CompareTo(endingPK) <= 0
                                                    select entity).AsTableServiceQuery<MetricsCapacityEntity>();

    // now we have the query set. Let us iterate over all entities and store into an output file.
    // Also overwrite the file
    Console.WriteLine("Writing to '{0}'", fileName);
    using (StreamWriter writer = new StreamWriter(fileName))
    {
        // write the header
        writer.WriteLine("Time, Category, Capacity (bytes), Container count, Object count");

        foreach (MetricsCapacityEntity entity in query)
        {
            writer.WriteLine("{0}, {1}, {2}, {3}, {4}",
                entity.PartitionKey,
                entity.RowKey,
                entity.Capacity,
                entity.ContainerCount,
                entity.ObjectCount);
        }
    }
}

The definitions for entities used are:

[DataServiceKey("PartitionKey", "RowKey")]
public class MetricsCapacityEntity {
    public string PartitionKey { get; set; }
    public string RowKey { get; set; }
    public long Capacity { get; set; }
    public long ContainerCount { get; set; }
    public long ObjectCount { get; set; }
}

[DataServiceKey("PartitionKey", "RowKey")]
public class MetricsTransactionsEntity {
    public string PartitionKey { get; set; }
    public string RowKey { get; set; }
    public long TotalIngress { get; set; }
    public long TotalEgress { get; set; }
    public long TotalRequests { get; set; }
    public long TotalBillableRequests { get; set; }
    public double Availability { get; set; }
    public double AverageE2ELatency { get; set; }
    public double AverageServerLatency { get; set; }
    public double PercentSuccess { get; set; }
    public double PercentThrottlingError { get; set; }
    public double PercentTimeoutError { get; set; }
    public double PercentServerOtherError { get; set; }
    public double PercentClientOtherError { get; set; }
    public double PercentAuthorizationError { get; set; }
    public double PercentNetworkError { get; set; }
    public long Success { get; set; }
    public long AnonymousSuccess { get; set; }
    public long SASSuccess { get; set; }
    public long ThrottlingError { get; set; }
    public long AnonymousThrottlingError { get; set; }
    public long SASThrottlingError { get; set; }
    public long ClientTimeoutError { get; set; }
    public long AnonymousClientTimeoutError { get; set; }
    public long SASClientTimeoutError { get; set; }
    public long ServerTimeoutError { get; set; }
    public long AnonymousServerTimeoutError { get; set; }
    public long SASServerTimeoutError { get; set; }
    public long ClientOtherError { get; set; }
    public long AnonymousClientOtherError { get; set; }
    public long SASClientOtherError { get; set; }
    public long ServerOtherError { get; set; }
    public long AnonymousServerOtherError { get; set; }
    public long SASServerOtherError { get; set; }
    public long AuthorizationError { get; set; }
    public long AnonymousAuthorizationError { get; set; }
    public long SASAuthorizationError { get; set; }
    public long NetworkError { get; set; }
    public long AnonymousNetworkError { get; set; }
    public long SASNetworkError { get; set; }
}

Case Study

Let us walk through a sample scenario how these metrics data can be used. As a Windows Azure Storage customer, I would like to know how my service is doing and would like to see the request trend between any two time periods.

Description: The following is a console program that takes as input: service name to retrieve the data for, start time for the report, end time for the report and the file name to export the data to in csv format.

We will start with the method “ExportMetrics”. The method will use the time range provided as input arguments to create a query filter. Since the PartitionKey is of the format “YYYYMMDDTHH00” we will create the starting and ending PartitionKey filters. The table name is $MetricsTransactions appended by the service to search for. Once we have these parameters, it is as simple as creating a normal table query using the TableServiceDataContext. We use the extension AsTableServiceQuery as it takes care of continuation tokens. The other important optimization is we turn off merge tracking in which the context tracks all the entities returned in the query response. We can do this here since the query is solely used for retrieval rather than subsequent operations like Delete on these entities. The class used to represent each row in the response is MetricsEntity and its definition is given below. It is a plain simple CSharp class definition exception for the DataServcieKey required by WCF Data Services .NET and has only subset of properties that we would be interested in.

Once we have the query, all we do is to iterate over this query which results in executing the query. Behind the scenes, this CloudTableQuery instance may lazily execute multiple queries if needed to handle continuation tokens. We then write this in csv format. But one can imagine importing this into Azure table or SQL to perform more reporting like services.

NOTE: Exception handling and parameter validation is omitted for brevity.

/// <summary>
/// Given a service name, start time, end time search for, this method retrieves the metrics rows for requests 
/// and exports it to CSV file
/// </summary>
/// <param name="tableClient"></param>
/// <param name="serviceName">The name of the service interested in</param>
/// <param name="startTimeForSearch">Start time for reporting</param>
/// <param name="endTimeForSearch">End time for reporting</param>
/// <param name="fileName">The file to write the report to</param>
static void ExportMetrics(CloudTableClient tableClient, string serviceName, DateTime startTimeForSearch, DateTime endTimeForSearch, string fileName)
{
    string startingPK = startTimeForSearch.ToString("yyyyMMddTHH00");
    string endingPK = endTimeForSearch.ToString("yyyyMMddTHH00");

    // table names are case insensitive
    string tableName = "$MetricsTransactions"+ serviceName;

    Console.WriteLine("Querying table '{0}' for PartitionKey >= '{1}' and PartitionKey <= '{2}'", tableName, startingPK, endingPK);

    TableServiceContext context = tableClient.GetDataServiceContext();

    // turn off merge option as we only want to query and not issue deletes etc.
    context.MergeOption = MergeOption.NoTracking;

    CloudTableQuery<MetricsEntity> query = (from entity in context.CreateQuery<MetricsEntity>(tableName)
                                            where entity.PartitionKey.CompareTo(startingPK) >= 0
                                            && entity.PartitionKey.CompareTo(endingPK) <= 0
                                            select entity).AsTableServiceQuery<MetricsEntity>();

    // now we have the query set. Let us iterate over all entities and store into an output file.
    // Also overwrite the file
    using (Stream stream= new FileStream(fileName, FileMode.Create, FileAccess.ReadWrite))
    {
        using (StreamWriter writer = new StreamWriter(stream))
        {
            // write the header
            writer.WriteLine("Time, Category, Request Type, Total Ingress, Total Egress, Total Requests, Total Billable Requests, Availability, Avg E2E Latency, Avg Server Latency, % Success, % Throttling, % Timeout, % Misc. Server Errors, % Misc. Client Errors, % Authorization Errors, % Network Errors");

            foreach (MetricsEntity entity in query)
            {
                string[] rowKeys = entity.RowKey.Split(';');
                writer.WriteLine("{0}, {1}, {2}, {3}, {4}, {5}, {6}, {7}, {8}, {9}, {10}, {11}, {12}, {13}, {14}, {15}, {16}",
                    entity.PartitionKey,
                    rowKeys[0], // category - user | system
                    rowKeys[1], // request type is the API name (and "All" for service summary rows)
                    entity.TotalIngress,
                    entity.TotalEgress,
                    entity.TotalRequests,
                    entity.TotalBillableRequests,
                    entity.Availability,
                    entity.AverageE2ELatency,
                    entity.AverageServerLatency,
                    entity.PercentSuccess,
                    entity.PercentThrottlingError,
                    entity.PercentTimeoutError,
                    entity.PercentServerOtherError,
                    entity.PercentClientOtherError,
                    entity.PercentAuthorizationError,
                    entity.PercentNetworkError);
            }
        }
    }
}

 [DataServiceKey("PartitionKey", "RowKey")]
public class MetricsEntity
{
    public string PartitionKey { get; set; }
    public string RowKey { get; set; }
    public long TotalIngress { get; set; }
    public long TotalEgress { get; set; }
    public long TotalRequests { get; set; }
    public long TotalBillableRequests { get; set; }
    public double Availability { get; set; }
    public double AverageE2ELatency { get; set; }
    public double AverageServerLatency { get; set; }
    public double PercentSuccess { get; set; }
    public double PercentThrottlingError { get; set; }
    public double PercentTimeoutError { get; set; }
    public double PercentServerOtherError { get; set; }
    public double PercentClientOtherError { get; set; }
    public double PercentAuthorizationError { get; set; }
    public double PercentNetworkError { get; set; }
}

The main method is simple enough to parse the input.

static void Main(string[] args)
{
    if (args.Length < 4)
    {
        Console.WriteLine("Usage: MetricsReporter <service to search - blob|table|queue> <Start time in UTC for report> <End time in UTC for report> <file name to export to>.");
        Console.WriteLine("Example: MetricsReporter blob \"2011-06-26T20:30Z\" \"2011-06-28T22:00Z\"");
        return;
    }

    CloudStorageAccount account = CloudStorageAccount.Parse(ConnectionString);
    CloudTableClient tableClient = account.CreateCloudTableClient();

    DateTime startTimeOfSearch = DateTime.Parse(args[1]).ToUniversalTime();
    DateTime endTimeOfSearch = DateTime.Parse(args[2]).ToUniversalTime();

    //ListTableRows(tableClient, timeOfRequest);

    ExtractMetricsToReports(tableClient, args[0], startTimeOfSearch, endTimeOfSearch, args[3]);
}

Once we have the data exported as CSV files, once can use Excel to import the data and create required important charts.

For more information, please see the MSDN Documentation

---

Jai Haridas, Monilee Atkinson and Brad Calder of explained Windows Azure Storage Logging: Using Logs to Track Storage Requests in an 8/2/2011 post to the the Windows Azure Storage Team Blog:

Windows Azure Storage Logging provides a trace of the executed requests against your storage account (Blobs, Tables and Queues). It allows you to monitor requests to your storage accounts, understand performance of individual requests, analyze usage of specific containers and blobs, and debug storage APIs at a request level.

What is logged?

You control what types of requests are logged for your account. We categorize requests into 3 kinds: READ, WRITE and DELETE operations. You can set the logging properties for each service indicating the types of operations they are interested in. For example, opting to have delete requests logged for blob service will result in blob or container deletes to be recorded. Similarly, logging reads and writes will capture reads/writes on properties or the actual objects in the service of interest. Each of these options must be explicitly set to true (data is captured) or false (no data is captured).

What requests are logged?

The following authenticated and anonymous requests are logged.

• Authenticated Requests:
  • Signed requests to user data. This includes failed requests such as throttled, network exceptions, etc.
  • SAS (Shared Access Signature) requests whose authentication is validated
  • User requests to analytics data
• Anonymous Requests:
  • Successful anonymous requests.
  • If an anonymous request results in one of the following errors, it will be logged
    • Server errors – Typically seen as InternalServerError or ServerBusy
    • Client and Server Timeout errors (seen as InternalServerError with StorageErrorCode = Timeout)
    • GET requests that fail with 304 i.e. Not Modified

The following requests are not logged

• Authenticated Requests:
  • System requests to write and delete the analytics data
• Anonymous Requests:
  • Failed anonymous requests (other than those listed above) are not logged so that the logging system is not spammed with invalid anonymous requests.

During normal operation all requests are logged; but it is important to note that logging is provided on a best effort basis. This means we do not guarantee that every message will be logged due to the fact that the log data is buffered in memory at the storage front-ends before being written out, and if a role is restarted then its buffer of logs would be lost.

What data do my logs contain and in what format?

Each request is represented by one or more log records. A log record is a single line in the blob log and the fields in the log record are ‘;’ separated. We opted for a ‘;’ separated log file rather than a custom format or XML so that existing tools like LogParser, Excel etc, can be easily extended or used to analyze the logs. Any field that may contain ‘;’ is enclosed in quotes and html encoded (using HtmlEncode method).

Each record will contain the following fields:

1. Log Version (string): The log version. We currently use “1.0”.
2. Transaction Start Time (timestamp): The UTC time at which the request was received by our service.
3. REST Operation Type (string) – will be one of the REST APIs. See “Operation Type: What APIs are logged?” section below for more details.
4. Request Status (string) – The status of request operation. See Transaction Status section below for more details.
5. HTTP Status Code (string): E.g. “200”. This can also be “Unknown” in cases where communication with the client is interrupted before we can set the status code.
6. E2E Latency (duration): The total time in milliseconds taken to complete a request in the Windows Azure Storage Service. This value includes the required processing time within Windows Azure Storage to read the request, send the response, and receive acknowledgement of the response.
7. Server Latency (duration): The total processing time in milliseconds taken by the Windows Azure Storage Service to process a request. This value does not include the network latency specified in E2E Latency.
8. Authentication type (string) – authenticated, SAS, or anonymous.
9. Requestor Account Name (string): The account making the request. For anonymous and SAS requests this will be left empty.
10. Owner Account Name (string): The owner of the object(s) being accessed.
11. Service Type (string): The service the request was for (blob, queue, table).
12. Request URL (string): The full request URL that came into the service – e.g. “PUT http://myaccount.blob.core.windows.net/mycontainer/myblob?comp=block&blockId=”. This is always quoted.
13. Object Key (string): This is the key of the object the request is operating on. E.g. for Blob: “/myaccount/mycontainer/myblob”. E.g. for Queue: “/myaccount/myqueue”. E.g. For Table: “/myaccount/mytable/partitionKey/rowKey”. Note: If custom domain names are used, we still have the actual account name in this key, not the domain name. This field is always quoted.
14. Request ID (guid): The x-ms-request-id of the request. This is the request id assigned by the service.
15. Operation Number (int): There is a unique number for each operation executed for the request that is logged. Though most operations will just have “0” for this (See examples below), there are operations which will contain multiple entries for a single request.
    1. Copy Blob will have 3 entries in total and operation number can be used to distinguish them. The log entry for Copy will have operation number “0” and the source read and destination write will have 1 and 2 respectively.
    2. Table Batch command. An example is a batch command with two Insert’s: the Batch would have “0” for operation number, the first insert would have “1”, and the second Insert would have “2”.
16. Client IP (string): The client IP from which the request came.
17. Request Version (string): The x-ms-version used to execute the request. This is the same x-ms-version response header that we return.
18. Request Header Size (long): The size of the request header.
19. Request Packet Size (long) : The size of the request payload read by the service.
20. Response Header Size (long): The size of the response header.
21. Response Packet Size (long) : The size of the response payload written by the service.
    NOTE: The above request and response sizes may not be filled if a request fails.
22. Request Content Length (long): The value of Content-Length header. This should be the same size of Request Packet Size (except for error scenarios) and helps confirm the content length sent by clients.
23. Request MD5 (string): The value of Content-MD5 (or x-ms-content-md5) header passed in the request. This is the MD5 that represents the content transmitted over the wire. For PutBlockList operation, this means that the value stored is for the content of the PutBlockList and not the blob itself.
24. Server MD5 (string): The md5 value evaluated on the server. For PutBlob, PutPage, PutBlock, we store the server side evaluated content-md5 in this field even if MD5 was not sent in the request.
25. ETag(string): For objects for which an ETag is returned, the ETag of the object is logged. Please note that we will not log this in operations that can return multiple objects. This field is always quoted.
26. Last Modified Time (DateTime): For objects where a Last Modified Time (LMT) is returned, it will be logged. If the LMT is not returned, it will be ‘‘(empty string). Please note that we will not log this in operations that can return multiple objects.
27. ConditionsUsed(string): Semicolon separated list of ConditionName=value. This is always quoted. ConditionName can be one of the following:
    1. If-Modified-Since
    2. If-Unmodified-Since
    3. If-Match
    4. If-None-Match
28. User Agent (string): The User-Agent header.
29. Referrer (string): The “Referrer” header. We log up to the first 1 KB chars.
30. Client Request ID (string): This is custom logging information which can be passed by the user via x-ms-client-request-id header (see below for more details). This is opaque to the service and has a limit of 1KB characters. This field is always quoted.

Examples

Put Blob
1.0;2011-07-28T18:02:40.6271789Z;PutBlob;Success;201;28;21;authenticated;sally;sally;blob;"http://sally.blob.core.windows.net/thumbnails/lake.jpg?timeout=30000";"/sally/thumbnails/lake.jpg";fb658ee6-6123-41f5-81e2-4bfdc178fea3;0;201.9.10.20;2009-09-19;438;100;223;0;100;;"66CbMXKirxDeTr82SXBKbg==";"0x8CE1B67AD25AA05";Thursday, 28-Jul-11 18:02:40 GMT;;;;"7/28/2011 6:02:40 PM ab970a57-4a49-45c4-baa9-20b687941e32"

Anonymous Get Blob
1.0;2011-07-28T18:52:40.9241789Z;GetBlob;AnonymousSuccess;200;18;10;anonymous;;sally;blob;"http:// sally.blob.core.windows.net/thumbnails/lake.jpg?timeout=30000";"/sally/thumbnails/lake.jpg";a84aa705-8a85-48c5-b064-b43bd22979c3;0;123.100.2.10;2009-09-19;252;0;265;100;0;;;"0x8CE1B6EA95033D5";Thursday, 28-Jul-11 18:52:40 GMT;;;;"7/28/2011 6:52:40 PM ba98eb12-700b-4d53-9230-33a3330571fc"

Copy Blob
Copy blob will have 3 log lines logged. The request id will be the same but operation id (highlighted in the examples below) will be incremented. The line with operation ID 0 represents the entire copy blob operation. The operation ID 1 represents the source blob retrieval for the copy. This operation is called CopyBlobSource. Operation ID 2 represents destination blob information and the operation is called CopyBlobDestination. The CopyBlobSource will not have information like request size or response size set and is meant only to provide information like name, conditions used etc. about the source blob.

1.0;2011-07-28T18:02:40.6526789Z;CopyBlob;Success;201;28;28;authenticated;account8ce1b67a9e80b35;sally;blob;"http://sally.blob.core.windows.net/thumbnails/lake.jpg?timeout=30000";"/sally/thumbnails/lakebck.jpg";85ba10a5-b7e2-495e-8033-588e08628c5d;0;268.20.203.21;2009-09-19;505;0;188;0;0;;;"0x8CE1B67AD473BC5";Thursday, 28-Jul-11 18:02:40 GMT;;;;"7/28/2011 6:02:40 PM 683803d3-538f-4ba8-bc7c-24c83aca5b1a"

1.0;2011-07-28T18:02:40.6526789Z;CopyBlobSource;Success;201;28;28;authenticated;sally;sally;blob;"http://sally.blob.core.windows.net/thumbnails/lake.jpg?timeout=30000";"/sally/thumbnails/d1c77aca-cf65-4166-9401-0fd3df7cf754";85ba10a5-b7e2-495e-8033-588e08628c5d;1;268.20.203.21;2009-09-19;505;0;188;0;0;;;;;;;;"7/28/2011 6:02:40 PM 683803d3-538f-4ba8-bc7c-24c83aca5b1a"

1.0;2011-07-28T18:02:40.6526789Z;CopyBlobDestination;Success;201;28;28;authenticated;sally;sally;blob;"http://sally.blob.core.windows.net/thumbnails/lake.jpg?timeout=30000";"/sally/thumbnails/lakebck.jpg";85ba10a5-b7e2-495e-8033-588e08628c5d;2;268.20.203.21;2009-09-19;505;0;188;0;0;;;;;;;;"7/28/2011 6:02:40 PM 683803d3-538f-4ba8-bc7c-24c83aca5b1a"

Table Batch Request or entity group requests with 2 inserts.
The first one stands for the batch request, and the latter two represent the actual inserts. NOTE: the key for the batch is the key of the first command in the batch. The operation number is used to identify different individual commands in the batch

1.0;2011-07-28T18:02:40.9701789Z;EntityGroupTransaction;Success;202;104;61;authenticated;sally;sally;table;"http://sally.table.core.windows.net/$batch";"/sally";b59c0c76-dc04-48b7-9235-80124f0066db;0;268.20.203.21;2009-09-19;520;100918;235;189150;100918;;;;;;;;"7/28/2011 6:02:40 PM 70b9cb5c-8e2f-4e8a-bf52-7b79fcdc56e7"

1.0;2011-07-28T18:02:40.9701789Z;InsertEntity;Success;202;104;61;authenticated;sally;sally;table;"http:// sally.table.core.windows.net/$batch";"/sally/orders/5f1a7147-28df-4398-acf9-7d587c828df9/a9225cef-04aa-4bbb-bf3b-b0391e3c436e";b59c0c76-dc04-48b7-9235-80124f0066db;1;268.20.203.21;2009-09-19;520;100918;235;189150;100918;;;"W/&quot;datetime'2011-07-28T18%3A02%3A41.0086789Z'&quot;";Thursday, 28-Jul-11 18:02:41 GMT;;;;"7/28/2011 6:02:40 PM 70b9cb5c-8e2f-4e8a-bf52-7b79fcdc56e7"

1.0;2011-07-28T18:02:40.9701789Z;InsertEntity;Success;202;104;61;authenticated;sally;sally;table;"http:// sally.table.core.windows.net/$batch";"/sally/orders/5f1a7147-28df-4398-acf9-7d587c828df9/a9225cef-04aa-4bbb-bf3b-b0391e3c435d";b59c0c76-dc04-48b7-9235-80124f0066db;2;268.20.203.21;2009-09-19;520;100918;235;189150;100918;;;"W/&quot;datetime'2011-07-28T18%3A02%3A41.0086789Z'&quot;";Thursday, 28-Jul-11 18:02:41 GMT;;;;"7/28/2011 6:02:40 PM 70b9cb5c-8e2f-4e8a-bf52-7b79fcdc56e7"

Operation Type: What APIs are logged?

The APIs recorded in the logs are listed by service below, which match the REST APIs for Windows Azure Storage. Note that for operations that have multiple operations executed (e.g. Batch) as part of them, the field OperationType will contain multiple records with a main record (that has Operation Number ‘0’) and an individual record for each sub operation.

Blob Service APIs

• AcquireLease
• BreakLease
• ClearPage
• CopyBlob
  • CopyBlobSource: Information about the source that was copied and is used only for logs
  • CopyBlobDestination: Information about the destination and is used only for logs
• CreateContainer
• DeleteBlob
• DeleteContainer
• GetBlob
• GetBlobMetadata
• GetBlobProperties
• GetBlockList
• GetContainerACL
• GetContainerMetadata
• GetContainerProperties
• GetLeaseInfo
• GetPageRegions
• LeaseBlob
• ListBlobs
• ListContainers
• PutBlob
• PutBlockList
• PutBlock
• PutPage
• ReleaseLease
• RenewLease
• SetBlobMetadata
• SetBlobProperties
• SetContainerACL
• SetContainerMetadata
• SnapshotBlob
• SetBlobServiceProperties
• GetBlobServiceProperties

Queue Service APIs

• ClearMessages
• CreateQueue
• DeleteQueue
• DeleteMessage
• GetQueueMetadata
• GetQueue
• GetMessage
• GetMessages
• ListQueues
• PeekMessage
• PeekMessages
• PutMessage
• SetQueueMetadata
• SetQueueServiceProperties
• GetQueueServiceProperties

Table Service APIs

• EntityGroupTransaction
• CreateTable
• DeleteTable
• DeleteEntity
• InsertEntity
• QueryEntity
• QueryEntities
• QueryTable
• QueryTables
• UpdateEntity
• MergeEntity
• SetTableServiceProperties
• GetTableServiceProperties

Transaction Status

This table shows the different statuses that can be written to your log. …

See the original post for an even longer (~ 10-foot) table.

Client Request Id

One of the logged fields is called the Client Request Id. A client can choose to pass this client perspective id up to 1KB in size as a HTTP header “x-ms-client-request-id” in with every request and it will be logged for the request. Note, if you use the optional client request id header, it is used in constructing the canonicalized header, since all headers starting with “x-ms” are part of the resource canonicalization used for signing a request.

Since this is treated as an id that a client may associate with a request, it is very helpful to investigate requests that failed due to network or timeout errors. For example, you can search for requests in the log with the given client request id to see if the request timed out, and see if the E2E latency indicates that there is a slow network connection. As noted above, some requests may not get logged due to node failures.

Turning Logging On

A REST API call, as shown below, is used to turn on Logging. In this example, logging is turned on for deletes and writes, but not for reads. The retention policy is enabled and set to ten days - so the analytics service will take care of deleting data older than ten days for you at no additional cost. Note that you need to turn on logging separately for blobs, tables, and queues. The example below demonstrates how to turn logging on for blobs.

Request: 
PUT http://sally.blob.core.windows.net/?restype=service&comp=properties 
HTTP/1.1
x-ms-version: 2009-09-19
x-ms-date: Fri, 25 Mar 2011 23:13:08 GMT
Authorization: SharedKey sally:zvfPm5cm8CiVP2VA9oJGYpHAPKxQb1HD44IWmubot0A=
Host: sally.blob.core.windows.net

<?xml version="1.0" encoding="utf-8"?>
<StorageServiceProperties>

    <Logging>
        <Version>1.0</Version>
        <Delete>true </Delete>
        <Read> false</Read>
        <Write>true </Write>
        <RetentionPolicy>
        <Enabled>true</Enabled>           
        <Days>7</Days>
    </RetentionPolicy>
    </Logging>

    <Metrics>
        <Version>1.0</Version>
        <Enabled>true</Enabled>           
        <IncludeAPIs>true</IncludeAPIs>
        <RetentionPolicy>
            <Enabled>true</Enabled>           
            <Days>10</Days>
        </RetentionPolicy>
    </Metrics>

</StorageServiceProperties >

Response: 
HTTP/1.1 202 
Content-Length: 0
Server: Windows-Azure-Metrics/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 4d02712a-a68a-4763-8f9b-c2526625be68
x-ms-version: 2009-09-19
Date: Fri, 25 Mar 2011 23:13:08 GMT

The above logging and metrics section shows you how to configure what you want to track in your analytics logs and metrics data. The logging configuration values are described below:

• Version - The version of Analytics Logging used to record the entry.
• Delete (Boolean) – set to true if you want to track delete requests and false if you do not
• Read (Boolean) – set to true if you want to track read requests and false if you do not
• Write (Boolean) – set to true if you want to track write requests and false if you do not
• Retention policy – this is where you set the retention policy to help you manage the size of your analytics data
  • Enabled (Boolean) – set to true if you want to enable a retention policy. We recommend that you do this.
  • Days (int) – the number of days you want to keep your analytics logging data. This can be a max of 365 days and a min of 1 day.

For more information, please see the MSDN Documentation. (this link will be live later today)

Where can I find the logs and how does the system store them?

The analytics logging data is stored as block blobs in a container called $logs in your blob namespace for the storage account being logged. The $logs container is created automatically when you turn logging on and once created this container cannot be deleted, though the blobs stored in the container can be deleted. The $logs container can be accessed using “http://<accountname>.blob.core.windows.net/$logs” URL. Note that normal container listing operations that you issue to list your containers will not list the $logs container. But you can list the blobs inside the $logs container itself. Logs will be organized in the $logs namespace per hour by service type (Blob, Table and Queue). Log entries are written only if there are eligible requests to the service and the operation type matches the type opted for logging. For example, if there was no table activity on an account for an hour but we had blob activity, no logs would be created for the table service but we would have some for blob service. If in the next hour there is table activity, table logs would be created for that hour.

We use block blobs to store the log entries as block blobs represents files better than page blobs. In addition, the 2 phase write semantics of block blobs allows our system to write a set of log entries as a block in the block blob. Log entries are accumulated and written to a block when the size reaches 4 MB or if it has been up to 5 minutes since the entries have been flushed to a block. The system will commit the block blob if the size of the uncommitted blob reaches 150MB or if it has been up to 5 minutes since the first block was uploaded – whichever is reached first. Once a blob is committed, it is not updated with any more blocks of log entries. Since a block blob is available to read only after commit operation, you can be assured that once a log blob is committed it will never be updated again.

NOTE: Applications should not take any dependency on the above mentioned size and time trigger for flushing a log entry or committing the blob as it can change without notice.

What is the naming convention used for logs?

Logging in a distributed system is a challenging problem. What makes it challenging is the fact that there are many servers that can process requests for a single account and hence be a source for these log entries. Logs from various sources need to be combined into a single log. Moreover, clock skews cannot be ruled out and the number of log entries produced by a single account in a distributed system can easily run into thousands of log entries per second. To ensure that we provide a pattern to process these logs efficiently despite these challenges, we have a naming scheme and we store additional metadata for the logs that allow easy log processing.

The log name under the $logs container will have the following format:

<service name>/YYYY/MM/DD/hhmm/<Counter>.log

• Service Name: “blob”, “table”, “queue”
• YYYY – The four digit year for the log
• MM – The two digit month for the log
• DD – The two digit day for the log
• hh – The two digit hour (24 hour format) representing the starting hour for all the logs. All timestamps are in UTC
• mm – The two digit number representing the starting minute for all the logs. This is always 00 for this version and kept for future use
• Counter – A zero based counter as there can be multiple logs generated for a single hour. The counter is padded to be 6 digits. The counter progress is based on the last log’s counter value within a given hour. Hence if you delete the last log blob, you may have the same blob name repeat again. It is recommended to not delete the blob logs right away.

The following are properties of the logs:

• A request is logged based on when it ends. For example, if a request starts 13:57 and lasts for 5 minutes, it will make it into the log at the time it ended. It will therefore appear in a log with hh=1400.
• Log entries can be recorded out of order which implies that just inspecting the first and last log entry is not sufficient to figure if the log contains the time range you may be interested in.

To aid log analysis we store the following metadata for each of the log blobs:

• LogType = The type of log entries that a log contains. It is described as combination of read, write and delete. The types will be comma separated. This allows you to download blobs which have the operation type that you are interested in.
• StartTime = The minimum time of a log entry in the log. It is of form YYYY-MM-DDThh:mm:ssZ. This represents the start time for a request that is logged in the blob.
• EndTime = The maximum time of a log entry in the log of form YYYY-MM-DDThh:mm:ssZ. This represents the maximum start time of a request logged in the blob.
• LogVersion = The version of the log format. Currently 1.0. This can be used to process a given blob as all entries in the blob will conform to this version.

With the above you can list of all of the blobs with the “include=metadata” option to quickly see which blob logs have the given time range of logs for processing.

For example, assume we have a blob log that contains write events generated at 05:10:00 UTC on 2011/03/04 and contains requests that started at 05:02:30.0000001Z until 05:08:05.1000000X, the log name will be: $logs/blob/2011/03/04/0500/000000.log and the metadata will contain the following properties:

• LogType=write
• StartTime=2011-03-04T05:02:30.0000001Z
• EndTime=2011-03-04T05:08:05.1000000Z
• LogVersion=1.0

Note, duplicate log records may exist in logs generated for the same hour and can be detected by checking for duplicate RequestId and Operation number.

Operations on $logs container and manipulating log data

As we mentioned above, once you enable logging, your log data is stored as block blobs in a container called $logs in the blob namespace of your account. You can access your $logs container using http://<accountname>.blob.core.windows.net/$logs. To list your logs you can use the list blobs API. The logs are stored organized by service type (blob, table and queue) and are sorted by generation date/time within each service type. The log name under the $logs container will have the following format: <service name>/YYYY/MM/DD/hhmm/<Counter>.log

The following operations are the operations allowed on the $logs container:

• List blobs in $logs container. (Note that $logs will not be displayed in result of listing all containers in the account namespace).
• Read committed blobs under $logs.
• Delete specific logs. Note: logs can be deleted but the container itself cannot be deleted.
• Get metadata/properties for the log.

An example request for each type of operation can be found below:

• List logs
  GET http://sally.blob.core.windows.net/$logs?restype=container&comp=list
• List logs generated for blob service from 2011-03-24 05:00 until 05:59
  GET http://sally.blob.core.windows.net/$logs?restype=container&comp=list&prefix=blob/2011/03/24/0500/&include=metadata
• Download a specific log
  GET http://sally.blob.core.windows.net/$logs/blob/2011/03/24/0500/000000.log
• Delete a specific log
  Delete http://sally.blob.core.windows.net/$logs/blob/2011/03/24/0500/000000.log

To improve enumerating the logs you can pass a prefix when using the list blobs API. For example, to filter blobs by date/time the logs are generated on, you can pass a date/time as the prefix (blob/2011/04/24/) when using the list blobs API.

Listing all logs generated in the month of March in 2011 for the Blob service would be done as follows: http://myaccount.blob.core.windows.net/$logs?restype=container&comp=list&prefix=blob/2011/03

You can filter down to a specific hour using the prefix. For example, if you want to scan logs generated for March 24^th 2011 at 6 PM (18:00), use
http://myaccount.blob.core.windows.net/$logs?restype=container&comp=list&prefix=blob/2011/03/24/1800/

It is important to note that log entries are written only if there are requests to the service. For example, if there was no table activity on an account for an hour but you had blob activity, no logs would be created for the table service but you would have some for blob service. If in the next hour there is table activity, table logs would be created for that hour.

What is the versioning story?

The following describes the versioning for logs:

• The version is stored in the blob metadata and each log entry as the first field.
• All records within a single blob will have the same version.
• When new fields are added, they may be added to the end and will not incur a version change if this is the case. Therefore, applications responsible for processing the logs should be designed to interpret only the first set of columns they are expecting and ignore any extra columns in a log entry.
• Examples of when a version change could occur:
  • The representation needs to change for a particular field (example – data type changes).
  • A field needs to be removed.

What is the scalability targets and capacity limits for logs and how does this related to my storage account?

Capacity and scale for your analytics account is separate from your ‘regular’ account. There is separate 20TB allocated for analytics data (this includes both metrics and logs data). This is not included as part of the 100TB limit for an individual account. In addition, $logs are kept in a separate part of the namespace for the storage account, so it is throttled separately from the storage account, and requests issued by Windows Azure Storage to generate or delete these logs do not affect the per partition or per account scale targets described in the Storage Scalability Targets blog post.

How do I cleanup my logs?

To ease the management of your logs, we have provided the functionality of retention policy which will automatically cleanup ‘old’ logs without you being charged for the cleanup. It is recommended that you set a retention policy for logs such that your analytics data will be within the 20TB limit allowed for analytics data (logs and metrics combined) as described above.

A maximum of 365 days is allowed for retention policy. Once a retention policy is set, the system will delete the logs when logs age beyond the number of days set in the policy. This deletion will be done lazily in the background. Retention policy can be turned off at any time but if set, the retention policy is enforced even if logging is turned off. For example: If you set the retention policy for logging to be 10 days for blob service, then all the logs for blob service will be deleted if the content is > 10 days. If you do not set a retention policy you can manage your data by manually deleting entities (like you delete entities in regular tables) whenever you wish to do so.

What charges occur due to logging?

The billable operations listed below are charged at the same rates applicable to all Windows Azure Storage operations. For more information on how these transactions are billed, see Understanding Windows Azure Storage Billing - Bandwidth, Transactions, and Capacity.

The capacity used by $logs is billable and the following actions performed by Windows Azure are billable:

• Requests to create blobs for logging

The following actions performed by a client are billable:

• Read and delete requests to $logs

If you have configured a data retention policy, you are not charged when Windows Azure Storage deletes old logging data. However, if you delete $logs data, your account is charged for the delete operations.

Downloading your log data

Since listing normal containers does not list out $logs container, existing tools will not be able to display these logs. In absence of existing tools, we wanted to provide a quick reference application with source code to make this data accessible.

The following application takes the service to download the logs for, the start time and end time for log entries and a file to export to. It then exports all log entries to the file in a csv format.

For example the following command will select all logs that have log entries in the provided time range and download all the log entries in those logs into a file called mytablelogs.txt:

DumpLogs table .\mytablelogs.txt “2011-07-26T22:00Z” “2011-07-26T23:30Z”

const string ConnectionStringKey = "ConnectionString";
const string LogStartTime = "StartTime";
const string LogEndTime = "EndTime";

static void Main(string[] args)
{
    if (args.Length < 3 || args.Length > 4)
    {
        Console.WriteLine("Usage: DumpLogs <service to search - blob|table|queue> <output file name> <Start time in UTC> <Optional End time in UTC>.");
        Console.WriteLine("Example: DumpLogs blob test.txt \"2011-06-26T22:30Z\" \"2011-06-26T22:50Z\"");
        return;
    }

    string connectionString = ConfigurationManager.AppSettings[ConnectionStringKey];

    CloudStorageAccount account = CloudStorageAccount.Parse(connectionString);
    CloudBlobClient blobClient = account.CreateCloudBlobClient();

    DateTime startTimeOfSearch = DateTime.Parse(args[2]);
    DateTime endTimeOfSearch = DateTime.UtcNow;

    if (args.Length == 4)
    {
        endTimeOfSearch = DateTime.Parse(args[3]);
    }

    List<CloudBlob> blobList = ListLogFiles(blobClient, args[0], startTimeOfSearch.ToUniversalTime(), endTimeOfSearch.ToUniversalTime());
    DumpLogs(blobList, args[1]);
}



/// <summary>
/// Given service name, start time for search and end time for search, creates a prefix that can be used
/// to efficiently get a list of logs that may match the search criteria
/// </summary>
/// <param name="service"></param>
/// <param name="startTime"></param>
/// <param name="endTime"></param>
/// <returns></returns>
static string GetSearchPrefix(string service, DateTime startTime, DateTime endTime)
{
    StringBuilder prefix = new StringBuilder("$logs/");

    prefix.AppendFormat("{0}/", service);

    // if year is same then add the year
    if (startTime.Year == endTime.Year)
    {
        prefix.AppendFormat("{0}/", startTime.Year);
    }
    else
    {
        return prefix.ToString();
    }

    // if month is same then add the month
    if (startTime.Month == endTime.Month)
    {
        prefix.AppendFormat("{0:D2}/", startTime.Month);
    }
    else
    {
        return prefix.ToString();
    }

    // if day is same then add the day
    if (startTime.Day == endTime.Day)
    {
        prefix.AppendFormat("{0:D2}/", startTime.Day);
    }
    else
    {
        return prefix.ToString();
    }

    // if hour is same then add the hour
    if (startTime.Hour == endTime.Hour)
    {
        prefix.AppendFormat("log-{0:D2}00", startTime.Hour);
    }

    return prefix.ToString();
}

/// <summary>
/// Given a service, start time, end time, provide list of log files
/// </summary>
/// <param name="blobClient"></param>
/// <param name="serviceName">The name of the service interested in</param>
/// <param name="startTimeForSearch">Start time for the search</param>
/// <param name="endTimeForSearch">End time for the search</param>
/// <returns></returns>
static List<CloudBlob> ListLogFiles(CloudBlobClient blobClient, string serviceName, DateTime startTimeForSearch, DateTime endTimeForSearch)
{
    List<CloudBlob> selectedLogs = new List<CloudBlob>();

    // form the prefix to search. Based on the common parts in start and end time, this prefix is formed
    string prefix = GetSearchPrefix(serviceName, startTimeForSearch, endTimeForSearch);

    Console.WriteLine("Prefix used for log listing = {0}", prefix);

    // List the blobs using the prefix
    IEnumerable<IListBlobItem> blobs = blobClient.ListBlobsWithPrefix(
        prefix,
        new BlobRequestOptions()
        {
            UseFlatBlobListing = true,
            BlobListingDetails = BlobListingDetails.Metadata
        });


    // iterate through each blob and figure the start and end times in the metadata
    foreach (IListBlobItem item in blobs)
    {
        CloudBlob log = item as CloudBlob;
        if (log != null)
        {
            // we will exclude the file if the file does not have log entries in the interested time range.
            DateTime startTime = DateTime.Parse(log.Metadata[LogStartTime]).ToUniversalTime();
            DateTime endTime = DateTime.Parse(log.Metadata[LogEndTime]).ToUniversalTime();

            bool exclude = (startTime > endTimeForSearch || endTime < startTimeForSearch);

            Console.WriteLine("{0} Log {1} Start={2:U} End={3:U}.",
                exclude ? "Ignoring" : "Selected",
                log.Uri.AbsoluteUri,
                startTime,
                endTime);

            if (!exclude)
            {
                selectedLogs.Add(log);
            }
        }
    }

    return selectedLogs;
}


/// <summary>
/// Dump all log entries to file irrespective of the time.
/// </summary>
/// <param name="blobList"></param>
/// <param name="fileName"></param>
static void DumpLogs(List<CloudBlob> blobList, string fileName)
{

    if (blobList.Count > 0)
    {
        Console.WriteLine("Dumping log entries from {0} files to '{1}'", blobList.Count, fileName);
    }
    else
    {
        Console.WriteLine("No logs files have selected.");
    }

    using (StreamWriter writer = new StreamWriter(fileName))
    {
        writer.Write(
            "Log version; Transaction Start Time; REST Operation Type; Transaction Status; HTTP Status; E2E Latency; Server Latency; Authentication Type; Accessing Account; Owner Account; Service Type;");
        writer.Write(
            "Request url; Object Key; RequestId; Operation #; User IP; Request Version; Request Packet Size; Request header Size; Response Header Size;");
        writer.WriteLine(
            "Response Packet Size; Request Content Length; Request MD5; Server MD5; Etag returned; LMT; Condition Used; User Agent; Referrer; Client Request Id");
        foreach (CloudBlob blob in blobList)
        {
            using (Stream stream = blob.OpenRead())
            {
                using (StreamReader reader = new StreamReader(stream))
                {
                    string logEntry;
                    while ((logEntry = reader.ReadLine()) != null)
                    {
                        writer.WriteLine(logEntry);
                    }
                }
            }
        }
    }
}

Case Study

To put the power of log analytics into perspective, we will end this post with a sample application. Below, we cover a very simple console application that allows search (e.g., grep) functionality over logs. One can imagine extending this to download the logs and analyze it and store data in structured storage such as Windows Azure Tables or SQL Azure for additional querying over the data.

Description: A console program that takes as input: service name to search for, log type to search, start time for the search, end time for the search and the keyword to search for in the logs. The output is log entries that match the search criteria and contains the keyword.

We will start with the method “ListLogFiles”. The method will use the input arguments to create a prefix for the blob listing operation. This method makes use of the utility methods GetSearchPrefix to get the prefix for listing operation. The prefix uses service name, start and end time to format the prefix. The start and end times are compared and only the parts that match are used in the prefix search. For example: If start time is "2011-06-27T02:50Z" and end time is "2011-06-27T03:08Z", then the prefix for blob service will be: “$logs/blob/2011/06/27/”. If the hour had matched then the prefix would be: “$logs/blob/2011/06/27/0200”. The listing result contains metadata for each blob in the result. The start time, end time and logging level is then matched to see if the log contains any entries that may match the time criteria. If it does, then we add the log to the list of logs we will be interested in downloading. Otherwise, we skip the log file.

NOTE: Exception handling and parameter validation is omitted for brevity.

/// <summary>
/// Given a service, start time, end time, and operation types (i.e. READ/WRITE/DELETE) to search for, this method
/// iterates through blob logs and selects the ones that match the service and time range.
/// </summary>
/// <param name="blobClient"></param>
/// <param name="serviceName">The name of the service interested in</param>
/// <param name="startTimeForSearch">Start time for the search</param>
/// <param name="endTimeForSearch">End time for the search</param>
/// <param name="operationTypes">A ',' separated operation types used as logging level</param>
/// <returns></returns>
static List<CloudBlob> ListLogFiles(
CloudBlobClient blobClient, 
string serviceName, 
DateTime startTimeForSearch, 
DateTime endTimeForSearch, 
string operationTypes)
{
    List<CloudBlob> selectedLogs = new List<CloudBlob>();
            
    // convert a ',' separated log type to a "flag" enum 
    LoggingLevel loggingLevelsToFind = GetLoggoingLevel(operationTypes);

    // form the prefix to search. Based on the common parts in start and end time, this prefix is formed
    string prefix = GetSearchPrefix(serviceName, startTimeForSearch, endTimeForSearch);

    Console.WriteLine("Prefix used = {0}", prefix);

    // List the blobs using the prefix
    IEnumerable<IListBlobItem> blobs = blobClient.ListBlobsWithPrefix(
        prefix, 
        new BlobRequestOptions()
        {
            UseFlatBlobListing = true,
            BlobListingDetails = BlobListingDetails.Metadata
        });
            
                           
    // iterate through each blob and figure the start and end times in the metadata
    foreach (IListBlobItem item in blobs)
    {                            
        CloudBlob log = item as CloudBlob;
        if (log != null)
        {
            DateTime startTime = DateTime.Parse(log.Metadata[LogStartTime]).ToUniversalTime();
            DateTime endTime = DateTime.Parse(log.Metadata[LogEndTime]).ToUniversalTime();
            string logTypes = log.Metadata[LogEntryTypes].ToUpper();

            LoggingLevel levelsInLog = GetLoggoingLevel(logTypes);

            // we will exclude the file if the time range does not match or it does not contain the log type 
            // we are searching for
            bool exclude = (startTime > endTimeForSearch 
                || endTime < startTimeForSearch 
                || (loggingLevelsToFind & levelsInLog) == LoggingLevel.None);

            Console.WriteLine("{0} Log {1} Start={2:U} End={3:U} Types={4}.",
                exclude ? "Ignoring" : "Selected",
                log.Uri.AbsoluteUri,
                startTime,
                endTime,
                logTypes);
                    
            if (!exclude)
            {
                selectedLogs.Add(log);
            }
        }
    }
       
    return selectedLogs;
}

/// <summary>
/// Given service name, start time for search and end time for search, creates a prefix that can be used
/// to efficiently get a list of logs that may match the search criteria
/// </summary>
/// <param name="service"></param>
/// <param name="startTime"></param>
/// <param name="endTime"></param>
/// <returns></returns>
static string GetSearchPrefix(string service, DateTime startTime, DateTime endTime)
{
    StringBuilder prefix = new StringBuilder("$logs/");
            
    prefix.AppendFormat("{0}/", service);
            
    // if year is same then add the year
    if (startTime.Year == endTime.Year)
    {
        prefix.AppendFormat("{0}/", startTime.Year);
    }
    else
    {
        return prefix.ToString();
    }

    // if month is same then add the month
    if (startTime.Month == endTime.Month)
    {
        prefix.AppendFormat("{0:D2}/", startTime.Month);
    }
    else
    {
        return prefix.ToString();
    }

    // if day is same then add the day
    if (startTime.Day == endTime.Day)
    {
        prefix.AppendFormat("{0:D2}/", startTime.Day);
    }
    else
    {
        return prefix.ToString();
    }

    // if hour is same then add the hour
    if (startTime.Hour == endTime.Hour)
    {
        prefix.AppendFormat("{0:D2}00", startTime.Hour);
    }

    return prefix.ToString();
}

Once we have a list of logs, then we just need to iterate and see if the log entry in the log contains the keyword.

/// <summary>
/// Displays all log entries containing a keyword
/// </summary>
/// <param name="blobList"></param>
/// <param name="keyword"></param>
static void DisplayLogs(List<CloudBlob> blobList, string keyword)
{
    Console.WriteLine("Log entries matching '{0}' are: ", keyword);

    foreach (CloudBlob blob in blobList)
    {
        using (Stream stream = blob.OpenRead())
        {
            using (StreamReader reader = new StreamReader(stream))
            {
                string logEntry;
                while((logEntry = reader.ReadLine()) != null)
                {
                    if (logEntry.Contains(keyword))
                    {
                        Console.WriteLine(logEntry);
                    }
                }
            }
        }
    }

    Console.WriteLine("End searching for '{0}'", keyword);
}


/// <summary>
/// Given a ',' separated list of log types aka operation types, it returns the logging level 
/// </summary>
/// <param name="loggingLevels"></param>
/// <returns></returns>
static LoggingLevel GetLoggoingLevel(string loggingLevels)
{
    string[] loggingLevelList = loggingLevels.ToUpper().Split(',');
    LoggingLevel level = LoggingLevel.None;
    foreach (string logLevel in loggingLevelList)
    {
        if (string.Equals("READ", logLevel))
        {
            level |= LoggingLevel.Read;
        }
        else if (string.Equals("WRITE", logLevel))
        {
            level |= LoggingLevel.Write;
        }
        else if (string.Equals("DELETE", logLevel))
        {
            level |= LoggingLevel.Delete;
        }
    }

    return level;
}

[Flags]
public enum LoggingLevel
{
     None = 0,
     Delete = 2,
     Write = 4,
     Read = 8,
 }

The main method then just invokes ListLogFiles and DisplayLogs.

static void Main(string[] args)
{
    if (args.Length < 4)
    {
        Console.WriteLine("Usage: LogFind <service to search - blob|table|queue> <Comma separated lit of operations> <Start time in UTC for search> <End time in UTC for search> <search string>.");
        Console.WriteLine("Example: LogFind blob \"WRITE,DELETE\" \"2011-06-26T22:30Z\" \"2011-06-26T22:50Z\" \"DeleteContainer\"");
        return;
    }

    CloudStorageAccount account = CloudStorageAccount.Parse(ConnectionString);
    CloudBlobClient blobClient = account.CreateCloudBlobClient();
           
    DateTime startTimeOfSearch = DateTime.Parse(args[2]).ToUniversalTime();
    DateTime endTimeOfSearch = DateTime.Parse(args[3]).ToUniversalTime();

    List<CloudBlob> blobList = ListLogFiles(blobClient, args[0], startTimeOfSearch, endTimeOfSearch, args[1]);
    DisplayLogs(blobList, args[4]);
}

Given the above methods, to search for log entries between 02:50 and 3:05 for DeleteContainer, the following can be executed. The spew from the console application is self-describing. It lists all the logs that are created between the two dates and then selects only the ones that have “Delete” in the LogType and then once it gets the eligible logs, it downloads it and then outputs any lines in the log that contains the search keyword:

LogFind blob "DELETE" "2011-06-27T02:50Z" "2011-06-27T03:05Z" "DeleteContainer"

This could give the following example results in the command line window:

This could give the following example results in the command line window:
Prefix used = $logs/blob/2011/06/27/
Selected Log http://gameusnorth.blob.core.windows.net/$logs/blob/2011/06/27/0200-000000 Start=Monday, June 27, 2011 2:58:06 AM End=Monday, June 27, 2011 2:59:45 AM Types=READ,WRITE,DELETE.
Ignoring Log http://gameusnorth.blob.core.windows.net/$logs/blob/2011/06/27/0300-000000 Start=Monday, June 27, 2011 3:00:25 AM End=Monday, June 27, 2011 3:53:42 AM Types=READ,WRITE.
Ignoring Log http://gameusnorth.blob.core.windows.net/$logs/blob/2011/06/27/0500-000000 Start=Monday, June 27, 2011 5:03:47 AM End=Monday, June 27, 2011 5:03:47 AM Types=READ.
Ignoring Log http://gameusnorth.blob.core.windows.net/$logs/blob/2011/06/27/0600-000000 Start=Monday, June 27, 2011 6:05:48 AM End=Monday, June 27, 2011 6:13:31 AM Types=READ.
Ignoring Log http://gameusnorth.blob.core.windows.net/$logs/blob/2011/06/27/0600-000001 Start=Monday, June 27, 2011 6:11:55 AM End=Monday, June 27, 2011 6:14:23 AM Types=READ.
Ignoring Log http://gameusnorth.blob.core.windows.net/$logs/blob/2011/06/27/0600-000002 Start=Monday, June 27, 2011 6:40:55 AM End=Monday, June 27, 2011 6:41:56 AM Types=READ.
Ignoring Log http://gameusnorth.blob.core.windows.net/$logs/blob/2011/06/27/0600-000003 Start=Monday, June 27, 2011 6:52:23 AM End=Monday, June 27, 2011 6:58:33 AM Types=READ.
Ignoring Log http://gameusnorth.blob.core.windows.net/$logs/blob/2011/06/27/0700-000000 Start=Monday, June 27, 2011 7:00:44 AM End=Monday, June 27, 2011 7:05:31 AM Types=READ.
Ignoring Log http://gameusnorth.blob.core.windows.net/$logs/blob/2011/06/27/0700-000001 Start=Monday, June 27, 2011 7:06:52 AM End=Monday, June 27, 2011 7:12:56 AM Types=READ.
Ignoring Log http://gameusnorth.blob.core.windows.net/$logs/blob/2011/06/27/0700-000002 Start=Monday, June 27, 2011 7:21:13 AM End=Monday, June 27, 2011 7:26:21 AM Types=READ,WRITE.

Log entries matching 'DeleteContainer' are:

1.0;2011-06-27T03:00:40.4636789Z;DeleteContainer;Success;202;18;18;authenticated;gameusnorth;gameusnorth;blob;"http://gameusnorth.blob.core.windows.net/photos?restype=container&timeout=30000";"/gameusnorth /photos";e09a61de-e47b-40aa-86e0-05fe620f818f;0;268.20.203.21;2009-09-19;364;0;125;0;0;;;;;;;;"ClientID f38d713b-7113-4fea-9173-9e9b00b22f71"


End searching for 'DeleteContainer'
Press any key to continue . . .

For more information, please see the MSDN Documentation.

---

<Return to section navigation list>

SQL Azure Database and Reporting

Cihan Biyikoglu offered a Quick Tip - Federations in SQL Azure: How to automatically maintain the federation distribution key in federated tables with FILTERING connections in an 8/3/2011 post:

In federations, federations member contain part of the data that is in a federation. The range of data contained in a federation member depends on the layout of the partitioning you have in your federations. There could be 10s or 100s of federation members covering the full range of values of the federation key. When connecting to federation member, apps use the USE FEDERATION command. USE FEDERATION takes a federation name (f1 below), a federation distribution key name (id below) and a atomic unit value (the guid value in the example below).

   1:  use federation f1(id='82855FD2-BA4C-4F72-BFB5-78523741C5A8')
   2:  with reset, filtering=off

USE FEDERATION provides 2 connection types.

#1 The unfiltering connection is just a regular connection to the whole federation member containing this atomic unit. This is no different than connecting to the db name of the federation member. In the example below, I express to connect to a GUID value, however when I query t1 for all rows, I get back all data that is in this federation member’s range.

    1:  use federation f1(id='00000000-0000-0000-0000-000000000000')
    2:  with reset, filtering=off
    3:  go
    4:  select * from t1
    5:  go

Returns: 

   1:  c1                                   c2
   2:  ------------------------------------ ------
   3:  08FFE74B-1BC0-460B-AA47-044E2DD7C0E0 b
   4:  732ABAAE-8727-4C49-B40B-8013DFB3C735 b

#2 The Filtering connection is a new concept that is available with federations. Filtering connection connect to a slice of a database that contains your atomic unit as opposed to the whole federation member (database). The filtering connection simply eliminates the need to express the federation key value in every query against federated tables. SQL Azure automatically injects the WHERE <federation_column>=federation_filtering_value(…) into all queries that target a federated table. This means the above example returns no rows since there is not rows present for '00000000-0000-0000-0000-000000000000’ in t1. If I connect to an existing value such as … I only get the rows that are in the atomic unit for t1 for my query. See the example below;

   1:  use federation f1(id='08FFE74B-1BC0-460B-AA47-044E2DD7C0E0')
   2:  with reset, filtering=on;
   3:  go
   4:  select * from t1
   5:  go

Returns: 

   1:  c1                                   c2
   2:  ------------------------------------ ------
   3:  08FFE74B-1BC0-460B-AA47-044E2DD7C0E0 b

If you establish a filtering connection to a federation member, we do provide the following handy function to return the value back to you in the federation member.

    1:  use federation f1(id='08FFE74B-1BC0-460B-AA47-044E2DD7C0E0')
    2:  with reset, filtering=on
    3:  go
    4:  select federation_filtering_value('id') as filtering_value
    5:  go

Returns:

   1:  filtering_value
   2:  ------------------------------------
   3:  08FFE74B-1BC0-460B-AA47-044E2DD7C0E0

You can also use the function to maintain the federation distribution column in federated tables for DML statements such as INSERTs. In the example below, t1 is created with a default based on this function. once a filtering connection is established, insert statements only specify the value to insert to column c2 and value of c1 is maintained automatically for you by the default property on table t1. Here is a full end to end example;

   1:  use federation root with reset
   2:  go
   3:  create federation f1(id uniqueidentifier range)
   4:  go
   5:  select 'use federation f1(id='''+cast(newid() as varchar(100))+''')
   6:  with reset, filtering=on'
   7:  go
   8:  use federation f1(id='82855FD2-BA4C-4F72-BFB5-78523741C5A8')
   9:  with reset, filtering=off
 10:  go
 11:  create table t1(
 12:  c1 uniqueidentifier primary key default federation_filtering_value('id'),
 13:  c2 nvarchar(100))
 14:  federated on (id=c1)
 15:  go
 16:  select 'use federation f1(id='''+cast(newid() as varchar(100))+''')
 17:  with reset, filtering=on'
 18:  go
 19:  use federation f1(id='732ABAAE-8727-4C49-B40B-8013DFB3C735')
 20:  with reset, filtering=on
 21:  go
 22:  -- value of c1 is set to the atomic unit value in the filtering connection
 23:  -- '732ABAAE-8727-4C49-B40B-8013DFB3C735'
 24:  insert into t1(c2) values('a')
 25:  go
 26:  select federation_filtering_value('id')
 27:  go
 28:  select 'use federation f1(id='''+cast(newid() as varchar(100))+''')
 29:  with reset, filtering=on'
 30:  go
 31:  use federation f1(id='08FFE74B-1BC0-460B-AA47-044E2DD7C0E0') 
 32:  with reset, filtering=on 
 33:  go 
 34:  -- value of c1 is set to the atomic unit value in the filtering connection 
 35:  -- '08FFE74B-1BC0-460B-AA47-044E2DD7C0E0' 
 36:  insert into t1(c2) values('b') 
 37:  go 
 38:  select federation_filtering_value('id') 
 39:  go 
 40:  use federation f1(id='00000000-0000-0000-0000-000000000000') 
 41:  with reset, filtering=off; 
 42:  go 
 43:  select * from t1 
 44:  go

Returns: 

   1:  c1                                   c2
   2:  ------------------------------------ ------
   3:  08FFE74B-1BC0-460B-AA47-044E2DD7C0E0 b
   4:  732ABAAE-8727-4C49-B40B-8013DFB3C735 b

Note: This functionality will be in the final version of the product however this does not work on the preview bits just yet. If you are in the technology preview program for federations today, you will get this functionality when we refresh the build to a more recent build.

---

Welly Lee reported the availability of a SSMA Feature Overview in a 8/3/2011 post to the SQL Server Migration Assistant blog:

TechNet Magize published an article which provide overview of SQL Server Migration Assistant (SSMA) features and how it can be used to support database migration to SQL Server.

SQL Server: Manage the Migration describes the main feature area of SSMA including migration assessment, converting database schema, migrating data, and testing database migration. If you prefer to watch video presentation, I recommend to watch the recording of the SSMA session at 2011 North America TechEd available from Channel 9.

---

<Return to section navigation list>

MarketPlace DataMarket and OData

AllComputers.us described Integrating DataMarket Data with a Visual Web Part : Create a WCF Service to Retrieve DATA.gov Crime Data (part 1) in a 7/21/2011 post (missed when published):

Although being able to manually explore the Windows Azure Marketplace DataMarket data is interesting, it’s arguably a little more interesting to consume that data programmatically from within other applications such as SharePoint Web Parts. Consuming the data in a Web Part involves data aggregation, creating custom filtering options, and of course, making other parts of SharePoint more meaningful through richer data sets.

In this section, you’ll create a WCF service that will retrieve the DATA.gov crime data from the Windows Azure DataMarket data programmatically. The reason for using a WCF service is that you can repurpose the service across different applications, thus mitigating the amount of disparate code you need to manage. (Note that you can also deploy the WCF service to Windows Azure and then use the data for Microsoft Office 365/SharePoint Online applications.) For example, you can build a SharePoint Visual Web Part and a Microsoft Silverlight application; both can use a common WCF service but consume and interact with it in different ways within the Microsoft .NET platform. For example, Silverlight application events must be fired and managed asynchronously.

Here’s the procedure to create a WCF service that retrieves the DATA.gov crime data.

Create a WCF Service to Retrieve DATA.gov Crime Data

1. Create a WCF Service to Retrieve DATA.gov Crime Data

1. Open Microsoft Visual Studio 2010.

2. Click Create | New Project | WCF, and select WCF Service Application. Name the project GetDallasData, and then click OK.

3. After the project has been created, right-click the project in the Solution Explorer, select Add, and then select Class.

4. Name the new class CrimeData, and add the properties in bold to the newly created class as shown in the following code:

   using System;
   using System.Collections.Generic;
   using System.Linq;
   using System.Web;
   namespace GetDallasData
   {
       public class CrimeData
      {
           public string City { get; set; }
           public string Year { get; set; }
           public string Population { get; set; }
           public string ViolentCrime { get; set; }
           public string Robbery { get; set; }
           public string PropertyCrime { get; set; }
           public string Burglary { get; set; }
       }
   }
   

   The preceding code represents the structure of your in-memory data object. The properties will be used to store specific elements of the incoming data stream. Note that you will not use every element or property in the incoming XML stream—you will use only certain parts of the DATA.gov data. That’s why this in-memory object (the class) does not directly map to all the properties that were displayed when you explored the data via the Marketplace DataMarket site.

5. Right-click the default Service1.svc file in the Solution Explorer, and rename it to DallasCrimeData.svc. Also, rename the IService1.cs interface file to IDallasCrimeData.cs.

   Note:

   Rename the service and service interface references in the Visual Studio project by clicking the reference in code (for example, Service1), and then click the Refactor menu option and select Rename. Provide the new name for the service and service interface.

   Your project folders should now look like the following image.

   

6. Double-click the DallasCrimeData.svc file, delete the starter methods that Visual Studio creates for you, and update the code as shown in bold in the following code. This will retrieve the Windows Azure data from the DataMarket by using a WebRequest object. You need to pass your Windows Live ID (for example, john_doe@hotmail.com) and your account key (which looks something like asaaKe8kisiK8leE+JkkswjSv2okwWy3zYAIcsbvC4TT=) with the WebRequest:

   using System;
   using System.Collections.Generic;
   using System.Runtime.Serialization;
   using System.ServiceModel;
   using System.ServiceModel.Web;
   using System.Text;
   using System.Net;
   using System.IO;
   using System.Xml.Linq;
   using System.Linq;
   
   namespace GetDallasData
   {
       public class DallasCrimeData : IDallasCrimeData
       {
           string myAccountKey = "myliveID@hotmail.com";
           string myUniqueUserId = "your key here";
           string myDallasURL = "https://api.datamarket.azure.com/Data.ashx/data.gov/
   Crimes/CityCrime?$filter=State%20eq%20%27Washington%27&$top=100";
           public List<CrimeData> GetCrimeData(string stateFilter)
           {
               List<CrimeData> listOfCrimesInWashington = new List<CrimeData>();
               WebRequest azureWebRequest = WebRequest.Create(myDallasURL);
               azureWebRequest.Credentials = new NetworkCredential(myAccountKey,
   myUniqueUserId);
               using (HttpWebResponse azureWebResponse = (HttpWebResponse)azureWebRequest.
   GetResponse())
               {
                   using (Stream AzureDataStream = azureWebResponse.GetResponseStream())
                   {
                       using (StreamReader reader = new StreamReader(AzureDataStream))
                       {
                           string responseFromAzure = reader.ReadToEnd();
                           XDocument xmlAzureResultData = XDocument.Parse(responseFrom
                               Azure);
                           XNamespace nsContent = "http://www.w3.org/2005/Atom";
                           XNamespace nsProperties = "http://schemas.microsoft.com/
                               ado/2007/08/dataservices/metadata";
                           XNamespace nsValue = "http://schemas.microsoft.com/
                               ado/2007/08/dataservices";
                           var result = (from q in xmlAzureResultData.
                               Descendants(nsContent + "entry")
                                         select new CrimeData
                                         {
                                             City = q.Element(nsContent + "content").
   Element(nsProperties + "properties").Element(nsValue + "City").Value.ToString(),
                                             Year = q.Element(nsContent + "content").
   Element(nsProperties + "properties").Element(nsValue + "Year").Value.ToString(),
                                             Population = q.Element(nsContent +
   "content").Element(nsProperties + "properties").Element(nsValue +
   "Population").Value.ToString(),
                                             ViolentCrime = q.Element(nsContent +
   "content").Element(nsProperties + "properties").Element(nsValue +
   "ViolentCrime").Value.ToString(),
                                             Robbery = q.Element(nsContent + "content").
   Element(nsProperties + "properties").Element(nsValue + "Robbery").Value.ToString(),
                                             PropertyCrime = q.Element(nsContent +
   "content").Element(nsProperties + "properties").Element(nsValue +
   "PropertyCrime").Value.ToString(),
                                             Burglary = q.Element(nsContent + "content").
   Element(nsProperties + "properties").Element(nsValue + "Burglary").Value.ToString()
                                         });
                           foreach (var c in result)
                           {
                               listOfCrimesInWashington.Add(c);
                           }
                       }
                   }
               }
               return listOfCrimesInWashington;
           }
       }
   }       

   The purpose of the preceding code is to use a predefined REST URI (one that retrieves crime data for Washington state), as shown here:

   https://api.datamarket.azure.com/Data.ashx/Data.gov/Crimes/CityCrime?$filter=State%20eq%20%27Washington%27&$top=100

   You then manage the data returned from the Marketplace DataMarket by loading it into an XDocument object that is then parsed and read into your custom object. Remember that you’re reading only specific parts of the XDocument object into the in-memory CrimeData object.

7. After adding the core service operation, you’ll need to add the interface contract. To do this, double-click the IDallasCrimeData.cs file, delete the starter methods, and edit it, adding the bold code as shown below:

   using System;
   using System.Collections.Generic;
   using System.Linq;
   using System.Runtime.Serialization;
   using System.ServiceModel;
   using System.ServiceModel.Web;
   using System.Text;
   
   namespace GetDallasData
   {
       [ServiceContract]
       public interface IDallasCrimeData
       {
           [OperationContract]
           List<CrimeData> GetCrimeData(string stateFilter);
       }
   }

8. To test the service, either press F5 on your keyboard or right-click DallasCrimeData.svc in the Solution Explorer and select View In Browser. The result should display a file listing in your browser. If you click your DallasCrimeData.svc file, the resulting page will be similar to the following image.

   

You have now created the core WCF service that you’ll be able to use to retrieve the Windows Azure data from the Marketplace DataMarket. However, before you can use the service, you’ll want to deploy it somewhere. This requires you to first publish the service from your Visual Studio solution to a virtual directory and then create an Internet Information Services (IIS) website.

In this exercise, you’ll deploy the service to your local IIS. Note that you could also deploy this service to Windows Azure and then call the service from there.

---

AllComputers.us continued its series with Integrating DataMarket Data with a Visual Web Part : Create a WCF Service to Retrieve DATA.gov Crime Data (part 2) of 7/21/2011 (missed when published):

2. Deploy a WCF Service to IIS

1. Open Windows Explorer, and create a new directory called DallasCrimeDataWCFService. Publish the WCF service project to the new directory by right-clicking the project, and selecting Publish. Then select File System in the Publish Method drop-down list, browse to the new directory location for the Target location, and finally click Publish.

2. Open IIS 7.0 Manager, and navigate to Sites. Right-click Sites, and select Add Web Site.

3. In the Add Web Site dialog box, add the Site name, select the physical path to the directory you just created (as shown in the following image), and then select Connect As and configure the access (for testing purposes only) using an account that has full administrator privileges on the server (such as the administratoraccount). Provide a password, click OK, and then click Test Settings. Be sure that you also select a separate port other than the default port 80 (for example, 2298).

   

4. Click OK when done.

5. Click the Content View tab, and right-click the DallasCrimeData.svc service file to ensure that it is working correctly from your local instance of IIS.

---

Note:

Ensure that your application pool is set to the Microsoft .NET Framework 4; if it’s set to an earlier version of the .NET Framework, you may get an error when trying to test the service from within IIS. To remedy this, navigate to Application Pools, select the DallasAzureCrimeData application pool, click Basic Settings, and then ensure that the .NET Framework version is similar to Figure 1.

Figure 1. Configuring the application pool in IIS.

---

After you’ve successfully tested the local deployment of your WCF service, you can use it in other applications. Copy the service URL from the test in your web browser to your Clipboard or a Notepad file.

3. Consume a WCF Service in a Visual Web Part

1. Open Visual Studio 2010.

2. From the menu, select File | New | SharePoint | Empty SharePoint Project. Name the project DallasCrimeDataWebPart.

3. Be sure to select Deploy As Farm Solution when prompted by the Customization Wizard. When you’re done, click Finish.

4. Right-click Reference, and select Add Service Reference.

5. Paste the URL you copied from the IIS service test (for example, http://blueyonderdemo:6622/DallasCrimeData.svc), and click Go.

6. When the service definition successfully returns, name the service reference DallasCrimeDataService, and then click OK.

7. After the project has been created, right-click the project in the Solution Explorer, select Add, and then select Class.

8. Name the new class ReturnCrimeData, and add the bold properties shown in the following code to the newly created class:

   ...
   
   namespace GetDallasData
   {
       public class CrimeData
       {
           public string City { get; set; }
           public string Year { get; set; }
           public string Population { get; set; }
           public string ViolentCrime { get; set; }
           public string Robbery { get; set; }
           public string PropertyCrime { get; set; }
           public string Burglary { get; set; }
       }
   }

9. Right-click the project, select Add, and then click New Item.

10. In the New Item dialog box, select SharePoint and Visual Web Part. After the Visual Web Part is added to your SharePoint project, rename it to CrimeDataVisualWebPart.

11. Right-click CrimeDataVisualWebPartUserControl.ascx, and select View Designer.

12. Type in the short title Dallas Crime Data, and then drag a GridView and a Button control onto the designer surface. Name the GridView control datagrdDallasData and the Button control btnGetData. Your UI should look similar to the following image.

    

13. Double-click the Button control to add an event handler.

14. If Visual Studio does not toggle to the code view, right-click CrimeDataVisualWebPart UserControl.ascx and select View Code.

15. Update the btnGetData_Click event (which was generated automatically when you double-clicked the button) with the following bold code:

    
    

    using System;
    using System.Web.UI;
    using System.Web.UI.WebControls;
    using System.Web.UI.WebControls.WebParts;
    using DallasCrimeDataWebPart.DallasCrimeDataSvc;
    using System.Collections.Generic;
    using System.Collections;
    using System.ServiceModel;
    
    namespace DallasCrimeDataWebPart.CrimeDataVisualWebPart
    {
        public partial class CrimeDataVisualWebPartUserControl : UserControl
        {
            protected void Page_Load(object sender, EventArgs e)
            {
            }
            protected void btnGetData_Click(object sender, EventArgs e)
            {
                BasicHttpBinding mySvcbinding = new BasicHttpBinding();
                UriBuilder serviceURI = new UriBuilder("http://localhost:7833/
    DallasCrimeData.svc");
                DallasCrimeDataClient myWCFProxy = new DallasCrimeDataClient(mySvcbinding,
    new   EndpointAddress(serviceURI.Uri));
    
                var myCrimeData = myWCFProxy.GetCrimeData(null);
                List<ReturnCrimeData> returnCrimeDataFromService = new
    List<ReturnCrimeData>();
                foreach (var item in myCrimeData)
                {
                    ReturnCrimeData tempEntity = new ReturnCrimeData();
                    tempEntity.City = item.City;
                    tempEntity.Year = item.Year;
                    tempEntity.Population = item.Population;
                    tempEntity.ViolentCrime = item.ViolentCrime;
                    tempEntity.Robbery = item.Robbery;
                    tempEntity.PropertyCrime = item.PropertyCrime;
                    tempEntity.Burglary = item.Burglary;
                    returnCrimeDataFromService.Add(tempEntity);
                }
    
                myWCFProxy.Close();
    
                datagrdDallasData.DataSource = returnCrimeDataFromService;
                datagrdDallasData.DataBind();
            }
        }
    }
    
    					  

    The preceding code takes the return XML package from the WCF service call (which has already been filtered to return only a specific set of elements—that is, city, year, population, violent crime, robbery, property crime, and burglary). The Visual Web Part code then uses its own in-memory object to populate a list collection, which it binds to the DataGrid control.

16. You can configure the properties of the Visual Web Part by using the .webpart and elements.xml files. For example, the bold code in Example 1 shows how you can change the Title and Description of the Web Part. In Example 2, the property in bold text shows how you can deploy the Web Part into a custom group within SharePoint called “SP And Azure.”

    Example 1. Changing the Web Part title and description

    <?xml version="1.0" encoding="utf-8"?>
    <webParts>
      <webPart xmlns="http://schemas.microsoft.com/WebPart/v3">
        <metaData>
          <type name=
            "DallasCrimeDataWebPart.CrimeDataVisualWebPart.CrimeDataVisualWebPart,
            $SharePoint.Project.AssemblyFullName$" />
          <importErrorMessage>$Resources:core,ImportErrorMessage;
    </importErrorMessage>
        </metaData>
        <data>
          <properties>
            <property name="Title" type="string">Crime Data Visual Web Part</
    property>
            <property name="Description" type="string">My visual web part that
               displays crime data from Dallas Azure.</property>
          </properties>
        </data>
      </webPart>
    </webParts>
    
    					  

    Example 2. Custom group for Web Part

    <?xml version="1.0" encoding="utf-8"?>
    <Elements xmlns="http://schemas.microsoft.com/sharepoint/" >
      <Module Name="CrimeDataVisualWebPart" List="113" Url="_catalogs/wp">
        <File Path="CrimeDataVisualWebPart\CrimeDataVisualWebPart.webpart"
           Url="CrimeDataVisualWebPart.webpart" Type="GhostableInLibrary" >
          <Property Name="Group" Value="SP And Azure" />
        </File>
      </Module>
    </Elements>

17. At this point, you can press F6 to build the Visual Web Part to ensure that it builds successfully. When it builds successfully, you can then press F5 to debug the Visual Web Part to SharePoint.

18. Assuming no errors, right-click the project, and select Deploy. This will deploy the Visual Web Part to SharePoint.

    ---

    Note:

    If you run into any issues with this Web Part, you can right-click the project and select Retract to clean and pull the Web Part from SharePoint.

    ---

19. After you deploy the Web Part, navigate to SharePoint to the Web Part page you created earlier. Click Site Actions and Edit Page.

20. Remove any previously added Web Parts from the page by selecting Delete from each of the Web Part’s Options menu.

21. Click Add A Web Part, navigate to the SP And Azure category, and click the Visual Web Part you just deployed. If you used the same properties as in this exercise, the name of the Web Part will be Crime Data Visual Web Part. Click Add.

    

22. When the Web Part loads, click the Get Data button. This will trigger the btnGetData_Click event, which calls the WCF service deployed into IIS. The WCF service will return the DATA.gov crime data from the Windows Azure DataMarket. The returned data will then be bound to the DataGrid and displayed in the Web Part.

    Figure 2 shows the final set of filtered Windows Azure DataMarket data that is displayed in the Visual Web Part.

Figure 2. Visual Web Part displaying data from Windows Azure DataMarket.

Although using the Visual Web Part may fit into your plans, you may also want to use Silverlight to present the data. Although Silverlight does provide an enhanced user experience, it comes with some programmatic differences that you need to manage .

---

<Return to section navigation list>

Windows Azure AppFabric: Apps, Access Control, WIF and Service Bus

Ron Jacobs interviewed Workflow program manager Josh Twist in a 00:33:56 Workflows in Azure AppFabric Applications video segment for AppFabric TV:

Workflow in the cloud... it's like a silver lining just waiting to happen. In this episode I'm joined by Josh Twist Workflow PM who will show you how you can use Workflow in Windows Azure AppFabric.

---

Valery Mizonov explained How to Simplify & Scale Inter-Role Communication Using Windows Azure AppFabric Service Bus in an 8/2/2011 post to the AppFabric CAT blog:

The following blog post demonstrates how Windows Azure developers can leverage the Windows Azure AppFabric Service Bus to enable asynchronous inter-role communication in cloud-based solutions using Service Bus topics and subscriptions. This post and its associated code sample offer a complete solution, one that enables simplified, scalable, loosely coupled communication between role instances on the Windows Azure platform.

Background

A few months ago, we published an article that described how useful inter-role communication can be in the Windows Azure world. The article articulated how to regain and amplify the benefits of inter-role communication by using Service Bus, and also provided guidance, such as using the Observer pattern to abstract an event-driven programming paradigm and Parallel LINQ to take advantage of multi-core computers. The original technical implementation of the inter-role communication scenario was based on the “direct” messaging pattern offered by Service Bus NetEventRelay binding.

A lot of ice cubes have melted since then. The Windows Azure AppFabric team recently released a CTP version of a set of “brokered” messaging features in Service Bus along with queues and topics, all of which make it easier for developers to build reliable, distributed, scalable, loosely coupled applications using the asynchronous messaging pattern. Queues support the one-to-one pattern in which each message is consumed by a single consumer while topics support the one-to-many pattern in which each message is consumed by one or more subscriptions that are registered with the topic. Subscriptions can use filtering to select a subset of the messages from the topic. Both queues and topics offer a rich set of capabilities, including support for correlation, sessions, large messages, duplicate detection, dead-lettering and extensible message properties.

In this post, I intend to refresh the original guidance with the implementation of the inter-role communication pattern leveraging the new publish/subscribe capability introduced in the latest release of Service Bus. The end goal is to demonstrate how Service Bus can help solve the challenges related to inter-role communication and make it easier to connect pieces of distributed solutions across deployment boundaries and geographies in a simple and scalable manner.

Real-World Customer Scenario

Consider a travel search website. Users provide basic criteria such as departure city, arrival city, dates, the number of travelers, other preferences, and the web site responds with options of airline, hotels, rental car and special deals.

The system architecture includes a web-based application with front-end logic and a user interface and some middle-tier services hosting business logic for processing search requests. In the business logic layer, there is a “Search Analytics & Result Aggregation” (SARA) service that is responsible for communicating with various back-end services and collecting intermediate results. It aggregates all responses from individual search services which deal only with a specific portion of the request – flight information, hotel details, car hire options and special deals. These services in turn communicate with LOB applications and other third-party services in order to fulfill the user requests.

A popular approach to handling the data exchange between distributed application services is to employ a loosely coupled messaging architecture. A user request with search criteria is received by the web application and is routed to the SARA service to initiate its processing. A message exchange pattern that involves sending a message to a single recipient is known as unicast. Next, the SARA service distributes a search request message to multiple back-end services, each performing search operations in their own domains: flights, hotels, car rentals, deals. In this scenario, a single message is intended to be received by multiple consumers. Such a message exchange pattern is known as multicast. When each individual search service fulfills a request, it responds back to the originating SARA service with partial search results for the specific section of the end-to-end itinerary. These responses are returned to the SARA service for the final stage of processing. Again, this is another example of unicast messaging. The SARA service collects all partial responses and composes them into a coherent result which then gets submitted back to the web tier so that it can be rendered to the user on a web page.

As a reference point, below is the visual representation of the above message flow.

As it can be seen in the above example, different pieces of the application logic run in different types of Windows Azure roles and span multiple hosted services. There are advantages to constructing an application as a collection of loosely-coupled components. Windows Azure supports this model by having distinct roles to host the different functions of the application (e.g. a Web role for presentation logic, a Worker role for middle-tier logic, etc.). The communication between the solution’s tiers and services needs to happen seamlessly across the entire deployment. A web role instance should be able to communicate to the worker role instance that is deployed in a different hosted service. By isolating the tiers of a cloud-based solution into individual hosted service deployments, one can simplify operations such as upgrading or re-configuring a specific tier without impacting other tiers that are not affected by a servicing operation.

Furthermore, for high-availability reasons, it is typical to run multiple instances within each tier of the application. In the customer scenario above, the web application and SARA service will ultimately become candidates for multi-instance deployment in order to be highly available. Such a topology imposes a requirement on the inter-role communication mechanism to support multiple role instances as “competing consumers” and ensure guaranteed delivery of messages to one and only one instance.

Extending the deployment resiliency further, and as far as disaster recovery (DR) is concerned, it is common for a fully DR-compliant solution to be deployed in different data centers. When all the role instances for a particular tier go down in one data center, a DR set of role instances running the same logic from another data center can process the message. For example, if all SARA service instances in North Central US datacenter are not available by whatever reason, the user request should be processed by a copy of the SARA service running in South Central US datacenter. Spanning data centers using traditional mechanisms for inter-role communication (e.g. external endpoints) is not straightforward; doing it securely and reliably makes it even more complicated. So, whatever other option for inter-role communication is available, it needs to seamlessly address communications across any boundaries, be that within a single datacenter or across datacenters.

And finally, in a hybrid deployment scenario, some services of a cloud-based solution may be hosted on the premises. These services require being reachable from theirs consumers while keeping a great degree of location virtualization so that consumers doesn’t have to be bound to a specific endpoint address. In the above example, the back-end services can be co-located along with the LOB applications and other systems that must stay on-premises and cannot be moved to the cloud. Such a deployment raises further requirements related to a secure communication between cloud-based and on-premises services.

In summary, these challenges lead to a need to provide a seamless way to support the inter-role communication regardless of the deployment topology. That was the original motivation behind this blog post which naturally brings us into a technical design discussion on how it can be implemented.

Technical Design Walkthrough

To follow along, download the full sample code from the MSDN Code Gallery. If you would like to skip the design discussion and jump over to the implementation details, please go to the next section.

In the previous section, our real-world customer scenario has combined multiple message exchange patterns to address a set of data flow requirements. First, there is unicast-based messaging that occurs between the web application and the Search Analytics & Aggregation service. Second, there is multicast-based message distribution which fans out a search request to a number of distributed back-end services for processing. In addition to supporting these two key message exchange patterns, a good technical design must meet the following goals:

• Generically simple which implies a minimum setup effort and no “excessive baggage” of features except those that are required to support the inter-role communication mechanism, ideally expressed in the familiar “publish/subscribe” notion.

• Fully abstracted to hide all the mechanical parts used in handling message exchange operations and remove client’s dependency on the underlying communication and messaging APIs.

• Inherently reliable to make inter-role communication resilient to intermittent connectivity and contention issues that may manifest themselves in multi-tenant, congested or highly distributed environments.

• Practically cost-efficient to decrease the operating costs related to running a solution leveraging the inter-role communication via Service Bus.

The following subsections intend to walk you through the main design concepts that I have applied in the technical implementation.

Messaging Entities

First, let me describe at a high level how Service Bus publish/subscribe messaging is plugged in into the design and implementation of the inter-role communication (or IRC for short) discussed in this post.

The key Service Bus messaging entities involved are topics, subscriptions and rules.

• Topics support the publish/subscribe message exchange pattern in which each published message is made available to all subscriptions registered with a given topic. By contrast, in a queue each message is only consumed by a single consumer.

• Subscriptions act like virtual queues which receive a copy of each message that is published into a topic. A subscription can have one or more rules defined to restrict the messages that it wants to receive.

• Rules are filter expressions that are used to select which messages are “copied” in to the subscription’s virtual queue. Rules can optionally have associated filter actions that can manipulate the properties of the message as it flows in to a subscription’s virtual queue.

For more information on topics, subscriptions and rules please refer to alternative sources listed in the “Additional Resources/References” section.

Inter-Role Communication Topic

In our design, the communication between Windows Azure role instances is done through a single topic that is provisioned ahead of time. In the current CTP of Service Bus, a single topic supports up to 500 concurrent publishers and subscribers, however this limit may be re-evaluated and could change in the future. The topic name is specified in the application configuration and is assumed to always exist when a Windows Azure solution is deployed. Only those role instances that share the same topic name in their configuration will be able to communicate together.

Subscription Management

The management of subscriptions and filters is the responsibility of the component implementing the inter-role communication (further referred to as the “IRC component”). By default, subscriptions that are internally maintained by the IRC component are dynamic in nature. This means that the subscriptions are created at runtime, and are removed upon graceful termination of the host (a Windows Azure role instance process). If a role instance fails over to another node or restarted, it will be re-attached to the original subscription after a successful reboot. The IRC component can also be configured to support static subscriptions which are created once and never removed by the IRC component.

Filters & Event Routing

The inter-role communication topic has one or more subscriptions (or “IRC subscription” for short) that are created automatically at runtime with a single filter. The filter contains an expression allowing the subscription to accept messages sent either to all role instances, all instances of a particular role or a specific role instance. This is implemented as follows. Each inter-role communication event (or “IRC event” for short) is published as a message with two custom properties; these are IRC_From and IRC_To. When IRC_To is set to a predefined value of “*” (an asterisk symbol), this tells the subscribers that an event shall be treated as a multicast message and can be received by all role instances. By contrast, when IRC_To is set to a value that corresponds to an internally computed globally unique ID of a particular Windows Azure role instance, only that specific instance will receive an IRC event. In addition to that, the IRC_To property can also be set to the name of a particular role. In this scenario, either all instances of the given role will receive a copy of the message or only one instance will receive the message depending on whether the inter-role communication is configured to allow competing consumers.

In addition, each publisher can receive a copy of the IRC events it publishes. Consider it as a carbon copy feature. It can be enabled on demand should this capability be required. Carbon copies may be useful if the communication needs to occur between decoupled services or processes hosted by the same role instance node. Such a communication pattern is sometimes referred to as “intra-application communication”. In the given context, the use of Service Bus topics and their durability help make intra-application communication reliable and guaranteed. Another potential use case for a carbon copy is to enable the sender to verify that the underlying publish/subscribe infrastructure is fully operational end-to-end. A receipt of a carbon copy indicates that the message successful went through the topic and came back. However, it should not be treated as an acknowledgement as this doesn’t warrant a successful receipt by all participating subscribers.

The filter expression that defines who can and cannot receive IRC events can be visualized as follows:

The interesting part in the above filter definition is that such a simple expression can effectively accommodate both unicast (single recipient) and multicast (many recipients) message exchange patterns, and also allows filtering out carbon copies if they are not desirable.

In order to reduce the coupling between the application code and the underlying communications infrastructure, I introduced an abstraction layer implemented via a generic contract, as shown below. Here I take advantage of the IObservable<T> interface which enables pushing the IRC events to one or more observers.

/// <summary>
/// Defines a generic contract for inter-role communication mechanism using publish/subscribe messaging infrastructure.
/// </summary>
public interface IInterRoleCommunicationExtension : IObservable<InterRoleCommunicationEvent>, IDisposable
{
    /// <summary>
    /// Publishes the specified inter-role communication event to one or more subscribers.
    /// </summary>
    /// <param name="e">The inter-role communication event to be delivered to the subscribers.</param>
    void Publish(InterRoleCommunicationEvent e);
}

It is expected that the implementations of the IInterRoleCommunicationExtension interface may possess some system resources such as communication objects that need to be explicitly disposed in a controlled manner. Therefore, the contract also supports the Dispose pattern by inheriting from IDisposable as shown above. In summary, the IInterRoleCommunicationExtension interface includes just 3 methods to be implemented: Publish, Subscribe and Dispose.

The next section provides examples of how and where to use the above methods.

Another code artifact that is worth mentioning in the context of the IRC framework is the InterRoleCommunicationEvent class which has surfaced in the inter-role communication interface above. This class implements a generic IRC event that carries user-defined payload, custom properties and event routing information. The implementation of this class is defined as follows:

/// <summary>
/// Represents a generic event for inter-role communication.
/// </summary>
[DataContract]
public class InterRoleCommunicationEvent
{
    /// <summary>
    /// Returns the event payload which is a DataContract-serializable object that carries the event's body.
    /// </summary>
    [DataMember]
    public object Payload { get; private set; } 

    /// <summary>
    /// Returns the ID of the event sender.
    /// </summary>
    [DataMember]
    public string From { get; internal set; }

    /// <summary>
    /// Gets or sets the ID of the event receiver.
    /// </summary>
    [DataMember]
    public string To { get; set; }

    /// <summary>
    /// Returns a collection of custom properties associated with the event.
    /// </summary>
    [DataMember]
    public IDictionary<string, object> Properties { get; private set; }
}

Let’s take a look at the event message flow. The diagram below depicts the inter-role communication between three roles in a Windows Azure solution, each deployed in its own hosted service spanning two different datacenters.

In the example above, there is a single topic with 8 subscriptions. Each subscription is created by the individual role instances at startup. Inside each subscription, there is a single rule with a filter expression similar to the one discussed earlier.

Sending a message to a particular role instance (a unicast pattern) involves setting up a correct value of the IRC_To message context property. This is performed by the IRC component so that the consumer doesn’t need to worry about dealing with these low-level concepts. In the above example, a unicast message is sent from role instance S1-1 to S3-1. The message is put into a topic with IRC_To equal to “S3-1” and matched to a single subscription created by role instance S3-1. The message is then reliably delivered to its destination.

To fan out a message to multiple role instances (a multicast pattern), the IRC_To message context property is set to an asterisk symbol. When the message is put into a topic, it is matched by all active subscriptions and therefore will be delivered to all role instances listening on their subscriptions.

Those are the main aspects that you need to be conceptually aware of. In the next section, these concepts are turned into concrete guidance around enabling inter-role communication for a Window Azure solution.

On a side note, it is worth mentioning that the technical solution discussed above represents a generic framework that supports a broad range of message exchange patterns. Such a generic approach has led to a certain degree of unification and standardization of how message exchange is implemented behind the scene. If you are building a messaging solution using Service Bus to address a specific problem, there will be alternative approaches with respect to a particular problem space.

Step-By-Step Implementation Guide

Let’s proceed with a step-by-step instruction on how to enable a Windows Azure solution to support the inter-role communication mechanism discussed in this post. For the purposes of clarification, it is worth pointing out that the technical implementation discussed in this post is based off the current CTP release of Service Bus. While the Service Bus is still in CTP, all of its management tasks such as creating namespaces and topics must be performed in the LABS environment at http://portal.appfabriclabs.com.

Step 1: Create a Service Bus Namespace

You will first need to provision a Service Bus namespace. If you already have an existing namespace, it can be reused for inter-role communication. In either case, you will need to note down the three key elements: namespace name, issuer name and issuer secret.

The three highlighted parameters will be required for configuring the inter-role communication component in step 3.

Step 2: Add an Inter-Role Communication Component to a Window Azure Solution

In this step, you will add a reference to Microsoft.AppFabricCAT.Samples.Azure.InterRoleCommunication.dll assembly to every project implementing either web or worker roles in a Visual Studio solution. In addition, add a reference to Microsoft.ServiceBus.dll and Microsoft.ServiceBus.Messaging.dll from the AppFabric SDK. The location of these assemblies depends on the version of the SDK. For June CTP, the path is %ProgramFiles%\Windows Azure AppFabric SDK – June 2011 CTP\ServiceBus\.

Important: It is very important to set the Copy Local property of the respective assembly references to True. Otherwise, the role instances may not initialize and spin themselves off successfully when deployed on Windows Azure.

Step 3: Configure the Inter-Role Communication Component

Once a reference to the assembly containing the IRC component has been added to the respective projects, the next step is to provide some configuration information to enable the IRC component to connect to the target Service Bus topic. Such a configuration will need to be supplied in the app.config file in each Visual Studio project implementing a web or worker role. The respective app.config file is to be modified as shown:

<configuration>
  <configSections>
    <!-- Add this line into your own app.config -->
    <section name="ServiceBusConfiguration" type="Microsoft.AppFabricCAT.Samples.Azure.InterRoleCommunication.Framework.Configuration.ServiceBusConfigurationSettings, Microsoft.AppFabricCAT.Samples.Azure.InterRoleCommunication" />
  </configSections>

  <!-- Add this entire section into your own app.config -->
  <ServiceBusConfiguration defaultEndpoint="IRC">
    <add name="IRC" serviceNamespace="irc-test" endpointType="Topic" topicName="InterRoleCommunication" issuerName="owner" issuerSecret="wjc/h98iL95XudZPJ1txaDB..." />
  </ServiceBusConfiguration>
</configuration>

In your version of app.config file, 3 parameters in the above fragment will have to be modified:

1. Service Bus namespace (see step 1).

2. Issuer name (see step 1).

3. Issuer secret (see step 1)

If required, you can also modify the name of the Service Bus topic that will be used to exchange the IRC events.

Note: If you have multiple Visual Studio projects implementing web/worker roles which need to participate in the inter-role communication, you will need to ensure that the app.config file is present in each project. Instead of creating multiple copies of app.config and placing them in each individual project, you can add app.config as a link rather than directly adding the file to your projects. Linked project items in Solution Explorer can be identified by the link indicator in its icon (a small arrow in the lower left corner). By linking to a file, you can capture ongoing changes to the configuration file without having to manually update a copy whenever changes are made. To create a link to a file, select the target project, open the Add Existing Item dialog box, locate and select the master app.config file you want to link. From the Open button drop-down list, select Add As Link.

In addition to configuration settings expressed inside an app.config file, the IRC component supports several useful runtime settings that are intended to be programmatically modified if their default values are not sufficient. These runtime settings are combined in the InterRoleCommunicationSettings object and are listed below:

Step 4: Initialize the Inter-Role Communication Component

In this step, you will extend the startup logic of your web or worker role implementation with a code snippet that is responsible for initializing the IRC component. The following example demonstrates how to initialize the IRC component from within a worker role.

using Microsoft.AppFabricCAT.Samples.Azure.InterRoleCommunication;
using Microsoft.AppFabricCAT.Samples.Azure.InterRoleCommunication.Framework.Configuration;

public class SampleWorkerRole : RoleEntryPoint
{
    private IInterRoleCommunicationExtension interRoleCommunicator;

    /// <summary>
    /// Called by Windows Azure service runtime to initialize the role instance.
    /// </summary>
    /// <returns>Return true if initialization succeeds, otherwise returns false.</returns>
    public override bool OnStart()
    {
        // ...There is usually some code here...

        try
        {
            // Read the configuration settings from app.config.
            var serviceBusSettings = ConfigurationManager.GetSection(ServiceBusConfigurationSettings.SectionName) as ServiceBusConfigurationSettings;

            // Resolve the Service Bus topic that will be used for inter-role communication.
            var topicEndpoint = serviceBusSettings.Endpoints.Get(serviceBusSettings.DefaultEndpoint);

            // Initialize the IRC component with the specified topic endpoint.
            // Instantiating the IRC component inside the OnStart method ensures that IRC component is a singleton.
            // The Singleton pattern prevents the creation of competing consumers for the same IRC events.
            this.interRoleCommunicator = new InterRoleCommunicationExtension(topicEndpoint);
        }
        catch (Exception ex)
        {
            // Report on the error through default logging/tracing infrastructure.
            Trace.TraceError(String.Join(ex.Message, Environment.NewLine, ex.StackTrace));

            // Request that the current role instance is to be stopped and restarted.
            RoleEnvironment.RequestRecycle();
        }

        // ...There is usually some more code here...
        return true;
    }

   // ...There is usually some more code here...
}

In summary, you will need to add a private field of type IInterRoleCommunicationExtension which will be initialized with an instance of the IRC component using the configuration information loaded from app.config.

In web roles, the IRC component initialization logic must be placed into the Global.asax.cs file, most specifically into its Application_Start method. The Application_Start method is called only once during the lifecycle of the IIS application pool that belongs to the web role instance. Another popular approach is to implement a lazy initialization whereby the IRC component instance will only be created when it’s first accessed as shown in the example below:

public class Global : System.Web.HttpApplication
{
    private static Lazy<IInterRoleCommunicationExtension> ircLazyLoader = new Lazy<IInterRoleCommunicationExtension>(() =>
    {
        var serviceBusSettings = ConfigurationManager.GetSection(ServiceBusConfigurationSettings.SectionName) as ServiceBusConfigurationSettings;
        var topicEndpoint = serviceBusSettings.Endpoints.Get(serviceBusSettings.DefaultEndpoint);

        return new InterRoleCommunicationExtension(topicEndpoint);
    },
    LazyThreadSafetyMode.ExecutionAndPublication);

    public static IInterRoleCommunicationExtension InterRoleCommunicator
    {
        get { return ircLazyLoader.Value; }
    }

    // ...There is usually some more code here...
}

Important: It is recommended that the IRC component is not to be instantiated by a role instance more than once per single topic. Under the hood, the component maintains a messaging client with an active subscription affinitized to a particular role instance ID. Multiple instances of the IRC component created by a role instance for the same topic will share the same subscription. This will create competing consumers for the subscription and may result in inter-role communication events being consumed by incorrect instances of the IRC component. Therefore, an instance of the InterRoleCommunicationExtension class must always be a singleton for a given topic. Make sure you implement a Singleton pattern when creating an instance of this class. Note that multiple instances of the IRC component in a single role instance are allowed, provided that you initialize the IRC component instances using different topic endpoints.

Step 5: Subscribe to Inter-Role Communication Events

In order to start receiving inter-role communication events, you will need to implement and activate a subscriber to these events. The subscriber is a class implementing the IObserver<InterRoleCommunicationEvent> interface with 3 methods: OnNext, OnCompleted and OnError. The following code is an example of the IRC event subscriber:

/// <summary>
/// Implements a subscriber for inter-role communication (IRC) events.
/// </summary>
public class InterRoleCommunicationEventSubscriber : IObserver<InterRoleCommunicationEvent>
{
    /// <summary>
    /// Receives a notification when a new inter-role communication event occurs.
    /// </summary>
    public void OnNext(InterRoleCommunicationEvent e)
    {
        // Put your custom logic here to handle an IRC event.
    }

    /// <summary>
    /// Gets notified that the IRC component has successfully finished notifying all observers.
    /// </summary>
    public void OnCompleted()
    {
        // Doesn't have to do anything upon completion.
    }

    /// <summary>
    /// Gets notified that the IRC component has experienced an error condition.
    /// </summary>
    public void OnError(Exception error)
    {
        // Report on the error through default logging/tracing infrastructure.
        Trace.TraceError(String.Join(error.Message, Environment.NewLine, error.StackTrace));
    }
}

Once a subscriber is implemented, the next step is to register it with the IRC component by letting the Subscribe method do all the heavy lifting behind the scene:

public class SampleWorkerRole : RoleEntryPoint
{
    private IObserver<InterRoleCommunicationEvent> ircEventSubscriber;
    private IDisposable ircSubscription;

    /// <summary>
    /// Called by Windows Azure after the role instance has been initialized. 
    /// This method serves as the main thread of execution for your role.
    /// </summary>
    public override void Run()
    {
        // ...There is usually some code here...

        // Create an instance of the component which will be receiving the IRC events.
        this.ircEventSubscriber = new InterRoleCommunicationEventSubscriber();

        // Register the subscriber for receiving inter-role communication events.
        this.ircSubscription = this.interRoleCommunicator.Subscribe(this.ircEventSubscriber);

        // ...There is usually some more code here...
    }
}

At this point, everything is ready to go for the very first inter-role communication event to be published. Let’s start with multicasting an IRC event to all active role instances.

Step 6: Publishing Multicast Inter-Role Communication Events

Publishing an IRC event requires creating an instance of the InterRoleCommunicationEvent class with the specified event payload, which can be either a simple .NET type or a custom DataContract-serializable object. The following fragment depicts a simplistic use case in which a string-based payload is packaged as an IRC event and published to all role instances participating in the IRC event exchange:

// Create an instance of the IRC event using a string-based payload.
InterRoleCommunicationEvent e = new InterRoleCommunicationEvent("This is a test multicast event");

// Publish the event. That was easy.
this.interRoleCommunicator.Publish(e);

In most cases, you will be publishing complex, strongly typed events carrying a lot more than a string. That is covered in step 8.

Step 7: Publishing Unicast Inter-Role Communication Events

To publish an IRC event that needs to be received by a specific role instance, you will need to know its role instance ID. If the target role instance resides in another hosted service deployment regardless of the datacenter location, you will also need to be in a possession of its deployment ID. To resolve the ID of a particular role instance that needs to receive an IRC event, you can query the Roles collection and walk through the list of instances for each role to find the recipient’s role instance ID.

The following code snippet is an example of how to send a unicast IRC event to the next instance succeeding the current instance in the role collection:

using System.Linq;

// Find a role instance that sits next to the role instance in which the code is currently running.
var targetInstance = RoleEnvironment.Roles.Where(r => { return r.Value == RoleEnvironment.CurrentRoleInstance.Role; }).
                        SelectMany(i => i.Value.Instances).
                        SkipWhile(instance => { return instance != RoleEnvironment.CurrentRoleInstance; }).
                        Skip(1).
                        FirstOrDefault();

if (targetInstance != null)
{
    // Create an instance of the IRC event using a string-based payload.
    InterRoleCommunicationEvent e = new InterRoleCommunicationEvent("This is a test multicast event", roleInstanceID: targetInstance.Id);

    // Publish the event. That was also easy.
    this.interRoleCommunicator.Publish(e);
}

If an IRC event needs to be received by a role instance that resides outside the current deployment – for instance, in another datacenter or in a different hosted service – you will need to resolve its deployment ID and specify it when creating the InterRoleCommunicationEvent object (see the class constructor’s signature for more details).

Important: The Windows Azure Managed Library defines a Role class that represents a role. The Instances property of a Role object returns a collection of RoleInstance objects, each representing an instance of the role. The collection returned by the Instances property always contains the current instance. Other role instances will be available via this collection only if you defined an internal endpoint for the role, as this internal endpoint is required for the role’s instances to be discoverable.

Step 8: Publishing Strongly Typed Inter-Role Communication Events

As noted in step 7, most real-world scenarios for inter-role communication will involve exchanging complex events carrying a lot more than just a single piece of data. For these reasons, we have enabled you to specify an instance of any custom type as a payload for the IRC events. You will need to decorate your custom type with a DataContract attribute with every member of the type to be marked as serializable using a DataMember attribute unless certain members should not be passed through the wire. Below is an example of a custom type that can be used as an IRC event payload:

/// <summary>
/// Implements an IRC event payload indicating that a new work item was put in a queue.
/// </summary>
[DataContract]
public class CloudQueueWorkDetectedTriggerEvent
{
    /// <summary>
    /// Returns the name of the storage account on which the queue is located.
    /// </summary>
    [DataMember]
    public string StorageAccount { get; set; }

    /// <summary>
    /// Returns a name of the queue where the payload was put.
    /// </summary>
    [DataMember]
    public string QueueName { get; set; }

    /// <summary>
    /// Returns a size of the queue's payload (e.g. the size of a message or the number of messages).
    /// </summary>
    [DataMember]
    public long PayloadSize { get; set; }
}

To publish a strongly typed event, create a new InterRoleCommunicationEvent passing the instance of the custom type and then invoke the Publish method as follows:

var ircEventBody = new CloudQueueWorkDetectedTriggerEvent() { QueueName = "MyQueue" };
var ircEvent = new InterRoleCommunicationEvent(ircEventBody);

this.interRoleCommunicator.Publish(ircEvent);

Strongly typed events require custom types to be shared between publishers and consumers. In other words, the respective .NET types need to be in a possession by the consumer so that it can successfully deserialize the event payload.

Important: The maximum size of a serialized IRC event must not exceed the maximum message size allowed by the Service Bus messaging infrastructure. As of writing, this message size limit is 256KB. To stay up-to-date with the latest limits and other important technical constraints that may apply, please visit the Windows Azure Service Bus FAQ on MSDN.

Step 9: Dispose the Inter-Role Communication Component

The implementation of the IRC component relies on messaging APIs provided by the Service Bus. Under the hood, the component instantiates and maintains communication objects which provide access to the underlying messaging infrastructure. These objects require being disposed in a graceful and controlled manner. You will need to ensure that the IRC component is explicitly told to shut itself down by invoking its Dispose method.

The following code sample disposes the IRC component upon termination of the hosting role instance:

public class SampleWorkerRole : RoleEntryPoint
{
    // ...There is usually some code here...

    /// <summary>
    /// Called by Windows Azure when the role instance is to be stopped.
    /// </summary>
    public override void OnStop()
    {
        if (this.interRoleCommunicator != null)
        {
            // Dispose the IRC component so that it gets a chance to gracefully clean up internal resources.
            this.interRoleCommunicator.Dispose();
            this.interRoleCommunicator = null;
        }
    }

    // ...There is usually some more code here...
}

The disposal of the IRC component is guaranteed not to throw any runtime exceptions; hence it’s safe to call into the Dispose method without any special precautions.

To conclude this section, it is worth mentioning that I’m going to drill down into additional examples in the upcoming days. These will include more advanced inter-role communication scenarios such as managing multiple subscribers and handling competing consumers.

Conclusion

This article showed how to build a versatile inter-role communication layer for Windows Azure solutions using new features of the Windows Azure AppFabric Service Bus. In a previous post, I showed how to use the direct messaging capabilities in Service Bus to send and receive messages between role instances in an asynchronous, loosely coupled fashion. In this post, I focused on the new durable topic-based messaging offered by the latest release of Service Bus. By using the publish/subscribe architecture, you can easily get Windows Azure role instances to exchange messages and events in either one-to-one, or one-to-many compositions. The new features leveraged by our implementation of the inter-role communication bring the following benefits:

• Reduction of complexity when connecting parts of cloud-based solutions that span multiple deployments, datacenters and geographies.
• Mitigation of constraints by reducing or completely eliminating some specific drawbacks that apply to the alternative means of inter-role communication such as internal endpoints.
• Ensured durability that guarantees that messages sent between role instances will never get lost in transit.
• Better scalability allows scaling to hundreds of role instances to meet the most demanding scale-out computing requirements.
• Complete abstraction by which all the mechanics of how the underlying messaging APIs work are made totally transparent to the client.

When would you use the Windows Azure Service Bus for inter-role communication? As a point of reference, the customer scenario discussed earlier is a good summary of potential use cases. In addition, consider if the following statements can apply to your solution:

• The solution employs multiple services working in parallel. The multicast nature of the inter-role communication architecture allows many services to work concurrently on one task.
• The solution uses loosely coupled services the location of which is dynamic or fully virtualized. The service location virtualization offered by Service Bus makes it easier to deliver such an architecture.

The accompanying sample code is available for download from the MSDN Code Gallery. Please note that the source code is governed by the Microsoft Public License (Ms-PL) as explained in the corresponding legal notices.

Additional Resources/References

For more information on the topic discussed in this blog post, please refer to the following:

• ”Windows Azure Service Bus CTP User Guide” whitepaper on the Service Bus Samples site.
• “Windows Azure Service Bus CTP FAQ” article in the MSDN library.
• ”An Introduction to Service Bus Topics” post on the Windows Azure AppFabric Team blog.
• “Exploring Topics and Queues by Building a Service Bus Explorer Tool – Part 1” post on the AppFabric CAT blog.
• “Implementing Reliable Inter-Role Communication Using Windows Azure Service Bus, Observer Pattern & Parallel LINQ” post on the AppFabric CAT blog.

Did this blog post help you? Please give us your feedback. Tell us on a scale of 1 (poor) to 5 (excellent), how would you rate this post and why have you given it this rating? For example:

• Are you rating it high due to having quality code samples, self-explanatory visuals, clear writing, or another reason?
• Are you rating it low due to poor examples, fuzzy screen shots, or unclear writing?

Your feedback will help us improve the quality of guidance we release. Thank you!
Authored by: Valery Mizonov
Contributed by: Abhishek Lal, Rama Ramani, Paolo Salvatori, David Ingham, Sidney Higa

(Visited 11 times, 11 visits today)

---

<Return to section navigation list>

Windows Azure VM Role, Virtual Network, Connect, RDP and CDN

No significant articles today.

---

<Return to section navigation list>

Live Windows Azure Apps, APIs, Tools and Test Harnesses

The Windows Azure Team (@WindowsAzure) posted Announcing the August 2011 Release of the Windows Azure Tools for Microsoft Visual Studio 2010 on 8/3/2011:

We are very pleased to announce the August 2011 release of the Windows Azure Tools for Microsoft Visual Studio 2010. You can download the tools here using the Web Platform Installer.

New in this release:

• Profile applications running in Windows Azure.
• Create ASP.Net MVC3 Web Roles.
• Manage multiple service configurations in one cloud project.
• Improved validation of Windows Azure packages.

With profiling support in the Windows Azure Tools you can easily detect performance bottlenecks in your application while it is running in Windows Azure.

The Tools now support creating MVC3 web roles. The new template includes the new universal ASP.Net providers that support SQL Azure and it will also make sure that MVC assemblies are deployed with your application when you publish to Windows Azure.

If you want to maintain different settings for different deployment environments, the Windows Azure tools now support multiple service configurations in the same Windows Azure Project. This is especially useful for managing different Windows Azure Storage connection strings for local debugging and running in the cloud.

Finally the new tools will help you avoid some of the common problems when you deploy your application to Windows Azure. If you are forgetting to include a local assembly in your package or you publish with a local Azure Storage connection string. The tools will let you know.

Read more about the new features here.

Related to this release:

• The Windows Azure Platform Training Kit has been updated for the new tools. The Windows Azure Platform Training Kit includes a comprehensive set of technical content including hands-on labs, presentations, and demos that are designed to help you learn how to use the Windows Azure platform. You can download it here.

---

Wade Wegner (@WadeWegner) reported the release of August 2011 Updates to the Windows Azure Platform Training Kit in an 8/3/2011 post:

Today we released the Windows Azure Platform Training Kit – August 2011. The Windows Azure Platform Training Kit (WAPTK) includes a comprehensive set of technical content including hands-on labs, presentations, and demos that are designed to help you learn how to use the Windows Azure platform.

In addition to the offline training kit above, we have pushed updates to the Windows Azure Platform Training Course on MSDN.

What’s new in this release?

Today announced the August 2011 release of the Windows Azure Tools for Microsoft Visual Studio 2010 (i.e. VS Tools for Windows Azure 1.4). New to these tools are the following capabilities:

• Profile applications running in Windows Azure
• Create ASP.NET MVC3 Web Roles
• Management Multiple service configurations in a single cloud project
• Improved validation of Windows Azure packages

For more information on the updates to the tools, see these excellent posts by Technical Evangelists Nick Harris and Nathan Totten. [See below]

We spent a significant amount of time reviewing and updating the training kit to make sure that the content works great with the 1.4 tools release. Additionally, we performed updates to hands-on labs and demos.

The August 2011 update of the WAPTK includes the following updates:

• Labs and Demos
  • All the labs and demos have been updated to leverage the new Window Azure Tools 1.4.
  • Updated SSMS dependency of affected labs to SSMS SP1.
• Windows Azure Deployment
  • Applied fix in the setup experience for the Windows Azure Platform Management Cmdlets.
  • Updated the Windows Azure Platform Management Cmdlets to the latest version.
• Building ASP.NET Applications with Windows Azure
  • Changed the way the membership password was encrypted to fix minor bug when deployed to azure.
• Exploring Windows Azure Storage
  • Updated the code to support deleting snapshots.
• Windows Azure CDN
  • Applied fix to a minor bug in the Overview section.

Feedback?

We are always looking to make improvements to the WAPTK. If you encounter any bugs or have any problems, please send us feedback at: azcfeed@microsoft.com. Additionally, if there’s additional content you’d like to see in the training kit, please let us know on the Windows Azure Platform Training Kit Forum – you’re feedback is highly valued.

---

Nick Harris explained Using the new Windows Azure Tools v1.4 for VS2010 in his first post as a newly minted Microsoft Developer Evangelist for Windows Azure:

The new Windows Azure Tools for v1.4 (August 2011) for VS2010 have just been released. You can download them using Web Platform Installer here. This latest version of the tools introduces several new features as follows:

1. Support for Multiple Service Configurations
2. Profiling support for Windows Azure apps running in Windows Azure
3. MVC 3 web role support
4. Package validation

Note: Profiling is only supported in VS 2010 Ultimate and Premium. All others are available from Visual Web Developer 2010 and up.

This post will take a brief look at the benefits each of these new features bring to you – the developer.

Multiple Service Configurations

Gone are the days of having to change your settings in your ServiceConfiguration.cscfg when you switch from debugging your local cloud+storage emulator to publishing up to Windows Azure. For example in your development environment for:

1. local debug you may want to:
   • utilize 1 instance of your web/worker role
   • use the local storage emulator
2. whereas in Windows Azure you may want to:
   • utilize 4 instances of your web/worker role
   • use a production Windows Azure storage account

To achieve this using the new multiple service configurations is easy.

1. Create a new Windows Azure project with an arbitrary web or worker role.
   • File > New Project > Cloud > Windows Azure Project
   • Select a web or worker role and press OK
2. Observe that the ServiceConfiguration.cscfg is now split by default into two files:
   • ServiceConfiguration.Local.cscfg – default used when debugging in VS
   • ServiceConfiguration.Cloud.cscfg – a config you can use on publish

   

3. To configure each individual configuration with the settings we desired above.
   • Double click on the WorkerRole1 in the roles folder of the cloud project

     

   • then and select the Service Configuration dropdown for Cloud

     

   • Set the desired settings for your Cloud profile ServiceConfiguration.Cloud.cscfg

     
     

4. You can then repeat the above for your Local configuration profile to setup and set the desired settings for ServiceConfiguration.Local.cscfg
5. The net result is that both ServiceConfiguration.Cloud.cscfg and ServiceConfiguration.Local.cscfg will now have their independent settings as follows:

   

   Independent config in ServiceConfiguration.Cloud.cscfg

   

   Independent config in ServiceConfiguration.Local.cscfg

6. When you hit debug now the ServiceConfiguration.Local.cscfg is used and when you hit publish you can select which Config you would like to use:
   

Overall its quite an easy experience to configure and use multiple Service Configuration profiles for your different environments. Please note that you can also rename and add additional Service Configuration profiles perhaps such that you would have a config for Local, Staging and Prod. For more detail on how to work with Service Configuration files please see Configuring a Windows Azure Application.

Profiling Support

This is an incredibly useful tool for any Windows Azure developer as it enables you to profile your Windows Azure application that’s running up in Windows Azure. The information gathered can help analyze any performance issues you may be facing. When you publish your application from VS you are able to specify your profiling options that will apply for the profiling session and results can be pulled for each instance.

The supported profiling options are as follows:

1. CPU Sampling – Monitor CPU-bound applications with low overhead
2. Instrumentation – Measure function call counts and timing
3. .NET Memory Allocation (Sampling) – Track managed memory allocation
4. Concurrency -Detect threads waiting for other threads

In this segment I will demonstrate how to configure a profiling session for a Windows Azure Application:

1. Open your existing cloud project and right click on the windows azure cloud project and select Publish

   

2. Select the Enable profiling option on the publish page and click settings

   

3. Select the type of profiling you wish to perform in this case .NET Memory Allocation

   

4. Note: Checking the Enable Tier Interaction Profiling option captures additional information about execution times of synchronous ADO.NET calls in functions of multi-tiered applications that communicate with one or more databases. With the absence of a SQL Profiler in SQL Azure this feature is useful for those developers who want to gain some insight into what queries or stored procedures are running slowly.
5. Press OK.
6. Once the deployment is complete and the application has been running for a period of time you can go and download the captured profiling report.
7. To download the profiling report
   • Select View > Server Explorer
   • Expand Windows Azure Compute
   • Expand the hosted service
   • Right click on the Instance that you want do download the profiling report from and press View Profiling Report

     

8. Once the report is downloaded it will open in VS and as you can see the CPU is maxing out

   

9. If we change the Current View dropdown to Allocation we can quickly identy a problem method that is using excessive amounts of memory

   

10. and finally if we right click and select View Source on the method of interest we can see the offending line causing the allocations

    

What an awesome tool. I look forward to digging deeper into the capabilities provided by for Windows Azure Profiling. In the meantime for more information please see – Profiling a Windows Azure Application

MVC 3 Web Role Support

ASP .NET MVC 3 web roles are now supported out of the box with the new tools. You can select ASP .NET MVC 3 from the new Windows Azure project dialog and the required assemblies used by ASP .NET MVC 3 are set to copy local for you. This results in these assemblies being deployed up to windows azure when you publish your application thus ensuring your MVC 3 application will start when deployed. I know a lot of you are probably thinking Eureka right now and those that may get a little bit too excited may even verbalize it, I know I did

To create a new Windows Azure ASP .NET MVC 3 application:

1. File > New Project > Cloud > Windows Azure Project
2. Select ASP .NET MVC 3 Web Role, press the > button followed by OK

   

3. In the ASP .NET MVC 3 project dialog select settings to suit your preferences and press ok
4. In solution explorer observe that all the references assemblies for ASP .NET MVC 3 that are not in the GAC in the current Windows Azure gues OS have had their Copy Local property set to true. The below image shows and example of one of the required reference assemblies that is automatically set to copy local so you dont actually have to do anything

   

From here on in all you have to do is start coding – For more information on ASP .NET MVC 3 see this and for a detailed walkthrough of ASP .NET MVC 3 on Windows Azure see this post by Nathan Totten

Package Validation

Last but definitely not least is improved package validation. When you select to create a package or publish your Windows Azure application. Additional warnings or errors are now provided in VS to enable you to fix the problem before you package or publish it. This as you know will be a great timesaver for details of what package validations are performed please see Troubleshooting Package Validation Errors and Warnings

Time to Download

All in all its an excellent new set of features that focus on improving your productivity and make your dev life a whole lot easier. If you have not already then now would be the time to download using Web Platform Installer.

---

Nathan Totten (@ntotten) described new support for MVC3 in his Windows Azure Tools 1.4 Released post of 8/3/2011:

Today, the Windows Azure Tools for Visual Studio 1.4 were released. These tools add a number of enhancements to the Windows Azure development and deployment experience that will make your life a bit easier. Below I am going to cover a few of the enhancements included with this release. For the full details you can see the release notes on MSDN here.

The biggest change for web developers is that the tools now include the latest MVC3 templates. You no longer have to create your cloud project and add an MVC3 application manually. You can see the updated template list with an MVC 3 Web Role selected.

After you select the MVC3 Web Role you will be prompted with the standard MVC3 project dialog asking for various template and configuration options.

Once the project is created you will have a standard MVC3 Web Application with the WebRole.cs class and the Windows Azure references automatically added.

Another time saving improvement is that the necessary MVC3 and Razor assemblies are added automatically to this template and are set to copy local. This has always been something you had to do manually in order to deploy an MVC3 web application to Windows Azure.

A final thing to note about this template (and all the web role templates) is that they now include the new ASP.NET Universal Providers. These new providers replace the old membership, role, session, and profile providers that have been included in ASP.NET since version 2.0. The new universal providers add native support for SQL Azure and SQL Compact. You can see the updated configuration file with the new providers below.

As you can see the new universal providers are enabled by default. The old providers are included, but these shouldn’t really be used for Windows Azure deployments.

IMPORTANT NOTE: If you attempt to deploy your unmodified solution to Windows Azure it will not work. You must either disable session state, change the DefaultConnectionString to point to a SQL Azure database, or change the session to use Windows Azure Caching. The reason the deployment will fail is because on the first load your site will check to make sure the session state database is configured. If it is not, it will attempt to create the database. By default the connection string is set to use .\SQLExpress which isn’t available on your Windows Azure Web Role.

The simplest fix is to just disable session state if you don’t plan on using it. To do so, just change the mode to “Off” as seen below.

It is also worth noting that there is another new feature in this release of the tools that actually warns you about this problem. When you deploy you would see this warning:

In my opinion this should actually register as an error rather than just a warning, because you can still ignore it, but when your deployment finishes you will just get the YSOD.

The final feature I will mention in this post is that you can now use multiple cloud configuration files. This makes it easy to have a separate configuration file for local development, staging, and production. These multiple configurations are similar to how Web.Config transforms allow you to have different configurations for different environments. The biggest difference is that with the cloud configuration files they aren’t transforms, but simply entire configuration files.

When you deploy your application you select which configuration file you want to use in the deployment dialog. You could also script this if you were automating the deployments.

I think you will find that this updated tools will improve your Windows Azure development experience. For the full set of features and changes read the post on MSDN and download the tools now to try them out.

---

Wilfried Schadenboeck reported Boing is using Windows Azure & Kinect to create a Virtual tour for Boing 737 in an 8/3/2011 post to the TechNet blog:

The commercial aircraft giant is using the software giant’s technology to create a virtual tour of the next-generation Boeing 737 plane, using Kinect, Silverlight Deep Zoom, and Windows 7 Touch and Azure.

Digital marketing agency Wire Stone created Boeing 737 Explained, an interactive marketing tool to help Boeing pitch the aircraft to potential buyers. While Wire Stone is based in the Silicon Valley, the Boeing Kinect work took place at the agency’s Seattle office.

In what is being billed as an early commercial non-entertainment use of Kinect, Wire Stone says it integrated Kinect and other Microsoft technology for Boeing to use in trade shows and other venues that can support massive displays where Boeing 737 Explained can be viewed in real-world dimensions.

From selling jetliners to training surgeons, it is already apparent that Kinect has applications beyond games.

The 737 project uses the technology behind Kinect motion controller for Xbox 360 to let a viewer move around and explore the 737.

With Kinect, Boeing is able to turn a dry, technical pitch into a virtual tour of the aircraft.

“If we look at all the approaches that we use to communicate about the 737, most of them are very analytical, enabling us to talk about the financial operating costs, maintenance costs and other attributes,” Diana Klug, Director of Marketing for Boeing Commercial Airplanes, said in a statement. “We wanted to take the marketing for the 737 to the next level, and the set of tools that we had did not allow us to convey the full range of new features and improvements that we’ve made to the product.”

Using Kinect and other Microsoft technology, stone created Boeing 737 Explained, an interactive marketing to help Boeing to pitch the aircraft to potential buyers.

Here is Microsoft's case study on the project.

The Deep Zoom element is old news; the Kinect slant is new.

---

<Return to section navigation list>

Visual Studio LightSwitch and Entity Framework 4.1+

ComponentOne released a live Web-based demo of their OLAP for LightSwitch Control on 8/3/2011:

From the OLAP for LightSwitch page:

Snap a pivoting data screen into Microsoft Visual Studio LightSwitch and instantly get in-depth business intelligence (BI) functionality with ComponentOne OLAP™ for LightSwitch. Create interactive tables, charts, and reports similar to those found in Microsoft Excel Pivot tables and charts. Drag-and-drop views give you real-time information, insights, and results in seconds. All this without cubes or code!

Highlights

OLAP for LightSwitch includes these tools for easy BI application development:

• OLAP Screen: An interface that includes a control panel and tabs to display a grid, chart, and raw data.
• OLAP Panel: A drag-and-drop interface for defining views, very similar to the Microsoft Pivot table interface.
• Pivoting Grid: Displays a pivoting table that groups data, applies conditional formatting, and allows clipboard actions and printing.
• Pivoting Chart: Displays a pivoting chart and controls to select chart type, theme, palette.

You can download a demo version of OLAP for LightSwitch there.

Thanks to LightSwitch MVP Michael Washington (@ADefWebserver) for the heads-up.

---

Return to section navigation list>

Windows Azure Infrastructure and DevOps

Lori MacVittie (@lmacvittie) said she is Making the case for a stateless infrastructure model in an introduction to her The Cloud Configuration Management Conundrum post of 8/3/2011 to F5’s DevCentral blog:

Cloud computing appears to have hit a plateau with respect to infrastructure services. We simply aren’t seeing even a slow and steady offering by providers of the infrastructure services needed to deploy mature enterprise-class applications. An easy answer as to why this is the case can be found in the fact that many infrastructure services while themselves commoditized are not standardized. That is, while the services are common to just about every data center infrastructure the configuration, policies and APIs are not. But this is somewhat analogous to applications, in that the OS and platforms are common but the applications deployed on them are not. And we’ve solved that problem, so why not the problem of deploying unique infrastructure policies and configuration too?

It’s that gosh-darned multi-tenant thing again. It’s not that infrastructure can’t or doesn’t support a multi-tenant model; plenty of infrastructure providers are moving to enable or already support a multi-tenant deployment model. The problem is deeper than that, at the heart of the infrastructure. It’s configuration management, or more precisely, the scale of configuration management necessary in a cloud computing environment.

CONFIGURATION MANAGEMENT

It’s one thing to enable the isolation of of network device configurations – whether in hardware or software form factor – as a means to offer infrastructure services in a multi-tenant environment. But each of the services requires a customer-specific configuration and, in many cases, per-application. Add to that the massive number of “objects” that must be managed on the infrastructure for every configuration and it takes little math skills to realize that the more customers and instances, the more configuration “objects”. At some point, it becomes too unwieldy and expensive a way to manage infrastructure services for the provider. A shared resource requires a lot of configuration regardless of the multi-tenant model. If you use virtualization to create a multi-tenant environment on a device using, then the support is either broad (one instance per customer) or deep (multiple configurations per instance). In either case, you end up with configurations that are almost certainly going to increase parallel to the increase in customers. Thousands of customers = many thousands of configuration objects. Many infrastructure devices have either imposed or have best practice limitations on the number of configuration “objects” it can support.

As a simple, oft-referenced, example consider the VLAN limitation. There is a “theoretical” maximum of 4096 VLANS per trunk. It is theoretical because of vendor-imposed limitations that are generally lower due to performance and other configuration-related constraints. The 4096, like most upper limiting numbers in networking, is based on primitive numerical types in programming languages comprised of a certain number of bits and/or bytes that impose a mathematical upper bound. These bounds are not easily modified without changes to the core software/firmware. These limitations are also, when related to transmission of data, constrained by specifications such as those maintained by the IEEE that define the data format of information exchanged via the network. Obviously changing any one of the pieces of data would change the size of the data (because they are strictly defined in terms of ranges of bit/byte locations in which specific data will be found) and thus require every vendor to change their solutions to be able to process the new format. This is at the heart of the problem with IPv4 –> IPv6 migration: the size of IP packet headers changes as does the location of data critical to the processing of packets. This is also true of configuration stores, particularly on networking devices where size and performance are critical. The smallest data type possible for the platform is almost always used to ensure performance is maintained. Thus, configuration and processing is highly dependent on tightly constrained specifications and highly considerate of speed, making changes to these protocols and configuration systems fraught with risk.

This is before we even begin considering the run-time impact in terms of performance and potential disruption due to inherent volatility in the cloud “back plane”, a.k.a. server/instance infrastructure. Consider the way in which iptables is used is some cloud computing environments. The iptable configuration is dynamically updated based on customer configuration. As the configuration grows and it takes longer for the system to match traffic against the “rules” and apply them, performance degrades. It’s also an inefficient means of managing a multi-tenant environment, as the process of inserting and removing rules dynamically is also a strain on the entire system because software-only solutions rarely differentiate between management and data plane activity.

So basically what we have is a configuration management problem that may be stifling movement forward on more infrastructure services being offered as, well, services. The load from management of the massive configurations necessary would almost certainly overwhelm the infrastructure.

So what’s the solution?

OPTIONS

The most obvious and easiest from a provider standpoint is architectural multi-tenancy. Virtual network appliances. It decreases the configuration management burden and places the onus on the customer. This option is, however, more complex because of the topological constraints imposed by most infrastructure, a.k.a. dependency on a stable IP address space, and thus becomes a management burden on the customer, making it less appealing from their perspective to adopt. A viable solution, to be sure, but one that imposes management burdens that may negatively impact operational and financial benefits.

A second solution is potentially found in OpenFlow. Using its programmable foundation to direct flows based on some piece of data other than IP address could alleviate the management burden imposed by topological constraints and allow architectural multi-tenancy to become the norm in cloud computing environments. This solution may, however, require some new network or application layer protocol – or extension of an existing protocol – to allow OpenFlow to identify, well, the flow and direct it appropriately. This model does not necessarily address the issue of configuration management. If one is merely using OpenFlow to direct traffic to a controller of some sort, one assumes the controller would need per-flow or per-application or per-customer configuration. Thus such a model is likely run into the same issues with scale of configuration management unless such extensions or new protocols are self-describing. Such self-describing meta-data included in the protocol would allow interpretation on the OpenFlow-enabled component rather than lookup of configuration data.

Basically, any extension of a protocol or new protocol used to dynamically direct flows to the appropriate services would need to be as stateless as possible, i.e. no static configuration required on the infrastructure components beyond service configuration. A model based on this concept does not necessarily require OpenFlow, although it certainly appears that a framework such as offered by the nascent specification is well-suited to enabling and supporting such an architecture with the least amount of disruption. However, as with other suggested specification-based solutions, no such commoditized or standardized meta-data model exists.

The “control plane” of a data center architecture, whether in cloud computing or enterprise-class environments, will need a more scalable configuration management approach if it is to become service-oriented whilst simultaneously supporting the dynamism inherent in emerging data center models. We know that stateless architectures are the pièce de résistance of a truly scalable application architecture. RESTful application models, while not entirely stateless yet, are modeled on that premise and do their best to replicate a non-dependence on state as a means to achieve highly scalable applications. The same model and premise should also be applied to infrastructure to achieve the scale of configuration management necessary to support Infrastructure as a Service, in the pure sense of the word. OpenFlow appears to have the potential to support a stateless infrastructure, if a common model can be devised upon which an OpenFlow-based solution could act independently.

A stateless infrastructure, freed of its dependency on traditional configuration models, would ultimately scale as required by a massively multi-tenant environment and could lay the foundation for true portability of applications architectures across cloud and data center environments.

---

<Return to section navigation list>

Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds

Adam Hall reported the Operations Manager 2012 Beta Community Evaluation Program Schedule in an 8/3/2011 post to the System Center Team Blog:

Hi everyone,

The Operations Manager 2012 Beta Community Evaluation Program (CEP) kicks off next week!

Throughout the CEP, you will be be taken on a guided tour of the Operations Manager 2012 Beta, with presentations from the Product Group and guidance on how to evaluate the different features. This is your chance to engage directly with the team!

If you have not signed up yet, it is not too late, just head on over to the CEP signup page, enter your details and we will send you all the details you need to join the calls, and give you access to the Connect site.

The initial topics and calls are listed below:

Note that all the calls are at 8am PST, and all will be recorded for those that cannot make the call.

We look forward to engaging with you on the CEP!

I’ve signed up for all the System Center CEPs on Microsoft Connect.

---

<Return to section navigation list>

Cloud Security and Governance

No significant articles today.

---

<Return to section navigation list>

Cloud Computing Events

Bruno Terkaly posted links to his Source Code for Azure Bootcamp in Bellevue on 8/3/2011:

Thank you to everyone who attended my Azure boot camp in Bellevue. I have spent some time packaging up the software for those of you who are interested and digging deeper. A couple of the links below also point to learning resources and/or videos pertaining to the download. I enjoyed meeting you all and I hope you make great progress adopting Azure as your next development effort.

---

Bill Wilder (@codingoutloud) posted July Boston Azure User Group - Recap on 7/31/2011 (missed when published):

The July Boston Azure User Group meeting had a tough act to follow: the June meeting included a live, energy-packed Rock, Paper, Azure hacking contest hosted by Jim O’Neil! The winners were chosen completely objectively since the Rock, Paper, Azure server managed the who competition. First prize was taken by two teenagers (Kevin Wilder and T.J. Wilder) whose entry beat out around 10 others (including a number of professional programmers!).

This month’s July Boston Azure User Group meeting was up for the challenge.

• In the opening act, Bill Wilder (that’s me) introduced the Windows Azure Toolkit for Social Games (see also the Cloud Cover episode 52 which also talks about the toolkit). You can find Bill on Twitter @codingoutloud.
• Mark Eisenberg of Microsoft then shared some great insights about the cloud and the Windows Azure Platform – what they really are, why they matter, and how they fit into the real world. You can find Mark on Twitter @azurebizandtech.
• Marshall Rosenstein then talked about building social games for Facebook with an Azure back-end. Marshall’s slides are on his skydrive here (the slides are here, the code download is here). You can find Marshall on Facebook (http://www.facebook.com/marshall.rosensteina) and on Twitter @mrosenstein.
• We wrapped up the meeting with a short live demonstration of the Windows Azure Portal doing its thing. Then a few of us retired to the Muddy.

Hope to see you at the Boston Azure meeting in August (Windows Phone 7 + Azure), two meetings in September (one in Waltham (first time EVER), and the “usual” one at NERD), and then kicking off a two-day Boston Azure Bootcamp!

Details on ALL upcoming Boston-area events of interest to Azure folks (that I know about) can be found in this blog post about Boston-events in August and September. Those hosted by Boston Azure are also at www.bostonazure.org and the upcoming events page.

---

<Return to section navigation list>

Other Cloud Computing Platforms and Services

Martin Tantow reported CumuLogic Offers Private PaaS Cloud Software on 7/2/2011:

CumuLogic, a Java Platform-as-a-Service (PaaS) software provider, today announced the public beta version of their new PaaS solution which enables enterprises, cloud providers and ISVs to build and manage Java PaaS in public, private and hybrid cloud environments. The product is based on a cloud application management platform, and includes cloud services automation, autoscaling, monitoring, policy-base workload deployment resource management and user management. The idea is to mix and match different cloud-based applications, no matter where they reside.

“Today’s clouds are complex and all different. There is almost no interoperability between cloud providers and between public and private clouds. I’m enthusiastic about CumuLogic’s PaaS cloud management solution as it utilizes the higher levels of abstraction inherent in the PaaS model to reduce the complexity of cloud management, provides targeted facilities for both developers and management, and erases the distinctions between the various clouds enabling transparent interoperability,” said James Gosling, Technical Advisory Board member at CumuLogic.

The service will include a cloud services catalog of various infrastructure elements that are available as part of the management framework. There are also a series of developer and administration API enables developers to push applications to the cloud. Initially, it will support Amazon Web Services, Cloud.com (acquired by Citrix), Eucalyptus Systems, OpenStack (next month), and on-premises VMware vSphere installations. The product allows cloud-based repositories from multiple vendors to be supported concurrently. The product also includes features such as IaaS abstraction, policy-based workload deployment, and the ability to mix-and-match infrastructure software which enables the deployment of modern applications, as well as consolidation of legacy Java applications to a single platform, substantially lowering the cost of managing various infrastructure assets. The beta version can be downloaded for free at their site.

“By enabling customers to leverage their existing investment in Java development, CumuLogic is in a unique position to help enterprises transition to the cloud with minimal risk and retooling,” said Peder Ulander, VP of product marketing, Cloud Platforms Group, Citrix Systems. “Their commitment to flexibility and choice when it comes to infrastructure choices makes them a strong contender in the enterprise PaaS market, and the combination of the Citrix Open Cloud portfolio with CumuLogic’s PaaS solution will ensure customers have access to a next-generation cloud infrastructure platform.”

Key features include:

• Flexibility to deploy modern applications on the clouds of choice;
• Automation of a myriad of tasks that allow developers to focus on rapidly deploying quality application which translates into business agility;
• Developer and administration API enables developers to push applications to the cloud and to expect all the platform service to be available to run those applications; and,
• Cloud Services Catalog allows organizations to acquire new infrastructure software and updates delivered and registered to their repositories in private clouds eliminating the need to build their own service images which can be a time consuming task.

“CumuLogic is unique in that we are one of the first companies to emerge with a full Java PaaS for the federated cloud,” said Rajesh Ramchandani, VP of Products at CumuLogic. “Instead of rewriting applications to fit new platforms and essentially giving up standardized application components, we sought to create a product that would give users the flexibility to keep using those components, from application platforms to databases.”

 

No significant articles today.

---

<Return to section navigation list>

Technorati Tags: Windows Azure,Windows Azure Platform,Azure Services Platform,Azure Storage Services,Azure Table Services,Azure Blob Services,Azure Drive Services,Azure Queue Services,SQL Azure Database,SADB,Open Data Protocol,OData,Windows Azure AppFabric,Azure AppFabric,Windows Server AppFabric,Server AppFabric,Cloud Computing,Visual Studio LightSwitch,LightSwitch,SharePoint,WebParts,Silverlight,Boeing,Kinect,Cumulogic,Windows Azure Platform Training Kit,SQL Azure Federations,AppFabric Workflows