Wednesday, November 17, 2010

Azure Storage Services Test Harness: Table Services 1 – Introduction and Overview

Update 11/17/2010: The source code is now available for download as SampleWebCloudService.zip from my Windows Live Skydrive account. See Updated Source Code for OakLeaf Systems’ Azure Table Services Sample Project Available to Download of 11/17/2010 for more details.

Update 11/16/2010: The project’s two-year-old source code has been updated to the Windows Azure SDK v1.2 (May 2010) from the CTP in effect in November 2008. Windows Azure Development and Production Fabrics no longer support code that uses the early Sample.StorageClient library. The new code uses the current Microsoft.WindowsAzure.* libraries.

Here’s a capture of the current CRUD (create, retrieve, update, delete) page at http://oakleaf.cloudapp.net (the create/insert feature is disabled in the public version):

image

Notice the “Powered by Windows Azure” logo at the top right of the capture. This application went through the “Microsoft Platform Ready” (MPR) test process, as indicated by the capture below:

image

Source code will be available shortly from my Windows Live Skydrive account. I’ll update this post when the code is available.

When time permits, I’ll update this and the six other parts of the series to reflect the changes required to accommodate the new Microsoft.WindowsAzure.* libraries.

This is the first of a series of articles about a ASP.NET 3.5 C# test harness for Azure Storage Services that is will be available for download from my “Retire Your Data Center” cover story for Visual Studio Magazine’s February 2009 issue (click the Get Code Download link) in the near future.

Here are links to the later episodes in this series:

Azure Storage Services are a component of the Azure Services Platform, which runs under the Windows Azure operating system. According to Microsoft’s Windows Azure page:

Windows® Azure is a cloud services operating system that serves as the development, service hosting and service management environment for the Azure Services Platform. Windows Azure provides developers with on-demand compute and storage to host, scale, and manage Web applications on the Internet through Microsoft® data centers.

Table Services are one of the three Azure Storage Services, each of which has a Representational State Transfer (REST) API. The following diagram shows how Storage Services fit into the Azure Services Platform (click the images to open a full-size screen capture):

Microsoft describes Azure Table Services as follows:

Windows Azure Tables provide scalable, available and durable structured storage in the form of tables. The tables contain entities, and the entities contain properties. The tables are scalable to billions of entities and terabytes of data, and may be partitioned across thousands of servers. The tables support ACID transactions over single entities and rich queries over the entire table. Simple and familiar .NET and REST programming interfaces are provided via ADO .NET Data Services [client library].

Azure Tables provide authenticated access and are replicated three times for data security. The Windows Azure Tables whitepaper and the following video sessions from the Microsoft Professional Developer Conference (PDC) 2008 provide additional technical details about table services:

Note: Azure Table Services have some features in common with SQL Data Services (SDS) but don’t offer SDS’s current and promised non-relational database management services, such as LINQ to REST Skip(), OrderBy() and Join() methods, and promised features such as aggregates, schemas, fan-out queries, and full-text search. SDS are expected to take on additional relational features in future versions, as described in the PDC 2008 session by Patrick McElroy: SQL Services: Futures. Although Microsoft hasn’t revealed pricing for Azure and SDS, you can expect to pay a premium for use of the SDS layer.

Prerequisites for the Test Harness Walkthrough

This walkthrough doesn’t depend on having access to the cloud-based Azure Services Platform because the Windows Azure SDK (October 2008 SDK) includes the Windows Azure Development Fabric and Azure Development Storage. You’ll need 32-bit or 64-bit version of the Windows Azure SDK and the Windows Azure Tools for Microsoft Visual Studio 1.0, as well as Visual Web Developer 2008 Express or Visual Studio 2008 Standard or higher.

For additional information on setting up for Windows Azure development see:

from the MSDN Library.

Setting Up an Azure Services Platform Storage Account and Hosted Services Project

If you’ve received your Microsoft Connect Invitation for the Azure Services Platform message and an Invitation Code to Windows Azure message from azinvite@microsoft.com with a GUID token to create an Azure Developer Portal Account for a project at http://lx.azure.microsoft.com, you’re ready to join the Windows Azure CTP.

The following illustrated topics in the Setting Up the Windows Azure Services Platform: An Illustrated Walkthrough post of 11/13/2008 take the mystery out of the CTP sign-up process:

  • Accepting the Microsoft Connect Invitation for the Azure Services Platform
  • Creating a Storage Account in the Azure Developer Portal
  • Creating a Hosted Services Project in the Azure Developer Portal

The SampleWebCloudService.sln Test Harness

The SampleWebCloudService.sln project is a single-page ASP.NET C# test harness for Azure Table Services that’s based on Jim Nakashima’s Windows Azure Walkthrough: Simple Table Storage post of 10/28/2008.

The following screen capture shows the page displaying entities with their RowKey property value set to the source table’s primary key (click image to display full-size capture):

Note: Calibri is the default font family and provides the best visual experience at 1024x768 resolution. If you don’t have Office 2007 installed and would like to use Calibri, which supports ClearType, download the PowerPoint Viewer 2007.

The Paging hyperlinks populate the DataView by capturing and sending a continuation token (ct) with NextPartitionKey/NextRowKey header values in the GET query string, as shown for on-premises deployment in the Address toolbar.

The buttons and check box at the bottom of the harness’s page let you:

  1. Count the number of Customer entities (nominally 91) by over a LINQ query entity set
  2. Delete all Customer entities by iteration or, with a code change, by deleting the table
  3. Recreate Customer entities from object initializers
  4. Select the primary key or a GUID as the RowKey
  5. Update the CompanyName value by adding “(Update)” (button not shown)

All operations are timed so you can evaluate the effect of deployment type on performance.

Here’s a screen capture of a modified page that assigns a default PartitionKey and automatically generates random GUID RowKey values:

The test harness lets you test Table Services’ performance in three deployment modes:

  1. On-premises with Visual Studio 2008 from the Development Fabric with Development Storage.
  2. On-premises but using Azure Table Storage services.
  3. In the cloud from the Azure Fabric with Azure Table Storage services.

Development Storage uses an SQL Server 2005 or 2008 Express instance to emulate Azure Storage and offers this simple UI to specify the name of the database that provides persistent storage of schemas and entities:

Deploying the test harness to my Windows Azure test harness and performing initial tests took less than 15 minutes.

The buttons at the bottom of the harness’s page let you:

  1. Count the number of Customer entities (nominally 91) by over a LINQ query entity set
  2. Delete all Customer entities by iteration or, with a code change, by deleting the table
  3. Recreate Customer entities from object initializers
  4. Select the primary key or a GUID as the RowKey
  5. Update the CompanyName value by adding “(Update)” (button not shown)

All operations are timed so you can evaluate the effect of deployment type on performance.

Following are some approximate times in seconds for iterative CRUD operations on 91 Customer entities averaged over 5 tries:

Deployment Mode Count (Query) Delete Insert Update
Developer Storage/Developer Fabric 1.277 3.393 3.191 4.003
Azure Storage/Developer Fabric 2.729 10.801 14.456 15.793
Azure Storage/Azure Fabric 1.423 5.085 5.204 5.775

The performance hit incurred by migrating the test harness from local execution and storage to Azure Hosted Services is surprisingly small, as indicated by the preceding timing data. I attribute the comparatively poor performance of the Azure Storage/Developer Fabric to a substantially greater amount of data on the wire (especially upload).

Note: My AT&T (PacBell) commercial DSL connection to the Windows Azure instance tests at 2,574 kbps download and 436 kbps upload.

Azure Tables, Entities, and Attributes (Properties)

The Azure Services Platform provides persistent storage for data in non-relational, semi-structed tables or unstructured blobs. Azure Tables, which are the subject of this series of articles, are similar to SQL [Server] Data Services (S[S]DS) in that they implement the Entity-Attribute-Value (EAV) model.

Azure tables have the following three required attributes, which Microsoft calls system properties and up to 252 additional attributes, which Microsoft calls flexible properties or a property bag. The three required attributes are:

System Property Description and Usage
PartitionKey A string up to 32 KB in size that defines the set of entities that are load balanced across storage nodes. (Similar to a SDS Container.)
RowKey A string up to 32 KB in size which, together with the PartitionKey, forms a composite primary key attribute (field) of the table and orders the sequence of entities
Timestamp A DateTime value that specifies the last date and time that an entity was modified, which the service adds and updates automatically to enable optimistic concurrency conflict management

Note: SDS system properties are called metadata properties: Id (primary key), Version (an integer value that increments for each modification), and Kind (user-defined entity type name)

Flexible properties must be one of the following data types:

Data Type Description
Binary Array of bytes up to 64 KB in size
Bool Boolean value
DateTime 64-bit value expressed as UTC time ranging from 1/1/1600 to 12/31/9999
Double

64-bit floating point value
GUID

128-bit globally unique identifier
Int

32-bit integer
Long 64-bit integer
String UTF-16-encoded value up to 64 KB in size

The maximum size of an entity is 2 MB.

Note: The current version of SDS supports a Decimal data type and GUIDs as strings, but not Double, INT, or Long. It’s likely that future versions of SDS and Table Services will converge to a single set of data types. For more details about SDS tables and data types, see SDS Data Model Overview (Authorities, Containers, Entities and Flexible Entities).

This series continues with Azure Storage Services Test Harness: Table Services 2 – the Table Services API of 11/14/2008.

Updated 11/17/2008 and 11/18/2008: Added paging features.

Updated 11/24/2008: Added links to later episodes, as well as “Prerequisites for the Test Harness Walkthrough” and “Setting Up an Azure Services Platform Storage Account and Hosted Services Project” sections.

Posted by Roger Jennings (--rj) at 1:03 PM

Labels: , , , , ,

0 comments:

Post a Comment

Subscribe to: Post Comments (Atom)