Monday, December 20, 2010

Windows Azure and Cloud Computing Posts for 12/20/2010+

A compendium of Windows Azure, Windows Azure Platform Appliance, SQL Azure Database, AppFabric and other cloud-computing articles.

AzureArchitecture2H640px3   
Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:

To use the above links, first click the post’s title to display the single article you want to navigate.


Cloud Computing with the Windows Azure Platform published 9/21/2009. Order today from Amazon or Barnes & Noble (in stock.)

Read the detailed TOC here (PDF) and download the sample code here.

Discuss the book on its WROX P2P Forum.

See a short-form TOC, get links to live Azure sample projects, and read a detailed TOC of electronic-only chapters 12 and 13 here.

Wrox’s Web site manager posted on 9/29/2009 a lengthy excerpt from Chapter 4, “Scaling Azure Table and Blob Storage” here.

You can now freely download by FTP and save the following two online-only PDF chapters of Cloud Computing with the Windows Azure Platform, which have been updated for SQL Azure’s January 4, 2010 commercial release:

  • Chapter 12: “Managing SQL Azure Accounts and Databases”
  • Chapter 13: “Exploiting SQL Azure Database's Relational Features”

HTTP downloads of the two chapters are available for download at no charge from the book's Code Download page.


Tip: If you encounter articles from MSDN or TechNet blogs that are missing screen shots or other images, click the empty frame to generate an HTTP 404 (Not Found) error, and then click the back button to load the image.

Azure Blob, Drive, Table and Queue Services

My (@rogerjenn) Speed ServiceContext.SaveChanges() Execution with the SaveChangesOptions.Batch Argument post (updated 12/20/2010) explained how to increase CRUD performance of bulk operations on Azure tables with batch updates:

imageMy (@rogerjenn) Adding Trace, Event, Counter and Error Logging to the OakLeaf Systems Azure Table Services Sample Project of 11/22/2010 described a set of load tests of a  Windows Azure Table Services demo app. The following table from the above article described the execution time in seconds for Count Customers, Delete All Customers, Create Customers, and Update Customers operations with three deployment configurations and the default ServiceContext.SaveChanges(SavechangesOptions.None) argument:

Table 1: Default execution times for 91 Customers records (seconds)

image

  • Dev/Dev = Development Fabric and Development Storage
  • Dev/Prod = Development Fabric and Production Storage
  • Prod/Prod = Production Fabric and Production Storage (fully deployed, single instance)

DELETE, INSERT and UPDATE load (POST) operations against the underlying Windows Azure Customer table were taking much longer to complete than expected when deployed to the Production Fabric, which has a high-speed (local) connection to Windows Azure storage.

Executing the ServiceContext.SaveChanges() method with the SavechangesOptions.Batch option resulted in substantially improved performance, as shown here:

Table 2: Batch execution times for 91 Customers records (seconds)

image

Performance improvement ranges from about 4:1 for deleting 91 customer entities to almost 30:1 for bulk UPDATE operations on these entities.

The OakLeaf Systems Azure Table Services Sample Project at http://oakleaf.cloudapp.net has been updated with a Batch Updates check box so users can compare batch versus conventional update operations:

image

Source code for the SampleWebCloudService.sln project will be uploaded to my Windows Live Skydrive account shortly. Stay tuned for an update.


<Return to section navigation list> 

SQL Azure Database and Reporting

imageNo significant articles today.

 


<Return to section navigation list> 

MarketPlace DataMarket and OData

imageSee Muhammed Arrabi announced on 12/20/2010 an Azure Development & OData Intro (Wed Dec 22nd, 6:30 @PSUT) technical talk in Jordan in the Cloud Computing Events section below.


<Return to section navigation list> 

Windows Azure AppFabric: Access Control and Service Bus

Alik Levin posted a 00:00:56 Video: Windows Azure AppFabric Access Control Service (ACS) v2 Key Components and Architecture tutorial on 12/20/2010:

In this short video I am covering ACS architecture and its key components. Related Videos:

Slides are here

Video: ACS Architecture

Related Books
Related Info

Attachment:


Alik Levin recommended a 00:00:55 Video: What Windows Azure AppFabric Access Control Service (ACS) v2 Can Do For Me? on 12/19/2010:

This is another short video covering what ACS can do for you. Consider watching What’s Windows Azure AppFabric Access Control Service (ACS) v2? first.

Slide deck is here.

Video: What ACS Can Do For Me?

Related Books
Related Info

Attachment: What ACS Can Do For Me.pptx


Gunnar Peipman (@gpeipman) explained Identifying AppFabric Access Control Service users uniquely in a 12/19/2010 post:

image In my last posting about AppFabric Labs Access Control Service I described how to get your ASP.NET MVC application to work with ACS. In this posting I will dig deeper into tokens and claims and provide you with some helper methods that you may find useful when authenticating users using AppFabric ACS. Also I will explain you little dirty secret of Windows Live ID.

What is inside token?

Token and claimsLet’s start with dissecting tokens. Token is like envelope that contains claims. Each claim carries some property of identity. By default we get the following claims from all identity providers:

  • name identifier – unique identifier for user that can be used to identify user (user can change his or her name in identity provider service but name identifier does not change),
  • identity provider – unique string that helps us detect identity provider that user used to authenticate to our web page.

As you can see from the image on right then token may contain more claims like claim for name and e-mail address. There can be also other claims if identity provider is able to provide them.

How claims are specified?

Claims are specified in web.config file when we add federation metadata definition to application using STS reference wizard. In my web.config there are following claims defined.

  • http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name
  • http://schemas.microsoft.com/ws/2008/06/identity/claims/role
  • http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier
  • http://schemas.microsoft.com/accesscontrolservice/2010/07/claims/identityprovider

You can find claims from microsoft.identitymodel block in web.config file.

Why Live ID gives us empty name?

image722322This is the question that I had when I added ACS support to my ASP.NET MVC web application and here is the answer. Live ID returns for user only the name identifier and it does not introduce it as user name to Windows Identity Foundation. As there is no name claim then user name is left empty and you find weird situation where user is authenticated but there is no username.

When user is authenticated over Live ID we get back these claims:

  • http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier
  • http://schemas.microsoft.com/accesscontrolservice/2010/07/claims/identityprovider

The values are something like this:

  • nameidentifier - QwxVt0jBnEUFw7B2HgfsaWeSQXIzAXC9yUKWR780UepE=
  • identityprovider - uri:WindowsLiveID

As you can see there is no claim for user name and that’s why user name is empty.

What Google returns?

Here is what Google returns when I authenticate myself:

  • nameidentifier - https://www.google.com/accounts/o8/id?id=AItOawlMC8j0cW8013mWBkEwXs6MsV5ixZUmxks
  • emailaddress – mygoogleusername@gmail.com
  • name - Gunnar Peipman
  • identityprovider - Google

Google asks user if he or she allows ACS to access my name and e-mail address. If I agree then my name and address are given to my web application and I have username filled correctly when asking for current user.

What data we should save for users?

This is the good question to ask because some services allow users to change their name. And names are not unique. Consider the name John Smith. There are hundreds of guys who have this name. There maybe also hundreds of Mary Smiths and as some of them get married they change their last name (if they don’t marry some John Smith, of course).

To be maximally fool-proof we have to save the values of identityprovider and nameidentifier claims to our users database. This way we can avoid the situation where two different identity providers give us same nameidentifier for different users.

Here is my claims extension class you can use to get the values of these fields easily.

image

And here is how we can use these extension methods.

image

What about names?

As some providers doesn’t support returning names we have two options:

  • avoid these providers (like I currently do),
  • ask new users for (unique) username or first and last name when they are joining.

Right now I am trying to work out solution for second option so I have easy way how to handle separate user names support easily so I don’t have to write tons of code for this simple thing.

Conclusion

Claims based authentication is powerful way to identify users and AppFabric ACS lets our sites to support different identity providers so we don’t have to write code for each one of them. As identity providers support different features and does not return us always all the data we would like to have we have to be ready to handle the situation (like Windows Live ID). As we saw it was still easy to identify users uniquely because data for this purpose is always given to us with token.


<Return to section navigation list> 

Windows Azure Virtual Network, Connect, and CDN

Alessandro Perilli reported Microsoft unlocks the IaaS component of Windows Azure on 12/16/2010 (missed when posted):

image After one year wait, almost two months ago Microsoft finally announced the upcoming availability of a new component of its public cloud computing platform Windows Azure.

So far Windows Azure only presented the characteristics of a Platform-as-a-Service (PaaS) cloud computing platform, but at the end of October the company said that it would extend Azure with a new VM Role, introducing some capabilities that are typical of an Infrastructure-as-a-Service (IaaS) cloud.

imageMicrosoft promised to deliver within the end of the year, and in fact the VM Role is available for all customers starting this week.

As confirmed during the announcement, there are some pretty significant limitations at the moment:

  • customers can only create virtual machines with Windows Server 2008 R2 as guest operating system
  • the VMs must be created, installed and configured inside an on-premises Hyper-V virtual infrastructure
  • there’s no way to control VMs hosted on Azure from an on-premises Hyper-V management console

On top of that, the whole process to have a VM up and running inside Azure seems overwhelmingly complex. A long tutorial published on MSDN explains step-by-step how to do so, and how to update the VM after it’s living on Azure. 
The complexity is especially striking when it’s compared with the relatively easy way to import VMware VMs inside EC2 that Amazon introduced just today. Once Amazon will introduce the plug-in for VMware vSphere, as just announced, the difference will be even more evident.

Along with the VM Role, Microsoft also unlocked the support for Remote Desktop, so Azure customers can use RDP to remotely connect to their VMs and administer the guest OS.

Windows_Azure_RDP.png

Also, Microsoft released the Windows Azure Connect component, which allows to bridge the corporate internal network with the virtual network assigned to Azure VMs, through an IPsec tunnel. 
The IPsec network bridge uses software clients, called endpoints, that customers have to install on each local machine (either physical or virtual) that they want to connect to the Azure VMs.
Afte this step, customers have to configure so called endpoint groups, to define which local machines can exchange packets with which Azure VMs.

Windows_Azure_Connect.jpg

According to previous Microsoft communications, this version of Windows Azure Connect should be an early preview, with the final version expected in the first half of 2011.

During 2011 Microsoft also plans to release additional features for the IaaS component of Azure, like the support for Windows Server 2003 and 2008 SP2 guest operating systems.


<Return to section navigation list> 

Live Windows Azure Apps, APIs, Tools and Test Harnesses

Wade Wegner (@wadewegner) reported on Twitter that the BidNow Azure Sample Application on CodePlex was updated on 12/20/2010:

BidNow Sample

image BidNow is an online auction site designed to demonstrate how you can build highly scalable consumer applications running in the Windows Azure Platform.

imageWhen BidNow was originally released, it was a sample built using Windows Azure and Windows Azure Storage, along with authentication provided via Live Id. Since the original release, a number of additional services and capabilities have been released. Consequently, BidNow now utilizes the following pieces of the platform:

  • Windows Azure
  • Windows Azure storage (e.g. blobs and queues)
  • SQL Azure
  • Windows Azure AppFabric Caching
  • Windows Azure AppFabric Access Control
  • OData
  • Windows Phone 7
These updates not only provide important and useful capabilities, but also highlight the ways in which to build applications in the Windows Azure Platform.
Get Started
Here are a few guides you can leverage to get started using BidNow.

For additional details, please see the post Significant Updates Released in the BidNow Sample for Windows Azure.

BidNowSample.png

Notice
The example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious. No association with any real company, organization, product, domain name, email address, logo, person, places, or events is intended or should be inferred.

Last edited Dec 3 at 2:18 PM by wwegner, version 37


BusinessWire published a Customer Solutions Group to Use Microsoft’s Cloud for New Multi-Channel Real-Time Shopping Platform press release on 9/20/2010:

Location-Agnostic Consumer Platform Uses Windows Azure To Give Cyber-Shoppers Instant Price Incentives

image Cloud-based consumer applications are becoming more and more familiar and now Customer Solutions Group, LLC (CSG), a Colorado-based start-up focused on consumer-facing retail solutions, has announced it will be launching a cloud-based shopping platform powered by Microsoft’s Windows Azure.

“We believe comparison shopping should be engaging, fun and economically rewarding. Our platform makes the entire Web your store, provides real-time re-pricing capabilities for retailers, and makes social shopping real-time. We make it visually rewarding and accessible at home, on your mobile phone, or in the store.”

The platform will allow retailers to offer features that organize shopping so consumers can find the best product at the best price easily and efficiently and retailers can motivate and influence buying decisions.

image“Retailers are continually seeking ways to reach and grow loyalty amongst consumers who are immersed in their digital lives, social networking and rich online interactions,” said David Gruehn, director, U.S. retail industry solutions, Microsoft Corp. “Cloud-based applications help bridge the gap between online and brick-and-mortar stores, providing a differentiated and connected shopping experience. For example, CSG’s platform powered by Windows Azure will provide real-time comparative pricing, giving retailers the ability to match or beat a competitor’s price instantly.”

This technology will provide an “always with you” resource to compare products, prices and receive retailer incentives, for the first time, all in real-time and based on shopping behavior. By utilizing the real-time “power of the crowd” across the Web and the power of the Windows Azure platform, this will provide customers with complete price surveys on products and allow retailers to promote and/or adjust their own prices in real-time to remain competitive.

“Comparison shopping can be overwhelming and existing on-line shopping tools require far too much effort and shopping ‘side trips’ to sites across the Web,” said Patrick Carter, founder and president, CSG. “We believe comparison shopping should be engaging, fun and economically rewarding. Our platform makes the entire Web your store, provides real-time re-pricing capabilities for retailers, and makes social shopping real-time. We make it visually rewarding and accessible at home, on your mobile phone, or in the store.

“That kind of dynamic experience and access can only really work using a cloud-based platform like Windows Azure.”

About Customer Solutions Group

Customer Solutions Group, LLC is a Denver-based customer experience innovation firm devoted to delivering the best customer experience through creative business solutions that cross multiple channels: on-site, on-line, and mobile. Established in 2002, CSG employs a unique combination of patent-pending functionally innovative technology, development expertise, and analytics to address each of its client’s unique customer experience enhancement goals. Whether you seek to improve customer acquisition, penetration, retention, or cost-to-serve, CSG has the expertise and acumen necessary to be your long-term partner for success.


Morebits continued his/her series with Walkthrough Windows Azure Service Part2: Creating the Service Project of 12/19/2010:

imageIn this post, you will create a Windows Azure (cloud) service project using Microsoft Visual Studio 2010.

To create the Windows Azure service project
  1. From the start menu, select All Programs and right-click Microsoft Visual Studio 2010.
  2. Choose Run as Administrator.If the User Account Control dialog appears, click Continue.
  3. From the File menu, click New Project.
  4. In the New Project dialog expand the language of your choice (in this walkthrough, we are using C#) and then select Cloud.
  5. In the templates list, select Windows Azure Cloud Service.
  6. In the Name box, enter GuestBook.
  7. In the Location box, enter the location where to store the project.
  8. Click OK to create the project.

    image

    Figure 2 Creating Windows Azure Service Project

  9. In the New Cloud Service Project dialog windows, in the .NET Framework Roles panel select ASP.NET Web Role.
  10. Click the right arrow to add an instance of the selected role to the solution.
  11. The role shows in the Cloud Service Solution right panel.
  12. Click the pencil icon.
  13. Rename the role GuestBook_WebRole.
  14. Click OK to create the cloud service solution.

    image

    Figure 3 Assigning Roles to the Service Project

  15. In Solution Explorer review the project structure.

    image

    Figure 4 GuestBook Service Project Structure

As shown in the previous illustration, the solution contains two separate projects.

  • GuestBook project. It contains the following files and folder:
    • ServiceDefinition.csdef. This is the application definition file that specifies the application requirements for the Windows Azure fabric such as which roles are used, their trust level, the end points exposed by each role, the local storage requirements and the certificates used by the roles. It also defines specific application settings. The following code shows the content of the definition file.
    <?xml version="1.0" encoding="utf-8"?>
    <ServiceDefinition name="GuestBook" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
      <WebRole name="GuestBook_WebRole">
        <InputEndpoints>
          <InputEndpoint name="HttpIn" protocol="http" port="80" />
        </InputEndpoints>
        <ConfigurationSettings>
          <Setting name="DiagnosticsConnectionString" />
          <Setting name="DataConnectionString" />
        </ConfigurationSettings>
      </WebRole>
      <WorkerRole name="GuestBook_WorkerRole">
        <ConfigurationSettings>
          <Setting name="DiagnosticsConnectionString" />
          <Setting name="DataConnectionString" />
        </ConfigurationSettings>
      </WorkerRole>
    </ServiceDefinition>
    • ServicesConfiguration.cscfg. This is the configuration file that specifies the number of instances for each role and assigns the values for the configuration settings as defined in the ServiceDefinition.csdef file. The following code shows the content of the configuration file.
    <?xml version="1.0"?>
    <ServiceConfiguration serviceName="GuestBook" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration">
      <Role name="GuestBook_WebRole">
        <Instances count="1" />
        <ConfigurationSettings>
          <Setting name="DiagnosticsConnectionString" value="UseDevelopmentStorage=true" />
          <Setting name="DataConnectionString" value="UseDevelopmentStorage=true" />
        </ConfigurationSettings>
      </Role>
      <Role name="GuestBook_WorkerRole">
        <Instances count="1" />
        <ConfigurationSettings>
          <Setting name="DiagnosticsConnectionString" value="UseDevelopmentStorage=true" />
          <Setting name="DataConnectionString" value="UseDevelopmentStorage=true" />
        </ConfigurationSettings>
      </Role>
    </ServiceConfiguration>

    The separation between service definition and configuration enables you to update the settings of a deployed application by just uploading a new service configuration file.

    • Roles node. It enables you to configure what roles the cloud service includes that are web, worker or both. It also enables you to specify which project to associate with these roles.

    Adding and configuring roles in the Roles node will update the definition and configuration files.

  • GuestBook_WebRole project. This is a standard ASP.NET web application modified for the Windows Azure environment. It contains an additional class that provides the entry point for the web role and contains methods for initializing, starting and stopping the role.

For related topics, see the following posts.

  • Walkthrough Windows Azure Service Part1:  Introduction
  • Walkthrough Windows Azure Service Part3: Creating the Table Storage Data Model   
  • Walkthrough Windows Azure Service Part4: Creating the Web Role to Handle UI
  • Walkthrough Windows Azure Service Part5: Creating the Worker Role to Handle Background Tasks
  • Walkthrough Windows Azure Service Part6: Configuring the Service
  • Walkthrough Windows Azure Service Part7: Testing the Service


Morebits began a new introductory blog series with Walkthrough Windows Azure Service Part1: Introduction on 12/19/2010:

image This is the first of several posts of a walkthrough whose goal is to create a guest book application while exploring Windows Azure service basic elements. The walkthrough demonstrates the use of compute and storage services, specifically:

  • Compute service: web and worker roles.
  • Storage service: blob, table and queues.

imageThe information presented here is an adaptation of a related article published on the Channel 9 site. For more information, see Introduction to Windows Azure.

In this walkthrough you will perform the following tasks:

  • Create a Windows Azure application using web and worker roles.
  • Use Windows Azure storage services which include blobs, tables and queues.
  • Deploy the Windows Azure application.

A Windows Azure compute service consists of one or more web roles and worker roles.

  • A web role is an ASP.NET Web application accessible via an HTTP or HTTPS endpoint and is commonly the front-end for an application.
  • A worker role is a background-processing application and is typically found in the back-end.

A Windows Azure compute service may be comprised of one or both types of roles and can run multiple instances of each type. Windows Azure can add or remove role instances on demand allowing an application to quickly and economically scale-up or down when the need arises. For more information, see Windows Azure Compute Service.

Windows Azure storage services provide storage in the cloud, which includes blob services for storing text and binary data, table services for structured storage that can be queried, and queue services for reliable and persistent messaging between services. For more information, see Windows Azure Storage Services.

The following illustration shows the GuestBook application architecture.

GuestBookArchitecture

Figure 1 GuestBook Application Architecture

  • A web role provides the front-end that enables the user to view the contents of the guest book and submit new entries. Each entry contains a name, a message, and an associated picture.
  • A worker role that generates thumbnails for the pictures submitted by the user.
  • When the user posts a new item, the web role uploads the picture to the blob storage and creates a new entry in the table storage. The entry contains the information entered by the user and a link to the blob with the picture. The web role renders this information through the browser so users can view the contents of the guest book.
  • After storing the image and creating the entry, the web role posts a work item to a queue to have the image processed in the back-end by the worker role.
  • The worker role fetches the work item from the queue, retrieves the image from the blob storage, and resizes it to create a thumbnail.

What we have described is a common pattern in cloud computing which enables the separation of compute-bound tasks from the front-end ones by using queues. The advantage of this approach is that front and back end can be scaled independently.

Prerequisites
  • The Windows Azure Tools for Microsoft Visual Studio are available on the Windows Azure Tools for Microsoft Visual Studio download page. The Windows Azure SDK is included with the installation, so there is no need to install the SDK separately.
  • You can also install the Windows Azure Tools using the Microsoft Web Platform Installer. The Web Platform Installer simplifies the process of installing and configuring IIS, the Windows Azure SDK, and the Windows Azure Tools.
  • The Windows Azure Tools can be used with these Microsoft products:
    • Visual Studio 2010
    • Visual Web Developer 2010 Express Edition
    • Visual Studio 2008 with Service Pack 1
    • Visual Web Developer 2008 Express Edition with Service Pack 1

For more information, see Windows Azure Tools for Microsoft Visual Studio.

For related topics, see the following posts.

  • Walkthrough Windows Azure Service Part2:  Creating the Service Project.
  • Walkthrough Windows Azure Service Part3: Creating the Table Storage Data Model   
  • Walkthrough Windows Azure Service Part4: Creating the Web Role to Handle UI
  • Walkthrough Windows Azure Service Part5: Creating the Worker Role to Handle Background Tasks
  • Walkthrough Windows Azure Service Part6: Configuring the Service
  • Walkthrough Windows Azure Service Part7: Testing the Service


Eric Nelson (@ericnel) described FREE goodies if you are a UK based software house already live on the Windows Azure Platform, the UK version of the US Microsoft Platform Ready program, on 12/20/2010:

image In the UK we have seen some fantastic take up around the Windows Azure Platform and we have lined up some great stuff in 2011 to help companies fully exploit the Cloud – but we need you to tell us what you are up to!

imageOnce you tell us about your plans around Windows Azure, you will get access to FREE benefits including email based developer support and free monthly allowance of Windows Azure, SQL Azure and AppFabric from Jan 2011 – and more! (This offer is referred to as Cloud Essentials and is explained here)

And… we will be able to plan the right amount of activity to continue to help early adopters through 2011.

Step 1: Sign up your company to Microsoft Platform Ready (you will need a windows live id to do this)

image

Step 2: Add your applications

image

For each application, state your intention around Windows Azure (and SQL etc if you so wish)

image

Step 3: Verify your application works on the Windows Azure Platform

image

Step 4 (Optional): Test your application works on the Windows Azure Platform

Download the FREE test tool. Test your application with it and upload the successful results.

image

Step 5: Revisit the MPR site in early January to get details of Cloud Essentials and other benefits

P.S. You might want some background on the “fantastic take up” bit:

  • We helped over 3000 UK companies deploy test applications during the beta phase of Windows Azure
  • We directly trained over 1000 UK developers during 2010
  • We already have over 100 UK applications profiled on the Microsoft Platform Ready site
  • And in a recent survey of UK ISVs you all look pretty excited around Cloud – 42% already offer their solution on the Cloud or plan to.

image

In the US, testing your application with the Platform-Ready test tool authorizes you to add the Powered by Widows Azure logo to your Azure project, as shown at the top right of the OakLeaf Systems’ Azure Table Services Sample Project - Paging and Batch Updates Demo page:

image    


Ryan Dunn (@dunnry) explained Using WebDeploy with Windows Azure in a 12/19/2010 post:

Update:  Looks like Wade also blogged about this, showing a similar way with some early scripts I wrote.  However, there are some important differences around versioning of WebDeploy and creating users that you should pay attention to here.  Also, I am using plugins, which are much easier to consume.

imageOne of the features announced at PDC was the coming ability to use the standard IIS7 WebDeploy capability with Windows Azure.  This is exciting news, but it comes with an important caveat.  This feature is strictly for development purposes only.  That is, you should not expect anything you deploy using WebDeploy to persist longterm on your running Windows Azure instances.  If you are familiar with Windows Azure, you know the reason is that any changes to the OS post-startup are not durable.

imageSo, the use case for WebDeploy in Windows Azure is in the case of a single instance of a role during development.  The idea is that you would:

  1. Deploy your hosted service (using the .cspkg upload and deploy) with a single instance with WebDeploy enabled.
  2. Use WebDeploy on subsequent deploys for instant feedback
  3. Deploy the final version again using .cspkg (without WebDeploy enabled) so the results are durable with at least 2 instances.

The Visual Studio team will shortly be supplying some tooling to make this scenario simple.  However, in the interim, it is relatively straightforward to implement this yourself and do what the tooling will eventually do for you.

If you look at the Windows Azure Training Kit, you will find the Advanced Web and Worker Lab Exercise 3 and it will show you the main things to get this done.  You simply need to extrapolate from this to get the whole thing working.

We will be using Startup Tasks to perform a little bit of bootstrapping when the role instance (not the singular) starts in order to use Web Deploy.  This will be implemented using the Role Plugin feature to make this a snap to consume.  Right now, the plugins are undocumented, but if you peruse your SDK bin folder, it won't be too hard to figure out how they work.

The nice thing about using a plugin is that you don't need to litter your service definition with anything in order to use the feature.  You simply need to include a single "Import" element and you are done!

Install the Plugin

In order to use this feature, simply extract the contents of the zip file into "%programfiles%\Windows Azure SDK\v1.3\bin\plugins\WebDeploy".  You might have to extract the files locally in your profile first and copy them into this location if you run UAC.  At the end of the day, you need a folder called "WebDeploy" in this location with all the files in this zip in it.

Use the Plugin

To use the plugin you simply need to add one line to your Service Definition:

 <Imports> 
      <Import moduleName="RemoteAccess" /> 
      <Import moduleName="RemoteForwarder" /> 
      <Import moduleName="WebDeploy"/> 
    </Imports>

Notice, in my example, I am also including the Remote Access and Remote Forwarder plugins as well.  You must have RemoteAccess enabled to use this plugin as we will rely on the user created here.  The Remote Forwarder is required on one role in your solution.

Next, you should hit publish on the Windows Azure project (not Web Site) and setup an RDP user.  We will be using this same user later in order to deploy because by default the RDP user is also an Admin with permission to use Web Deploy.  Alternatively, you could create an admin user with a Startup Task, but this method is easier (and you get RDP).  If you have not setup the certificates for RDP access before, it is a one time process outlined here.

image

image

Now, you publish this to Windows Azure using the normal Windows Azure publishing process:

image

That's it.  Your running instance has WebDeploy and the IIS Management Service running now.  All you had to do was import the WebDeploy plugin and make sure you also used RDP (which most developers will enable during development anyway).

At this point, it is a pretty simple matter to publish using WebDeploy.  Just right click the Web Site (not the Cloud Project) and hit publish:

image

You will need to type in the name of your .cloudapp.net project in the Service URL.  By default, it will assume port 8172; if you have chosen a different public VIP port (by editing the plugin -see *hint below), here is where you need to update it (using the full https:// syntax).

Next, you need to update the Site/Application.  The format for this is ROLENAME_IN_0_SITENAME, so you need to look in your Service Definition to find this:

image

Notice, in my example, my ROLENAME is "WebUX" and the Site name is "Web".  This will differ for each Web Role potentially, so make sure you check this carefully.

Finally, check the "Allow untrusted certificate" option and use the same RDP username and password you created on deploy.  That's all.  It should be possible for you to use Web Deploy now with your running instance:  Hit Publish and Done!

How it works

If you check the plugin, you will see that we use 2 scripts and WebPI to do this.  Actually, this is the command line version of WebPI, so it will run without prompts.  You can also download it, but it is packaged already in the plugin.

The first script, called EnableWebAdmin.cmd simply enables the IIS Management service and gets it running.  By default, this sets up the service to run and listen on port 8172.  If you check the .csplugin file, you will notice we have opened that port on the Load Balancer as well.

start /w ocsetup IIS-ManagementService
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WebManagement\Server /v EnableRemoteManagement /t REG_DWORD /d 1 /f
net start wmsvc
sc config WMSVC start= auto
exit /b 0

*Hint: if you work at a place like Microsoft that prevents ports other than 443 to host SSL traffic, here would be a good place to map 443 to 8172 in order to deploy.  For most normal environments, this is not necessary.

Next, we have the proper WebDeploy installation command using WebPI command line.

@echo off
ECHO "Starting WebDeploy Installation" >> log.txt
"%~dp0webpicmd\WebPICmdLine.exe" /Products: WDeploy /xml:https://www.microsoft.com/web/webpi/2.0/RTM/WebProductList.xml /log:webdeploy.txt

ECHO "Completed WebDeploy Installation" >> log.txt

net stop wmsvc

net start wmsvc

You will notice one oddity here - namely, we are using the WebPI 2.0 feed to install an older version of WebDeploy (v1.1).  If you leave this off, it will default to the 3.0 version of WebPI that uses the Web Deploy 2.0 RC.  In my testing, Web Deploy 2.0 RC only works sporadically and usually not at all.  The 1.1 version is well integrated in VS tooling, so I would suggest this one instead until 2.0 works better.

Also, you will notice that I am stopping and restarting the IIS Management Service here as well.  In my testing, WebDeploy was was unreliable on the Windows Server 2008 R2 family (osFamily=2).  Starting and stopping the service seems to fix it.

Final Warning

Please bear in mind that this method is only for proto-typing and rapid development.  Since the changes are not persisted, they will disappear the next time you are healed by the fabric controller.  Eventually, the Visual Studio team will release their own plugin that does essentially the same thing and you can stop using this.  In the meantime, have fun!


Cumulux posted Windows Azure Management (with ManageAxis) on 12/19/2010:

image The advent of cloud computing has forced IT departments to look at hardware and software from an investment model to a On-demand model. The flexibility and advantage of paying for only what is used makes it all the more attractive. It cuts down a lot of redundant maintenance costs. It also lets the enterprises focus on its core businesses.

In an investment model, the customer would purchase the software and hardware and there is a one time fixed cost associated with the purchase. Then, the variable costs such as maintenance, service and update follows. In the on-demand model, the customer would subscribe to services, say like Windows Azure and based on the subscription, the customer can use the application/platform/infrastructure services.Later based on the usage, the customer would be billed.

The bill for windows azure cloud would reflect usage of

  • Number of hours for which a VM of reserved— Even if application isn’t currently running, you would be charged for the deployed application
  • Number of CPUs in a VM
  • Bandwidth, measured per GB in / out
  • Amount of GB storage used
  • Number of transactions on storage
  • Database size on SQL Azure
  • Number of connections on Windows Azure platform AppFabric

With new features announced in PDC 2010, and more services are available for use, depending upon customer subscription, the above list would get altered.

So, it is very essential to do meticulous planning for deploying in the cloud to optimize resource utilization and costs. There are readily available tools like Cumulux Cloud TCO tool to evaluate the total cost of ownership. But before that, the customer must evaluate and arrive at their demands and requirements with respect to number of VMs, Storage Usage, Database requirements, High availability scenarios and estimation of number of transactions, bandwidth/network requirements.

Virtual Machines :

There are two roles, web role and the worker role. Each role is a VM provisioned to cater to the appropriate requests. While the web role serves to process and forward requests, worker role is responsible for the execution of business logic. It is not mandatory to design it that way and based on the type of application, business logic can also be processed under web role.

To ensure scalability, it is suggested to provision at least 2 instances of web role to scale accordingly based on the demands. So, it becomes essential to closely monitor CPU utilization, Memory Usage, Amount of HTTP requests to increase or decrease the number of instances serving the requests.

ManageAxis from Cumulux provides an easy to use Cloud management and monitoring tool using Windows Azure Diagnostics API, to give illustrative information about the various performance parameters as mentioned above.

image

The customers can define and deploy rules to automatically scale based on the performance observed. Look out this blog for more details of ManageAxis and the its features for better Azure cloud management.

To [learn] about the pricing details of various options in Windows Azure, [click] here.

<Return to section navigation list> 

Visual Studio LightSwitch

Kunal Chowdhury posted Pune DevCon Event: Download Visual Studio LightSwitch Slides on 12/19/2010:

It was a nice time to present myself to the audience to deliver my presentation on Visual Studio LightSwitch for Silverlight application development in Pune Developer Conference (DevCon) event organized by Pune User Group. Today my session was from 2:45pm. Though the count of attendees was low than the expected count but they really enjoyed to know about LightSwitch.

Thanks to all who attended my session. Those who were unable to attend it for any reason, can view the PPT online. You can also download it for your future reference.

image Today, 19th December 2010, I delivered 1 hour 30 minutes session on Visual Studio LightSwitch. Lots of demos, which was really appreciated by the attendees. Please feel free to go thru the slides embedded in this post here:

You can directly download it from here. I uploaded both the PPTX and PPT versions. In case, you don’t have the latest Power point, can download the old PPT file. Click any of the below icons to save it in your Local drive:

image2224222I also have a bunch of Tutorials on LightSwitch hosted here: Beginners Guide to Visual Studio LightSwitch. More to come. Enjoy reading those chapters and learn more about it. Please feel free to revert back to me if you have any queries.


The LightSwitch Developer Center added a 00:06:30 #10 – How Do I: Create and Control Lookup Lists in a LightSwitch Application? video tutorial:

image

image2224222In this video you will learn how Lookup Lists work and how LightSwitch automatically creates a Modal Window Picker for referenced data when creating screens with the Screen Designer. You also learn how to write a query using the Query Designer to sort and filter the data and use that query when selecting from the list.


The LightSwitch Developer Center added a 00:12:32 #9 – How Do I: Create a Screen that can Both Edit and Add Records in a LightSwitch Application? video tutorial:

image

image2224222In this video you will see how you can use the Screen Designer to modify a default Edit Details screen to also allow adding new records. You will also see how to write code to enable and disable buttons depending on the edit or add mode of the screen.


Return to section navigation list> 

Windows Azure Infrastructure

James Urquhart listed The top 12 gifts of cloud from 2010 in a 12/20/2010 post to C|Net News’ The Wisdom of Clouds blog:

What a year 2010 has been for cloud computing.

image We've seen an amazing year of innovation, disruption, and adoption--one I personally think will go down in history as one of the most significant years in computing history. Without a doubt, a significant new tool has been added to the IT toolbox, and its one that will eventually replace most of the tools we know today.

image Don't agree with me? Well, with the help of my generous Twitter community--and in the spirit of the season here in the US--I've assembled 12 innovations and announcements from 2010 that had big impact on the IT market. Take a look at these with an open mind and ponder just how much cloud computing changed the landscape through the course of the year:

1. The growth of cloud and cloud capacity
The number of cloud computing data centers skyrocketed in 2010, with massive investment by both existing and new cloud providers creating a huge burst in available cloud capacity. You might have noticed new services from Verizon, IBM, Terremark, and others. Tom Lounibos, CEO of "test in the cloud" success story SOASTA, notes that the number of data centers offered by Amazon, Microsoft, and IBM alone grew from four to 10, with that number slated to grow to more than 20 in the coming year.

2. The acceptance of the cloud model
A number of studies showed a dramatic switch in the acceptance of the cloud model by enterprise IT between 2009 and 2010. Two that I often quote are one by the Yankee Group that noted that 60 percent of "IT influencers" now consider cloud computing services an enabler, versus 40 percent who considered it "immature"--a complete reversal from 2009. The second, published by Savvis, claimed that 70 percent of IT influencers were using or planning to use cloud services within two years. Since then, most studies show cloud computing becoming an increasingly high priority for CIOs everywhere.

3. Private cloud debated...to a truce
Heated debates between representatives of public cloud computing companies and more traditional IT computing vendors raged over the summer of 2010, with the former arguing that there is no such thing as a "private cloud," and the latter arguing that there definitely is, and it should be an option in every IT arsenal. The argument died down, however, when both sides realized nobody was listening, and various elements of the IT community were pursuing one or the other--or both--options whether or not it was "right."

4. APIs in--and out--of focus
As we started 2010, there were something like 18 APIs being proposed to define cloud operations interfaces. As the year wore on, it appeared that the AWS EC2 and S3 APIs were the de facto standards for compute and storage, respectively. However, VMware's vCloud, Red Hat's Delta Cloud, and Rackspace's Cloud Servers and Cloud Files APIs each still have legitimate chances to survive in the wild, as it were, and without Amazon's express support for implementations of their APIs by others, standardization around their offerings remains a risk.

5. Cloud legal issues come to the forefront
Toward the end of the year, Amazon's decision to unilaterally shut down WikiLeaks' presence on their site demonstrated one of the true risks of the public cloud today: if a customer builds their business on a public cloud environment, and the provider terminates that relationship without warning or recourse (with or without cause), what are those customer's rights? The scenario was repeated in a different form with the shutdown of low-cost content-distribution provider SimpleCDN by a reseller of a third-party hosting environment. The issue here isn't whether the terminated party was doing right or wrong but what rights the law establishes for all parties in a public cloud operations scenario.

6. Cloud economics defined
Two seminal works of cloud economic analysis had significant impact on our understanding of the forces behind cloud computing adoption. Joe Weinman's "Mathematical Proof of the Inevitability of Cloud Computing," published in November 2009, treated us to a carefully thought out "proof" of why pay-per-use pricing will eventually capture such a large chunk of our application workloads. James Hamilton's "Cloud Computing Economies of Scale" analyzes of the economics of the data center, and why a large scale provider with a wide customer base has tremendous advantages over a smaller provider with a smaller customer base.

7. The rise of DevOps
Those of us studying cloud computing for some time have noted that cloud computing models are both the result of and driver for changes in the way we design, deploy, and operate applications in a virtualized model. We've seen a shift away from server-centric models to application-centric alternatives, and a rapid change from manual processes to automated operations. This, in turn, has driven several software methods that combine development and operations planning and execution. The result is automation packaged *with* the application at deployment time, rather than developed after-the-fact in a reactive fashion.

8. Open source both challenged and engaged
At the beginning of the year, open-source developers were watching developments in the cloud computing space with a wary eye and rightfully so. For infrastructure projects (one of the most successful classes of open-source software), the model threatens to change the community nature of open source. However, the year also showed us that open source plays a critical role in both cloud provider infrastructure and the software systems we build and run in the cloud. All in all, OSS seems to have found peace with the cloud, though much has yet to be worked out.

9. Introducing OpenStack
One of the great success stories in open source for the cloud this year was that of the partnership between cloud provider Rackspace and NASA, producing the popular OpenStack project for compute and storage services. Drawing more than 300 attendees to its last design summit, OpenStack is quickly attracting individual developers, cloud start-ups, and established IT companies to its contributor list. While the long-standing poster child of open-source cloud infrastructure, Eucalyptus, seemed threatened initially, I think there is some truth to the argument that Eucalyptus is tuned for the enterprise, while OpenStack is being built more with public cloud providers in mind.

10. Amazon Web Services marches on
Amazon Web Services continues to push an entirely new model for IT service creation, delivery, and operation. Key examples of what Amazon Web Services has introduced this year include: an Asia-Pacific data center; free tiers for SQS, EC2, S3 and others; cluster compute instances (for high-performance computing); and the recently announced VMDK import feature. Not to mention the continuous stream of improvements to existing features, and additional support for well-known application environments.

11. Platform as a Service steps up its game
VMware announced its Cloud Application Platform. Salesforce.com introduced Database.com and its acquisition of Ruby platform Heroku. Google saw demand for developers with App Engine experience skyrocket. Platform as a Service is here, folks, and while understanding and adoption of these services by enterprise IT still lags that of the Web 2.0 community, these services are a natural evolutionary step for software development in the cloud.

12. Traditional IT vendors take their best shots
2010 saw IBM introduce new services for development and testing, genome research and . HP entered the cloud services game with its CloudStart offering. Oracle jumped into the "complete stack" game with Exalogic (despite Ellison's derision of cloud in 2009). My employer, Cisco Systems, announced a cloud infrastructure service stack for service providers and enterprises. Even Dell acquired technologies in an attempt to expand its cloud marketshare. Of course, none of these offerings pushed the boundaries for cloud consumers, but for those building cloud infrastructures, there are now many options to choose from--including options that don't directly involve any of these vendors.

Of course, these are just a few of the highlights from cloud's 2010. Not mentioned here were the plethora of start-ups in the space, covering everything from application management to cloud infrastructure to data management to...well, you get the idea. What does 2011 hold for cloud? I don't know, but I would hazard a guess that it will be at least as interesting as cloud's historic 2010.

James is a market strategist for cloud computing at Cisco Systems and an adviser to EnStratus.



Dick Weisinger reported Cloud Computing: Where the Biggest Shifts to the Cloud Are Occurring in a 12/20/2010 post to the Formtek blog:

image Cloud Computing is taking Enterprise IT by storm, but certain areas are seeing heavier rains than others.  The move to the cloud has been accelerated by the across-the-board cutbacks in budgets as businesses seek to shift/slash their internal IT costs.  The results from a survey of company executives taken by Bloomberg Businessweek identifies cloud application areas that are seeing the largest growth.

Not too unexpectedly, the largest growing area in the cloud is with non-critical business application, particularly in the area of HR.  Until issues around security and loss of control of data are fully resolved, this is to be expected.   The percentage of executives that think that they will have moved an application fully to the cloud by 2012 are as follows:

  • 44%  Benefits Administration
  • 42%  Recruitment Systems
  • 39%  Talent Management
  • 39%  Payroll
  • 37%  Customer Service
  • 36%  Sales Automation
  • 36%  Supply Chain Management
  • 35%  HR System of Records
  • 34%  Time and Attendance
  • 33%  Marketing
  • 32%  Treasury
  • 31%  Accounts Payable/Receivable

The report showed clear trends by industry, with high tech and professional service companies making the move to the cloud faster than manufacturing companies and nonprofits.

The report also asks executives, on a scale of 1-5 with 5 being the highest, what their biggest worry around cloud computing is.  As expected, security tops out at number one, followed closely by “difficult integration” and governance.

Dick is Chief Technologist at Formtek, Inc.


<Return to section navigation list> 

Windows Azure Platform Appliance (WAPA), Hyper-V and Private Clouds

Yasser Abdel Kader continued his blog series with Private Cloud Architecture - Part 2: Principles of 12/20/2010:

image

In the first part of this series (here and here), I started by some discussion regarding the basic definition that we will build upon toward achieving the Private Cloud Promises.

In this post, I will discuss the main principles behind Private Cloud

Let’s agree first some terminology; the cloud promise is to provide computing as a utility. Let me explain, we need to reach a model where the user of the cloud service will use the computing power as if he uses electricity. When he needs it, he will find it… and he will get as much or as little as he wants. So, let’s name the user who uses this service as a consumer.

So, the principles are:

Infinite Capacity: Users will see the service appears as if it has unlimited capacity. Consumers use the service as much or as little as he wants. To achieve that by not using the cloud model, you will need a great amount of resources and will end up - for sure - with an underutilized servers / infrastructure resources and will cost you a huge amount of money.

Continuous Availability: the service will appear to be available when the consumer wanted it with no interruption of services. We will achieve this by providing a mature approach for governance and management in combination with resiliency at the application level and redundancies at the infrastructure level (computing, storage and network)

Drive Predictability: Consumer should find the service consistence with the same quality and functionality any time they are used. From the service provider view, standardization will be the key for services and processes manageability.

Service Provider’s approach to delivering infrastructure: the current data center approach for providing IT Services to business is on per need bases. This will lead to longer time to provision servers and services (more cost due to duplicate infrastructure).

Resiliency over redundancy: Normally, It provides high availability by H/W redundancy at all the layers of the stack by providing another H/W component that will start to serve the request if the original component failed. This comes at a prime cost (consider active/passive cluster were the passive server is just there doing nothing). In contrast, Private Cloud provides that through resiliency. The design depends on the fact that a failure will happen. It focuses instead on rapidly restoring the service through virtualization and real-time detection and automated response to health states that will allow the workload to be moved to another working node.

Minimize human involvement: by dynamically detect and respond to failure and elastically add or reduce capacity per the need

Optimization of resource usage By allowing resource sharing, optimization of resources usage can be achieved.

Influence Good Consumer Behavior by using a cost per use bases, consumer will pay according to his actual use. For example, in the electrical utility world, consumers are charged a lower multiplier when utilization is below an agreed upon threshold. If they reach the upper bounds of the threshold, a higher multiplier will be used as additional resources are consumed.

In the next blog I will discuss the concepts behind Private Cloud.


<Return to section navigation list> 

Cloud Security and Governance

image

No significant articles today.


<Return to section navigation list> 

Cloud Computing Events

Muhammed Arrabi announced on 12/20/2010 an Azure Development & OData Intro (Wed Dec 22nd, 6:30 @PSUT) technical talk in Jordan:

Part of the Community Technical Talks series (JordanRuns.NET)

image

Developing on Azure & Intro to OData

By Suliman Battat (Evanteck) and Mohammad Saleh (Devosis, MVP)

Future Sessions in the Community Technical Series:

About the Speakers

Suliman Battat (LinkedIn, Blog, @SulimanBattat) Suliman is Technical Consultant, Trainer and Speaker. A Co-founder of Evanteck Inc., Founder and Leader of Jordan Azure User group and board member in Jordan.NET User Group (Jordev).

Mohammad Saleh (Linkedin, Blog, MVP profile, @mohkanaan) Mohamed Saleh is a Platforms Consultant at Devosis (www.devosis.com) in Amman, Jordan. Microsoft MVP in Microsoft SharePoint Server for his contribution to the technical communities in the Arabic region, and he is member of the Global IT Community Association (GITCA) Technical Team.


Rainer Stropek will keynote the Basta! 2011 conference at the Maritim Rhein-Main Hotel in Darmstadt, Germany on 2/24/2010. Following is the Google translation of the abstract:

image

We believe market research organizations such as Gartner, a release from Cloud Computing in 2010, the area of virtualization as the No. 1 IT trend. At the same time the matter is controversial. Positive scale effects resulting from the combination of many small IT infrastructure to huge data centers are not disputed. Against this are many, however, concerns about privacy and security. One question however is often given too little attention: Are our current software solutions at all ready for Cloud Computing? As the requirements change in software architectures, when a planned operation in a cloud computing infrastructure such as Windows Azure? Rainer Stropek represents the keynote of the challenges faced by software manufacturers are seeking to operate their products in Windows Azure. [Emphasis added.]


<Return to section navigation list> 

Other Cloud Computing Platforms and Services

Todd Hoff posted Netflix: Use Less Chatty Protocols in the Cloud - Plus 26 Fixes to the High Scalability Blog on 12/20/2010:

image

In 5 Lessons We’ve Learned Using AWS, Netflix's John Ciancutti says one of the big lessons they've learned is to create less chatty protocols:

In the Netflix data centers, we have a high capacity, super fast, highly reliable network. This has afforded us the luxury of designing around chatty APIs to remote systems. AWS networking has more variable latency. We’ve had to be much more structured about “over the wire” interactions, even as we’ve transitioned to a more highly distributed architecture.

There's not a lot of advice out there on how to create protocols. Combine that with a rush to the cloud and you have a perfect storm for chatty applications crushing application performance. Netflix is far from the first to be surprised by the less than stellar networks inside AWS.

A chatty protocol is one where a client makes a series of requests to a server and the client must wait on each reply before sending the next request. On a LAN this can work great. LAN's are typically fast, wide, and drop few packets.

Move that same application to a different network, one where round trip times can easily be an order of magnitude or larger because either the network is slow, lossy or poorly designed, and if a protocol takes many requests to complete a transaction, then it will make a dramatic difference in performance.

My WAN acceleration friends says Microsoft's Common Internet File System (CIFS) is infamous for being chatty. Transferring a 30MB file could tally something like 300msecs of latency on a LAN. On a WAN that could stretch to 7 minutes. Very unexpected results. What is key here is how the quality characteristics of the pipe interacts with the protocol design.

OK, chatty protocols are bad. What can you do about it? The overall goal is to reduce the number of times a client has to wait for a response and reduce the number of packets sent.

  1. eduPERT has a great example of how sending mail using SMTP was reduced from 9 to 4 rountrips through pipelining.
  2. Great paper on improving X Window System Network Performance using protocol-level performance analysis combined with passive network monitoring.

Some possible strategies are:

  1. Identify and remove unnecessary roundtrips. This may be obvious, but take a look at your protocols and see if any of the rountrips can be combined or eliminated. There may be a lot of low hanging fruit here with just a little thought. 
  2. Think services, not objects. One of the reasons protocols become so chatty is programmers are thinking in terms of fine-grained distributed objects rather coarser-grained services. This leads to chatty protocols because there is a lot of back and forth for each method call as methods tend to be pretty low in granularity. What you want is a service interface that does as large an amount of work for each request as possible. To transform an image, for example, you want to make one call and pass in all the transforms so they can be done in that one call rather than making a separate remote method invocation for each transform. This is a classic and common mistake when first using CORBA, for example. 
  3. Aggregate data to reduce the number roundtrips.
    1. Use token buckets. Token buckets are a technique for allowing a certain amount of burstiness while imposing a limit on the average data transmission rate.
    2. Batch/pipeline requests. Send more than one operation in a request and receive a single response.
    3. Batch via integration timers. Queue up requests for X milliseconds so they can be sent in a batch rather than one by one.
    4. Use fat pings. Fat pings reduce the amount of data that needs to be sent, especially compared to constant polling.
  4. Data Reduction Techniques. Keep data as small as possible so more work can fit inside a single TCP segment.
    1. Data compression
    2. Use a binary protocol. Please, no flamewars, but a custom binary protocol will generally be more space efficient than text + compression.
    3. Do not transmit redundant information.  This is more effective than compressing the information.
    4. Use high level descriptions rather than low level descriptions. Can you create a DSL or other way of describing your data and what you want done using a more compact code?
    5. Allow requests to reuse recently transmitted information. If data has already be sent can later day refer back to it so it doesn't have to be transmitted again?
  5. Keep a local cache. Simply avoid going over the network by caching data locally. This of course pushes the problem out to become one of maintaining consistency.
  6. Use Asynchronous APIs. Don't pause an application waiting for replies. This doesn't fix the protocol necessarily, but may lessen the effect.
  7. Use ackless protocols. Don't even worry about replies. Fire off requests to a message queue. When a worker processes a message it can forward the response on to the next stage of the processing pipeline.
  8. Lazy and bulk initialization. When applications start they sometimes need to go through an elaborate initial configuration phase. Try to minimize the amount of configuration an application needs. When configuration is needed try to do it in bulk so that a program will get all it's configuration attributes in one big structure rather than making a lot of individual calls. Another approach is to use lazy initialization so you just get configuration as needed rather than all at once. Maybe you won't need it?
  9. Move functionality to the client and out of the server. Is it possible for the client to do more work so nothing has to go back to the server? The X Window System saved a lot by moving font storage to the client.
  10. Realize more bandwidth is usually not a fix. Latency and packet loss are usually the problems so simply adding more bandwidth won't solve the problem, unless of course a lack of bandwidth is really the problem.
  11. Use rack affinity and availability zones. Latency and RTT times are increased by going through a long series of routers, switches etc. Take care that data stays within the same availability zone as much as possible. Some databases like Cassandra even let you specify rack affinity to further reduce this overhead.
  12. Avoid fine grained operations.  Don't, for example, implement a drag and drop operation over the network that can take 100s of requests to implement. Older versions of Windows would go get the directory on the remote side and for every file it would perform individual file stat operations for each file attribute. What a WAN accelerator does is get all the file data at once, create a local cache, and serve those requests from the local cache. A good transparent work around, but of course the idea is to avoid these fine grained type operations completely.
  13. Use persistent connections. Create a connection and keep it nailed up. This avoids the overhead of handshaking and connection establishment. On the web consider using Comet
  14. Get right to business. Pass real data back and forth as soon as possible. Don't begin with a long negotiation phase or other overhead. 
  15. Use stateful protocols. Do you need to send the same state all the time? For an application protocol the answer is probably no. For something more general like HTTP statelessness is probably the higher goal.  
  16. Tune TCP.  This is one of the typical functions of a WAN accelerator. TCP was not developed with high latency  in mind, but you can fake it into behaving. On a WAN you could try creating your own MPLS network, but inside a cloud you don't have a lot of options. Hopefully in the future the L2/L3 network fabrics inside datacenters will improve as will datacenter interconnects.
  17. Colocate behavior and state. Collocating behavior and state reduces the number of networks hops necessary to carry out operations.
  18. Use the High Performance Computing (HPC) cluster. HPC cluster is backed by a high performance network, and if your use case fits, it could be a big win. 
  19. Measure the effect of changes in a systematic fashion. Profile your application to see what's really going on. Surprises may abound. When a change is made verify an improvement is actually being made.

Most of these recommendations apply equally to Windows Azure. For example, my Speed ServiceContext.SaveChanges() Execution with the SaveChangesOptions.Batch Argument post (updated 12/20/2010) describes how use batch updates to Windows Azure tables can improve performance by a factor of almost 30 by reducing “chattiness.”


Adam DuVander reported Google Base Axed in Favor of New Shopping APIs in a 12/20/2010 post to the ProgrammableWeb blog:

Google BaseThe database that tried to be everything to everyone is being replaced by more specific tools. The Google Base API, launched in 2005, has been deprecated. In its place, Google suggests using its new Google Shopping Search API for finding products and Shopping Content API for merchants to add products. The new APIs will help Google focus on the most common uses of Google Base. However, the change means that developers will have to update code and that some non-shopping applications will be left without a replacement.

Google Base launched with broad goals, as described within Google Base help pages:

Google Base is a place where you can easily submit all types of online and offline content, which we’ll make searchable on Google (if your content isn’t online yet, we’ll host it for you). You can describe any item you post with attributes, which will help people find it when they do related searches.

Because of the broad nature, many felt Google was reaching too far with its launch of Base. Some thought Google wanted to be in its own search results, while others worried that Google shouldn’t own the structured web. Others thought Google Base could go on to disrupt entire industries, writing about Google Base’s potential impact to real estate

With Google Base there is very little justification for sellers to use a broker because they can do a FSBO, market their own homes and get the same amount traffic through Google because of its strong brand presence. And sellers don’t have to pay a dime.

RealGeni Real Estate Search Engine

In the end, Google Base was probably too broad to achieve these fears. While real estate was a popular user of Base (often from the selling agents), it didn’t replace the tried and true local methods. In fact, real estate is one of the areas that is left behind by the move to the new APIs, according to Google’s announcement:

There are a few non-shopping data types that won’t be supported with new Shopping APIs, such as jobs, real estate, events, and activities. We hope this won’t cause too much disruption, and we recommend developers use Google Site Search and other search technologies for those use cases.

Our directory lists a number of Google Base real estate mashups, including RealGeni shown above. We also list a few mashups for the other unsupported categories, but overwhelmingly the most use we’ve seen of the Google Base API are in Google Base shopping mashups. Shopping makes up 40% of the 63 Google Base mashups that we list. Those developers will still need to update their mashups, but it appears that Google has chosen the right segment upon which to focus its structured search.

I posted about GoogleBase during its early days:


<Return to section navigation list> 

0 comments: