Windows Azure and Cloud Computing Posts for 7/7/2010+

Windows Azure, SQL Azure Database and related cloud computing topics now appear in this daily series.

 
Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:

• Azure Blob, Drive, Table and Queue Services
• SQL Azure Database, Codename “Dallas” and OData
• AppFabric: Access Control and Service Bus, and CDN 
• Live Windows Azure Apps, APIs, Tools and Test Harnesses
• Windows Azure Infrastructure
• Cloud Security and Governance
• Cloud Computing Events
• Other Cloud Computing Platforms and Services

To use the above links, first click the post’s title to display the single article you want to navigate.

• Updated 7/9/2010: You can now freely download by FTP and save the following two online-only PDF chapters of Cloud Computing with the Windows Azure Platform, which have been updated for SQL Azure’s January 4, 2010 commercial release:

• Chapter 12: “Managing SQL Azure Accounts and Databases”
• Chapter 13: “Exploiting SQL Azure Database's Relational Features”

HTTP downloads of the two chapters are available for download at no charge from the book's Code Download page.

Cloud Computing with the Windows Azure Platform published 9/21/2009. Order today from Amazon or Barnes & Noble (in stock.)

Read the detailed TOC here (PDF) and download the sample code here.

Discuss the book on its WROX P2P Forum.

See a short-form TOC, get links to live Azure sample projects, and read a detailed TOC of electronic-only chapters 12 and 13 here.

Wrox’s Web site manager posted on 9/29/2009 a lengthy excerpt from Chapter 4, “Scaling Azure Table and Blob Storage” here.

Azure Blob, Drive, Table and Queue Services

Adatum changes from SQL Azure to Windows Azure table storage in Phase 2: Automating Deployment and Using Windows Azure Storage of the Windows Azure Architecture Guide – Part 1:

This chapter walks you through the changes Adatum made to the aExpense application during the second phase of the project. You'll see how Adatum extended the build process for aExpense to include a step that deploys the packaged application directly to Windows® Azure™ technology platform. You'll also see how Adatum changed the aExpense application to use Windows Azure table storage instead of SQL Azure and how the development team met some of the challenges they encountered along the way. The user-perceived functionality of the application didn't change from the previous phase.

The Premise

At the end of the first phase of the project, Adatum now had a version of the aExpense application that ran in the cloud. When the team at Adatum developed this version, they kept as much as possible of the original application, changing just what was necessary to make it work in Windows Azure.

The team does most of its testing against the local development fabric, which makes it easy for them to debug issues with the code. They also deploy the application to Windows Azure for additional testing in a staging environment in the cloud. They have found that manually deploying the application to Windows Azure through the Windows Azure Developer Portal was error-prone, especially editing the configuration files with the correct connection strings.

Note: Chapter 7, "Application Life Cycle Management for Windows Azure Applications," discusses testing applications for Windows Azure in more detail.

A simple cost analysis of the existing solution has revealed that SQL Azure would account for about one third of the annual running costs of the application. Because the cost of using Windows Azure table storage is much lower than using SQL Azure, Adatum is keen to investigate whether it can use Windows Azure table storage instead.

Goals and Requirements

In this phase, Adatum has two specific goals. The first is to evaluate whether the aExpense application can use Windows Azure table storage instead of SQL Azure. Data integrity is critical, so Adatum wants to use transactions when a user submits multiple business expense items as a part of an expense submission.

Note: You should evaluate whether Windows Azure table storage can replace SQL Azure in your application.

The second goal is to automate the deployment process to Windows Azure. As the project moves forward, Adatum wants to be able to deploy versions of aExpense to Windows Azure without needing to manually edit the configuration files, or use the Windows Azure Developer Portal. This will make deploying to Windows Azure less error-prone, and easier to perform in an automated build environment. …

The chapter continues with detailed descriptions of the migration steps.

<Return to section navigation list> 

SQL Azure Database, Codename “Dallas” and OData

• Updated 7/9/2010: You can now freely download by FTP and save the following two online-only PDF chapters of Cloud Computing with the Windows Azure Platform, which have been updated for SQL Azure’s January 4, 2010 commercial release:

• Chapter 12: “Managing SQL Azure Accounts and Databases”
• Chapter 13: “Exploiting SQL Azure Database's Relational Features”

HTTP downloads of the two chapters are available for download at no charge from the book's Code Download page. (Repeated from above the fold.)

Wayne Walter Berry describes how to use SQL Azure Data Sync to create a Data Distribution Network that’s similar to the Windows Azure CDN in this 7/8/2010 post:

For certain scenarios it makes sense to distribute your data and your web site closer to the end user; reducing the network latency and providing a better user experience. You can think about it like your own data delivery network, very similar to the Windows Azure Content Delivery Network. This data delivery network can be built with SQL Azure and Data Sync Serv[ice] for SQL Azure, we will talk about how in this blog post.

Geo Distributed Web Sites

When you deploy your Windows Azure web role (your web site) to multiple data centers around the world the site can be closer to the end user and allowing the responses to travel over the network faster. This can be accomplished with some intelligent routing; for example users in the US get sent to the US data center. Currently, Windows Azure doesn't provide a mechanism for this; you would need to do this yourself at the DNS layer. Another way to direct users to the closest web site to them is to have separate subdomains with an entrance page on the domain that prompts the user to select their country of origin.

It makes sense that if you move your web sites closer to the end user; you will want to move your databases closer to the web sites. Web sites that make heavy use of the database will need the faster performance of having that data collocated with the application. Because you have multiple web sites in different data centers, you will have multiple databases which might have exactly the same data. This is where Data Sync Services for SQL Azure comes into play, the basics of Data Sync Services for SQL Azure is covered in our previous blog post.

Finding the Right Fit

This isn’t a solution for all types of web sites; Data Sync Services works best when all the databases don’t have to be in synchronized 100% of the time. For example, an airline ticketing web site couldn’t use the solution that I am proposing here; data sync services doesn’t provide transaction replication that modifies all the member databases on a commit.

However, if you are running a publishing web site, like a magazine or newspaper that has schedule deployments of content and that content is stored in a database, running a data distribution network to push data to localized datacenters would work very well.

Publishing a Magazine

Let’s say that you were running a web site for a print magazine. All of the magazines content is stored in the hub database (refer to this blog post for Data Sync Service terminology), and that content is published once a week.

Because of the nature of the web site, the database usage tends to be high read, low write. Management wants a Windows Azure centric solution, and it has to be fast – new search engine optimization rules favor web sites that return contently quickly.

You know that the majority of your traffic comes from the United States, so you decide to use three SQL Azure databases to support the read queries; these are read-only member databases of the Data Sync Services synchronization group. They exist in the same data center as the hub database. Then you create two more member databases one in Asia and one in Europe, they are in the same data centers as your Windows Azure web roles.

Note: this is fictional data, and not a reflection of MSDN Magazine traffic, nor does MSDN magazine use SQL Azure – currently.

Your content team would always update the hub database; maybe you would provide a private edition of the web site so they could see how the content looks before you published it. Then when you are ready to publish, you would tell Data Sync Service services to synchronize the group, and all the member databases would be updated with hub data pushed out to the remote data centers. For the rest of the week, the Azure web roles would call to member database that was local to them.

Notice that there is a potential cost savings to you because you have reduced the transfer charges for cross data center queries. Anytime you cross the data centers there is a data transfer fee, this includes querying SQL Azure databases that are not in the same data center as the web role. If you have a low write, high read scenario it benefits you to use Data Sync Services to push the data to the datacenter where the SQL Azure web role is hosted. The caveat is that the price of running another SQL Azure database is less than the potential transfer charges.

Writes

What about writes? Even the simplest front facing web sites want to do some writes. If you have a low write scenario you can always send your writes directly to the hub database. The issue becomes that those writes are not synchronized with the member databases until the content is published on a weekly basis. If you write them directly to the localized member database (Data Sync Service supports bi-directional synchronization) you have the same issue, plus a chance of merge conflicts.

One potential solution is to use a separate database for writes, one that is hosted in the same data center as your hub database. Worldwide web roles read and write data directly to this separate database to support forums, comments, and user generated content.

In the future I will blog about a more complicated technique that provides real time writes with Data Sync Services that provides geographic redundancy.

Rob Tiffany’s Here Comes SQL Server Compact 4.0 post of 7/8/2010 describes the new features of the SQL Server Compact 4.0 CTP:

Microsoft SQL Server Compact 4.0 is the next generation of embedded database from the SQL Server family, and it’s coming to an IIS/ASP.NET web server in your neighborhood.  I wanted to share the news first announced by Scott Guthrie and Ambrish Mishra on their respective blogs.

SQL Server Compact 4.0 will provide an easy and simple to use database storage for starter websites, and the features of Compact that wooed the mobile devices and desktop developers will now be available to the ASP.NET web application developers. In addition, the features and enhancements in SQL Server Compact 4.0 like the new Transact-SQL syntax for OFFSET & FETCH, higher reliability, seamless migration of schema and data to the higher SKUs of SQL Server, support for code-first & server generated keys in ADO.NET Entity Framework 4.0, and the ability to use Visual Studio 2010 to develop ASP.NET web applications for Compact 4.0 etc. will also excite the existing development community.

The first Community Technology Preview (CTP1) release of SQL Server Compact 4.0 is integrated with the WebMatrix Beta, which is a free ASP.NET web application development tool for Windows, and provides development & database support, and can easily publish/deploy websites to the web hosting providers. SQL Server Compact 4.0 CTP1 plays an important role in WebMatrix Beta, and enables WebMatrix Beta to be a small download, with a simple development experience, and contributes to the seamless end-to-end experiences from development through to publish. Scott Guthrie’s (ScottGu) blog has detailed information about the WebMatrix Beta and its different components. WebMatrix Beta can be installed in one-click through the Microsoft Web Platform Installer (Web PI) 3 Beta. In addition to the Web PI 3 Beta, SQL Server Compact 4.0 CTP1 is also available for download from the Microsoft download center.

1.    Specialty of SQL Server Compact:

SQL Server Compact has many specialties that are well liked by the developer community. The specialties of Compact, among others, are as listed below:

• Small Size: Easy to download with a small size of 2 to 2.5 MB and installs fast with no configuration needed.
• Program with .NET Framework APIs: Simple to use because of the familiar .NET Framework APIs, including ADO.NET and ADO.NET Entity Framework, and similar programming patterns.
• Supports Transact-SQL Syntax: Supports a rich subset of the Transact-SQL syntax supported by higher SKUs of SQL Server.
• Private (xcopy) Deployable: Xcopy deployable within the application folders and no admin privileges are needed. The database file contains no code and is safe to deploy along with the database. 
• In-process Database: Loads in the application’s process memory space and requires no inter-process communication.
• Integrated with Visual Studio: Applications can be rapidly developed using the integrated development experience in Visual Studio.
• Easy Migration to SQL Server: Smooth on-ramp to SQL Server when the applications requirements grow and there is a need to handle large amounts of data and high concurrent loads due to multiple users.
• Free Product with Free Tools: Compact is a free product, and is integrated with the free Express versions of Visual Studio and WebMatrix Beta, and requires low investment to start building and deploying applications.
• Simple Solutions for Syncing data with SQL Server: Schema and data can be synced easily with SQL Server using inbuilt and programmable technologies like Sync Framework (Sync FX) and merge replication.

2.    New Features in SQL Server Compact 4.0 CTP1

The first CTP release of SQL Server Compact 4.0 has a lot of features that will be useful to all of the development community of Compact, including the latest entrants, the ASP.NET developers.

Rob continues with detailed descriptions of new features in Fundamentals, Improvements for ASP.NET Web Applications, Develop Compact 4.0 CTP1 Applications with WebMatrix Beta and Visual Studio 2010, New Transact-SQL Syntax, and API Enhancements topics.

Ambrish Mishra’s Introducing SQL Server Compact 4.0, the Next Gen Embedded Database from Microsoft post of 7/7/2010 is the source of most of Rob Tiffany’s post above.

Abrish is a Microsoft  Program Manager for SQL Server Compact.

Miguel de Icaza announced OData’s .NET Client will be included in Mono 2.8 according to his Microsoft MS-PL code in Mono post of 7/7/2010:

Over the past couple of years Microsoft has been open sourcing some key .NET libraries under the MS-PL or Apache 2 license.

We are tremendously grateful to Microsoft for making these components open source. This has allowed us to distribute this in the past, and we are going to be bundling a lot more of it with Mono 2.8:

In Mono 2.8, the following assemblies and code come from Microsoft:

• Managed Extensibility Framework, a framework to create extensible applications (blogged about their open sourcing).
• The Dynamic Language Runtime is the engine that powers IronPython and IronRuby, but also third party efforts like IronJS and RemObject's Javascript compiler. This lives largely in the System.Core assembly in .NET 4. Some of the code is also used in Mono's version of System.Numerics.
• OData's .NET client (System.Data.Services.Client), I blogged about this here [on 3/22/2010].
• ASP.NET MVC both MVC1 and MVC2.
• Microsoft's AJAX library for use with ASP.NET WebForms.
• Silverlight's Controls, these are used by Moonlight.

With Mono 2.8 we are going to default to the .NET 4.0 profile. So from the list above MEF, the DLR, OData and MVC2 become first class citizens in Mono-land, no longer third-party packages that you needed to install.

Michael Hodgson posted Distributed Architecture: Part 1 - WCF Data Services ... A RESTful way of doing things to the Syndicated Methods blog on 7/7/2010:

One of our strong practices here at Syndicated Methods is design and implementation of Service Oriented Architectures (SOA).  When people immedietly hear SOA, they think SOAP and bloated contracts.  Yes, SOA boasts some grandious goals with its standards, and with those goals comes a complex implementation. I have seen SOA make sense of many enterprises, and like any loyal developer ready to fight for their technology,  this mantra has always rubbed me the wrong way.   As of the release of .NET 4.0 however, and the rise of REST services, it has become quite clear that SOA isn’t the only way.  It has actually changed how I think of our practice.  So much so that I make sure to tell clients and folks that I talk to that our practice specializes in “Distributed Architectures”.  I have come to realize that we can take all of those lessons learned in SOA and apply them to any number of manners for exposing enterprise data in a distributed fashion applying the most suitable manner for clients. 

After reading a recent article on WCF Data Services, it got me thinking about how to spread the word for such technologies.  Oh yeah, did you know Syndicated Methods has a blog!  So, I want to start a series on the Syndicated Blogs titled”Distributed Architectures”.  The first candidate … you guessed it.  WCF Data Services.

WCF Data Services

There are certain scenarios where the development overhead associated with SOA is just not necessary.  For example, you have an analytics group that needs to have open access to a data store.  Or, you have an application that allows the user to shape and export data how they see fit.  Maybe you do a lot of prototyping and are looking for a turn key solution for getting to your data. Such business problems have been solved by the Open Data Protocal (i.e. OData).  Those familiar with RESTful services will immedietly understand.  The idea is to expose a repository of data over a standard pathing system, such as urls.  WCF Data Servies takes these concepts and allows you to define a WCF Service (without the contracts of course) over an Entity Data Model.  You can expose your data in just a couple of easy steps. …

Michael continues with step-by-step instructions with C# sample code and shows the result in a screen capture of Fiddler2 output.

Marcelo Lopez Ruiz explains Custom Query Options on the WCF Data Services Client as they relate to the OData protocol in this 7/6/2010 post:

So, the OData Protocol talks about custom query options, but how exactly do you use them from the WCF Data Services client?

If you've used service operations at all, you already know the answer: use the AddQueryOption method. There is a subtle difference though, and things are actually easier for custom query options.

For example, here is a console program that will set up a service and query for all customers that have a "2" in their name.

First, we'll set up the classes to handle data and include a service operation to filter the customers (normally you'd just use a $filter option, but I'm keeping the operation simple). …

C# source code excised for brevity.

Now when you run this, the service operation will query 'out-of-band' directly to the WCF operation context for the URL, intercept the request, and you will get the following. …

Querying for http://localhost:3000/CustomOps()?name='2'
Customer: Customer #2
Customer: Customer #12
Customer: Customer #20
Querying for http://localhost:3000/CustomOps()?name='3'&worldcup=awesome
Customer: The world cup is awesome

<Return to section navigation list> 

AppFabric: Access Control and Service Bus, and CDN

Microsoft uses Bing Maps as a case study for the Windows Azure CDN in its Mapping Service Increases Performance by 80 Percent with Global Data Center Network post of 7/2/2010:

Microsoft Corporation operates a content delivery network for its Internet-based services. Bing Maps took advantage of the Windows Azure Content Delivery Network, to cache and deliver content at strategically placed, global nodes. By bringing content closer to customers, Bing Maps improved performance by 80 percent, attracted new customers, and gained the ability to cost-effectively scale.

Business Needs
Microsoft Corporation operates in more than 100 countries, offering technology solutions to consumers and organizations. Bing Maps is no exception and delivers high-resolution aerial photography and maps to customers around the world. Bing Maps offers both consumer and business-to-business products and has gained more traction among the geographic information system industry competitors, including Google Maps and MapQuest, due in part to its excellent map coverage and high-quality images.

Though Bing Maps serves a global audience, it originally did so through just one Microsoft data center in Virginia, with a failover data center in California. However, Bing Maps is client-driven—that is, when a customer pulls up a map using Bing Maps, the data request is sent from wherever the client computer is located over the Internet to the data center in Virginia. With only one data center in the United States to process requests from around the world, customers who used Bing Maps sometimes experienced latency issues. “You can imagine the amount of time it takes to send data over the Internet from half-way around the world,” says Chris Pendleton, Bing Maps Technology Evangelist, Microsoft Corporation. “The bottom line was that we needed to improve the performance for our customers.”

Beyond performance issues, and despite the recognized quality of Bing Maps content, the latency resulted in lost business for Bing Maps, especially for its business-to-business offering. Performance, and the resulting loss of business, was such a concern that the Bing Maps team even considered building its own global data centers, but the cost was exorbitant.

The Bing Maps team wanted to bring mapping data closer to customers—no matter where they are located—to reduce latency and improve application performance. “Ideally, we wanted a data center with a global presence to which we could replicate content to and spread out across multiple nodes that are geographically dispersed—just like our customers,” explains Pendleton.

Solution
In 2009, the Bing Maps team started using the Windows Azure Content Delivery Network. The network, which initially started as a service for internal teams at Microsoft to use, enables customers to cache and deliver content at strategically placed locations around the globe. The Content Delivery Network, which also hosts content for microsoft.com, MSN, and other Microsoft services, is now available to the public as part of the Windows Azure platform.

The Bing Maps team uses the Content Delivery Network to deliver its image tiles—more than 2 petabytes of aerial images that are each 256 pixels wide and 256 pixels high. Each of the image tiles are cached across the Content Delivery Network so that at run time, when a customer requests a map, that request is sent to the nearest Content Delivery Network node and the tiles are compiled and sent back for quick map-rendering.

Bing Maps also stores its licensed application programming interface (API) on the Content Delivery Network. The JavaScript API control, which is 242 kilobytes, enables developers to embed maps in web-based applications. Each time a customer visits a webpage that uses the API control, the large file is pulled from a Content Delivery Network node and executed at run time as close as possible to the client computer.

While the Content Delivery Network already delivers 2 petabytes of data for Bing Maps, the team plans to scale out and cache even more data at similar levels as it continues to expand its map coverage.

Benefits
As a result of using the Windows Azure Content Delivery Network, Bing Maps resolved its latency and performance issues; and by doing so, it also reversed customer attrition and attracted new customers. At the same time, Bing Maps gained cost-effective scalability.

Chris Pendelton, Bing Maps Technology Evangelist, Microsoft Corporation, put it this way: “By using the Windows Azure Content Delivery Network, we proved that our service was reliable. We retained our current customers, won back customers who had previously left, and gained new customers.”

<Return to section navigation list>

Live Windows Azure Apps, APIs, Tools and Test Harnesses

Karsten Januszewski (@irhetroric) explains How The Archivist Polls Twitter in this 7/7/2010 post to the MIX Online Labs blog:

You may be wondering how frequently The Archivist updates archives. Well, the answer to the question is more complicated that it may first appear. Let’s dig in.

The Archivist interacts with Twitter using the Twitter Search API, which it polls at variable intervals based on the frequency with which a particular archive is updated. We call this the elastic degrading polling function. This algorithm helps The Archivist be a good Twitter citizen, allowing us to poll Twitter conservatively while at the same time maintaining archives with the latest tweets.

Here’s how the algorithm works: When a user makes an archive ‘active’, the polling process begins. Every archive is inspected once an hour to determine how ‘hot’ it is. We determine how hot an archive is by recording how many results we get back each time we poll Twitter (the maximum we can pull at any one time is 1500).  We use this number to determine how frequently to poll Twitter for that archive. Depending on the number, we either hold off on polling for a given interval or query again, based on the following buckets:

So, let’s look at an example. Say we have an archive going for the term ‘Wittgenstein’. When the Archivist checks on this archive at 10 AM, it discovers that the last query for Wittgenstein only returned 10 tweets. It also discovers that this archive was last updated at 9 AM. The Archivist won’t poll Twitter for this archive, because the tweet count isn’t high enough and the archive had been queried within 24 hours. Since the archive is in the 24 hour bucket, the same thing will happen when The Archivist checks on this archive each hour.

Once 9 AM rolls around on the next day, since 24 hours have passed, Twitter will be polled for the Wittgenstein archive .

Now, let’s say for some reason there’s a flurry of tweets about Wittgenstein—when that archive was updated at 9am, it pulled 600 tweets. In this case, the archive adjusts because it has become hot. It is now in the 1 hour bucket instead of the 24 hour bucket.  So, when 10 AM rolls around, the Wittgenstein archive gets updated again. 

But let’s say at 10 AM it pulls only 250 tweets. Well, now the archive moves to the 8 hour bucket. So,the Wittengenstein archive will not be polled again again until 6 PM. Let’s say it pulls 1000 tweets. Well, it goes back to the 1 hour bucket, since it appears to be hot. At 7 PM the term is checked again.  This time, the response is only 10 tweets. It seems to have cooled off quickly, so we’ll move it back to the 24 hour bucket.  

Some of you may notice that there’s a chance that The Archivist could possibly miss tweets when a term becomes hot. This is a reality of our architecture and is justified by the following: First, once a term gets hot, the amount of data can grow quickly. Ultimately, in that scenario,The Archivist becomes a statistical sample as opposed to a true historical record.  Second, Twitter itself doesn’t guarantee that all tweets will be returned for a given search. See http://help.twitter.com/entries/66018-my-tweets-or-hashtags-are-missing-from-search and http://dev.twitter.com/doc/get/search for more on this. Consequently there is no way that The Archivist can ever claim to be a true historical record. Third, The Archivist is optimized for following non-trending topics over a long period of time, as opposed to trending topics over a short time.  For a tool optimized for the latter scenario, see Archivist Desktop. Another option would be to run your own instance of The Archivist Web and tweak the polling algorithm, which would be trivial to do. Contact me if you are interested in doing so.

For more details about The Archivist and links to three (the maximum per person) public archives, see my Archive and Mine Tweets In Azure Blobs with The Archivist Application from MIX Online Labs post to the OakLeaf blog of 7/4/2010.

The Windows Azure team posted  on 7/8/2010 Real World Windows Azure: Interview with Francois Bochatay, CEO and Cofounder, Sobees, a variation on the Microsoft Case Study Team’s Social Media Service Provider [Sobees] Expands Offering, Attracts Customers with Cloud Services post of 6/27/2010:

As part of the Real World Windows Azure series, we talked to Francois Bochatay, CEO and Cofounder at Sobees, about using the Windows Azure platform to deliver the company's social-media aggregator and the benefits that Windows Azure provides. Here's what he had to say:

MSDN: Tell us about Sobees and the services you offer.

Bochatay: Sobees develops software that people can use to manage all of their social-media information streams with a single application. Our aggregator is popular among consumers who use applications such as Twitter and Facebook, and it recently earned a spot as the leading application on the Yahoo! Mail portal in the United Kingdom and later in other countries in Europe. We also have a real-time search platform that aggregates data from multiple search engines.

MSDN: What was the biggest challenge Sobees faced before implementing the Windows Azure platform?

Bochatay: With the exposure we've received through Yahoo!, and the potential to acquire millions of new customers, we weren't confident that the small hosting company we used for our web services and applications could provide the reliability and scalability we needed. We have seven employees, and as we grow, we want to spend time developing new services and focusing on what we do best. We don't want to have to worry about managing IT.

MSDN: Can you describe the solution Sobees built with the Windows Azure platform to address your need for scalability and reliability?

Bochatay: We migrated all of our web services, applications, and real-time search platform to the Windows Azure platform in just a few days, creating Web roles in Windows Azure to host the Microsoft ASP.NET webpages. We also used the Windows Azure platform for our new business-to-business service, which we are launching in June 2010. The new service sits on top of the social-networking application networks, Twitter and Facebook, and some news feeds. With full access to the Twitter and Facebook application programming interfaces (APIs), Sobees developed its own API and application to enable businesses to monitor Twitter and Facebook feeds and manage Twitter posts based on user-set criteria. In addition, we use Microsoft SQL Azure to store relational data. …

The interview continues with the usual “MSDN: What makes your solution unique?” and “MSDN: What are some of the key benefits Sobees has seen since implementing the Windows Azure platform?” Q&A topics.

TechNet Blogs presented in late June 2010 a series of four Microsoft Blog posts about the cloud in vertical industries, three of which appear to have been lost in cloud noise:

• Cloud computing in healthcare – private, public, or somewhere in between by Steve Aylward, 6/28/2010 (see story by Nicole Harris for InformationWeek in the Cloud Security and Governance section below.)
• Cloud Computing is a Perfect Fit for Retailers by Fred Bentfield, 6/25/2010
• Collaboration Is Job No. 1 in Manufacturing by Craig Hodges, 6/24/2010
• Wall Street’s Real Fuel for Positive Change: Cloud Computing by Coleen Healy, 6/23/2010

The Technical Blog posted Microsoft Azure – Paul Lappas, a 00:03:00 video interview on 7/5/2010:

Paul Lappas, VP Engineering – GoGrid (San Francisco, CA). GoGrid allows its customers to launch Linux or Windows based virtual machines in minutes using a web browser or API. GoGrid provides services for hosting, batch processing, development, and testing. GoGrid is extending services to help Azure development teams save time and money by using GoGrid infrastructure for development, testing, and backup while using Windows Azure.

Andrew Kisslo reported Two More Customers Pick Microsoft Over Google in his 6/30/2010 TechNet Blogs post (just showed up in my reader on 7/8/2010):

Over the last several weeks, I've been proud to highlight why customers of all sizes, geographies and industries have picked Microsoft over Google Apps. Some have left Google after realizing 'it's more pony than horse', others gave it a test drive and found out quickly it was 'more showroom than track ready'.  (Sorry, I couldn't resist) . Despite the hype machines running overtime at the GooglePlex in Mountain View, customers are voting with their feet to pick Microsoft.

This is why I am excited to highlight two more customers who have decided to share their reasons why Microsoft has become their trusted software vendor. As always, if you want more customer examples or content that outlines how our solutions differ to Google, you can always visit this website.

Leaving Google.  Why Even Free Google Apps Wasn't Worth It

Jared Morgan from Bradshaw and Weil in Paducah, KY has written a guest post over on the MS Online blog about their experience leaving Google Apps Standard Edition for MS Online. Standard edition is a free offering from Google that is limited to 50 users. They see the tremendous value in BPOS to now pay for the suite of services. Jared shares why as a small business the cloud offers a cost effective way to remain agile while gaining access to technology previously out of reach due to cost and on site company support. My favorite part about Jared's post is his singling out of SharePoint Online as a key benefit of the BPOS Suite, "What I thought I was getting as a simple throw-in with BPOS, SharePoint Online has turned out to be as valuable as Exchange Online, if not more so."

SharePoint is the fastest growing product in the history of Microsoft. It's received numerous accolades from industry analysts including top placement in all related Gartner Magic Quadrants. Imagine being a small business with under 50 employees and having access to such a product via the cloud? It's no surprise why SharePoint is a winner for Bradshaw and Weil and 'Why Microsoft' is an affirmative statement and not question at all.

Future Proofing Your Business - Cloud on Your Terms with Microsoft

On the Exchange Blog, David Aird, Head of IT for MITIE, a strategic outsourcing and asset management company, shares his insights into why they selected Exchange over Google Apps. He explains how Microsoft provides flexibility they need and how Google is an ultimatum full of hidden costs. We are the only company in the industry that allows you to run your solutions on prem, in the cloud, hosted with a partner or a hybrid. That means that companies like Bradshaw and Weil who are 'All In' can embrace the cloud immediately and for companies like MITIE, they can still run the systems themselves but by being on Microsoft, they have a product that is 'future proofed' because it allows them an 'on ramp' to go to the cloud on their terms.

"We looked at Google. They were competitively priced but with only three years in the productivity space, they lacked maturity and seemed like a risky investment… Ultimately, we choose Exchange 2007 for its manageability, reliability and enterprise class support. Despite all the recent focus on the cloud, we're not quite ready to move our data outside of our immediate control. When we do, we'll do it on our terms rather than being forced into a fit that's not right for our business." (see post for full transcript. I pulled from two paragraphs)

Stay tuned for more customer stories in the coming weeks as we continue to highlight 'Why Microsoft'.

Firedancer uploaded a Layered Architecture Sample for Azure to CodePlex on 6/26/2010:

Layered Architecture Sample for Azure takes the Expense Sample application that was originally developed in Layered Architecture Sample for .NET and ports it into the Windows Azure platform to demonstrate how a carefully designed layered application can be easily deployed to the cloud. It also illustrates how layering concepts can be applied in the cloud environment.

Layered Architecture Sample for .NET was designed to showcase various .NET Technologies such as Windows Presentation Foundation (WPF), Windows Communication Foundation (WCF), Windows Workflow Foundation (WF), Windows Form, ASP.NET and ADO.NET Entity Framework working in conjunction with the Layered Architecture Design Pattern.

Layered Architecture Sample for Azure extends on that vision by demonstrating how to perform them in the cloud.

The current version of Layered Architecture Sample for Azure uses Windows Azure and SQL Azure completely to show how to deploy all the layers to the cloud while retaining a small portion of the desktop clients for on-premise communication with the cloud service. Future versions of the sample will demonstrate hybrid on-premise and cloud scenario by leveraging on Windows Azure AppFabric Service Bus to communicate with on-premise services.

Full cloud scenario with minimal on-premise bits (Non Azure AppFabric scenario)

Return to section navigation list> 

Windows Azure Infrastructure

Lori MacVittie (@lmacvittie) claims No, it’s not global server load balancing or GeoLocation. It’s something more… because knowing location is only half the battle and the other half requires the ability to make on-demand decisions based on context in her Location-Aware Load Balancing post of 7/7/2010 to F5’s DevCentral blog:

In most cases today, global application delivery bases the decision on which location should service a given client based on the location of the user, availability of the application at each deployment location and, if the user is lucky, some form of performance-related service-level agreement. With the advent of concepts like cloud bursting and migratory applications that can be deployed at any number of locations at any given time based on demand, the ability to determine not just the user location accurately but the physical location of the application as well is becoming increasingly important to address concerns regarding regulatory compliance.

Making the equation more difficult is that these regulations vary from country to country and the focus of each varies greatly. In the European Union the focus is on privacy for the consumer, while in the United States the primary focus is on a combination of application location (export laws) and user location (access restrictions). These issues become problematic for not just application providers who want to tap into the global market, but for organizations whose employee and customer base span the globe.

Many of the benefits of cloud computing are based on the ability to tap into cloud providers’ inexpensive resources not just at any time its needed for capacity (cloud bursting) but at any time that costs can be minimized (cloud balancing). These benefits are appealing, but can quickly run organizations afoul of regulations governing data and application location.

In order to maximize benefits and maintain compliance with regulations relating to the physical location of data and applications and ensure availability and performance levels are acceptable to both the organization and the end-user, some level of awareness must be present in the application delivery architecture.

Awareness of location provides a flexible application delivery infrastructure with the ability to make on-demand decisions regarding where to route any given application request based on all the variables required; based on the context. Because of the flexible nature of deployment (or at least the presumed flexibility of application deployment) it would be a poor choice to hard-code such decisions so that users in location X are always directed to the application at location Y. Real-time performance and availability data must also be taken into consideration, as well as capacity of each location.

THE RIGHT RESOURCE at the RIGHT TIME from the RIGHT LOCATION

This information becomes increasingly critical as we look to future capabilities surrounding mobility and the anticipated “on-demand” deployment of applications to meet capacity and performance demands. As load increases on an application, the ability to auto-scale is paramount to the successful use of cloud computing in an application delivery strategy. But the ability to auto-scale and still meet any restrictions on location based on user demand and application restrictions is vital to maintaining compliance and avoid the possibility of fines – or worse – for failure. Thus there must be included in such a strategy the means by which all concerns – legal, performance, and operational cost – can be met. This requires an understanding of all the factors involved and a way to act on the data in real-time intelligently; within the context of the operating parameters provided.

The challenge lies in gathering that data, in having a holistic view of user demand, application deployments, and current performance across all locations in real-time or at least in near real-time. It cannot be data from an hour ago – the demand now is surely different than it was then. 

While debate continues regarding what solution or device should be the “authoritative control” over provisioning of applications, one thing stands as true: it is the global application delivery infrastructure that has available all the information required regarding location, demand, and performance. It is the strategic point of control for a multi-location application delivery strategy. Whether that means the global application delivery infrastructure should itself initiate provisioning of additional resources as required or simply provide that information to the controlling system is – for now – irrelevant. As long as the information is provided back to the global application delivery infrastructure so that it can route user requests appropriately, of course.

It is simply not enough to “spin up capacity” in any old cloud when regulations may be in play. The key to successful leverage of cloud computing will be the ability to “spin up capacity” in the right place at the right time, and then to direct the right users to the right application. That must include location, for while the concept of clouds having no boundaries may be true as far as the ability to access, clouds must respect that the physical boundaries imposed by legal jurisdictions do exist and have an impact on the way in which applications are deployed and accessed by end-users.

This is why it is called location aware and not location based. The decision regarding which application instance or cloud environment should serve a given user is not based on location, but the decision-making infrastructure must certainly be aware of location and balance requirements regarding location with requirements for performance in that decision.

Load balancing is – or should  be when properly applied to the problems at hand – becoming more about balancing business requirements than it is about just balancing load. 

John Treadway’s Open Source Cloud Bits post of 7/6/2010 to the CloudBzz blog begins:

Last week I got into a nice discussion on Twitter regarding the role of open source in an infrastructure as a service (IaaS) stack.  With open source cloud stacks from Eucalyptus, Cloud.com, Abiquo and others competing against proprietary source solutions from Enomaly, VMware and others, this can get fairly confusing quickly.

For clarity, here is my position on open source vs. proprietary source in this aspect of the market:  both have a role to play and natively one is not better or more advantaged than the other.  However, when you get into the details there are factors that might favor one model over the other in specific cases. I will look at this from the perspective of the service providers and enterprises who use cloud stacks.  In a future post I may touch on factors that vendors should when choosing between open source and closed source models.

For service providers, margins are critical.  Any increase in capital and operating costs must enable a corresponding increase in value provided in the market.  Amazon and Google have the scale and ability to build a lot of capabilities from scratch, trading a short-term increase in R&D against a long-term decrease in operating costs.

While some cloud providers may attempt to match the low-cost giants on pricing, they know that they need to differentiate in some other material way (e.g. performance, customer service, etc.).   For these providers, the more “free open source” technology that they can leverage, the lower their operating costs may be.

This low-cost focus must permeate their decision making, from the physical infrastructure (commodity servers, JBOD/DAS storage, etc.) to the hypervisor (Xen or KVM vs. VMware), to the cloud provisioning/automation layer, and more.  Open source CMDBs (example), monitoring technologies (e.g. Nagios) and other technologies are often found in these environments.

There are trade-offs, of course.  Open source can often be more difficult to use, lack key functionality, or suffer from poor support – all of which increases costs in often material and unintended ways (note that proprietary solutions can have many of the same issues, and do more often than most people realize).

Other service providers may target the enterprise and focus on highly-differentiated offerings (though I really haven’t see much differentiation yet, at least at the IaaS level).  For these providers, the benefits of enterprise-grade storage (EMC, NetApp, HP), VMware’s HA and fault-tolerant capabilities, and other capabilities gained from using tools from HP, IBM, BMC and other vendors, may be well worth the increase in cost.  And make no mistake, the cost increase from using these technologies can be quite substantial.

Newer vendors, such as Enomaly, are having some success despite their closed-source nature (Enomaly started as open source but changed models in 2009).  Further, even when a provider uses a solution from Cloud.com or Abiquo, both of them with open source models, they will often choose to pay for premium editions in order to get functionality or support not available via open source.  In reality, anybody serious about this market will want a mix of open-source (though not necessarily free) and closed-source technologies in their environment.

In the enterprise, the story is a bit different.  If you’re already paying VMware for an all-you-can-eat enterprise license agreement (ELA), the marginal cost to use vSphere in your private cloud is zero.  KVM or Xen are not less expensive in this case.  Same is true for tools from HP, IBM, BMC and others.

The primary question, then, is whether or not these are the right solutions.  Does BMC have a better answer for private clouds than Eucalyptus?  Is IBM CloudBurst better than Abiquo for development and test? 

Open source for open source’s sake is not rational. …

John continues with his open-source vs. proprietary analysis and concludes with these recommendations:

1.  Insist on open source (or at least full source access – not escrow) when one or more of the following situations exist:

- the supplier is small or thinly funded (VCs can and do pull the plug even after many million$ have been invested)
- the capability/functionality provided by the technology is strategically important to you, especially when investment must be maintained to remain leading-edge in a fast-moving and intensely competitive market
- migration costs to a different technology are very high and disruptive

2.  Consider closed-source/proprietary solutions when at least two or more of the following factors are present:

- the functionality provided by the software is not core to your competitive positioning the market
- replacement costs (particularly internal change costs) are moderate or low
- the functionality and value is so much higher than open source alternatives that you’re willing to take the risk
- the technology is so widely deployed and successful that the risks of abandonment is very low
- the costs are low enough so as not to make your offering uncompetitive or internal environment unaffordable

Balancing risk, capability and control is very difficult – even more so in a young and emerging market like cloud computing.  The decisions made in haste today can have a profound impact on your success in the future – especially if you are a cloud service provider.

While open source can be a very potent source of competitive advantage, it should not be adopted purely on philosophical grounds.  If you do adopt closed source, especially at the core stack level, work hard to aggressively manage your exposure and make sure you work hard to ensure that those “unforeseen events” don’t leave you high and dry.

<Return to section navigation list> 

Cloud Security and Governance

Lori MacVittie (@lmacvittie) asserts As the majority of an application’s presentation layer logic moves to the client it induces changes that impact the entire application delivery ecosystem in a preface to her The New Distribution of The 3-Tiered Architecture Changes Everything post of 7/8/2010:

The increase in mobile clients, in demand for rich, interactive web applications, and the introduction of the API as one of the primary means by which information and content is shared across applications on the web is slowly but surely forcing a change back toward a traditional three-tiered architecture, if not in practice then in theory. This change will have a profound impact on the security, delivery, and scalability of the application but it also forces changes in the underlying network and application network infrastructure to support what is essentially a very different delivery model.

What began with Web 2.0 – AJAX, primarily – is continuing to push in what seems a backward direction in architecture as a means to move web applications forward. In the old days the architecture was three-tiered, yes, but those tiers were maintained almost exclusive on the server-side of the architecture, with the browser acting only as the interpreter of the presentation layer data that was assembled on the  server. Early AJAX applications continued using this model, leveraging the out-of-band (asynchronous) access provided by the XMLHTTPRequest object in major  browsers as a means to dynamically assemble smaller pieces of the presentation layer. The browser was still relegated primarily to providing little more than rendering support.

Enter Web 2.0 and RESTful APIs and a subtle change occurred. These APIs returned not presentation layer fragments, but data. The presentation layer logic required to display that data in a meaningful way based on the application became the responsibility of the browser. This was actually a necessary evolution in web application architecture to support the increasingly diverse set of end-user devices being used to access web applications. Very few people would vote for maintaining the separation of presentation layer logic used to support mobile devices and richer, desktop clients like browsers. By forcing the client to assemble and maintain the presentation layer that complexity on the server side is removed and a single, unified set of application logic resources can be delivered to every device without concern for cross-browser, cross-device support being “built in” to the presentation layer logic. 

This has a significant impact on the ability to rapidly support emerging clients – mobile and otherwise – that may not support the same robust set of capabilities available on a traditional browser. By reducing the presentation layer assembly on the server side to little more than layout – if that – the responsibility for assembling all the components and their display and routing data to the proper component is laid on the client. This means one server-side application truly can support both mobile and desktop clients with very little modification. It means an API provided by a web application can not only be used by the provider of that API to build its own presentation layer (client) but third-party developers can also leverage that API and the data it provides in whatever way it needs/chooses/desires.

This is essentially the point to which we are almost at today. …

Lori continues with THE IMPACT on EVERYTHING and NEW BURDENS, NEW CHALLENGES topics.

Doug posted 12 Ways to Secure the Cloud – Resistance is Futile to Learning Tree’s Perspectives on Cloud Computing blog on 7/8/2010:

I keep reading that people are resisting the cloud because of security. I’m not really getting this. Moving your application to Windows Azure or Google App Engine doesn’t mean you are abdicating your security responsibilities and handing them to Microsoft or Google.

In the cloud or not, it is your data and your applications and your responsibility to make them secure. Here are 12 things you can do to keep your applications secure:

1. The best way to secure confidential data is to not store it in the first place. This seems obvious, but start by asking if the value of collecting the data outweighs the cost of keeping it safe. 
2. If you decide to store the data, then it should be encrypted. You might be thinking, “I only have to encrypt it, if I put it in the cloud where Microsoft employees can see it.  Right?”  Uh… wrong.
3. If you’re going to transmit confidential data over the Internet you need to use HTTPS for transmission (cloud or not).
4. Configuration information should be encrypted. “But Doug, on my network I don’t have to worry about that because only my programmers can see it.  Right?”  Uh… wrong.
5. If you’re using a database, connect to it with a restricted account. 
6. Only allow access to data through stored procedures. Do not allow access to tables directly. Many people think stored procedures are used solely for better performance.  Stored procedures should be used for security as well.
7. Never use string concatenation to build a query – using parameterized queries or stored procedures will prevent SQL injection.
8. All input from a Web application, whether the user typed it or not, needs to be validated and cleaned. Characters such as “-”, “;”, “<”, “>” need to be removed. These characters are used for code injection.
9. If you are using .NET, pre-compile your applications with strong names.  There’s a misconception that strong names are only needed to deploy to the GAC. Strong names prevent tampering and enable versioning. “But Doug, I only have to worry about that if I’m using the cloud, right?“  Uh… wrong.
10. Strong names prevent tampering, but the source code can still be decompiled and read. To prevent that, obfuscate it. 
11. In a Web application make sure to turn off detailed error pages. In ASP.NET, use the Application_Error event as your last line of defense for unhandled exceptions.
12. Keep your servers patched and up-to-date.

If you don’t understand what I’m talking about, come to Learning Tree course 940: Securing Web Applications, Services and Servers. Everything in that course applies whether you use the cloud or not. In the cloud or not, your applications need to be secured by you!

If you’re already doing the things above, your application will be pretty secure in the cloud or on your network.

If you’re not doing the things above, your application is already not secure. So, move it to the cloud and save some money. It might come in handy for the lawsuit.

David Movshovitz claims “Data governance concerns hinder the adoption of SaaS applications” in a preface to his SaaS Without the Data Security and Regulatory Concerns post of 7/8/2010:

The business benefits of using business applications delivered via software-as-a-service (SaaS) are compelling, and it's clear that this segment of the software market is growing steadily. Compared with traditional information technology systems, SaaS offers organizations lower costs (for deployment, operations and support), pay-as-you-go usage metering, faster implementation, less support overhead and effortless scalability.

Despite the compelling benefits of moving to SaaS, however, many enterprises are reluctant to make the move to the "cloud" due to serious concerns in the areas of data security, data privacy and regulatory compliance. Surveys conducted by analyst firms and publications consistently indicate that concerns regarding data security and regulatory compliance are among the top issues preventing the wider adoption of SaaS by enterprises.

It's no surprise that enterprises are so concerned about storing their business data with third parties as remote data hosting is sometimes legally prohibited in particular industries or jurisdictions, and leaks of sensitive data can prove embarrassing (or worse) to any company. Beyond fines and the other possible legal repercussions of a data breach, organizations face significant business risks. It doesn't matter who or what caused the breach; what matters is that the organization is perceived to have failed in its duty to care for its customers' confidential information. Furthermore, when an organization chooses to store its corporate data at a third party, the organization may be waiving its right to fourth amendment protection from unreasonable search and seizure; law enforcement could potentially seize a company's data from a SaaS vendor's servers without probable cause and without even informing the company that actually owns the data.

Current Data Security Solutions Fail to Address the Concerns
None of the data security solutions commonly used today satisfactorily addresses all of these concerns. While transmission of sensitive data is typically encrypted while in transit, the same sensitive data is stored in SaaS provider databases in plain text, highly vulnerable to various kinds of breaches, whether malicious or accidental. Firewalls provide some protection against outside hackers, but SaaS provider employees can still access the data, as could anyone who gains access to the application via the application layer (e.g., as occurs with identity theft).

The recent security breach of Apple's data stored on AT&T servers is a case in point. 114,000 e-mail addresses of iPad owners, including members of the U.S. military and executives at media, technology and finance companies, were exposed via an unsecured application-level back door. No matter how advanced the security mechanisms in place, no matter how diligent the company and its employees, and no matter how effective the security auditors, data stored in an unencrypted state may always be vulnerable to some kind of unauthorized exposure.

Why Not Store All SaaS Data in an Encrypted State?
One possible solution to this situation would be to encrypt all data for storage on a third-party provider's servers. Even if data were leaked, it would be meaningless, rendering the leak harmless.

While this method is useful for data archives, which do not need to be regularly accessed or processed, it's not viable for operational data because certain application processing would no longer be able to function on encrypted data. For example, an application would be unable to search through encrypted data or to sort it.

Implementing on-the-fly data decryption as part of the application's processes is a possible solution, but it poses two problems of its own. First, the computational overhead required for large-scale encryption-decryption operations is cost and performance prohibitive. Second, as long as the application is decrypting the data, all application-level data security vulnerabilities remain in full force.

A Revolutionary Solution: SaaS Without the Data Security Risks
A comprehensive approach to addressing this situation would be one in which all sensitive data is stored in an encrypted state on the cloud server, while unmodified SaaS applications function normally. This would ensure that no sensitive data could ever be accessed or leaked while in the cloud, completely eliminating the data security, data privacy and regulatory concerns of using SaaS applications.

However, as mentioned earlier, if the data remains encrypted at all times, how can the SaaS application function normally? And if the SaaS application remains unchanged, when and where is the data being encrypted?

A revolutionary solution involves an encryption scheme based on unique implementations of NIST-standard encryption algorithms (such as AES), which allow sensitive data elements to be encrypted in such a way that search and sort operations performed by the application remain unaffected.

This encryption occurs in a fully transparent background process (running on a reverse proxy server) on the customer side, ensuring that the SaaS application and those with access to it never have access to the encryption keys necessary to decrypt the data. When the encrypted data is returned to the client by the application server, all encrypted data elements are automatically identified and decrypted inside the organization's firewall.

The result is that end users are completely unaware of this background process, yet their data is completely unreadable (and therefore meaningless) when stored on the SaaS provider's servers. Database theft, accidental leaks, law enforcement subpoenas to the SaaS provider, and even identity theft all become harmless, and regulatory compliance is ensured, since all sensitive data remains undecipherable when transmitted and stored outside the enterprise firewall.

Of course, the technical details of implementing this approach are not trivial. Interpreting and processing all data traffic between any SaaS application and its Web clients in order to identify the data elements for encryption/decryption is challenging. Implementing encryption algorithms that allow an unmodified application to successfully search and sort encrypted data is even more so. Nevertheless, this approach already exists and promises to change the security landscape of cloud applications.

If, despite the advantages of SaaS, you are holding off moving your organization to the cloud due to data security or regulatory concerns, you owe it to yourself to take a good look at an encryption  solution along the lines of the one described in this article.

Chris Hoff (@Beaker) wrote a retrospective The Security Hamster Sine Wave Of Pain: Public Cloud & The Return To Host-Based Protection… post on 7/7/2010:

This is a revisitation of a blog I wrote last year: Incomplete Thought: Cloud Security IS Host-Based…At The Moment.

I use my ‘Security Hamster Sine Wave of Pain” to illustrate the cyclical nature of security investment and deployment models over time and how disruptive innovation and technology impacts the flip-flop across the horizon of choice.

To wit: most mass-market Public Cloud providers such as Amazon Web Services rely on highly-abstracted and limited exposure of networking capabilities.  This means that most traditional network-based security solutions are impractical or non-deployable in these environments.

Network-based virtual appliances which expect generally to be deployed in-line with the assets they protect are at a disadvantage given their topological dependency.

So what we see are security solution providers simply re-marketing their network-based solutions as host-based solutions instead…or confusing things with Barney announcements.

Take a press release today from SourceFire:

Snort and Sourcefire Vulnerability Research Team(TM) (VRT) rules are now available through the Amazon Elastic Compute Cloud (Amazon EC2) in the form of an Amazon Machine Image (AMI), enabling customers to proactively monitor network activity for malicious behavior and provide automated responses.

Leveraging Snort installed on the AMI, customers of Amazon Web Services can further secure their most critical cloud-based applications with Sourcefire’s leading protection. Snort and Sourcefire(R) VRT rules are also listed in the Amazon Web Services Solution Partner Directory, so that users can easily ensure that their AMI includes the latest updates.

As far as I can tell, this means you can install a ‘virtual appliance’ of Snort/Sourcefire as a standalone AMI, but there’s no real description on how one might actually implement it in an environment that isn’t topologically-friendly to this sort of network-based implementation constraint.

Since you can’t easily “steer traffic” through an IPS in the model of AWS, can’t leverage promiscuous mode or taps, what does this packaging implementation actually mean?  Also, if  one has a few hundred AMI’s which contain applications spread out across multiple availability zones/regions, how does a solution like this scale (from both a performance or management perspective?)

I’ve spoken/written about this many times:

Where Are the Network Virtual Appliances? Hobbled By the Virtual Network, That’s Where… and

Dear Public Cloud Providers: Please Make Your Networking Capabilities Suck Less. Kthxbye

Ultimately, expect that Public Cloud will force the return to host-based HIDS/HIPS deployments — the return to agent-based security models.  This poses just as many operational challenges as those I allude to above.  We *must* have better ways of tying together network and host-based security solutions in these Public Cloud environments that make sense from an operational, cost, and security perspective.

Comments (2)

Chris Hoff (@Beaker) posted The Classical DMZ Design Pattern: How To Kill Security In the Cloud on 7/7/2010:

Every day I get asked to discuss how Cloud Computing impacts security architecture and what enterprise security teams should do when considering “Cloud.”

These discussions generally lend themselves to a bifurcated set of perspectives depending upon whether we’re discussing Public or Private Cloud Computing.

This is unfortunate.

From a security perspective, focusing the discussion primarily on the deployment model instead of thinking holistically about the “how, why, where, and who” of Cloud, often means that we’re tethered to outdated methodologies because it’s where our comfort zones are.

When we’re discussing Public Cloud, the security teams are starting to understand that the choice of compensating controls and how they deploy and manage them require operational, economic and architectural changes.  They are also coming to terms with the changes to application architectures as it relates to distributed computing and SOA-like implementation.  It’s uncomfortable and it’s a slow-slog forward (for lots of good reasons,) but good questions are asked when considering the security, privacy and compliance impacts of Public Cloud and what can/should be done about them and how things need to change.

When discussing Private Cloud, however, even when a “clean slate design” is proposed, the same teams tend to try to fall back to what they know and preserve the classical n-tier application architecture separated by physical or virtual compensating controls — the classical split-subnet DMZ or perimeterized model of “inside” vs “outside.” They can do this given the direct operational control exposed by highly-virtualized infrastructure.  Sometimes they’re also forced into this given compliance and audit requirements. The issue here is that this discussion centers around molding cloud into the shape of the existing enterprise models and design patterns.

This is an issue; trying to simultaneously secure these diametrically-opposed architectural implementations yields cost inefficiencies, security disparity, policy violations, introduces operational risk and generally means that  the ball doesn’t get moved forward in protecting the things that matter most.

Public Cloud Computing is a good thing for the security machine; it forces us to (again) come face-to-face with the ugliness of the problems of securing the things that matter most — our information. Private Cloud Computing — when improperly viewed from the perspective of simply preserving the status quo — can often cause stagnation and introduce roadblocks.  We’ve got to move beyond this.

Public Cloud speaks to the needs (and delivers on) agility, flexibility, mobility and efficiency. These are things that traditional enterprise security are often not well aligned with.  Trying to fit “Cloud” into neat and tidy DMZ “boxes” doesn’t work.  Cloud requires revisiting our choices for security. We should take advantage of it, not try and squash it.

Comments (3)

Chris Hoff (@Beaker) continues his 7/7/2010 roll with CLOUDINOMICON: Idempotent Infrastructure, Survivable Systems & Bringing Sexy Back to Information Centricity at Black Hat 2010:

I’m hurrying to polish up the next in my series of virtualization and cloud computing security presentations which I’m going to give at this year’s Black Hat conference in Las Vegas on July 29th.  I’m speaking from 10-11am on day two up next to folks like Jeremiah Grossman, Moxie Marlinspike, Ivan Ristic, Haroon Meer…quite the “power hour” as someone said on the Twitter.

At any rate, I started the series a couple of years ago with the following progression:

1. The Four Horsemen of the Virtualization Security Apocalypse
2. The Frogs Who Desired a King: A Virtualization & Cloud Computing Fable Set To Interpretative Dance
3. Cloudifornication: Indiscriminate Information Intercourse Involving Internet Infrastructure

I proudly present numero quatro:

CLOUDINOMICON: Idempotent Infrastructure, Survivable Systems & Bringing Sexy Back to Information Centricity

Mass-market, low-cost, commodity infrastructure-as-a-Service Cloud Computing providers abstract away compute, network and storage and deliver hyper-scaleable capabilities.

This “abstraction distraction” has brought us to the point where the sanctity and security of the applications and information transiting them are dependent upon security models and expertise rooted in survivable distributed systems, at layers where many security professionals have no visibility.

The fundamental re-architecture of the infostructure, metastructure and infrastructure constructs in this new world forces us back to the design elements of building survivable systems focusing on information centricity — protecting the stuff that matters most in the first place.

The problem is that we’re unprepared for what this means and most practitioners and vendors focused on the walled garden, perimeterized models of typical DMZ architecture are at a loss as to how to apply security in a disintermediated and distributed sets of automated, loosely-coupled resources.

We’re going to cover the most salient points relating to how IaaS Cloud architecture shifts how, where and who architects, deploys and manages security in this “new world order” and what your options are in making sustainable security design decisions.

It’s progressing nicely.  Hope to see you there (and at Defcon)

Phil Wainwright claims “Cloud users shouldn’t just worry about security. SLAs should cover performance, availability, privacy, cost and many other criteria. In an ideal world, those service levels should be reconfigurable on demand, too. But the technology’s not there yet” in a summary of his Policy and the cloud post to ZDNet’s Sofware as Services blog:

Every time I get into a discussion about security and trust in cloud computing these days, I end up talking about service level agreements. People considering cloud computing rightly worry about whether their data is going to be secure, and private, and accessible when they need it. The umbrella term they use for that is ’security’, but their worries encompass a broad range of performance, security, privacy, operational and cost criteria. That’s why I end up talking about SLAs — the contracts that govern the provider’s commitment to meet all those various criteria. It turns out that, once you drill down into what people really want, the answer is much more granular and textured than a single metric about security, privacy, or whatever. We’re actually talking about a framework for governance across a broad range of policy settings.

The discussion then rapidly leads into the realization that service levels as they’ve traditionally been defined and measured aren’t fit for purpose in this new environment. SLAs, like everything else in the classic IT realm, have been designed on the assumption of a one-off, upfront determination of a set of static requirements that will remain the same throughout the lifetime of the contract. To make matters worse, those requirements are defined in terms of the technology infrastructure, specifying feeds and speeds for engine-room components that may in the end have very little relevance to the ability to conform to business policy objectives.

These fixed SLAs are out of kilter with the dynamic, elastic nature of the cloud environment. If the cloud is all about delivering IT on demand, then why can’t the service levels be on-demand, too? “You don’t need to be paying top-whack five-nines every time,” I opined in discussion the other day with Eddie Budgen, VP of technology services at Sensible Cloud, a start-up that specializes in business-driven SLAs for cloud computing. While some applications are so mission critical that anything less than five-nines reliability is out of the question, others can get by with much lower levels of continuous uptime. There may be differential requirements for other criteria, ranging from security and privacy to response times, and they can vary not only by application but also by user, by geography or by date and time of day.

The trouble is, such dynamic SLAs are only possible with automation. A traditional SLA will set static limits, and then the provider or the customer (often both independently) can program their monitoring tools to send out alerts as those limits get close. But if the limits are constantly changing in response to an array of interlocking policies, the monitoring tools have to be constantly reprogrammed to react to those changes. That level of responsiveness rules out manual processes in favor of configurable automation.

The infrastructure has to be flexible enough to respond to those policy changes, too, if we want an application to ramp up and down along a variable cost-availability matrix. At present, cloud providers don’t offer much in the way of service level choice — many of them even avoid specifying any SLAs at all. In a discussion at a CompTIA workshop at last week’s Cloud World Forum in London, one speaker said that cloud computing is bringing the disciplines of assembly line manufacturing to IT, but my response was that it’s still a primitive process in which the customer gets, in Henry Ford’s words, “any color … so long as it is black.”

At present, the only way to change your cloud computing service levels is to move from one cloud provider to another. Without interoperability standards and a common language to describe service levels, that’s a custom process that’s hard to automate. Nor is it in the interests of providers to rush to create standards that make it easier for customers to shop around for cloud services on the fly. Yet customers will want that flexibility and so it’s only a matter of time before providers start to offer enough visibility and control to give them real choice over service levels — at first on a proprietary basis within individual cloud infrastructures, and later on across multiple clouds, as standards gradually evolve.

Much of this is still at a research stage — there’s an EU sponsored program called SLA@SOI that’s looking into some of the technologies that may be required. In the meantime, customers of cloud computing are pretty much stuck with paper contracts and fixed SLAs (and often a struggle to get satisfaction even at that level). But next time you’re wondering what security guarantees you should be asking your cloud provider for, just remember there may be one or two other policy criteria you should be worrying about too.

Nicole Harris asserts “Healthcare providers still have many reservations about the security of cloud computing for electronic medical records and mission-critical apps” as a preface to her Microsoft Aims To Alleviate Health IT Cloud Concerns post of 7/6/2010 to InformationWeek’s HealthCare blog:

Many healthcare providers have questions and doubts about adopting cloud computing for administration and hosting of their healthcare information. Steve Aylward, Microsoft's general manager for U.S. health and life sciences, has encouraged healthcare IT decision makers to embrace the technology, which he said could help them improve patient care and provide new delivery models that can increase efficiency and reduce costs.

"Just about everyone I know in healthcare is asking the same question: "What can cloud computing do for me?" Aylward writes in a June 28 blog post. [Link added.]

"Plenty," Aylward answers. "The cloud can allow providers to focus less on managing IT and more on delivering better care: It can, for instance, be used to migrate e-mail, collaboration, and other traditional applications into the web. It can also be used to share information seamlessly and in near-real-time across devices and other organizations," Aylward explains.

Generally defined as anything that involves delivering hosted services over the internet, a cloud computing model that offers a software-as-a-service platform is increasingly being offered to healthcare delivery organizations, especially small and medium-size physician practices that are budget constrained and have few technical administrators on staff.

What has helped information technology managers at health delivery organizations take a closer look at cloud computing, however, is the Obama administration's objective to move medical practices and hospitals away from paper-based systems and onto digitized records. Primarily through the Health Information Technology for Economic and Clinical Health (HITECH) Act, the government has established programs under Medicare and Medicaid to provide incentive payments for the "meaningful use" of certified electronic medical record (EMR) technology.

The government's drive to have every American provided with an EMR by 2014 will mean that digitized clinical data is expected to grow exponentially. However, several doctors contacted by InformationWeek say that, even with those considerations, they are in no rush to outsource the maintenance of their important records and they have delayed their decisions to put sensitive information, such as their EMR systems, onto a cloud-based computing model.

Dr. Michael Lee, a pediatrician and director of clinical informatics for Atrius Health, an alliance of five nonprofit multi-specialty medical groups with practice sites throughout eastern Massachusetts, said that while he recognizes that cloud computing can be a cheaper and more practical model, especially for non-mission-critical applications, he is waiting to see what improvements in security will take place over the next five to 10 years before supporting a decision to put high-level data on cloud computing technology.

"The only cloud computing that we would contemplate at the moment is in the personal health record space, so that patients would own the dimension in the cloud in terms of where they want to store or access information," Lee said. …

Page 2:  Cloud Is Not Secure Enough
1 | 2 | 3 Next Page »

When you consider the almost weekly reports of hundreds of thousands of patient records being lost from on-premises EHR and related storage, the security of cloud-based storage is likely to be better than that of current private data centers. (See Health Net Settles Breach Suit of 7/7/2010 as an egregious example.)

<Return to section navigation list> 

Cloud Computing Events

See Chris Hoff’s (@Beaker) CLOUDINOMICON: Idempotent Infrastructure, Survivable Systems & Bringing Sexy Back to Information Centricity for Black Hat 2010 on 7/29/2010 in the Cloud Security and Governance section above. The session schedule here.

The CloudTweaks blog reported on 7/7/2010 a Live Webinar From CloudSwitch on Using the Cloud Instead of Scaling Your Colo scheduled for 7/13/2010:

CloudSwitch, Inc. today announced a live webinar on July 13, 2010 to demonstrate how cloud computing offers dramatic cost savings and flexibility improvements compared to traditional hosting providers or colocation facilities. The cloud enables rapid provisioning and scaling to meet peak usage, while eliminating the overhead of managing power, network connectivity, and physical hardware. Until now, enterprises considering the cloud have been faced with concerns about the security of their data and communications, and the hurdles of re-building applications and management tools for a specific cloud environment.

In this live webinar, you’ll learn how CloudSwitch’s breakthrough software lets enterprises kick the colo habit by running applications in the cloud with the security, confidence, and management tools they rely on. CloudSwitch will present several customer stories showcasing how real enterprises are using CloudSwitch to deploy and run applications in the cloud for rapid development and peak-period scaling — relieving them from the headaches and costs of maintaining traditional hosting environments. In a live demonstration, you’ll see how CloudSwitch automatically creates secure connectivity into the cloud allowing cloud-based applications to remain integrated with backend systems, and learn how to host your applications in the cloud with point-and-click simplicity and no modifications.

Join us for a free webinar to learn how CloudSwitch enables you to:

• Move hosting infrastructure into the cloud with no modifications
• Dramatically lower operational costs
• Automatically encrypt all network traffic and storage
• Secure and control access to hosted applications
• Maintain compliance with internal security standards

Presenter: Ellen Rubin, Founder & VP Products
Presenter: Damon Miller, Director of Technical Field Operations.

Register here.

Kevin McLaughlin cites 4 Things You'll Hear At Microsoft's Partner Conference, three of which are about Windows Azure and BPOS, in his 7/6/2010 post to Channel Web:

Microsoft (NSDQ:MSFT) is gearing up for its annual Worldwide Partner Conference, to be held from July 11-15 in Washington, D.C. WPC is Microsoft's channel Super Bowl, but this one will carry arguably greater importance than previous WPCs because it'll give Microsoft executives a chance to articulate what the company's new "all-in" stance on cloud computing means for channel partners.

Given that cloud computing and the channel are viewed in some IT industry circles as diametrically opposed forces, Microsoft partners will be hanging on every word. They'll also learn more about Microsoft's overhauled partner competency structure and how it could convey competitive advantages to partners who've long been searching for the Holy Grail of differentiation.

The nation's capital also happens to be birthplace of departing Microsoft Channel Chief Allison Watson, who's moving to a new position as corporate vice president of the U.S. Business & Marketing Organization. Watson, leader of the Microsoft channel program since 2002, recently took time out from WPC preparations to chat with CRN about what partners can expect to hear at the event.

1. Microsoft Is All-In With Cloud Computing

"All-in" has replaced "Software Plus Services" as Microsoft's marketing mantra. The change reflects Microsoft's desire to establish itself as a cloud pioneer and perhaps a bit of overzealousness that stems from its late arrival to the cloud market.

At WPC, partners can expect to hear Microsoft cite examples of channel partners moving their IT operations and software development to the cloud, although Watson acknowledges that some VARs are still trying to figure out how the cloud fits with their businesses.

"There's a tremendous amount of partner interest in the cloud," Watson said. "But in a room of traditional small and midmarket VARs, I'd say 30 percent will get it, and 70 percent aren’t sure yet about the opportunities. That's why it's important for partners to connect with other partners, to see in every solution there is a cloud story."

Microsoft says it's not abandoning Software Plus Services entirely, but on-premise software probably isn't going to get much attention at WPC. Instead, Microsoft will show its newfound dedication to the cloud with glimpses into its R&D roadmap. For example, inside Microsoft, 70 percent of developers are actively developing cloud technologies, a figure that will grow to 90 percent by the end of fiscal 2011 next June, according to Watson.

Windows Azure, which launched as a paid service in February, now has more than 10,000 customers, as well as 4,900 apps that developers have ported over or built entirely on the Azure platform, Watson said.

"Azure is ideal for spatial data infrastructure solutions that require huge amount of data processing, because it’s able to take advantage of scale-out and scale-up infrastructure for managing large volumes of data," she said.

For customers that don't have huge data consumption needs, the costs of hosting on Azure are 30 percent less than traditional hosting solutions, because customers are only charged for what they consume, Watson added.

Next: The Cloud And SMB Partners

1 | 2 | Next >>

tbtechnet reported Virtual Boot Camp VII – This Time it’s TWO will start on 7/7/2010:

Virtual Boot Camp USA

• Learn the Windows Azure platform. It’s new, yet still familiar. No steep learning curve.
• With Azure you pay only for what you use, scale up when you need capacity and pull it back when you don’t.

NO credit card required.

Windows Azure pass is provided valid for July 7th until July 20th 2010 so you can put Windows Azure and SQL Azure through their paces.

IMPORTANT: The Windows Azure virtual boot camp pass is valid ONLY for July 7th until July 20th 2010 .

The pass will automatically expire on July 20th 2010

Follow these steps:

1. Request a Windows Azure One Month Pass here. (This link will start your email client)
   Please: only one request per person - allow other developers to also try out Windows Azure.

2. Sign in to the Windows Azure Developer Portal and use the pass to access your Windows Azure account.

3. Please note: your Windows Azure application will automatically de-provision at the end of the virtual boot camp.

   • Since you will have a local copy of your application, you will be able to publish your application package on to Windows Azure after the virtual boot camp using one of the Windows Azure Offers.

   • See the Azure Offers here

4. For USA developers, no-cost phone and email support during and after the Windows Azure virtual boot camp with the Front Runner for Windows Azure program

5. Startups - get low cost development tools and production licenses with BizSpark - join here

6. Get the Tools

   • Download Microsoft Web Platform Installer

   • Download Windows Azure Tools for Microsoft Visual Studio

7. Learn about Azure

   • Why Windows Azure - learn why Azure is a great cloud computing platform with these fun videos

   • Learn how to put up a simple application on to Windows Azure

   • Learn about PHP on Windows Azure

   • Take the Windows Azure virtual lab

   • Read about Developing a Windows Azure Application

   • View the series of Web seminars designed to quickly immerse you in the world of the Windows Azure Platform

8. Dig Deeper into Windows Azure

   • Download the Windows Azure Platform Training Kit

Just in case you missed the message: The pass will automatically expire on July 20th 2010.

<Return to section navigation list> 

Other Cloud Computing Platforms and Services

Audrey Watters reported Amazon S3 Bucket Policies Allow Increased Control and Security on 7/7/2010:

One of the lingering problems with adoption of cloud computing has been the issue of facilitating access - both for the end-user and for the IT professional.

In a move that addresses these concerns, Amazon Web Services announced yesterday that it had added support for Bucket Policies. These policies will provide a single mechanism for managing access to the Amazon S3 buckets and for the objects stored in them. These policies are expressed using Amazon's Access Policy Language, which will centralize and refine permissions management.

The Old Controls

Prior to the announcement, there were two access control mechanisms for Amazon S3: query string authentication and the Access Control List. The former creates a URL that will grant temporary access to a bucket. The latter provides for selective access, with certain permissions - read, write, read ACL, write ACL - designated for certain people. One of the drawbacks to either of these methods was that new objects added to a bucket required their access controls be set individually.

The New Controls

So while these ACLs grant permission on an object-by-object basis, the new bucket policies allow a much more granulate level of control. Permissions can be added or denied across all or a subset of the objects within a single bucket. The policies can include references to IP addresses and ranges, dates, the HTTP referrer, and transports (http and https).

As AWS notes in their explanation of the new bucket policies, this allows you to, for example, allow write access to a particular S3 bucket only from your corporate networ during business hours from your custom application (as identified by a user agent string).

These new bucket policies are designed to facilitate the way in which information is stored and accessed in the cloud, adding to IT's security and management toolkit

Nicole Hemsoth interviews Eucalyptus CEO Marten Mickos in her Eucalyptus Chief Talks Future After $20 Million Infusion post of 7/7/2010 to the HPC in the Cloud blog:

The holiday weekend that just ended was probably just a little peppier for private cloud software vendor, Eucalyptus Systems, following a fresh infusion of cash to the tune of $20 million. This dwarfs the first round of funding the company received, which was $5.5 million, most of which sits untouched to date.

According to CEO of Eucalyptus Systems, Marten Mickos, during a phone interview on Friday, “We had several venture capitalists who were knocking on our door every day but we only spoke with a small group of them, mostly they were the VCs who came recommended or we knew. We very quickly settled on New Enterprise Associates (NEA) but I will say, I’ve never raised capital this quickly and efficiently before—it was a breeze.”

Not all companies can call the funding process a breeze, but not all are as uniquely positioned as Eucalyptus Systems since their software, which is used widely in its open source incarnation, has been gaining popularity. Eucalyptus has its roots directly in HPC, where it began as a research project led by Rich Wolksi who was then at UC Santa Barbara. The open source project grew wings in mid-2009 following its first round of capital from Benchmark Capital, which allowed Wolksi to take his project from open source to the enterprise. Far from being a direct mention of a tree that the founder particularly liked, Eucalyptus is an acronym for “Elastic Utility Computing Architecture for Linking Your Programs to Useful Systems” which is, well, pretty much exactly what it does—hence its adoption in both its open source and commercial form.

While it might seem that this kind of injection of funding would spur a new mission or vision statement on the part of a CEO with a high profile, this is not necessarily the case—at least not to the degree one might think. Trying to talk about a company’s vision with Mickos is complicated since he does not believe that vision is what makes a company succeed, especially as he noted, since he already agreed with the trajectory the company was taking when he took the position in March. According to Mickos, “A CEO doesn’t need a vision, a CEO needs ears to listen to customers and markets. That’s much more important than vision. If you look at some of the most spectacular failures in IT they were led by very visionary people—but those people didn’t listen. That’s my philosophy.”

So with the vision statement all taken care of, we were left to focus on more important matters as they relate to the influx of funding and the roadmap for the next two years that the Eucalyptus CEO discusses in some detail.

During the course of our 45-minute discussion, Mickos was asked several specific questions about the current state and future of the company now that it is in a better competitive position. We also spend time toward the end of our discussion talking about some of the lesser-known concepts that form the backbone of Eucalyptus and address what Mickos calls “misconceptions” about Eucalyptus, especially in terms of a few of its more recent decisions.

HPCc: The tenure of your involvement with Eucalyptus is relatively short; how did you come to the company—what foundational ideas did you bring to the table and how might these be shaping Eucalyptus, especially now?

Mickos: So when I left Sun over a year ago I asked everyone when I left, “so what’s bigger than open source” and some were just joking and said “closed source” (laughs) but the two serious ones were the mobile internet and the other was cloud computing. At the time, I didn’t pay much attention but over successive months I came to realize these are really two massive shifts in IT. At one point, I realized I needed to be involved with the cloud but on the infrastructure side, which is close to my heart. I visited the Eucalyptus team and fell in love with the team and knew the market opportunity and original DNA of the team are wonderful. I applied and begged for them to bring me on and they did.

I think the client-server paradigm has been one of the biggest historical IT shifts up until now, one which happened in the late 80s—it was a great paradigm that worked for a long time. Then came the web, which replaced the client part of client-server, which means we stopped having thick clients and instead used web browsers. But on the server side we stuck to the same architecture. Now with cloud computing, we are replacing and shifting away from the server part and building a new infrastructure for running applications or services that are scalable in a way never seen before. It is a major, major shift. It could take a long time to fulfill, maybe five to ten years, but it will be massive.

HPCc: If we are to look at your involvement with the business model, how have you shifted the vision or focus of the company since when you took over in March, if at all?

Page:  1  of  4
1 | 2 | 3 | 4 All »

Redmonk’s James Governor adds to the analysis stack with his EMC Big Data Play Continues: Greenplum Acquisition post of 7/7/2010:

I wrote recently about VMware’s emerging Data Management play after the announcement the firm was hiring Redis lead developer Salvatore Sanfillipo.

While [CEO Paul] Maritz may say VMware isn’t getting into the database business, he means not the relational database market. The fact is application development has been dominated by relational- Oracle on distributed, IBM on the mainframe – models. Cloud apps are changing that. As alternative data stores become natural targets for new application workloads VMware does indeed plan to become a database player, or NoSQL player, or data store, or whatever you want to call it.

We have been forcing round holes into square pegs with object/relational mapping for years, but the approach is breaking down. Tools and datastores are becoming heterodox. something RedMonk has heralded for years.

Now comes another interesting piece of the puzzle. EMC is acquiring Greenplum – and building a new division around the business, dubbed Data Computing Product Division. While Redis is a “NoSQL” data store, Greenplum represents a massively parallel processing architecture designed to take advantage of the new multicore architectures with pots of RAM: its designed to process data into chunks for parallel processing across these cores. While Greenplum has a somewhat traditional “datawarehouse” play – it also supports MapReduce processing. EMC will be competing with the firms like Hadoop packager Cloudera [client] and its partners such as IBM [client]. Greenplum customers include Linkedin, which uses the system to support its new “People You May Know” function.

There is a grand convergence beginning between NoSQL and distributed cache systems (see Mike Gualtieri’s Elastic Cache piece). It seems EMC plans to be a driver, not a fast follower. The Hadoop wave is just about ready to crash onto the enterprise, driven by the likes of EMC and IBM. Chuck Hollis, for example, points out Greenplum would make a great pre-packaged component VBlock for VMware/EMC/Cisco’s VCE alliance – aimed at customers building private clouds. Of course Cisco is likely to make its own Big Data play anytime soon… That’s the thing with emergent, convergent markets- they sure make partnering hard. But for the customer the cost of analysing some types of data is set to fall by an order of magnitude, while query performance improves by an order of magnitude. Things are getting very interesting indeed.

Julie Bort asserted “Over the past month, App Engine was speedier than both Azure and EC2, but Azure is fighting back” in her Microsoft's cloud is slower than Google's, Amazon's, benchmark says post of 7/6/2010 to NetworkWorld’s Microsoft Subnet blog:

Over the past month, Google's cloud, App Engine, performed faster than all of the other major clouds, including Microsoft's Azure. Azure was also consistently slower than at least one of Amazon's EC2 data centers, according to a live benchmarking service known as CloudSleuth.com.

Ironically, I was poking into cloud benchmarking hoping to learn that Microsoft Azure was faster than both Amazon and Google. I learned about the CloudSleuth.com from a blog post on MSDN when a Microsoft employee was bragging that Azure was outperforming the others this week. That result must have been a blip in the data because as I sliced the data, Azure never landed on top.

Google's average was about 1 second faster than Azure's, at least for the last 30 days.

^{Click to enlarge graphic.}

CloudSleuth was created as a free online service by Compuware. These are the same folks that built the Gomez benchmarking tests that monitor Web app performance metrics such as comparing the same Web site loading into different browsers. (Compuware is a vendor of application performance monitoring tools.) Ergo, CloudSleuth uses the Gomez Performance Network (GPN) to measure the performance of an identical sample application running on several popular cloud service providers.

One day soon, CloudSleuth hopes to let users upload and compare their own cloud app to be benchmarked across the participating cloud vendors.

While playing with this site, I noticed that in the past few hours and days, Azure has been performing faster than all the other clouds except OpSource. (By the way, CloudSleuth names OpSource as a partner, though I can't say that this partnership affects the benchmarking results. The 30-day result clearly showed Google App Engine as faster than OpSource, but much of the time, OpSource lands on top.)

CloudSleuth shares all the details about the app used to benchmark the tests. It uses the default recommended configurations for each cloud service, although there are inherent differences between "old fashioned" hosting providers today known as Infrastructure as a Service (IaaS) and Platform as a Service providers (PaaS) which includes Azure and App Engine. The sample app is an e-commerce Web site.

Speed isn't the only consideration when comparing cloud services. But it is interesting to see that during any given period, an IaaS isn't always faster than a PaaS and vice versa.

<Return to section navigation list> 

Technorati Tags: Windows Azure,Windows Azure Platform,Azure Services Platform,Azure Storage Services,Azure Table Services,Azure Blob Services,Azure Drive Services,Azure Queue Services,SQL Azure Database,SADB,Open Data Protocol,OData,Azure AppFabric,Server AppFabric,Cloud Computing,EMC,Greenplum,Amazon Web Services,AWS,Eucalyptus,SQL Server Compact 4.0,Worldwide Partner Conference 2010,WPC 2010,GoGrid,The Archivist,Twitter,Sobees,Bing Maps