OakLeaf Systems: Windows Azure and Cloud Computing Posts for 7/16/2010+

Sunday, July 18, 2010

Windows Azure and Cloud Computing Posts for 7/16/2010+

A compendium of Windows Azure, Windows Azure Platform Appliance, SQL Azure Database, AppFabric and other cloud-computing articles.

Update 7/17/2010: Added update to My Using the SQL Azure Migration Wizard v3.1.3/3/1/4 with the AdventureWorksLT2008R2 Sample Database in the SQL Azure Database, Codename “Dallas” and OData section and added Bruno Terkaly’s Leverage Cloud Computing with Windows Azure and Windows Phone 7 – Step 1 to Infinity in the Windows Azure Infrastructure section, both marked ••.

Update 7/17/2010: Added items marked •.

Update 7/15/2010: Diagram above updated with Windows Azure Platform Appliance, announced 7/12/2010 at the Worldwide Partners Conference and Project “Sydney”

Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:

Azure Blob, Drive, Table and Queue Services
SQL Azure Database, Codename “Dallas” and OData
AppFabric: Access Control and Service Bus
Live Windows Azure Apps, APIs, Tools and Test Harnesses
Windows Azure Infrastructure
Windows Azure Platform Appliance (WAPA)
Cloud Security and Governance
Cloud Computing Events
Other Cloud Computing Platforms and Services

To use the above links, first click the post’s title to display the single article you want to navigate.

Cloud Computing with the Windows Azure Platform published 9/21/2009. Order today from Amazon or Barnes & Noble (in stock.)

Read the detailed TOC here (PDF) and download the sample code here.

Discuss the book on its WROX P2P Forum.

See a short-form TOC, get links to live Azure sample projects, and read a detailed TOC of electronic-only chapters 12 and 13 here.

Wrox’s Web site manager posted on 9/29/2009 a lengthy excerpt from Chapter 4, “Scaling Azure Table and Blob Storage” here.

You can now freely download by FTP and save the following two online-only PDF chapters of Cloud Computing with the Windows Azure Platform, which have been updated for SQL Azure’s January 4, 2010 commercial release:

Chapter 12: “Managing SQL Azure Accounts and Databases”
Chapter 13: “Exploiting SQL Azure Database's Relational Features”

HTTP downloads of the two chapters are available for download at no charge from the book's Code Download page.

Azure Blob, Drive, Table and Queue Services

Ryan Dunn explains Getting the Content-Type from your Registry for Windows Azure blobs in this 7/16/2010 post:

For Episode 19 of the Cloud Cover show, Steve and I discussed the importance of setting the Content-Type on your blobs in Windows Azure blob storage. This was was especially important for Silverlight clients. I mentioned that there was a way to look up a Content Type from your registry as opposed to hardcoding a list. The code is actually pretty simple. I pulled this from some code I had lying around that does uploads.

Here it is:

SQL Azure Database, Codename “Dallas” and OData

•• Updated My Using the SQL Azure Migration Wizard v3.1.3/3.1.4 with the AdventureWorksLT2008R2 Sample Database post of 1/23/2010 on 7/18/2010 to reflect upgrade from v.3.1.3 to v3.3.3:

George Huey’s SQL Azure Migration Wizard (SQLAzureMW) offers a streamlined alternative to the SQL Server Management Studio (SSMS) Script Wizard for generating SQL Azure schemas that conform to the service’s current Data Definition Language (DDL) limitations. You can download the binaries, source code, or both from CodePlex; a Microsoft Public License (Ms-PL) governs use of the software. Detailed documentation in *.docx format is available. The recent versions use SQL Server’s Bulk Copy Protocol (BCP) utility to populate SQL Azure tables with data from an on-premises database.

• Update 7/18/2010 9:00AM: George Huey updated SQLAzureMW from v.3.3.2 to v.3.3.3 on 7/17/2010 with 5,227 downloads of the new version on its first day. The last documentation release on 7/6/2010 covers v.3.3.

AppFabric: Access Control and Service Bus

• The “Geneva” Team reports the availability of the AD FS 2.0 Step-by-Step Guide: Federation with CA Federation Manager guide in a 7/16/2010 post:

We have published a step-by-step guide on how to configure AD FS 2.0 and CA Federation Manager to federate using the SAML 2.0 protocol. You can view the guide either as a web page or in docx format. This is the first in a series of these guides; the guides are also available on the AD FS 2.0 Step-by-Step and How To Guides page.

• Eugenio Pace’s What’s next? Tailspin goes mobile… post of 7/15/2010 reports the availability and content of the Windows Azure Architecture Guide – Part 2:

Now that the Windows Azure Architecture Guide – Part 2 is “done” (Done = content is complete and production is taking place. Production = index, covers, PDF conversion, etc), our team is ready for the next adventure.

Our fictitious ISV, Tailspin, is very innovative and wants to stay on the latest and greatest. Now they want to extend their flagship application: Tailspin Surveys, to mobile users. The essential idea being to allow its customers (Adatum, Fabrikam, etc) to publish surveys to people with Windows Phone 7 devices, so they can capture information from the field.

This is a “mobile front-end” interacting with a “cloud backend” scenario.

Note: Tailspin backend is mostly covered in the Windows Azure Guide, but we will extend it to support new user stories that are specific to the mobile client. So expect a different version included in the new guide.

Detailed scenario (hidden agenda items in red [in the original post])

TailSpin: the “SaaS” ISV (this is covered in WAAG Part 2)

Develops a multitenant “survey as a service” for a wide range of companies (with “Big IT” and with “Small IT”)

Hosts the application on Windows Azure (service endpoint for WP7 app to interact: REST? SOAP? Chunky? Chatty?)

Develops a WP7 application for “surveyors” (people taking surveys) (packaging the app)

Publishes the WP7 application to the Application Marketplace (Publishing the app, certification process, versions, trials and betas)

Fabrikam & Adatum (companies subscribing to TailSpin’s service)

(Self) design and launch surveys using TailSpin’s app (surveys now will have a target geography, used for “get surveys near me”)

Waits for collected information

Browses and analyzes results (added data-points captured by the phone: geo-location, voice, pictures, barcodes)

People with WP7: “surveyors”

Own a WP7 device (browse the Tailspin mobile website, download the app from the Application Marketplace)

They “work from home”

They subscribe to surveys that are targeted to the place they are (location based search, like “Surveys 40 miles from Redmond”)

They are notified of new surveys available that match their subscription criteria (push notifications)

They complete the surveys (UI design, app state, connectivity management, sending results back to the service-> service design, authenticating, being a “good citizen app”, resource management like battery levels)

In this scenario, the “surveyors” could answer for themselves, but our current preference would be that they’ll interview others or would provide multiple answers (think someone surveying homes or traffic patterns in different places or someone going from home to home in a neighborhood collecting information).

You can imagine a mechanism were there’s compensation for surveyors from the companies that submit the survey: Adatum could pay for every “x submitted surveys” or there might even be a “coupon” system: “every 10 surveys you get a discount coupon for the supermarket”.

As usual, the surveys application is just an excuse to discuss the design patterns behind the solution (all the red things above). The entire app will be functionally incomplete, but technically sound.

The current (draft) structure for the guide is the following:

Feedback of course, is greatly welcome!

• Zane Adam links brief video interviews in his Partners at WPC discuss benefits of SQL Azure and AppFabric post of 7/15/2010:

The Worldwide Partner Conference (WPC) is a great chance for us to sit down with industry partners and understand their latest plans for adopting cloud technologies and the opportunities they see with SQL Azure, Windows Azure AppFabric, and “Dallas” –as they adopt the Windows Azure Platform and deliver value to customers in new ways. I’ll be sharing some of these stories and usage scenarios with you by linking to some very short partner interviews, presentations, and demos captured onsite here at the conference.

First up, here is a link to Brian Rose at Infor, discussing the benefits they see with SQL Azure and Windows Azure AppFabric. Infor is a 2 Billion dollar per year company with more than 70,000 customers worldwide. We are pleased to see Infor adopt the Windows Azure Platform!

Other interviewees are Darren Guarnaccia from Sitecor, Jason Popillion from GCommerce, Rainer Stropek from Software Architects and Rob McGreevy from Invensys.

Come learn how SQL Azure, AppFabric, and the Windows Azure platform will enable you to grow your revenue and increase your market reach. Learn how to build elastic applications that will reduce costs and enable faster time to market using our highly available, self-service platform. These apps can easily span from the cloud to the enterprise. If you are either a traditional ISV looking to move to the cloud or a SaaS ISV who wants to get more capabilities and a larger geo-presence, this is the session that will show you how.

Note that video is missing from 00:00:00 to 00:03:30; use the slider. (Repeated from the SQL Azure Database, Codename “Dallas” and OData section.)

• Pluralsight’s Matt Millner wrote an 11-page Service Bus Technologies from Microsoft – Better Together whitepaper, which became available for download on 7/16/2010 for Microsoft. From the abstract:

Messaging for the enterprise and beyond

Applies to: The BizTalk ESB Toolkit 2.0 and the Windows Azure AppFabric Service Bus

Abstract: This whitepaper provides an overview of the Microsoft^® BizTalk^® Enterprise Service Bus Toolkit (BizTalk ESB Toolkit) 2.0 and the Windows® Azure AppFabric Service Bus. The BizTalk ESB Toolkit is a set of tools built on top of Microsoft BizTalk Server (starting with BizTalk Server 2009) that provide a more loosely coupled messaging and service composition. The AppFabric Service Bus is a message relay service hosted in the cloud, and is one part of the Windows Azure AppFabric. In this paper, you will be introduced to each of these technologies and provide examples of how each can be applied. For a more detailed discussion of each technology, see the Additional Resources section at the end of this paper.

Aaron Skonnard presented Introducing AppFabric: Moving .NET to the Cloud by Aaron Skonnard to Microsoft Dev Days on 4/11/2010. Here’s the description of the 01:15:02 session:

Companies need infrastructure to integrate services for internal enterprise systems, services running at business partners, and systems accessible on the public Internet. And companies need be able to start small and scale rapidly. This is especially important to smaller businesses that cannot afford heavy capital outlays up front. In other words, companies need the strengths of an ESB approach, but they need a simple and easy path to adoption and to scale up, along with full support for Internet-based protocols. These are the core problems that AppFabric addresses, specfically through the Service Bus and Access Control services.

This session is presented by Aaron Skonnard during Microsoft DevDays 2010 in The Hague in The Netherlands.

<Return to section navigation list>

Live Windows Azure Apps, APIs, Tools and Test Harnesses

•• Bruno Terkaly’s Leverage Cloud Computing with Windows Azure and Windows Phone 7 – Step 1 to Infinity post of 5/17/2010 begins:

I am about to discuss a series of technologies that will be around for the next 5 to 10 years. I am talking about Cloud and Mobile Computing.

You will see detailed code about how to build applications. I want to discuss how to create and expose data through service oriented approaches, hosted in Windows Azure. I want to show you how to consume SQL Azure. …

The Value Proposition of Windows Azure. Businesses can rent access to applications and IT infrastructure that reside on the Internet, pay for them on a subscription or per-use basis and provide employees with access to information from anywhere at any time with nothing more than a connected device. …

The World of Private Clouds. Private cloud services run in datacenters on company property managed by corporate IT staffs. Private clouds address the security concerns of large enterprises. They're scalable, growing and shrinking as needed. They're also managed centrally in a virtualized environment.

…

What are we going to cover? I am going to begin a series of posts that illustrate how to leverage power of cloud computing on Windows Phone 7. My posts will show you how to make use of many technologies:

SQL Azure

Windows Azure

WCF Services

Silverlight LINQ

And various framework and language technologies

I’d like to hear from you. Shoot me an email at bterkaly@microsoft.com and let me hear your thoughts.

Bruno’s a Microsoft developer evangelist for the San Francisco Bay Area. Read his complete post. Here’s a link to Design Resources for Windows Phone (as of 7/12/2010) from MSDN.

•• See Phani Raju announced Windows Phone OData sample application updated for July 2010 Beta release on 7/13/2010 in the SQL Azure Database, Codename “Dallas” and OData section.

Brian Hitney brings developers up to date with @home with Windows Azure: Behind the Scenes of 7/16/2010:

As over a thousand of you know (because you signed up for our Webcast series during May and June), my colleague Jim O’Neil and I have been working on a Windows Azure project – known as @home With Windows Azure – to demonstrate a number of features of the platform to you, learn a bit ourselves, and contribute to a medical research project to boot. During the series, it quickly became clear (…like after the first session) that the two hours was barely enough time to scratch the surface, and while we hope the series was a useful exercise in introducing you to Windows Azure and allowing you to deploy perhaps your first application to the cloud, we wanted (and had intended) to dive much deeper.

So enter not one but two blog series. This introductory post appears on both of our blogs, but from here on out we’re going to divide and conquer, each of us focusing on one of the two primary aspects of the project. Jim will cover the application you might deploy (and did if you attended the series), and I will cover the distributed.cloudapp.net application, which also resides in Azure and serves as the ‘mothership’ for @home with Windows Azure. Source code for the project is available, so you’ll be able to crack open the solutions and follow along – and perhaps even add to or improve our design.

You are responsible for monitoring your own Azure account utilization. This project, in particular, can amass significant costs for CPU utilization. We recommend your self-study be limited to using the Azure development fabric on your local machine, unless you have a limited-time trial account or other consumption plan that will cover the costs of execution.

So let’s get started. In this initial post, we’ll cover a few items

Project history

Folding@home overview

@home with Windows Azure high-level architecture

Prerequisites to follow along

Project history

Jim and I have both been intrigued by Windows Azure and cloud computing in general, but we realized it’s a fairly disruptive technology and can often seem unapproachable for many of your who are focused on your own (typically on-premises) application development projects and just trying to catch up on the latest topical developments in WPF, Silverlight, Entity Framework, WCF, and a host of other technologies that flow from the fire hose at Redmond. Walking through the steps to deploy “Hello ~~World~~ Cloud” to Windows Azure was an obvious choice (and in fact we did that during our webcast), but we wanted an example that’s a bit more interesting in terms of domain as well as something that wasn’t gratuitously leveraging (or not) the power of the cloud.

Originally, we’d considered just doing a blog series, but then our colleague John McClelland had a great idea – run a webcast series (over and over… and over again x9) so we could reach a crop of 100 new Azure-curious viewers each week. With the serendipitous availability of ‘unlimited’ use, two-week Windows Azure trial accounts for the webcast series, we knew we could do something impactful that wouldn’t break anyone’s individual pocketbook – something along the lines of a distributed computing project, such as SETI.

SETI may be the most well-known of the efforts, but there are numerous others, and we settled on one (http://folding.stanford.edu/, sponsored by Stanford University) based on its mission, longevity, and low barrier to entry (read: no login required and minimal software download). Once we decided on the project, it was just a matter of building up something in Windows Azure that would not only harness the computing power of Microsoft’s data centers but also showcase a number of the core concepts of Windows Azure and indeed cloud computing in general. We weren’t quite sure what to expect in terms of interest in the webcast series, but via the efforts of our amazing marketing team (thank you, Jana Underwood and Susan Wisowaty), we ‘sold out’ each of the webcasts, including the last two at which we were able to double the registrants - and then some!

For those of you that attended, we thank you. For those that didn’t, each of our presentations was recorded and is available for viewing. As we mentioned at the start of this blog post, the two hours we’d allotted seemed like a lot of time during the planning stages, but in practice we rarely got the chance to look at code or explain some the application constructs in our implementation. Many of you, too, commented that you’d like to have seen us go deeper, and that’s, of course, where we’re headed with this post and others that will be forthcoming in our blogs.

Overview of Stanford’s Folding@Home (FAH) project

Stanford’s http://folding.stanford.edu/ was launched by the Pande lab at the Departments of Chemistry and Structural Biology at Stanford University on October 1, 2000, with a goal “to understand protein folding, protein aggregation, and related diseases,” diseases that include Alzheimer’s, cystic fibrosis, CBE (Mad Cow disease) and several cancers. The project is funded by both the National Institutes of Health and the National Science Foundation, and has enjoyed significant corporate sponsorship as well over the last decade. To date, over 5 million CPUs have contributed to the project (310,000 CPUs are currently active), and the effort has spawned over 70 academic research papers and a number of awards.

The project’s Executive Summary answers perhaps the three most frequently asked questions (a more extensive FAQ is also available):

What are proteins and why do they "fold"? Proteins are biology's workhorses -- its "nanomachines." Before proteins can carry out their biochemical function, they remarkably assemble themselves, or "fold." The process of protein folding, while critical and fundamental to virtually all of biology, remains a mystery. Moreover, perhaps not surprisingly, when proteins do not fold correctly (i.e. "misfold"), there can be serious effects, including many well known diseases, such as Alzheimer's, Mad Cow (BSE), CJD, ALS, and Parkinson's disease.

What does Folding@Home do? Folding@Home is a distributed computing project which studies protein folding, misfolding, aggregation, and related diseases. We use novel computational methods and large scale distributed computing, to simulate timescales thousands to millions of times longer than previously achieved. This has allowed us to simulate folding for the first time, and to now direct our approach to examine folding related disease.

How can you help? You can help our project by downloading and running our client software. Our algorithms are designed such that for every computer that joins the project, we get a commensurate increase in simulation speed.

FAH client applications are available for the Macintosh, PC, and Linux, and GPU and SMP clients are also available. In fact, Sony has developed a FAH client for its Playstation 3 consoles (it’s included with system version 1.6 and later, and downloadable otherwise) to leverage its CELL microprocessor to provide performance at a 20 GigaFLOP scale.

As you’ll note in the architecture overview below, the @home with Windows Azure project specifically leverages the FAH Windows console client.

@home with Windows Azure high-level architecture

The @home with Windows Azure project comprises two distinct Azure applications, the distributed.cloudapp.net site (on the right in the diagram below) and the application you deploy to your own account via the source code we’ve provided (shown on the left). We’ll call this the Azure@home application from here on out.

… Brian continues with more details about the project.

The Windows Azure Team reported OCCMundial Named Second Most-Innovative Company [in Mexico] with Windows Azure Solution in a case study of 7/16/2010:

InformationWeek Mexico just named OCCMundial as the second most innovative company in Mexico in its annual "50 Most Innovative Companies" issue. OCCMundial developed the largest job-listing website in the Mexican employment market and has more than 15 million unique visitors annually and posts more than 600,000 jobs each year.

The company earned its innovator distinction from InformationWeek Mexico for its recommendation system, OCCMatch, that uses a powerful algorithm to match job openings to candidate resumes. Each day the algorithm that OCCMatch uses compares approximately 90,000 active job offers to a candidate pool represented by up to four million resumes. The system matches qualified applicants to job openings and sends email notifications to potential candidates. OCCMatch generates hundreds of thousands of email messages every single day.

To achieve the computing power necessary to process the powerful algorithm, and the storage capacity required for the popular service, OCCMundial relies on the Windows Azure platform. With on-premises infrastructure, the company was limited in the number of operations it could run each day, but by innovating with a cloud-based solution on the Windows Azure platform, it can now execute millions of OCCMatch operations in parallel-with as much processing power as the company needs. It currently uses more than 200 instances of Windows Azure for several hours each day, but can quickly scale up, or down, when the need arises; plus, it achieved impressive levels of scalability while still avoiding approximately U.S.$600,000 in capital and operational costs.

Read the full story on how OCCMundial uses the Windows Azure platform at: www.microsoft.com/casestudies/Case_Study_Detail.aspx?CaseStudyID=4000005802

Read more Windows Azure customer success stories, visit: www.windowsazure.com/evidence.

The OnWindows site posted a Connecting retailers to the cloud case study on 7/15/2010:

BEDIN Shop Systems has unveiled version 2.0 of aKite, a software services suite for retail stores developed on Windows Azure and SQL Azure. This announcement was made at the Microsoft Worldwide Partner Conference in Washington.

“An excellent example of Microsoft cloud momentum in the retail industry ecosystem, BEDIN’s service-oriented architecture solutions for retailers and the Windows Azure platform complement well and together deliver the benefits of the cloud and optimise the connected customer experience,” said Bill Gonzalez, general manager, Worldwide Distribution and Services Sector, Microsoft.

“Through alliances with industry leaders such as BEDIN, we are able to empower retailers to become ‘Connected Retailers’ by providing capabilities that enhance the connection between retailers and their customers, provide the critical connection needed between retailers and their suppliers, while greatly improving internal collaboration from the desktop to the data centre, and into the cloud,” addded Gonzalez.

BEDIN Shop Systems smart clients are light Internet-centric desktop applications, deployed from a Web page. They are automatically updated from Retail Web Services, the intelligent ‘hub in the cloud’, designed as a service-oriented architecture for straightforward cooperation inside and outside a retail store chain. Retail Web Services fully leverages SQL Azure.

Another important new feature of BEDIN’s cloud-based retail stores services suite is aKite Document Interchange, a software service based on the Windows Azure platform AppFabric Service Bus. It simplifies integration with legacy enterprise resource planning systems and other on-premises.

“aKite from BEDIN Shop Systems is an example of how a partner can generate value for the end customers by leveraging the Windows Azure platform to create innovative solutions. These solutions enable fast time-to-market, simplicity to deploy, flexibility, expected service level agreements and instant integration with the existing on-premises assets,” said Anders Nilsson, Director Developer and Platform Evangelism, Microsoft Italy.

“The aKite reduction of store servers, the energy efficiency of Microsoft’s cloud computing platform, coupled with our capability to use the right level of resources, is our contribution to greentailing,” said Giorgio Betti, BEDIN Shop Systems sales and marketing manager.

Check out the rest of our WPC 2010 news in our special event section

Adron Hall’s Microsoft White Papers post of 7/15/2010 describes two new Windows Azure migration white papers written by Logic 20/10:

Recently Microsoft updated their Windows Azure Site with some new whitepapers. The cool thing is, some of these papers I was directly involved with helping to create! That makes me a little biased in saying, I’m stoked that they’ve been released!

So if you’re in the market for some cloud technology and happen to be researching Windows Azure, check these out. I’ve included the links to the specific whitepapers I had considerable involvement in. Thanks Logic20/20 for providing the opportunity, it was tons of fun working on and researching Windows Azure! I look forward to future opportunities to work with the cloud more.

Here are the Microsoft links:

Custom E-Commerce (Elasticity Focus) Application Migration Scenario

E-commerce applications that deal with uneven computing usage demands are frequently faced with difficult choices when it comes to growth. Maintaining sufficient capacity to expand offerings while continuing to support strong customer experiences can be prohibitively expensive—especially when demand is erratic and unpredictable. The Windows Azure Platform and Windows Azure Services can be used to help these organizations add computing resources on an as-needed basis to respond to uneven demand, and when it makes sense, these organizations can even implement programmatic system monitoring and parameters to help them achieve dynamic elasticity, which it is only possible when organizations become unconstrained by the costs and complexity of building out their on-premises computing infrastructure. While programmatic elasticity may be non-trivial to achieve, for certain organizations it can become a competitive advantage and a differentiating capability enabling faster growth, greater profitability, and improved customer satisfaction. Learn about migrating custom e-commerce (elasticity focus) applications in this whitepaper.

Custom Web (Rapid Scaling Focus) Application Migration Scenario
Organizations hosting custom web applications that serve extremely large numbers of users frequently need to scale very rapidly. These organizations may have little to no warning when website traffic spikes, and many organizations that have launched this type of custom application were unprepared for subsequent sudden popularity and explosive growth. For some organizations, their resources became overburdened, and their sites crashed; instead of riding a wave of positive publicity, their upward trajectory was halted, and they needed to repair their reputations and assuage angry customers and site visitors. The Windows Azure Platform and Windows Azure Services can help organizations facing these challenges achieve web application architecture that is both highly scalable and cost effective. Learn about migrating custom web (rapid scaling focus) applications in this whitepaper.

The Financial Post blog reported iStore to Launch Digital Oilfield Software as a Service
[powered by Azure] in this 7/15/2010 post:

The Information Store (iStore) today announced Digital Oilfield Express (DOX),the first Digital Oilfield offering delivered online through a subscription price model. The new service will bring key exploration and production (E&P) business functionality to the thousands of small businesses that comprise the U.S. independent oil and gas producer market. This announcement was made at Microsoft’s Worldwide Partner Conference (WPC) 2010 in Washington, D.C.

“American independent producers are the backbone of domestic energy production,” said Barry Irani, CEO of iStore. “Because the vast majority of these companies are small businesses they typically lack the infrastructure and IT expertise of the big players which has kept Digital Oilfield solutions out of reach. Recognizing that the Digital Oilfield concept would allow these companies to better manage and use exploration and production data, we looked to Microsoft’s Azure Cloud computing platform as the means to deliver essential and affordable technology to this largely underserved market. And because independents produce 68 percent of the oil and 82 percent of the natural gas consumed domestically what is good for them is good for our national energy security.”

iStore is creating DOX as a streamlined set of key E&P functions, such as document management and data visualization, utilizing SQL Azure and other components of the Windows Azure Services Platform. The Azure cloud platform provides Digital Oilfield technology in a software-as-a-service (SaaS) pricing model, eliminating the barriers created by the up-front capital requirements of a large software implementation and requisite hardware platform that can be prohibitive for a smaller company. Delivering Digital Oilfield in the cloud also allows iStore to bypass the long implementation cycles intolerable for smaller enterprises.

“An excellent example of Microsoft cloud momentum in the oil and gas industry partner ecoysystem, iStore envisions a roadmap for E&P companies moving into the cloud,” said Simon Witts, corporate vice president, Enterprise & Partner Group (EPG), Microsoft.“iStore is leading the migration of data-rich Digital Oilfield applications from the data center to the cloud, which is aimed at creating a cost-effective software-as-a-service pricing model for end users. Microsoft’s Azure platform enables iStore to bring the power of the Digital Oilfield to a market tier that lacks the IT infrastructure of the larger players, allowing smaller E&P companies to focus on managing their business, not IT infrastructure.”

For larger oil & gas businesses the cloud provides an essential fabric for Digital Oilfields of the future on which to combine both software and services, enabling petroleum enterprises to create the mix of online and on-premises data and software that fit the business purpose.

The Microsoft Windows Azure platform addresses the IT requirements of the Digital Oilfield by:

Leveraging cloud services to act as the inter-organization integration layer.

Providing a secured worldwide distributed run-time environment that is always on and supports auto provisioning and hosting of these services.

Taking advantage of the built-in services available (compute, storage, access control, etc.) and the common development toolset to seamlessly connect to existing or new on-premise systems.

The Windows Azure platform, consequently, provides oil and gas solution providers and enterprise customers with the ability to extend and deploy their business services into the cloud and enables improved partner integration. These capabilities are poised to deliver greater levels of innovation and cooperation, further digitization of processes across global value chains and high levels of interactivity and engagement with end consumers.

About iStore

iStore helps petroleum companies access exploration and production (E&P) data wherever it resides and presents it in a useful form. As a result, customers improve asset performance, reduce cycle time and maximize return on investment. Most importantly, iStore's suite of software solutions puts the information E&P asset management teams need to make good decisions at their fingertips. Founded in 1994, the privately held company's headquarters are in Houston. For more information, visit www.istore.com.

Tim Aidlin described The Evolution of The Archivist in this 7/12/2010 second post of a series to MIX Online’s Opinion blog:

In this post, part II of a series, I'll take a step back and explain how we concepted and executed The Archivist Web (alpha)and talk about what came before the design and code.

In an earlier post I talked about working on The Archivist Desktop version, and how the software began its evolution to the web and the cloud. In this post, part II of a series, I'll take a step back and explain how we concepted and executed The Archivist Web (alpha) and talk about what came before the design and code.

Audience Research

Once we decided to take on the project, we turned to our established group of experts for guidance. We already had a prototype of a similar application in The Archivist Desktop, so we had a cadre of fellow-Microsoft employees, friends and colleagues in other companies, and current users we could tap for solid user-testing. We focused on two primary groups of users: our internal Microsoft team and users of The Archivist Desktop.

Getting Stakeholders and Internal Microsoft Team Members On Board

We started by discussing the project's intricacies and goals with our internal team members and stakeholders. We set up short individual meetings with people, gathered feedback on how they used The Archivist Desktop and asked what new features they wanted to see.

“Conversation” with current users

Again, because we had already released The Archivist Desktop, we had an already-established user-base. Over the years there had been a few “vocal” users – who either wrote in to thank us, or report problems and failures – who we felt could provide valuable feedback from real-world use-cases. At the beginning stages of building The Archivist Web we simply emailed a few of the users with whom we had been in contact and asked them a simple set of questions surrounding specific topics such as types of visualizations they would use, how often the tool needed to update, the way they were using the data and the like. By asking specific questions rather than asking for general feedback, users were able to focus on key issues and needs that we could act upon. Often, by asking a generalized question such as “What would you like to change,” you get very generalized answers that are hard to turn into actionable tasks.

We also communicated our plans openly, which helped us make contacts in other parts of the company and take advantage of Microsoft's existing technologies and knowledge-base. Discussing our plans with the Windows Azure team, for example, helped us overcome some serious hurdles with our data-storage methodology. Without their help, we might have been derailed at the beginning.

What Current Users Want

At the beginning stages of the project, we simply emailed a few of our most vocal users and asked them questions about what types of visualizations they might use, how often the tool needed to update, how they were using data, etc. Asking specific questions instead of relying on general feedback helped us identify key issues and needs we could easily act on.

User personas

From our sit-downs and email conversations, we were able to segment our audience into 'personas', defined as representations of real users, including their habits, goals and motivation.

There are many ways to approach user personas. Some designers spend a long time fleshing out the behaviors and market segmentations of a target audience. We, on the other hand, took a more rudimentary approach, following the example of Smashing Magazine great article on the same subject.

Drive-by vs. Return Users

We know that most users will come by the site once, do a quick search, check out some visualizations, and then leave. That’s just the nature of building websites, and especially one geared toward a niche audience.

We wanted to provide an engaging experience with few barriers to entry for the drive-by user, but it was more important to concentrate on bringing people back. So, we decided to focus on our return users, Marketing Managers and academics.

The Marketing Manager

Marketing Managers primarily care about tracking sentiment—who is tweeting about what, how often, and how people feel about a brand or site over time. But up until recently, there were only a few ways to gain access to targeted Twitter data, most of which was presented in a very raw format that was difficult to extrapolate from.

Our plan was to change all this with The Archivist, and make it easy to find, share and interpret large swaths of data suited to the Marketing Manager's needs.

The Academic

Because of our experience with The Archivist Desktop, we knew that there was a surprisingly large demand for the Archivist in academia. During the last year or so, we’ve talked with with numerous professors and students who have found The Archivist Desktop useful for their research.

Whereas the Marketing Manager wants to track 'sentiment', the academic is interested in raw data that's accurate, easy to access and appropriate for use in research.

Wireframes

I thought it might be useful to provide the full PowerPoint walk-through of our wireframe set, which gave our team an understanding of how different users might come to and interact the site.

What we couldn't do (Feature Scoping)

Coming to agreement on the wireframes was easy, but brought up questions about what data we could actually display to users. As I noted earlier, I spent a lot of time thinking about the frequent requests for filtering by date, secondary keywords and other items. Because of how Twitter works, some of this data wasn't consistently available. Other items were so gnarly we couldn't scope them into this release.

Visualization explorations

As Karsten and I were wrapping up the early Archivist Desktop version, I explored additional visualizations I thought we might be able to incorporate into the downloadable WPF application. Since we were working in WPF at that point, I felt free to explore visualizations that might be useful to our audience. Some that I came up with:

For a few specific reasons, we decided to ship The Archivist using the ASP.NET charting contols, and made the decision to feature the following charts.

Tweets over time

Top Users

Top Words

Top URLs

Tweet vs. Retweet

Top Sources (software)

Use ASP.Net Charting Controls

In my next post, I'll talk about taking the site from wireframes and explorations to a functional HTML + CSS3 + jQuery site, and how I worked with Karsten, who made the beauty of the backend happen in the cloud.

Do you use the same process as us, moving from problem to persona to wireframes to design? Leave a comment and be sure to follow us on Twitter @Mixonline

See my Archive and Mine Tweets In Azure Blobs with The Archivist Application from MIX Online Labs (updated 7/11/2010) for more details about the architecture of The Archivist and live links to archives of my tweets and those for SQL Azure and OData.

Return to section navigation list>

Windows Azure Infrastructure

• Tom Simonite asserts “Tools that benchmark performance promise to reveal the strengths and weaknesses of competing cloud providers” as a preface to his Pitting Cloud against Cloud post of 7/16/2010 to the MIT Technology Review blog:

New software developed to measure the performance of different cloud computing platforms could make it easier for prospective users to figure out which of these increasingly popular services is right for them.

Right now, developers have little means of comparing cloud providers, which lease access to computing power based in vast and distant data centers. Until actually migrating their software to a cloud service, they can't know exactly how fast that service will perform calculations, retrieve data, or respond to sudden spikes in demand. But Duke University computer scientist Xiaowei Yang and her colleague Ang Li are trying to make the cloud market more like the car market, where, as Yang says, "you can compare specifications like engine size or top speed."

Working with Srikanth Kandula and Ming Zhang of Microsoft Research in Redmond, WA, Yang and Li have developed a suite of benchmarking tools that make it possible to compare the performance of different cloud platforms without moving applications between them. These tools use algorithms to measure the speed of computation, and shuttle data around to test the speed at which new copies of an application are created, the speed at which data can be stored and retrieved, the speed at which it can be shuttled between applications inside the same cloud, and the responsiveness of a cloud to network requests from distant places. The researchers used the software to test the services offered by six providers: Amazon, Microsoft, Google, GoGrid, RackSpace and CloudSites. Results of those tests were combined with the providers' pricing models to allow for quick comparisons. [Emphasis added.]

The results are among the first attempts to compare the performance of several clouds platforms, says Yang. "We found that it's very hard to find a provider that is best in all metrics," she says. "Some are twice as fast for just 10 percent extra cost, which is a very good deal, but at the same time their storage service is actually very slow and has a lot of latency variation." Another provider showed good computation speeds but was less quick at spawning new instances of an application--something that might be necessary for a service that experiences peaks in demand, as a video site does when some of its content goes viral. "It seems like in today's market it is hard to pick a provider that is good at everything," says Yang.

Windows Azure Platform Appliance

• Gavin Clark reported about Rackspace adopting WAPA in Cloud fluffer ready to lighten Microsoft's load in a 7/15/2010 article for the Channel Register blog:

Web-hosting giant Rackspace could be among the first to deliver private versions of Microsoft's Azure cloud, free of Redmond's control.

Rackspace has had discussions with Microsoft to run Azure in its own data centers, the company's told The Reg.

Already a Microsoft hosting and cloud partner, Rackspace believes the world is too big and too complicated with different security standards and regional requirements for Microsoft to deliver everything on its own.

Microsoft this week announced the Windows Azure Platform Appliance containing Windows Azure compute and SQL Azure storage.

Rackspace chief technology officer John Engates told us: "I think it's a great idea to allow private versions of Azure because realistically, Microsoft can't run all the world's IT in its own data centers."

He added: "If and when we get strong demand for Azure in our datacenter, we'll certainly consider offering it."

The service provider also believes it's got the experience in hosting and .NET to make Azure usable and to help ease customers concerns over the newness of the architecture – most, for example, won't even know what SQL Azure is or be willing to commit at this stage..

Lew Moorman, Rackspace's president for cloud and chief strategy officer, said: "It's like house hunting and buying the house and every single element in it. You can't customize or do your own thing, and that's a big commitment for people... In time we could host it, but people are nervous about giving everything to Microsoft."

Windows Azure Platform Appliance came to light after The Reg broke the news on Microsoft's Project Talisker, a plan to let Microsoft customers and service providers run their own, private versions of Azure. Until Talisker, Microsoft's said only its datacenters would host Azure.

The appliance software will be tested on servers by Dell, Hewlett-Packard, and Fujitsu in their data centers, with plans to deliver the appliance on hardware with services from Dell and HP once they've got a clearer idea of what the product should be, how it might work, pricing, and licensing.

Engates said the appliance concept would let customers run an Azure-like capability to meet the specific needs of their own environments or let Rackspace run Azure for them in its own datacenters.

Rackspace claims 80,000 cloud customers, but installing Azure would mean a change. Rackspace seems to define cloud by server virtualization, as it offers VMware's vSphere with its Windows Server and Red-Hat Linux stacks in addition to services and support.

Rackspace has been agnostic on the software side: while it hosts specific applications as services, such as Microsoft's email and SharePoint, Rackspace also offers vanilla .NET and Linux/Apache/MySQL/PHP (LAMP) stacks that customers can launch code on.

Running Azure, though, would mean commitment to the complete Azure compute and storage fabric and Rackspace would not be allowed to customize the code.

Azure comprises a layer to cluster servers running on Microsoft's hypervisor, SQL Azure storage, and the .NET Framework using Microsoft's Common Language Runtime and ASP.NET.

Engates said he had "high confidence" Rackspace can run Azure when it's ready to ship, noting the company's already very familiar with hosting Microsoft's applications and operating systems.

• Krishnan Subramanian answered Why Windows Azure Is Not Available For On-Premise Deployments? with an ancient Microsoft post in a 7/15/2010 post to the Cloud Ave blog:

When Microsoft announced Windows Azure appliance few days back at PDC10, I was critical of their plans to sell the technology bundled with hardware from their partners.

What I expected from Microsoft was the availability of Azure stack to be used with the existing infrastructure, something like what one can do with Eucalyptus, for example. Expecting the enterprises to buy physical compute infrastructure along with the platform stack is a big downer for me. I guess they are taking the approach to protect their existing cash cow (the sales of Windows Server OSes and other tools). Forcing the potential customers to buy hardware appliance defeats the very idea of drastically cutting down the capital costs with the move to cloud. Even though private clouds are capital intensive compared to public clouds, I feel that letting the enterprises use their existing infrastructure while embracing cloud like features is a more sensible approach than forcing them to buy new hardware along with the platform stack. Their decision to not offer the stack to install on the existing compute infrastructure is the biggest disappointment from this announcement.

Immediately after I posted this one, James Urquhart, the author of the famous Wisdom Of The Clouds blog told me that the reason why they are not offering it could be linked to a possibility that Azure is a combination of software and data center architecture. My initial position on my post was based on the fact that Microsoft has not explicitly talked about the issue. However, James' comment made me think about that possibility. Now, it has officially come out of Microsoft.

According to a blog post by Steve Martin, a senior manager at Microsoft, Azure cannot run on any datacenter. He clearly highlights how the implementation of Windows Azure needs some of their global data-center hardware design and large scale multi-tenancy. Since these cannot be done on many of the existing datacenters, an offering similar to Eucalyptus doesn't make any sense.

We don’t envision something on our price list called “Windows Azure” that is sold for on-premises deployment. Some implementation details aren’t going to be practical for customers, such as our global data-center hardware design and large scale multi-tenancy features which are integral to Windows Azure and the Azure Services Platform.

Also, he clearly highlights how they are going to address this market in the same post

We will continue to evolve Windows Server and System Center focusing significantly on technologies like virtualization, app and web server capabilities, single-pane management tools for managing on-premise and cloud in the same way, etc. which extend the enterprise data center in significant ways. We’ll continue to license Windows Server and System Center (and therefore the shared innovation derived from Windows Azure) to hosters through our SPLA program.

If you are interested in understanding Microsoft's strategy on the private cloud side, I urge you to read his post.

Update: The above post by Steve Martin was more than a year old. I overlooked the date stamp and assumed it was a clarification after the announcement of Azure appliance. It was an oversight on my part. However, the post does offer an official clarification for my comments after the announcement of Azure appliance. My apologies for making it appear as if the blog post is a recent one.

A year is an eternity in the cloud computing business.

Microsoft’s LukaD puts the interdependence of software and hardware into more recent perspective in his Bringing the Windows Azure Cloud On-Premise! post of 7/16/2010:

Yep, it looks like, by popular demand Microsoft is enabling the Windows Azure Platform inside of your datacenter.

Windows Azure Appliance was announced on the keynote of the Worldwide Partner Conference, here is a brief description of the functionality:

“Windows Azure™ platform appliance is a turnkey cloud platform that customers can deploy in their own datacenter, across hundreds to thousands of servers. The Windows Azure platform appliance consists of Windows Azure, SQL Azure and a Microsoft-specified configuration of network, storage and server hardware. This hardware will be delivered by a variety of partners.”

Nice!

Cloud Security and Governance

• David Tesar posted a 00:28:52 Cloud Security Panel Interview at TechEd NA 2010 Channel9 video segment to Microsoft Technet on 7/16/2010:

Mark Russinovich, Andy Malone, and Patrick Hevesi give us insight into the cloud security space at TechEd North America with Ch9 Live. We dig into quite a few areas such as:

What are some of the tradeoffs with security in regards to the different type of cloud offerings?

What are the biggest concerns in regards to security with moving to the cloud?

Can you trust the data security boundaries with cloud technologies like Windows Azure?

What are some tools to help with having visibility into security in the cloud?

What are some areas where we need improvements in the cloud security space?

What are some things which are strong or weak with security forensics in the cloud?

Use of Bitlocker and data encryption in the cloud datacenter space?

Cloud Computing Events

• Maurene O’Gara introduces:

Bentley Radcliff, who used to work in marketing at Apple and Sun - no Microsoft fanboy he but curious about Redmond's approach to the cloud - wanted to go to its Worldwide Partner Conference this week so we handed him a reporter's notebook and sent him on his way. Miracle of miracle, he returned impressed. This is what he had to say ..

in her Microsoft: O Cloud, O Cloud, O Cloud post of 7/16/2010:

Sunday was the end of the World Cup so Microsoft decided to play off the theme with videos of fans in the pubs singing OLAY, OLAY, OLAY but instead sang O Cloud, O Cloud, O Cloud, not exactly your bonding IBM theme song.

Click image for link to 00:01:48 O Cloud!, O Cloud! video segment from ReadWriteCloud.

Just to be clear this is a partner conference full of people who buy Microsoft products and resell them in their markets. And the cloud SCARES THE CRAP OUT OF THEM.

What came across loud and clear is that Microsoft is not going to miss this round of IT transformation.

Steve Ballmer gave the keynote and reiterated the key dimensions of the cloud, something that he spoke about back in March but for most of the partners this was brand new.

Later I found out from a group of Microsoft executives that Microsoft has mandated that 90% of its products be cloud-focused by the end of the year. They felt that they had reached 70% by the conference, a lot of it focusing on Azure and pushing key products to be SaaS.

The key dimensions of the cloud according to Ballmer are:

Creates opportunities and new responsibilities.

Cloud learns and helps you to learn, decide and take action

Cloud enhances social and professional interactions

Cloud is going to drive server advances that run on the cloud

Cloud wants smarter devices - which was a big play for fat clients!!!

What's important about this is that Microsoft is defining what the cloud will be for many of its customers. And it's doing everything it can to bring the partner community along with them. If you look at the competition, Amazon, Rackspace, Google there's no room for the partner. This is what scares them. Microsoft is giving partners a chance to jump on board.

The big announcement was the Azure appliance, which is a preconfigured Azure platform to be installed at a client's physical site alleviating the concern for security and data proximity. There was not a lot of information and I got the distinct impression the thing was only baked over the last week or two but a number of big vendors committed to building or using the Azure appliance namely HP, Fujitsu, Dell, and - surprise - eBay.

The Azure appliance lets Microsoft extend its cloud solutions to its partners. And by announcing that some larger partners would open Azure-hosted data centers they fulfilled the vision. This is where Microsoft has made an amazing move on the cloud chessboard, at least Check!

They are going to give partners of all sizes and levels a way to host, collocate with a hosting company or buy from Microsoft cloud services for their clients. With the installed base of Microsoft products this could rapidly propel them to the front of the cloud line especially in the SMB areas where partners play.

Still the mood was skeptical because the cloud changes the game on the partner. Where they used to compete only at a local level against other VARs or VADs now they have to compete on a national, even international level with every cloud provider. Welcome to the brave new world. Either get on board or become the next BlockBuster video stunned by Netflix!

Of course there were plenty of the traditional upgrade announcements, the most important being Windows 7 and Windows Server. During the discussion a new message about Windows caught my ear - "Make the Windows experience a vital and LOVED part of users' lives!" Something Vista didn't do!! According to Microsoft 1.1 billion PCs run Windows and 70% still run XP at work but at home they run Windows 7 (of course they didn't mention Mac). The user experience at home is superior to the user experience at work. So Windows 7 upgrades for 2010-2011 is a high priority.

During a demo session Microsoft showed different flavors of laptops, desktops and of course had an iPad-like device they called a slate. Not sure why this got so much attention in the press other than the clear message that there will be full-blown slates (Pads) that run Windows 7, phones that run Windows 7 Mobile, and that life will be great with Windows on a Pad. There was nothing too shocking as there was very little to talk about.

As far as ISVs, a lot of them are coming on board with the Azure and cloud strategy. Some of course are waiting to see what happens and the ones that have to provide secure access and on-site premise solutions are moving slowly. But according to Microsoft 10k customers are using the Azure platform. And for something that's only been shipping this year that's an impressive number.

On the show floor after the keynotes, I talked with a number of hosted Azure providers from all niches. They mostly support the cloud direction and the Azure program. Of course Microsoft gave them the software free to run and opened a marketplace for applications so why not jump on board. I had an interesting conversation with New World App's about its niche play in the government with managed security and disaster recovery. It's extending its partnership with Microsoft by offering Azure for government applications.

It appears that Microsoft buttering up ISVs and resellers by pledging Business Investment Funds (BIFs) - development dollars - to get them to come over. This is the key - apps bring volume and volume brings apps. So a jumpstart is a great model.

Microsoft intends to take some of the billions it's got in the bank and become a leader if not the leader in the cloud. Everyone's jumping on the cloud but they've jumped out in front. Well done.

Didn’t Bentley intend “¡Olé!, ¡Olé!, ¡Olé!” for Spain rather than “OLAY, OLAY, OLAY” for cosmetics in his first paragraph?

• Jean-Christophe Cimetiere noted in the Interoperability @ Microsoft blog on 7/16/2010 that Microsoft [will be] at OSCON next week:

Microsoft returns to the O’Reilly Open Source Convention (OSCON) in Portland, Oregon next week. For us, OSCON is a great opportunity to meet and interact with developers, programmers, systems engineers and administrators who live and breathe open source every day. We’re excited to participate in this forum, as it gives us a chance to hear and see firsthand what exciting developments are happening and how and what bridges we can build to improve interoperability between platforms.

Jean Paoli, Microsoft General Manager for Interoperability Strategy, will be keynoting on Thursday, talking about “Open Cloud, Open Data.” Jean has played a pivotal role in Microsoft’s interoperability efforts over the past few years and he continues to explore opportunities to create greater interoperability in new technical areas and with emerging platforms. In his keynote, Jean plans to address how interoperability is at the core of Microsoft’s vision of a cloud that can capture the imaginations of developers, unleash their creativity and enable them to build new breakthrough applications.

If you won’t be at OSCON in person, you can still watch the keynote presentations through a streaming feed on the O’Reilly OSCON site.

A crew of Microsoft people will be on the ground and would love to talk all things interop and open source with you. If you have a question, an idea, an observation, just poke us @openatmicrosoft or me @jccim, we’ll see you there. We will also have several speakers presenting track sessions throughout the week and I encourage you to check them out. Links to the session summaries are provided below.

CoApp: Bringing Open Source Package Management to Windows
Garrett Serack, Software Development Engineer for Microsoft
Date: Wednesday, July 21
Location: E143/E144
Discover the changing landscape of Open Source on Windows, and how the introduction of the Common Opensource Application Publishing Platform is driving performance, stability and quality into OSS applications on Windows.

Build Mission Critical Cloud Applications on Windows Azure Platform using Open Source Technologies
Vijay Rajagopalan, Principal Architect for Microsoft
Date: Thursday, July 22
Location: E143/E144
The purpose of this session is to demonstrate Microsoft's commitment to openness and interoperability by practical Cloud Application development scenarios on Windows Azure using Open Source Technologies and Languages

Private Cloud Deployment with Microsoft and Novell: Driving Transformational Architecture in the Enterprise
Fabio Da Cunha, Senior Manager, Interoperability Alliances for Microsoft
Frank Rego, Senior Product Manager for Novell
Date: Thursday, July 22
Location: E143/E144
The evolution of cloud computing promises significant efficiency gains and cost savings over today’s distributed architecture model. Many IT leaders, however, face tough choices about what to deploy to evolve their data center while, at the same time, leveraging existing assets and managing to a flat or often declining budget.

Reactive Extensions For JavaScript
Erik Meijer, Architect for Microsoft
Date: Thursday, July 22
Location: Portland 255
The "A" in "AJAX" stands for "Asynchronous" and indeed almost all Web and mobile applications have to be written in an asynchronous and event-driven style. Reactive Extensions for JavaScript is a library for coordinating and orchestrating asynchronous and concurrent computations in a high-level and declarative way.

Advancing Interoperability, Patient Safety, and Efficiency with the Microsoft Connected Health Platform Open Toolkits
Teddy Bachour, Sr. Technical Strategist for Microsoft
Date: Friday, July 23
Location: F151
The Microsoft Connected Health Platform (CHP) provides open toolkits and guidance for the information and communication technology (ICT) community to help them speed architecture, design and deployment of interoperable, efficient, and scalable e-Health infrastructures and solutions for the health industry.

See you there!

Jean-Christophe is a Senior Technical Evangelist in Microsoft’s Interoperability Technical Strategy team lead by Jean Paoli.

Alex Williams posted CloudCamp at OSCON: 5 Topics We'd Like to Discuss to the ReadWriteCloud on 7/15/2010. CloudCamp will start on Monday, 7/18/2010, at 7 PM at the Oregon Convention Center in Portland, OR:

We are thinking of how to get the most out of CloudCamp at OSCON on Monday night as the discussions will be exclusively about open-source and the cloud.

So we decided why not write down some topics that we'd like to explore. These are in no particular order but just generally what we are thinking about. A lot of these questions come from trends and topics of discussion we are seeing on the Web.

Let's get started:

The Basics

A simple discussion on the different components of the open cloud.

Questions: What is the anatomy of the open cloud? What are its components at the node and system infrastructure level? What are the dynamics of the current market that is affecting its overall structure?

On Azure and Proprietary Systems

A general discussion to explore Windows Azure's significance and how other proprietary systems are affecting the direction of cloud computing.

Open-Source Database Explosion!

According to Kin Lane, a boom in open-source database offerings is evident in looking at the exhibitor list at OSCON. What's happening is a disruptive change that the database world has not seen in the past 20 years. What are the dynamics of this trend?

Interoperability: Will It Ever Happen?

Interoperability between service providers can help avoid vendor lock-in. But will interoperability ever happen? Are there any brewing open source standards so data can be easily moved between services? Are standards needed?

On Open-Source SaaS

SaaS services are proliferating. What are the emerging principles of open-source SaaS environments? What will open-source SaaS services look like in the years ahead?

That's a start. There are any number of discussions to be had at an event like CloudCamp.

I am the local organizer for CloudCamp Portland. Hope you can make it if you are in town. CloudCamp Portland will start on Monday at 7 pm in the Oregon Convention Center. It is free to attend.

Matthew Weinberger reported Microsoft: We Want Hosting Providers To Make Cloud Moves to the MSPMentor blog on 7/15/2010:

Another dispatch from the Microsoft Worldwide Partner Conference (WPC10): Microsoft is striving to help transform hosting providers and telecommunications providers into full-on IT service providers. The secret sauce, predictably, involves the cloud. But what are the implications for smaller MSPs and VARs? Here’s some perspective.

“Microsoft sees service providers becoming more important as the cloud becomes more predominant. Given their experience in deploying and selling infrastructure and software as a service, businesses will depend on them for IT as a service,” said John Zanni, general manager of Worldwide Hosting for the Communications Sector at Microsoft, in a prepared statement.

Key developments on that front, as per Microsoft’s press release, include: NetApp providing enterprise disaster recovery to SMBs by way of the Dynamic Data Center Alliance and a newly published whitepaper that indicates enterprises are half as likely to change hosted messaging and collaboration service providers as they are on-premise.

To prove their point, Microsoft’s using WPC10 to showcase their hosting partners who are syndicating Business Productivity Online Suite (BPOS).

Microsoft’s shown a huge focus on how partners of all shapes and sizes can benefit from the cloud at WPC10, and this just continues that momentum.

Still, each Microsoft move also raises its share of questions. A prime example: Microsoft is allowing big service providers to receive key discounts if they wrap additional services around BPOS (Business Productivity Online Suite). But some of those BPOS benefits are not available to smaller MSPs because the small players lack economies of scale.

Also of note: Several small MSPs with whom we’ve spoken are still calling on Microsoft to change its terms of service with small MSPs. In particular, the small MSPs want to manage billing and branding for BPOS, rather than handing that service off to Microsoft. So far, Microsoft seems unwilling to budge on that point.

Next up, we’ll be watching to see if Microsoft makes more moves during HostingCon, which is slated to start July 19 in Austin, Texas.

Additional reporting by Joe Panettieri.

Read More About This Topic

Fujitsu And Microsoft Teaming Up In The Cloud

Microsoft Channel Chief: Potential BPOS Marketplace?

Ballmer’s Cloud Computing Message to Channel Partners

SaaS: Microsoft Launches BPOS Partner Resource Page

Microsoft Channel Chief Concedes: I Need to Learn About MSPs

Other Cloud Computing Platforms and Services

• Wolfgang Gentzsch analyzes Amazon CCI in his New HPC in the Cloud: Good Marketing or Real Progress? post to the HPC in the Cloud blog of 7/16/2010:

A few days ago Amazon announced that it added HPC capabilities to EC2. This is great news indeed for the HPC community, because it further paves the way for HPC to becoming mainstream, which indicates there is cloud money to be made with HPC. Obviously, Amazon did a careful market analysis, and certainly got some requests from important users, and perhaps felt some pressure after rumors surfaced that Google’s server farmers are playing with Infiniband.

In more detail, Amazon added so-called Cluster Compute Instances (CCI) to EC2, each consisting of a pair of quad-core Intel X5570 (Nehalem) processors with a total of 33.5 ECU (EC2 Compute Units), 23 GB of RAM, and 1690 GB of local instance storage. CCIs are interconnected using a 10 Gbps Ethernet network. Within this network you can create one or more placement groups of type "cluster" and then launch CCIs within each group. Instances within each placement group of this type benefit from non-blocking bandwidth and low latency node to node communication. First benchmark results from LBNL show their HPC applications on CCIs ran 8.5 times faster than on the previous (vanilla) EC2 instance types.

So far so good. To me, in this context, there are two aspects which seem interesting: performance and price. Let’s look at performance first:

To achieve high performance, many HPC application programs have been optimized in the past for high execution speed, e.g. through parallelization of numerical algorithms, speeding up communication, overlapping communication with computation, and other sophisticated tricks. Thus an application programmer’s limits are mostly set by physical boundaries: e.g. if processors or interconnects are slow. Now that Amazon has added fast processors and Ethernet to its server farm, there is no surprise to see some interesting speedups over the standard EC2 servers and interconnection. But Ethernet is not Infiniband, and you still face the cloud’s virtualization layer, which may cause different parallel (virtual) processes sitting on different cluster compute instances which still may cause communication delays. Fair enough, Amazon admits that the only way to know if you got a genuine HPC setup for your specific application is to benchmark it, which is anyway a general wisdom in HPC (We should ask Ed Walker to repeat his NAS Parallel Benchmark tests from 2008 now on the new CCIs).

Looking closer into the TOP500 list, Amazon’s Linpack on 880 CCIs (7040 cores) and 41.82 TeraFLOPS is giving them the 146th position. Other Ethernet based supercomputers with a similar position have similar numbers of cores, no surprise. BUT, those with a similar position and with Infiniband interconnect need only about 4800 cores to achieve the same performance.

Therefore, my guess is that most of the average real HPC capability computing applications (e.g. in electronic design automation, automotive applications, or finite-element based material analysis) won’t show a big performance improvement over vanilla EC2 instances, especially those which really demand low latency and high bandwidh. But, fortunately, not all of the HPC applications have this demand; especially the many ones under the umbrella of Capacity Computing with more loosely coupled parallelization (and thus moderate to no communication) might benefit from this improvement. And the fact that the Berkeley LBNL expert team has been involved in early beta, and certainly in consulting AWS and doing a lot of HPC benchmarking, gives the whole project real credibility. I am sure we will soon see some good results, lessons learned, and recommendations from LBNL; at the latest when Kathy Yellick from LBNL will give her keynote at the ISC Cloud Conference in Frankfurt on October 29. …

• Nicole Hemsoth claims Cloudy Costs for Amazon’s New HPC Offering in this 7/16/2010 article for the HPC in the Cloud blog:

Unless you’ve been living in a cave, (and I am well aware that some of you find that environment to be most suitable for your personal productivity) you’ve probably already been pinged with the news that Amazon has proffered an EC2 instance specifically for the needs of HPC users. If you need a moment to blink, dazed by the light of day, refresh here via an article on the announcement.

Now that we’re moving beyond looking at this news outside of the surprise it presented to many at the beginning of this week, we’re seeing it in the light of what it means in the broader context. Accordingly, there are serious questions mounting about that most critical issue outside of performance--pricing.

Without any use cases yet on the Cluster Compute Instances front to draw from in order to examine the model as it relates to different applications or different use scenarios (cloud-bursting versus continuous utilization, etc.) there is rampant speculation that relies on employing hypothetical scenarios that yield some strange, scary arithmetic—scary because those costs still look pretty high for sustained use.

While the numbers involved with purchasing a cluster outright are enough to set any small startup research shop shaking, the solution has been to send the embarrassingly parallel stuff to EC2, since it is generally accommodating for such workloads or to turn to any number of firms offering clusters on a rental basis. The benefit of EC2 has been cost for bursting needs in particular and the benefit of the latter (rentable HPC—which is not to be confused with “cloud” by the way, no matter what their marketing managers tell you) has been the support that comes with having specialized HPC clusters run by specialists.

So even though it’s still easy to contend that in the long run, this announcement was a kick in the behind for the ecosystem of HPC cloud vendors and users alike, it still comes down to a cost question—specifically, one that revolves around buying, renting, or clouding. And there are certainly differences between those last two.

Amazon on the Dollars and Sense Behind Cluster Compute Instances

Amazon was open about the pricing to begin with, but to get more in-depth answers in context, I put a few questions to Business Development Manager at Amazon Web Services, Deepak Singh about how the pricing works in practical context. His fully developed answers might help us to shed some light on that ages-old question: to buy a cluster, to rent one, or to automatically spin up nodes as needed in a pay-as I go fashion? Well, it shed light on the questions, but it’s still hazy as to how pricing will be understood for individual users with wildly varying needs.

The CCI model itself is not difficult to understand, especially if one is already familiar with EC2 instance cost evaluation. It’s almost surprisingly uncomplicated on the surface, which alone prompted some speculation. Like any other EC2 instance, Cluster Compute Instances (CCI) run in an on-demand fashion by the hour. For CC Instances, the price is $1.60 per core hour. Fair enough? I asked Deepak Singh to put this in practical context.

“Let's say we have a 32-node cluster which is running a simulation task for 20 hours a week. Using the on-demand pricing, the customer would pay $1024 per run (not counting any additional storage costs). If the customer needs to do this run twice a month, they would end up paying $2,048 per month for their EC2 usage (or < $25K/yr).”

Singh continued with the statement that this is an attractive model for several HPC use cases that are driven by bursting versus sustained use. This is especially true because if there is a need to run applications one time in a single strong flurry, there would otherwise be the queue issue—waiting in line to get access to a shared resource with no guarantee about when the job would run. Conversely, it is a viable alternative to “acquiring the hardware, which means not only the capital costs of purchasing the hardware and the facilities in which the hardware is run, but also the operational cost of running the hardware 24-7 when you might need it for a fraction of the time.”

Page: 1 of 3
Continue Reading: 1 | 2 | 3 All »

David Linthicum posits “Amazon Web Services' new cluster computing offering clearly targets high-performance data center needs” in a preface to his 7/15/2010 Amazon.com's big move into high-end computing analysis for InfoWorld’s Cloud Computing blog:

As I pointed out a few weeks ago, cloud computing is not always the right tool for the job -- specifically, when there are high-end computational requirements and the processors need to be more tightly coupled to support high-performance computing standards such as MPI.

However, infrastructure service providers, such as Amazon Web Services, are perhaps finally putting the right tools in the shed. That seems to be Amazon.com's intent with its new high-performance-computing (HPC) cloud service offering, which could provide a home for those special applications that require supercomputing power.

Amazon.com's Cluster Compute Quadruple Extra Large is a 64-bit platform with 23GB of memory, 1,690GB of instance storage, and 10Gbps of Ethernet I/O performance composed of 33.5 Elastic Compute Cloud (EC2) compute units. The default usage limit for the Cluster Compute instance type is eight instances, or 64 cores, although customers can request more, the company says.

That's hardly the kind of genetic-workload infrastructure platform Amazon Web Services is best known for. Even critics are warming up to the upmarket offering. For example, although the National Energy Research Scientific Computing Center (NERSC) at the Lawrence Berkeley National Laborator found Amazon.com's EC2 wanting, NERSC has now determined that its HPC applications run 8.5 times faster than on previous EC2 instance types, according to Amazon.com.