Windows Azure and Cloud Computing Posts for 6/28/2010+
 | Windows Azure, SQL Azure Database and related cloud computing topics now appear in this daily series. |
Updates of 6/29/2010 are marked •
Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:
- Azure Blob, Drive, Table and Queue Services
- SQL Azure Database, Codename “Dallas” and OData
- AppFabric: Access Control and Service Bus
- Live Windows Azure Apps, APIs, Tools and Test Harnesses
- Windows Azure Infrastructure
- Cloud Security and Governance
- Cloud Computing Events
- Other Cloud Computing Platforms and Services
To use the above links, first click the post’s title to display the single article you want to navigate.
Cloud Computing with the Windows Azure Platform published 9/21/2009. Order today from Amazon or Barnes & Noble (in stock.)
Read the detailed TOC here (PDF) and download the sample code here.
Discuss the book on its WROX P2P Forum.
See a short-form TOC, get links to live Azure sample projects, and read a detailed TOC of electronic-only chapters 12 and 13 here.
Wrox’s Web site manager posted on 9/29/2009 a lengthy excerpt from Chapter 4, “Scaling Azure Table and Blob Storage” here.
You can now download and save the following two online-only chapters in Microsoft Office Word 2003 *.doc format by FTP:
- Chapter 12: “Managing SQL Azure Accounts and Databases”
- Chapter 13: “Exploiting SQL Azure Database's Relational Features”
HTTP downloads of the two chapters are available from the book's Code Download page; these chapters will be updated in June 2010 for the January 4, 2010 commercial release.
Azure Blob, Drive, Table and Queue Services
• Matias Woloski warns “That solution doesn’t scale” in a 6/27/2010 sequel to his Poor man’s memcached for Windows Azure post of 6/26/2010:
I posted yesterday about a poor man’s distributed caching solution using Windows Azure queues and ASP.NET cache. I’ve got an interesting comment in twitter:
My short answer is that there aren’t solutions that scale or doesn’t scale. The scalability is a quality attribute of a system and it varies depending on the context. A good system is the one that is easily adaptable to new contexts and a good solution is the one that is the most convenient in a local context.
Putting things in context
So here is the analysis of the solution proposed and the context where it might applies.
I used the following parameters in the calculation:
- Time to dequeue 32 messages at a time: 1600 milliseconds (source)
- Time spent notifying ASP.NET cache the dependency changed: 300 milliseconds (this is a worst case scenario, it is actually in the nanosecs scale)
Some conclusions you can take from this
- From 1 to 1000 items updated per second (i.e. items invalidated) there is a penalty of 3 minutes (in the worst case) to invalidate the cache of those 1000 items. We are talking about items invalidated PER SECOND. Even having 1 update per second is a lot, in certain systems, so this seems to be good enough for many applications.
- Passing the 1000 items barrier, the time to invalidate all the cache items could be unacceptable (e.g.: you don’t want to wait hours to invalidate the cache). However, if you have more than 1000 updates per seconds, you are probably having other problems.
So what is the drawback compared to a distributed cache?
If you compare this solution to something like memcached, the main difference resides when you have lots of web servers in a farm. This is because memcached will replicate and keep synchronized the cache items between nodes. So when you insert an item in memcached it is available to all the web servers almost immediately. However, when using the ASP.NET Cache and the cache dependency mechanism, when the item is invalidated, EACH web server will have to retrieve the item again and insert it in the cache. Retrieving the resource is the expensive operation, but not invalidating the cache.
I hope this clarifies a bit what I meant by “poor man’s memcached”.
• RBonini’s Thoughts on Windows Azure Storage post of 6/29/2010 makes the case for a built-in connection between Azure table rows and blobs:
I’m sitting here planning what to do with the re-write of the Feedreader I wrote for university. Naturally, the storage solution is always going to be a contentious issue. The NoSql versus SQl debate plays a lot into it.
But, for the most part, i used Windows Azure tables and blobs for its simplicity over SQL Azure. In saying that, SQL is not my favourite thing in the world, so make of that what you will. But also, for a demonstrator application, the use of something completely new played in my hands very well.
So the re-write is also meant to be a demonstrator application. so the Windows Azure storage is staying.
But, not so fast. because Windows Azure Storage needs something. The way I used Tables and Blobs essentially was as a poor mans object database. Thus meant that there was a lot of leg work involved in this, not to mention tedious plumbing code. The fact is, I think that the is the most logical use case for Windows Azure Storage – where in metadata is stored in the tables and the object themselves are in the blobs.
What I would like to be added, then, is the ability to formalize this relationship in code. Some way of saying “hey, this value in this TableRow actually points to a blob in this container”. So I can call “getBlob()”, or something on a Row an get the blob back. Now, to be clear, I don’t mean this to be a foreign key relationship. I don’t want to cascade updates and deletes. And i certainly don’t want my hands tied by making the Blob Attribute column (or whatever) mandatory.
Now, this could be added right now. And in fact am considering doing that. But support for this in the Storage Client library would be nice. But weather backend support is needed, or in fact a good idea, is another question entirely. The implications of making such a fundamental change on the back end. For example, say you’ve exposed a table via OData. How do you expose the blob as well? And given the nature of the use case, the fact that it is needed on a table by table basis makes it much easier to limit any such functionality to the Tools library only.
I can hear you asking yourself why I’m asking for support in the Storage Client library if I can gin up and duct tape together some Extension Methods? I’m a big proponent of the idea that anything that we use in intact with software, be that actual applications or libraries we use in our code, has a Natural user interface. Think of it, the API methods that are exposed for us as developers to use are in fact a User Interface. so I’m asking for a better user interface that supports this functionality without me having to do the legwork for it. In so delivering support, it is perfectly possible, indeed likely, that the library code that Microsoft ships will be more efficient than whatever code I can write.
My final report on the project did call out VS Windows Azure Tools, mainly for not making my life easier. So I’m looking forward to the new version (1.3) and seeing how it goes, particularly with regard to storage.
Now performance-wise, the version I wrote last wasn’t exactly fast at retrieving data. I suspect that this was due to a) my own code in-efficiencies and b) the fact that my data wasn’t optimally (obviously) normalized. Its also probable that better use of MVC viewdata (actually, I think that should be “better use of MVC, period”) and caching will improve things a lot as well.
Rbonini is “a Computer Science Student at the University of the West of Scotland. I'm starting my forth year (Honours).”
• Jon G. Elerath and Michael Pecht claim RAID 5 isn’t as reliable as vendors claim in their A Highly Accurate Method for Assessing Reliability of Redundant Arrays of Inexpensive Disks (RAID) research report of March 2009 (sponsored by NetApp) plublished online by the IEEE Computer Society. From the Abstract:
The statistical bases for current models of RAID reliability are reviewed, and a highly accurate alternative is provided and justified. This new model corrects statistical errors associated with the pervasive assumption that system (RAID group) times-to-failure follow a homogeneous Poisson process, and it corrects errors associated with the assumption that the time-to-failure and time-to-restore distributions are exponentially distributed. Statistical justification for the new model uses theories of reliability of repairable systems. Four critical component distributions are developed from field data.
These distributions are for times to catastrophic failure, reconstruction and restoration, read errors, and disk data scrubs. Model results have been verified to predict between 2 and 1,500 times as many double disk failures as estimates made using the mean time-to-data-loss (MTTDL) method.
Model results are compared to system-level field data for a RAID group of 14 drives and show excellent correlation and greater accuracy than either MTTDL or Markov models.
This is an obvious reason why most high-reliability cloud file systems, including Windows Azure storage and SQL Azure, are based on multiple replicas, not RAID.
• Robin Harris comments on the preceding paper in his RAID 5: theory & reality post of 6/28/2010 to ZDNet’s Storage Bits blog.
In theory, RAID 5 protects your data. In reality, RAID 5 is often a painful failure. Why? Mean time-to-data-loss (MTTDL) is a fraud: actual rates of double-disk failures are 2 to 1500 times higher than MTTDL predicts.
What’s behind MTTDL’s failure?
In A Highly Accurate Method for Assessing Reliability of RAID researchers Jon G. Elerath of NetApp - a major storage vendor - and Prof. Michael Pecht of the University of Maryland, compared RAID theory against actual field data. They found that MTTDL calculations inaccurate for 3 reasons:
- Errors in statistical theory of repairable systems.
- Incomplete consideration of failure modes.
- Inaccurate time-to-failure distributions.
By repairing MTTDL’s theoretical basis, adding real-world failure data and using Monte Carlo simulations they found that today’s MTTDL estimates are wildly optimistic. Which means your data is a lot less safe with RAID 5 than you know.
Repairable systems
The typical MTTDL assumption is that once repaired - i.e. a disk is replaced with a new one - the RAID system is as good as new. But this isn’t true: at best, the system is only slightly better than it was right before the failure.One component is new - but the rest are as old and worn as they were before the failure - so the system is not “like new.” The system is no less likely to fail after the repair than it was before.
The problem is that in RAID arrays repairs take time: the disk fails; a hot spare or a new disk is added; and the data rebuild starts - a process that can take hours or days - while the other components continue to age.
Net net: MTTDL calculations use the wrong failure distributions and incorrectly correlate component and system failures.
<Return to section navigation list>
SQL Azure Database, Codename “Dallas” and OData
The other day a good friend of mine asked me if it was possible to send appointments created in Outlook into a SharePoint calendar list. It’s easy enough to grab this data from Outlook by building an add-in with Visual Studio, but what also turned out to be really easy was sending this information to SharePoint using its built-in WCF data services, also known as OData. OData is an open REST-ful protocol for exposing and consuming data on the web. I’ve written before about how we could consume these SharePoint 2010 data services and do some analysis in Excel. Today I’d like to show you how easy it is to update data through these services as well.
Creating the Outlook Add-in and Adding the SharePoint Data Service Reference
I’m going to use Visual Studio 2010 to build an Outlook 2010 add-in, but you could also choose to build an add-in for Outlook 2007. As a matter of fact, you could use Visual Studio 2008 to build Outlook 2007 add-ins and access data services, just make sure you target the .NET 3.5 framework or higher.
File –> New –> Project, select the Office 2010 node and select Outlook 2010 add-in. I named this example project UpdateSharePointCalendar. First thing we need to do is add a reference to our SharePoint 2010 data service. You do this the same way you add references to other WCF services. Right-click on the project in the Solution Explorer and select “Add Service Reference”. If you have SharePoint 2010 installed, you can navigate to the data service for the site that contains the data you want to consume. To access the data service of the root site you would navigate to http://<servername>/_vti_bin/ListData.svc. I have SharePoint 2010 installed on my local dev box so for this example I’ll use the data service located at http://localhost/_vti_bin/ListData.svc and name the service reference SPService.
Once we add the service reference, the client entity types will be generated for us and an assembly reference to the System.Data.Services.Client is added. All the SharePoint list data and type information is exposed via the data service. We can browse these types visually using the Open Data Protocol Visualizer. Once you install the visualizer, right-click on the SPService service reference and select “View in Diagram”. Expand the Entity Types and then drag the CalendarItem onto the diagram to explore its properties. A lot of these properties are internal fields to SharePoint, but you can see things like Title, StartTime, EndTime and Description related to the calendar item.
Adding a Form Region and Ribbon to Outlook Appointments
Now that we have our data service reference set up let’s go ahead and add some UI. For this example, I want to display an adjoining form region when a user creates a new Appointment item in Outlook. It should display all the events on the SharePoint calendar for that day. I’ll use a simple read-only data grid for this example but you could display anything here, even WPF controls like I showed in this post. I also want to add a button to the Ribbon that the user can click to send the appointment data to the SharePoint calendar list.
So right-click on the project and select Add –> New Item and select Outlook Form Region, I named it AddToSharePoint, and click Add. Next select “Design a new Form Region”. Next select the type of Form Region, in this case Adjoining. Next, for the name I decided on “SharePoint Calendar Events”. At the bottom of this screen it asks which display modes this region should appear in and by default all three modes are checked. If we were adding this form region to a mail item then this would control when the region would display. However there is no reading pane or separate compose or read mode for an appointment item, so we will need to determine at runtime if the user is creating a new appointment or not. Click Next and finally select the Appointment message class as the class to associate the form region with, then click Finish. …
Beth continues with several feet of more detailed instructions.
Wayne Walter Berry explains Managing Connections in PowerPivot in a fully illustrated tutorial of 6/29/2010:
A good friend of mine that analyzes databases with business intelligence for a living has told me that the majority of his job is knowing the data. PowerPivot makes it very easy to add additional tables, and bring together multiple data sources –allowing you to discover your data as you build reports. This article will discuss how to manage your connections to SQL Azure from PowerPivot, add additional tables, and build relationships.
After creating the initial connection (which I blogged about here), I always find myself needing to add tables, manage inner database relationships and bring in other data sources. One of the reasons is that once I correlate the data into a PowerPivot report, I realize that I want to generate another variation of that report that requires an additional lookup table or relationship.
A lot of business intelligence is taking two dissimilar data points and relating them in a manner that brings new insight into your data. Typically, these data points are inner related via a network of tables and relationships, which need to be brought into PowerPivot; this might take several trips to the table selection dialog to get right. Being able to access this dialog again is critical to success with PowerPivot.
Wayne continues with sections about Finding Your Connection and Building Relationships, and concludes:
To be successful with PowerPivot take what you know about your data and use it to import the correct tables and establish the relationships need to generate your PivotTables for your reporting.
Sanjay Jain reported Pervasive Data Integrator Universal CONNECT! offers OData integration on 6/29/2010 in a post to the US ISV Evangelism blog:
Pervasive Software(R) Inc. (NASDAQ: PVSW), a Microsoft Gold Certified Partner and recent “Best of SaaS Showplace” award winner, enables the integration and connectivity to the Open Data Protocol (OData) using the Microsoft Dynamics certified Pervasive Data Integrator ™. Pervasive Data Integrator Universal CONNECT! has an unparalleled range of connectivity and integration capabilities to a variety of on-premises, on-demand and cloud-based applications and sources, including the Open Data Protocol (OData). OData, a web protocol for querying and updating data, applies web technologies such as HTTP, Atom Publishing Protocol (AtomPub) and JSON to provide access to information from a variety of applications, services, and stores. There is a growing list of products that implement OData. Microsoft supports the Open Data Protocol in SharePoint Server 2010, Excel 2010 (through SQL Server PowerPivot for Excel), Windows Azure Storage, SQL Server 2008 R2, and Visual Studio 2008 SP1.
Enjoy the show!
Wayne Walter Berry suggested Download: Troubleshooting and Optimizing Queries with SQL Azure on 6/28/2010:
This paper provides guidelines on the Dynamic Management Views that are available in SQL Azure, and how they can be used for troubleshooting purposes.
Download it here.
Wayne Walter Berry explains WinForm Application Streaming from SQL Azure and provides source code in this 6/28/2010 post:
In this blog post, we are proving code for a sample WinForm application that streams images from SQL Azure to a PictureBox control. Our goal is to show how you can stream large BLOB data from a varbinary(max) column in SQL Azure to the WinForm application, in this case, an image. One of the nice things about streaming data is that you can update a process bar control to give the user some indication of how long it will take to download all the data. This blog post is an extension of this blog post that introduces the SqlStream class.
Our goals for this code sample:
- Show example code of calling SQL Azure from a WinForm application.
- Stream the response into the PictureBox control.
- Update a progress bar control as the download takes place.
- Prompt for login and password, a best practice of connecting to SQL Azure, as discussed in this blog post.
Adventure Works Viewer
The sample application views the thumbnail images in the SalesLT.Products table in the Adventure Works for SQL Azure database. You can download this database from here. When the application is started, it prompts the user for their login and password to the SQL Azure database. Then it fills in the drop down list with all the product names from the database that contain images. If the user clicks on the download button, the application queries the database to determine the size of the image; the size is needed to accurately configure the progress bar. The image is then streamed from SQL Azure and displayed. During the download, the status bar is updated to give the user an indication of the download speed and size of the image.
Wayne continues with sample code for Filling the Drop Down List, Prompting the User for Credentials, and Streaming the Image. Download the sample code from a link at the end of Wayne’s post.
• Dave Kearns added “Ten years ago I wrote: 'Using a directory for relational data or a relational DBMS for directory data is like using a hammer to cut wood: It can be done, but a real carpenter wouldn't do it.'” as a preface to his Relational database: An ongoing debate post of 6/25/2010 to NetworkWorld’s Security blog:
Over the years I've been writing about identity services it's always been taken as a given that the data store for identity information is a directory system (such as eDirectory, Active Directory, iPlanet, etc.). But there have always been those who espoused a relational database (RDBMS) as a "better" data store. It's an argument that's been going on for the entire 10 years this newsletter has been published, and one that is more political in nature than technological.
In June, 2000, for example, I wrote: "Every special-purpose database is designed and structured to perform its special purpose optimally. A relational DBMS is designed to let items of data be related to one another either permanently or on an ad hoc basis, while a directory service uses the object-oriented style of database (with its built-in hierarchy) to create a database supporting a tree-like view of the data." That's still true today, of course -- choose the best data structure for the work you want to do. As I concluded: "Using a directory for relational data or a relational DBMS for directory data is like using a hammer to cut wood: It can be done, but a real carpenter wouldn't do it."
What's interesting about the discussion from 10 years ago is a follow-up edition I wrote ("More information on object-oriented databases"). Interesting, because a new argument is raging today -- whether relational databases should be replaced by XML structures with the reason given that the XML structures are more flexible. Interesting, because of a pointer I provided to an authority on "object-oriented" databases (OODB). XML databases are merely an outgrowth of OODBs. Check the work here and see how today's "hot topic" really shows how there's rarely something new in the world.
The object-oriented, hierarchical directory service is still the best place to store identity data. Can that be "XML-ized"? Sure, it doesn't change the nature of the data store, merely it's representation. If that allows the data to be more easily used by today's applications and services, then that would also encourage coders to use existing identity stores for their apps and services. And that should benefit all of us.
I don’t believe that the NoSQL crowd is promoting XML (or OODBs) as a replacement for relational DBMSs in the cloud. Hadoop, Hive and Cassandra appear to be the top contenders at the moment.
• Mike Flasko recommended that you Join the OData Mailing List in this 6/28/2010 post to the WCF Data Services Team Blog (formerly the Astoria Team blog):
Today we released an OData Mailing List. This is the list to use if you have OData questions, comments on OData or want to discuss how OData should evolve over time.
To sign up to the list go here and follow the instructions provided. The list is fully open so anyone can subscribe and participate. Be sure to read the terms of use on the signup page to understand how your feedback on the mailing list may be used to enhance OData over time.
You can see a read only archive of all the messages from the mailing list here: http://www.odata.org/mailing-list
We look forward to hearing your thoughts, comments and feedback on the list!
Deep Fried Bytes posted Episode 53: A Lap Around OData with Mike Flasko of their podcasts on 4/26/2010, which I missed at the time:
<Return to section navigation list>
AppFabric: Access Control and Service Bus
• Vittorio Bertocci (@vibronet) reported The June 2010 Identity Training Kit Contains Powerpoint Decks, Videos on 6/28/2010:
As promised, here there’s a version of the Identity Developer Training Kit which contains the decks from the WIF Workshops and embedded players for all the video recordings of the Redmond event; alongside, of course, with the usual hands-on labs. After more than one year, I was finally able to get rid of the sentence “subsequent releases may contain presentations, samples and demos” from the training kit download page…
If you want to do a presentation on WIF, now you have slides covering a pretty comprehensive range of topics to chose from; in fact, you can ever redeliver the workshop in its entirety if you choose so. If you do, please let me know and I’ll make sure to advertise your event (and publish pictures afterwards!) from this pages.
I get asked to give intros to WIF every day: knowing your way through claims-based identity is a skill in high demand, and I hope that all the assets packaged in the June 2010 version of the kit will help your business!
• The Windows Azure Team reminds you that a New Training Video Outlines Use of WIF in Windows Azure in this 6/29/2010 post:
Don't miss the latest session of the WIF Workshop recordings, which covers the use of Microsoft Windows Identity Foundation (WIF) in Windows Azure and provides practical advice on aspects of distributed development such as handling NLB sessions, certificate management, dealing with volatile application URI, handling tracing, metadata and generation considerations. The discussion covers both Web roles and WCF roles.
This is the final unit of the 10-part WIF Workshop recordings, which cover how to take advantage of technologies such as WIF to easily solve authentication, authorization and identity-driven personalization challenges. If you want to take the entire course, you should have experience developing in Visual Studio and the .NET framework. This session is also available, along with a related lab, as part of the "The Id Element" show, which also offers Identity and Access Management-related interviews, developer how to's, news, aggregated resources and more.
• Jackson Shaw’s Great ADFS video and InfoWorld reading of 6/28/2010 praises Matt Steele’s video and an InfoWorld article:
I saw a reference to this video elsewhere and wanted to let you know about it. The video shows Matt Steele who is a program manger at Microsoft who works on ADFS V2. Matt gives a clear explanation of how you “project” your Active Directory identity to an outside-the-firewall application; where and why you might want to purchase a cheap, trusted, SSL certificate; high availability of ADFS; how a federation “broker” would eliminate needing 1:1 relationships with every federated application that you might want to connected to and how authorization policies and decisions are made within an ADFS environment.
If you’d like a excellent introduction to ADFS V2 check out Matt’s video – it’s well worth it. Here’s the synopsis of the video:
Matt Steele walks us through on a whiteboard all of the steps required on how to federate your identity to Windows Azure using ADFS 2.0 for single-sign-on. This video is a great way to learn how ADFS works and to help you get started to deploy this scenario before you dig into deeper whitepapers. We will help you answer questions like:
- What kind of SSL certificate should we get and when to get it?
- Should we open up the firewall to the ADFS server or just manually copy over the certificates to establish the initial trust relationship?
- Should we use an ADFS broker or not?
Once you've watched the Matt's video you might want to read this article that recently appeared in InfoWorld: Does ADFS 2.0 deliver on its single sign-on promise?
Ron Jacobs reported the availability of endpoint.tv - Workflow Messaging Architecture with Dave Cliffe on 6/28/2010:
In this episode, Dave Cliffe, Program Manager for Workflow Services, describes the architecture of the Workflow Services infrastructure.
For more information [and earlier workflow video segments], see the Workflow Foundation developer center on MSDN.
Workflow is in this section because .NET Services (Windows Azure AppFabric’s predecessor) originally included workflow, but removed it before changing its name.
<Return to section navigation list>
Live Windows Azure Apps, APIs, Tools and Test Harnesses
• Steve Marx released a live Azure demo, Pivot View of Netflix Instant Watch Movies, on 6/29/2010:
• Steve Marx’s Pivot, OData, and Windows Azure: Visual Netflix Browsing post of 6/29/2010 describes how he created the project:
The PivotViewer Silverlight control shipped this morning, which means you can now embed a Pivot collection (with great UI) directly in a web page. Pivot is fantastic for sorting, filtering, and browsing large numbers of items.
I’ve put together my own example of using the new PivotViewer control at http://netflixpivot.cloudapp.net. It lets you browse the top ~3,000 movies that Netflix has available to stream online. I really encourage you to click through to the demo… it’s really a fantastic way to find a movie to watch.
Technical Overview
The demo is built on Windows Azure and consists of a web role (which serves the web page itself), a worker role (which creates the Pivot collection once every hour or so), and blob storage, which hosts the collection and the Silverlight control (all behind the Windows Azure CDN). The data comes from Netflix’s OData feed.
I only had to write about 500 lines of code to make this all happen, and I suspect that number would go down if I used the Pauthor library (which I didn’t have access to when I wrote this demo).
Creating the Pivot Collection
The Pivot collection is created by a worker role that only has a single instance. It takes more than an hour to process the latest Netflix feed into the form needed for Pivot. I could have parallelized some of this and spread the load across multiple instances, but the feed changes infrequently, so I’m not in any particular rush to get the work done. Using a single instance makes the code very simple, because everything happens locally on a single disk, but I have also built Pivot collections in the past using a large number of instances. …
Steve provides sample source code for Creating the Pivot Collection, Storing the Collection in Blob Storage, and Serving the Collection. He concludes:
Download the Code
You’ve now seen nearly all of the code involved, but you can download the full Visual Studio 2010 solution at http://cdn.blog.smarx.com/files/NetflixPivot_source.zip.
If you want to run it, you’ll also need:
And note that the collection takes quite some time to create, so expect to run this for at least an hour before you can see anything.
Thanks for another Azure tour de force, Steve!
• Kevin Kell’s Worker Role Communication in Windows Azure – Part 1 post for Learning Tree of 6/28/2010 is the first of a two-part series:
In an earlier post we talked about the “Asynchronous Web/Worker Role Pattern” in Windows Azure. In this pattern the web roles are “client facing” and expose an http endpoint. In Azure a web role is essentially a virtual machine running IIS while a worker role is a background process that is usually not visible to the outside world. The web roles communicate with the worker roles via asynchronous messages passed through a queue. Even though it is relatively simple this basic architecture is at the heart of many highly scalable Azure applications.
There are, however, other ways in which roles can communicate internally or externally. This allows for considerable flexibility when implementing Azure based solutions. A worker role, for example, could be directly exposed to the outside world (through a load balancer, of course!) and communicate over tcp (or other) protocol. This might be useful in a solution which required, for some reason, a highly customized application server exposed over the Internet. It is also possible for worker roles to communicate directly with other worker roles internally. Often the internal communication is done over tcp but other protocols can be used internally as well.
In the first of this two part series we will explore the basics of exposing an external (i.e. available over the Internet) endpoint to a worker role over tcp. We will use this to implement a simple client/server version of a “math service”. Note that this is not something you would ever necessarily want to do in practice! It is used here simply as an example of how to enable the communication. In the real world the “math service” could be replaced by whatever custom application service was required.
As usual to get started, we use Visual Studio 2010 to create a Cloud Service project. We add a single worker role into that project. We can then define an “InputEndpoint” (which is Microsoft’s term for an endpoint that is visible to the external world). An InputEndpoint has a name, a protocol and a port number.
This definition is done in the ServiceDefinition.csdef file (some details omitted for clarity):
<?xml version=“1.0″ encoding=“utf-8″?>
<ServiceDefinition … >
<WorkerRole name=“Square.Service”>
<ConfigurationSettings>
<Setting … />
</ConfigurationSettings>
<Endpoints>
<InputEndpoint name=“SquareEndpoint” protocol=“tcp” port=“2345″ />
</Endpoints>
</WorkerRole>
</ServiceDefinition>Note that the port number (in this case I have arbitrarily chosen 2345) is the port number that the Load Balancer is listening on. You need to make an API call to get the actual internal to Azure endpoint as follows:
RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["SquareEndpoint"].IPEndpoint;
Hopefully a demo will make this clearer:
http://www.youtube.com/watch?v=DsDyoZOAkR8
Note that this demo is very minimal and does not necessarily demonstrate good design or programming practice. The sole purpose is to show how an Azure worker role can communicate with the external world over tcp.
By opening up role communication in this way Microsoft has given us lots of options to implement a wide variety of design architectures in Azure. In a future post we will examine how roles can also directly communicate with other roles internally.
• SnoBlaze introduced CloudBlaze, an “ISV Application Framework for the Microsoft Windows Azure™ Platform” on 6/28/2010:
CloudBlaze is the industry’s first application framework designed specifically to help ISVs migrate their traditional desktop and Web based applications to the Microsoft Windows Azure™ Platform.
CloudBlaze is a “Solution Accelerator” that eliminates the non trivial burden on the development team to create the underlying “plumbing code” required to SaaS-enable your product(s) for running in Azure. It allows the team to focus on building the core software required to meet the business needs of your customer. The CloudBlaze framework enables the ISV's transition from an existing business application to a SaaS model on the Azure platform, or to build a SaaS-enabled application from scratch.
• Bob Familiar reported ARCast.TV - Neil Palmer of Sungard on the Business Impact of Cloud Computing and RIA in a 6/28/2010 post to the Innovation Showcase blog:
ARCast.TV - Neil Palmer of Sungard on the Business Impact of Cloud Computing and RIA [a 00:14:35 Channel9 video segment].
In this episode, Hong Choing talks to Neil Palmer, Partner for Sungard Consulting Service about the impact of Cloud Computing and new Rich Internet Application technologies such as Silverlight 4.
Hong covers this solution in detail in this blog post [below].
• Hong Choing’s Sungard Light Up Gas-Trading Application with Silverlight 4 post of 6/25/2010 explains:
Rich Interactive Application technologies such as Silverlight 4 adds a whole new dimension to the way users interact with data intensive applications. For Sungard, their Gas Trading is the perfect application that benefit of such technologies.
What does the application do?
The Gas Trading application provides either tree based navigation of the area / zone / location and pipelines or directly by overlays on the Bing map. Push pins represent transfer points on pipelines and can be used to zoom in, display position at that location, and enter new trades, and to dynamically change the actual gas routing.
• Marjorie Censer reported Federal government seeks bids for cloud computing services in a 6/28/2010 article for the Washington Post:
Local companies are hoping to help the federal government move into cloud -- or Internet-based -- computing, as the General Services Administration prepares to select approved vendors.
GSA has issued a request for quotes, open to companies on its information technology schedule, to provide the cloud infrastructure. After reviewing the bids -- due June 30 -- GSA will select contractors that federal agencies can then draw on for cloud services.
Through cloud computing, agencies can access a pool of Internet-based resources, such as networks, servers and applications, rather than invest in computing infrastructure. The technology can then be used on an as-needed basis, making it cheaper and quicker to set up.
"Cloud computing is inevitable, said David L. McClure, associate administrator for GSA's Office of Citizen Services and Innovative Technologies. "It's the speed and pace and direction that we're always uncertain of, but it's an inevitable move because the computing industry itself is moving in this direction."
Chantilly-based Apptis -- which is partnering with Amazon Web Services -- is among those companies that plan to compete.
"This is a very important program for the government because it addresses a couple things that have been inhibitors to cloud, the first being ease of procurement," said Phil Horvitz, the company's chief technology officer.
Additionally, he said, government concerns about security are addressed in the cloud requirements.
Falls Church-based CSC also plans to bid. In a statement, the company said it already has extensive experience in cloud projects of similar scope.
McClure said he expects federal agencies to use the selected vendors, but the agencies are not limited to buying cloud computing through this program. …
Return to section navigation list>
Windows Azure Infrastructure
• David Linthicum asserts “Asking whether hybrid clouds are real or not ignores a simple truth: They've been around in some form for years” in his Hybrid clouds: A new twist on an old trick post of 6/29/2010 to InfoWorld’s Cloud Computing blog:
Information Week's John Soat had an interesting post entitled "Cloud computing: What's a hybrid cloud and where can I get one?" As John puts it: "Hybrid clouds are the Big Foot of cloud computing: tantalizing but difficult to conceive of and frustratingly elusive to the point of arousing deep skepticism on the part of potentially interested parties. Are they real or a myth?"
This kind of stuff drives me crazy. We've been using architectures like this for some time, including when I was in college. Back then compute resources, such as supercomputers, were both expensive and in short supply. Thus, many of us building solutions would use hybrid architectures to leverage remotely hosted supercomputers only when needed. For the most part we processed locally, but when supercomputing MIPS were required, we'd send jobs to some supercomputer thousands of miles away. This included virtualization and multitenancy, albeit old-school stuff.
Hybrid clouds, as we're defining them today, are really nothing different. There are two major components of the architecture: the on-premise systems, or the private cloud, and the off-premise systems, or the public cloud.
In IT, we consider the on-premise or private cloud systems to be the most desirable place for our processing because we've already invested in the hardware and software. Then, if we near the saturation point in our private clouds, we move processing to the public cloud until the need for additional capacity has ended. In other words, we're "cloudbursting."
Of course it's not that easy to create a private cloud these days. You have to deal with the interfaces used by both clouds, as well as with heuristics and security. It's a systems development and systems integration challenge. To solve this problem, up-and-coming companies such as Nimbula are looking at creating operating systems that span public and private clouds, and standards are being created around interface compatibility between private and public clouds as well. However, to say that the technology and standards are mature would not be true currently.
Hybrid clouds? Nothing scary and new here. This is a very old architectural trick that has some new cloudy potential.
There’s no question that hybrid cloud hype is heating up.
• OnWindows presented an Interview: Cloud computing: going green article from the June 2010 issue of Prime:
From sustainable supply chains to scalable infrastructures, the cloud offers many environmental benefits, say thought leaders in cloud services.
Cloud computing is fast becoming ‘the’ buzzword in the business world. For many, it represents the platform for the next generation enterprise, promising low total cost of ownership (TCO), high scalability and an easy pay-as-you-go cost structure. These advantages and more are driving its adoption across businesses around the world and are causing leading IT companies to focus their energies on the cloud and the applications that can be derived from it.
Microsoft is no exception. Addressing an audience of several hundred students and professors at the University of Washington earlier this year, the company’s CEO Steve Ballmer gave his strongest indication yet of the company’s commitment to cloud computing: “As I like to say at Microsoft, for the cloud we’re all in.” With 70 per cent of its employees already focused on innovation in the cloud and its major software offerings already geared up to take advantage of it, the company is confident that this is where the future lies. “There is incredible opportunity in the cloud, incredible opportunity,” says Ballmer. “There will be more inventors creating more interesting innovations and having better access to customers globally. The opportunity, and the opportunity for companies like ours to build the infrastructure to enable that is quite high.” At Microsoft, our goal is to reduce the impact of our operations and products, and to be a leader in environmental responsibility
Colin Masson, Microsoft: While there are already signs of companies looking to move to the cloud, Microsoft and its counterparts still have some way to go to convince their customers of its true potential. One major benefit that isn’t highlighted as much as it could be, for example, is the green aspect. By moving data centres to the cloud, companies can significantly reduce their power consumption – something that will have a considerable impact on their carbon footprint.
“Few companies have the capability of tapping into US$9.5 billion of research and development that Microsoft supports (company-wide) per year,” says Kevin Timmons, Microsoft’s general manager of data centre services. “And we hope that by sharing our key learnings and best practices that we can help the industry, as a whole, work together to drive greater efficiencies through our cloud services infrastructures and collectively reduce our carbon wastes.”
In this roundtable, we address some misconceptions about the cloud, and find out more from Microsoft and some of its key partners about why this is the path to a greener future and how it will help the manufacturing and resources sector. …
Following are the roundtable interviewees:
- Rohit Bhargava, worldwide industry technology strategist for manufacturing, Microsoft
- Colin Masson, worldwide director for ERP and supply chain, Microsoft
- Bernd Kosch, green IT advisor, Fujitsu Technology Solutions
- Steven Smith, co-founder and president, GCommerce
- David Gustovich, founder and CEO, IQity Solutions
- James Butler, head of client services and strategy, Program Framework
• John Brodkin asserts “Microsoft official [Tim O’Brien] says infrastructure and platform cloud lines will blur” in his Microsoft Windows Azure and Amazon EC2 on collision course article of 6/25/2010 for Network World’s Data Center blog:
Microsoft's Windows Azure and Amazon's Elastic Compute Cloud tackle two very different cloud computing technology problems today, but are destined to emulate each other over time, Microsoft cloud official Tim O'Brien says.
Whereas Windows Azure is a platform-as-a-service cloud, giving developers the tools they need to build and deploy Web applications, Amazon EC2 is primarily an infrastructure-as-a-service cloud, offering on-demand access to customizable virtual machine instances.
Azure simplifies the building of web applications in a way that Amazon does not, but Amazon's cloud-based virtual machines have the benefit of working with multiple programming models, O'Brien says, predicting that over time Microsoft will move more into infrastructure-as-a-service and Amazon will cross over into platform-as-a-service (PaaS).
O'Brien, senior director of Microsoft's Platform Strategy Group, discussed his take on the cloud market in an interview with Network World, as well as a public presentation at the recent Cloud Leadership Forum, hosted by IDC and IDG Enterprise.
"It's a double edged sword," O'Brien said in the interview. "The reason people like infrastructure-as-a-service is because it's programming model agnostic. The bare metal VM doesn't care what language you wrote the application in, it doesn't matter what tools you use and what runtimes you've targeted. If it runs on Windows or Linux, give it a virtual machine and it will run just fine. The problem is it's a lot of extra work. You're responsible for that virtual machine the same way you're responsible for a server sitting under your desk. You're responsible for turning it on. You're responsible for turning it off. You're responsible for applying a patch or an update. If Red Hat applies a Linux patch, and you have a Linux VM running on Amazon, you have to apply that patch yourself. They won't do that for you."
But there are shortcomings in the platform-as-a-service model as well, O'Brien acknowledges. The biggest problem with PaaS may be difficulty migrating existing applications from the internal data center to the cloud.
"Platform-as-a-service has a different set of tradeoffs," O'Brien says. "All of that stuff is completely abstracted away, it's a friction-free development, you basically code up an application, you hit deploy and it'll go run on the platform that's supplied by those runtimes. So in our case its PHP, C Sharp, in the case of Google [App Engine] it's Python and Java." While building new applications is easy, and removes the need for owning internal hardware and software, other than a Web browser, "part of the challenge there is it's not necessarily optimal for migrating existing applications."
Microsoft has already announced that "at some point [in the next 12 months] we will be offering the ability to provision a bare-metal VM, and run your application on that," O'Brien says.
While Amazon provides a variety of Windows and Linux virtual machine images through EC2, the company's Web Services business offers a variety of other tools that might be useful to developers, including databases, storage services and load balancing.
John continues his story on page 2. See also stories about Amazon Web Services in the enterprise in the Other Cloud Computing Platforms and Services section below.
<Return to section navigation list>
Cloud Security and Governance
• Lori MacVittie asserts Security risks are not always indicative of a lack of faith in the provider’s competency but about, well, risk in her Risk is not a Synonym for “Lack of Security”:
IDC recently conducted another cloud survey and [feign gasp of surprise here] security risks topped a healthy list of concerns that, according to the survey, outweighed cloud computing benefits.
While growing numbers of businesses understand the advantages of embracing cloud computing, they are more concerned about the risks involved, as a survey released at a cloud conference in Silicon Valley shows. Respondents showed greater concern about the risks associated with cloud computing surrounding security, availability and performance than support for the pluses of flexibility, scalability and lower cost, according to a survey conducted by the research firm IDC and presented at the Cloud Leadership Forum IDC hosted earlier this week in Santa Clara, Calif.
[…]
However, respondents gave more weight to their worries about cloud computing: 87 percent cited security concerns, 83.5 percent availability, 83 percent performance and 80 percent cited a lack of interoperability standards.
It would be parsimonious (but altogether commonplace) to assume that “security concerns” or “security risks” translate directly into a lack of security on the part of cloud providers. Ockham’s razor might not draw blood from such an assumption but it does lead to the dismissal of what are certainly legitimate concerns on the part of would-be cloud computing customers.
Risk is not a synonym for “lack of security.” Respondents to surveys asking about cloud computing adoption inhibitors are not necessarily concerned that cloud providers are lax in their implementations of security. Rather it is more likely that because cloud computing impacts the ability of organizations to quantify some of the risks and properly address those risks through processes and technology that it becomes problematic to justify the benefits despite the risk because the latter is unknown.
LET’S DEFINE RISK, SHALL WE?
Risk is about the unknown, about the possibility, the potential for a negative outcome. There is for every organization what is considered an acceptable
level of risk and then there’s, well, an unacceptable level of risk. Generally speaking in order to determine whether some initiative or strategy falls under the organization’s acceptable level of risk the risks must somehow be measurable in terms of impact as well as probability of occurrence.
This is easy to do with availability. Loss of revenue or customers can be quantified based on the last outage, so you have a quantifiable monetary risk. Historical uptime of a cloud provider is becoming easier to find at this point, so the probability of a cloud computing environment’s outage can be calculated. Even security risks can be quantified using industry data (Ponemon’s annual study often helps here) and the probability of attack can be determined in a number of ways, including academic research.
Now, to understand why survey respondents continue to cite “security” as the primary reason they are hesitant to “go cloud” consider what cloud computing today does to an organization’s ability to address security risks through technology, and then examine how deployment in a cloud computing environment impacts the ability to quantify risk and its potential negative outcome. …
Lori continues with a detailed analysis of security risks in cloud computing environments and concludes:
It isn’t necessarily the case that citing “security risk” means customers think cloud computing providers are lax or inattentive to security concerns. It may simply be the case that they are well-aware that there are known and unknown security risks associated with new technology that have not yet been addressed to their satisfaction and because as an organization they are highly sensitive to risk they just cannot justify taking unknown and unquantifiable risks with their (and their customers’) data and information. The organizational processes and solutions put in place to address those risks in a typical localized data center deployment are not necessarily available for deployment in a public cloud computing environment, which means the risks must be borne by the organization – and its customers – when using public cloud computing. Some organizations may find that burden too high for them and their customers.
I’m more than okay with that because the potential cost to me if there’s a loss of data is much higher than the savings for the organization from a couple pieces of a hardware.
• The Times of India reported Deloitte says Security an area of concern in cloud computing in this unbylined 6/28/2010 article:
NEW DELHI: Even as an increasing number of Indian firms look at embracing cloud computing, data security continues to remain a primary concern for them, said research firm Deloitte.
Cloud computing allows companies to use services, softwares and applications on a pay-per-use model without actually owning any hardware infrastructure or licence. This helps companies save on setting up IT infrastructure or purchasing licences.
“A lot of chief information officers (CIOs) express concern over data security in a non-cloud environment. This concern is only heightened when their data is hosted on a server elsewhere,” Deloitte senior director Sundeep Nehra said.
However, he declined to comment on the investments being made for cloud computing, or the potential opportunities for security solution providers in this regard. Citing an example, he said that in a 500-bed hospital, hundreds of medical images and medical records may be stored. Any data loss could hamper the diagnosis of patients.
The situation gets complicated in the case of banking and financial services, where data on millions of transactions has to be stored everyday. “Information is critical, and, as companies would invest in securing data on their own premises, they also need to ensure that their data on the cloud is secure,” he said.
Companies like Amazon.com, Microsoft and Google offer cloud computing services globally. Analysts peg the Indian cloud computing market to be worth more than $200 million. While small and medium-sized enterprises seem to be adopting cloud-computing faster, large companies are also keen on moving in that direction for optimising costs. These companies have the option of moving to either a private cloud or a public cloud (where solutions and data of more than one company are stored).However, security becomes a primary concern when the data is hosted on public clouds. “Since data of more than one company is hosted on public clouds, it is but natural that firms become apprehensive about the security of their data,” Nehra said.
Data loss on a cloud can happen when it is accessed without proper authorisation, or through hacking.
“Similar to normal data centres, clouds are also vulnerable to data loss, either deliberately or by mistake. Hence, it is important that companies invest in data security,” he said. “We advise clients to invest in data leakage protection (DLP) solutions from various vendors like CA, McAfee and Symantec,” Nehra added.
Old news in the USA, but perhaps it’s a new issue in India.
<Return to section navigation list>
Cloud Computing Events
• Vanessa Alvarez reported on 6/24/2010 from Structure 2010 [about the] Economics Of Cloud session in an InformationWeek Plug Into the Cloud article:
I'm at the Structure 2010 conference, where panelists on the "Cloudonomics: Economics of the Cloud" session were focused on what they are seeing happening in the cloud, from their customers' perspectives.
Some of the more interesting topics at the session were:
Evolving Cloud Computing Value Proposition
While cost still continues to be the primary reason for enterprises to think about deploying this model, it's no longer the only reason. The benefits and value are starting to sink in with enterprises.
As organizations look to remain competitive, a highly scalable, flexible, and on-demand IT environment is critical. The pertinent question is how to start the process of transforming the IT environment, and what applications and workloads can effectively and efficiently benefit from cloud.
Changing CIO & IT Roles
Many panelists pointed out that the disconnect between IT and line-of-business (LOB) managers continues to be challenging. IT is really not aware of who is using public cloud services and therefore, doesn't really see the need for deploying a cloud computing model that will deliver the features and functionalities that LOB managers are turning to the public cloud for.
No "One-Size Fits All"
The question is not about one cloud or the other, but about having the right mix. I've written before about this topic. The problem is that the solutions today are not focusing on the specific needs of the business. In order to help an enterprise choose the right model, it's necessary to understand what the business needs are. It may not be about cost efficiency more than it is about high performance or scalability.
Transformation Of The Data Center
Enterprises looking to leverage their IT environments for competitive advantage are, in fact, beginning to transform their data centers to deliver more resources and overall cost efficiencies through cloud computing models.
The market now realizes that it is not a "one cloud fits all" world. Individual enterprises have unique needs which must be understood before a solution can be chosen that fits their needs. This requires a combination of technological and business acumen, as well as an understanding of the enterprise's specific vertical.
<Return to section navigation list>
Other Cloud Computing Platforms and Services
• Alin Irimie reported Amazon Relational Database Service (RDS) Now Supports SSL Encrypted Connections on 6/29/2010:
RDS now generates an SSL certificate for each DB Instance. If you need a certificate for an existing instance youll need to reboot it using the AWS Management Console, the RDS command-line tools, or the RDS APIs.
Here are a few interesting things things to keep in mind:
- SSL encrypts the data transferred ‘over the wire’ between your DB Instance and your application. It does not protect data ‘at rest.’ If you want to do this, youll need to encrypt and decrypt the data on your own.
- SSL encryption and decryption is a compute-intensive task and as such it will increase the load on your DB Instance. You should monitor your database performance using the CloudWatch metrics in the AWS Management Console (pictured at right), and scale up to a more powerful instance type if necessary.
- The SSL support is provided for encryption purposes and should not be relied upon to authenticate the DB Instance itself.
- You can configure your database to accept only SSL connections by using the GRANT command with the REQUIRE SSL option. You can do this on a per-user basis so you could, for example, require SSL requests only from users connecting from a non-EC2 host.
You can learn more about this new feature in the RDS Documentation on Database Instances and in the forum post.
I’m surprised I haven’t received an e-mail message from AWS about this.
• Joab Jackson asserted “Red Hat's Cloud Foundations, a complete package for running hybrid clouds, includes software and services” in his Red Hat launches rival to Windows Azure post of 6/28/2010 to InfoWorld’s Cloud Computing blog:
Red Hat has launched a comprehensive package, called Red Hat Cloud Foundations, that will allow organizations to run applications in both public clouds and their own private clouds.
With this release, Red Hat is one of only two companies that offer a complete package for running a hybrid cloud, said Scott Crenshaw, vice president and general manager of Red Hat's cloud business unit. The other company is Microsoft, with its Azure platform.
The announcement was one of a number of cloud-related announcements that the company made during its Red Hat Summit last week in Boston. The company also has added new partners to its Red Hat Certified Cloud Provider Program. It has released version 2.2 of its Red Hat Enterprise Virtualization (RHEV) package, and has integrated Cisco's Virtual Network Link (VN-Link) technology within the RHEV package.
The first edition of the Red Hat Cloud Foundations package includes a set of Red Hat programs, a reference architecture, and a number of consulting services and training classes. The Red Hat programs include Red Hat Enterprise Linux (RHEL), Red Hat Network Satellite, RHEV, JBoss and the company's messaging software.
The company also announced that IBM, NTT Communications, and Savvis are new partners in the Certified Cloud Provider Program. They join Amazon Web Services, which was the program's first partner in the program, which certifies that users can run Red Hat cloud software in these environments with no additional customization.
The new release of RHEV includes, for the first time, support for virtual desktop infrastructure (VDI), said Navin Thadani, who is a senior director for Red Hat's virtualization business. It will use the company's Spice remote rendering technology. A conversion tool to move VMware or Xen virtual machines over to Red Hat's Kernel-based Virtual Machine (KVM) format has also been added to RHEV 2.2. Virtual machines on RHEV 2.2 can now use up to 256GB of memory each, up from 64GB in the previous iteration.
Red Hat also took the day to announce a new partnership with Cisco. The two companies have integrated Cisco VN-Link within the RHEV kernel. This integration will allow data center operators to control their virtual servers in the same way they control their physical servers, simplifying management and speeding throughput.
I think it’s a bit premature to endow Microsoft with an RTM hybrid cloud solution, although that’s what folks are expecting to see at its Woldwide Partners Conference 2010.
• Paul Greenberg analyzes Zuora and the Subscription Economy in his 6/28/2010 post to ZDNet’s Social CRM: The Conversation blog:
Every now and then a vendor does something that genuinely surprises and even delights me. They come up with a new feature at the least but at the most, they define a concept, which while it certainly pushes the vendor’s agenda, it also adds a new way of thinking to the overall discussion and even has a practical merit. The latter is something that comes once in a great while, not frequently, but when it comes, there is something of a “hmmm, that’s very interesting…very, very interesting.”
Last week, Zuora released its Z-Commerce for the Cloud service, an upgrade to its billing and payment service that is built for, guess who, cloud providers. What makes it interesting as a service is that its the first external service (SaaS) that actually understands the complex billing models of the cloud providers which account for not only monthly subscription fees, but also have to think about automated metering, pricing and billing for products, bundles, and highly individualized/specific configurations. So for example, storage as a service - one of the components of cloud platforms - is able to charge for terabytes of storage used, or IP address usage, or data transfer charges. Or, cloud providers can structure a per CPU instance charge or per application use charge. It can take complexities like peak usage into account. In other words, Zuora as they always do, figured it out. In fact, they’ve provided 20 pre-configured templates for the billing and payment models that cloud providers use - which tells you how confusing a market this can be. Not to Zuora though.
But honestly, what makes this as well as other aspects of the Zuora business model so interesting is what Zuora is using for the underlying rationale for their success (125 customers, 75 employees, profitable, etc.). They call it the “subscription economy” and while normally, I think that ideas like that are just vendor self-justification - in this case, though of course there is certainly an element of that, they might just well be onto something.
The Subscription Economy
Paul continues with a long quote from “Tien Tzuo, the CEO of Zuora, who is also the former Chief Strategy Officer of salesforce.com” and resumes:
Let’s translate this. What he is saying is that the model for future business is not the purchase of goods and services, but a price provided to a customer for an ongoing relationship to the company. The customer gets to structure the relationship in a way that provides that customer with what they need to accomplish the job(s) that the company can help them with. That can be an array of services, products, tools and even structured experiences.
What also makes this interesting is that your business is measuring the customer’s commitments to you and vice versa in operational terms even though the business model is shifting to more interactions than ever before. So you’re looking at some traditional CRM metrics like CLV, churn, share of wallet, adoption rates, etc. as they apply to a business model that has been evolving away from pure transactions for awhile now. What Tien is saying is that payment/billing etc. to him is the financial infrastructure for this new customer centered economic model - a subscription economy.
Denis Pombriant, as usual, wrote a great piece on this on his blog last week. He points out that just because a business uses a subscription model is no guarantee of success. How well the business manages it or has it managed (Zuora to the rescue here) has significant bearing on the success or failure of that business.
The same can be said for the subscription economy. Zuora is recognizing what they’ve seen coming - more and more companies are moving their business models to subscription based pricing. This is the same model that supports free software and hardware - give them the box, charge them by the month for it. How it gets managed is a whole other thing but for now, Zuora has done a service by recognizing that the customer-driven companies are realizing that the customers are willing to pay for the aggregate capabilities of the company in an ongoing way as long as the company continues to support what that customer needs to solve whatever it is they need to solve.
And that’s a good thing.
Sounds to me as if Microsoft’s original annuity model for software revenue was a bit ahead of the “Subscription Economy,” but it’s here in earnest with the Windows Azure Platform and BPOS.
James Urquhart described The intersection of open source and cloud computing in his 6/28/2010 article for CNet’s The Wisdom of Clouds blog:
Cloud computing and open-source software have been intertwined since the early days of the cloud. Vendors such as Amazon.com, SugarCRM, Rackspace, and many, many others, utilized open-source choices for everything from virtualization to data stores to user interfaces.
Today, it is fair to say that much of the cloud was made possible by both the economics and malleability of open-source software. However, despite the widespread adoption of open-source software by the cloud community, the future of open-source software is greatly affected by the cloud operations model itself.
Take the interesting discussion about the future of the LAMP stack recently. LAMP--a software stack consisting of Linux, Apache Web Server and/or Tomcat, MySQL (or another open-source database engine) and the Perl, Python and/or PHP scripting languages--plays a critical role in the world of Web applications, but as I noted recently, it may not be as critical to the cloud.
The ways in which open source is most affected by cloud computing mostly revolve around the changing roles in the IT operations sphere that I talked about in my series on DevOps. Briefly, I pointed out that cloud drives operations teams from a server focus to an application focus. This means that different teams now operate the infrastructure components that an application depends on versus the application itself.
(Credit: James Urquhart)
Now, the definition of IT infrastructure has shifted significantly in the last 15 years or so, and much of that term now encompasses software as well as hardware. Operating systems have long been considered infrastructure in the world of client-server. Middleware and data bases, such as J2EE application servers and relational database management systems have also been largely described as common infrastructure.
In fact, a very common practice in enterprise IT organizations is to create standard builds of key software infrastructure stacks, to create a common operations framework on which application code is the only variant--at least in theory.
As many of these infrastructure components shifted to open-source options, they received a tremendous amount of attention from application developers. The reason for this was two-fold. The first was the fact that these projects were available for download for free--a characteristic the average developer loves in tools and infrastructure. The second is that developers were free to manipulate the entire software infrastructure stack if they so chose--though most rarely, if ever, actually did so.
Here's the thing. Developers who wanted to play with infrastructure code were able to do so for two reasons:
The source code and instructions for building the software were freely available for manipulation on the developer's own system.
The developer could then build and deploy the software on said system to test and then utilize any changes.
What changes in cloud computing is that deployment of infrastructure software is strictly under the control of the cloud service provider. If I'm a user of Google App Engine, for example, I can't go into the source code for their management systems, change something to suit me, and push it out to the wider Google service environment.
Of course, we want it that way--it would be ridiculous to allow anyone who wants to change the way App Engine works to affect all other users of that environment. The security implications alone make that completely unreasonable, much less the other operational problems it would present.
Which means that the only users of open-source infrastructure projects in the public cloud are the cloud providers. They may see themselves as responsible users of open source and contribute back, or they may not. In any case, the incentive for the average application developer to delve into infrastructure code is weakened, if not removed outright.
The good news is that "infrastructure as a service" companies like Rackspace, Terremark, or Amazon leave so much of the software infrastructure up to their customers (such as operating systems and middleware) that it will be quite some time before most projects see this effect. In fact, it might accelerate interest in the short term.
However, as "platform as a service" offerings proliferate, and enterprise developers increasingly go for the path of least resistance, it may only be a matter of time until most cloud infrastructure is supported only by the professional operations teams behind cloud services.
Ultimately, I think one of two things will happen. Either the cloud community will find ways to ensure that open-source infrastructure projects are highly visible to end users and encourage innovation by the larger community, or most such projects will be supported only by cloud providers who--as competitive businesses--will seek opportunities for differentiation. That, in turn, may actually kill the advantages of open source for those organizations, and cause increased forking or even abandonment of open-source projects.
I'm not sure how all of this will work out, but I am fascinated by the possibility of increased competition between a shrinking pool of active contributors to common open-source infrastructure projects. Will open source suffer, or will the developer community find innovative new ways to keep open-source infrastructure accessible to application developers using cloud services?
James Hamilton reviewed Amazon SimpleDB Developers Guide by Prabhakar Chaganti and Rich Helms in his Amazon SimpleDB Developer Guide post of 6/27/2010:
The NoSQL movement continues to gain momentum. I don’t see these systems as replacing relational systems for all applications but it is also crystal clear that relational systems are a poor choice for some workloads. See One Size Does Not Fit All for my take on the different types of systems that make up the structured storage market.
The Amazon Web Services entrant in the NoSQL market segment is SimpleDB. I’ve posted on SimpleDB in the past starting back in 2007 Amazon SimpleDB Announced and more recently in I Love Eventual Consistency but… I recently came across a book by Prabhakar Chaganti and Rich Helms on SimpleDB.
Wait a second, I know that name. Rich and I worked together more than 20 years ago at the IBM Toronto Software Lab where he was Chief Image Technology Architect and I was lead architect on DB2. It’s been a long time.
The book, Amazon SimpleDB Developers Guide is a detailed guide for developers with examples in PHP, Java, and Python. Very recent features like BatchPutAttributes() are covered. Towards the end of the book, the authors show an application of Memcached with SimpleDB. The table of contents:
Preface
Chapter 1: Getting to Know SimpleDB
Chapter 2: Getting Started with SimpleDB
Chapter 3: SimpleDB versus RDBMS
Chapter 4: The SimpleDB Data Model
Chapter 5: Data Types
Chapter 6: Querying
Chapter 7: Storing Data on S3
Chapter 8: Tuning and Usage Costs
Chapter 9: Caching
Chapter 10: Parallel Processing
IndexSimpleDB really does have a developer guide from the Amazon Web Services SimpleDB team but more examples and more data is always good. If you interested in SimpleDB, check out: Amazon SimpleDB Developers Guide.
• Larry Dignan reported 10 things: Using Amazon Web Services in the enterprise in a 6/28/2010 article for ZDNet’s Between the Lines blog:
Amazon Web Services recently held a powwow for potential enterprise customers and a bevy of details emerged ranging from contracts to security to procedures to ensure employees don’t procure a cloud servers en masse for giggles.
Here’s a reporter’s notebook from Amazon Web Services’ enterprise powwow and 10 things you may not have known:
- AWS customers mentioned that there were frequently three cloud platforms they evaluated leading up to a move to the cloud. Those players included AWS, Rackspace and Microsoft’s Azure. Marc Dispensa, chief enterprise architect at MediaBrands World Wide, offered a few details about his bakeoff. Microsoft’s Azure was an easy fit for MediaBrands’ developers, but had limited SQL storage. Rackspace had a grid option, but APIs were limited and its on-demand server business was less than a year old. AWS won the deal based on features and experience with other similar customers. Here’s Dispensa’s comparison slide:
- Watch your budget when you move to AWS. A handful of AWS customers said that cloud computing is less expensive, but can be too easy to use and blow your budget. Simply put, any developer with a credit card can get provision a machine. If too many people use AWS you have cloud sprawl quickly and blow your computing budget. “It’s too easy and that can hurt your cost controls,” said Dispensa. “It’s cheaper, but can get unwieldy.” Dispensa said he put in a process where managers have to approve a developer’s request to use an AWS server and there are financial thresholds. That process is why it takes 15 minutes for an AWS server instead of 2 seconds. Pfizer’s Michael Miller, senior director of research, high performance computing, had a similar beef. “Allocate money upfront and then run the meter to avoid big surprises,” said Miller. “There are challenges when doing AWS at scale for a large number of users. Pay as you go is nice, but a debit model would even be better so it’s not so easy to spend more than you have.”
- The linchpin of Amazon’s reliability case revolves around “availability zones.” When you get an AWS computing resource it’s assigned by region. Regions include U.S. (east and west), EU (Ireland) and Asia Pacific (Singapore). These regions include at least three availability zones—a data center hub roughly speaking. AWS is architected so two availability zones can fail concurrently and data is still protected. Amazon’s aim is to eliminate any single point of failure, because IT fails all the time. AWS recommends that customers spread their assets around multiple availability zones in a region.
- Phased implementations make more sense. Amazon customers across the board said they shied away from big bang projects when moving to AWS. Jennifer Boden, director of IT at Amazon, is moving the company’s internal systems—financial, email and calendar, HR applications and knowledge management tools—to AWS, but the projects are phased. “Take a phased approach, make it easy and have no big bang projects,” said Boden.
- Security remains a top concern for CIOs. Adam Selipsky, vice president of AWS, says Amazon has to spend a lot of time talking security. The common concern is that “inside four walls is somehow more secure,” says Selipsky. Usually, AWS talks certifications, assessments and access points and the concerns go away after about an hour. Nevertheless, Steve Riley, AWS’ technology evangelist, had some animated banter back and forth with the enterprise-focused audience. Simply put, security is still a hang-up for enterprise customers, but the conversations are getting much easier. Boden said her security group put AWS through its paces and recommended that any company evaluating cloud computing bring their security team into the loop early.
AWS’ virtual private cloud service may be its most valuable product. In a nutshell, VPC sets up a virtual private network connection to a data center (right). Customers use their own IP addresses and AWS appears as an extension to current computing assets. VPC is the conversation starter for many enterprises. Pfizer’s Miller said AWS wouldn’t even be a consideration without VPC. - Think about the internal work you need to do to make AWS scale for the enterprise. Multiple AWS customers said that their applications—especially the legacy ones—weren’t built for cloud computing. Everything from development to security needs to be rethought. Pfizer’s Miller said that one move is to separate applications from operating systems with a provisioning layer so they can be managed independently. If applications and the OS are intertwined Amazon Machine Instances may be more difficult. Miller said Linux machines worked well with AWS, but Pfizer had some struggles with Windows. The provisioning layer is designed to take care of those OS hangups.
- There’s a difference between hybrid cloud and private cloud approaches. IT executives and AWS were working to put some definitions around cloud computing. The hybrid cloud approach is one where you run your own gear—after all it is already depreciated—but focus new development in the cloud. The private cloud approach is more of an equivalent to a data center. You build the assets and then deliver a cloud-like service internally. AWS executives panned the private cloud approach since you’re still buying servers, allocating labor to maintenance and still struggle to get full utilization. The theme at AWS’ powwow: Hybrid cloud is reality. Private cloud is a sales pitch. Also: Who is pushing the private cloud: Users or vendors?
- AWS may become the backbone of your friendly neighborhood software as a service provider. Many smaller SaaS companies are relying on AWS for their infrastructure and that’s no surprise. However, Lawson is using AWS for its on-demand strategy and SAP has an increasingly tight partnership with Amazon’s infrastructure as a service unit. Both Lawson and SAP are likely to bring more enterprises into the AWS party.
- AWS is still getting used to enterprise agreements. It’s one thing to charge a credit card for cloud computing. It’s quite another to put everything an enterprise wants in a comprehensive contract. Joseph Galarneau, CIO of Newsweek, moved Newsweek.com entirely to AWS’ cloud, but doesn’t have an enterprise agreement. He anticipates there will be one in the next month. Pfizer’s Miller also said he doesn’t have an enterprise agreement from AWS and put no timeline on getting one. All parties involved said there’s good back and forth about enterprise nuances and scaling enterprise-friendly agreements at a later date.
• Chuck Hollis takes on Carl Brooks’ Amazon touts Pfizer and other wins in the enterprise (see below) article in a Why Public Clouds Will Never Win The Enterprise post of 6/28/2010 to EMC’s Service Provider Insider blog:
One of the most interesting storylines in the cloud debate has to do with Amazon.
While they were undoubtedly one of the first "name brand" public cloud services (and perhaps the best known), they continually struggle to adapt their offer to meet the needs of large enterprises.Last Friday, Amazon held an invitation-only event to proclaim their "readiness" for large enterprises. The event was duly reported on by Carl Brooks for SearchCloudComputing.com
But -- to my eye -- the event had precisely the opposite effect: it amply demonstrated why public cloud operators like Amazon will never have more that a small sliver of enterprise business until they change their ways.To Start With
I'd encourage you to go read the article first, since I'm going to spend the rest of this post commenting on what was said.
Most of the article (and Amazon's marketing to enterprises) center around Pfizer's use of AWS.
"Michael Miller, senior director at Pfizer Global Research and Development, said that his firm had embraced Amazon Web Services (AWS) only after the advent of Amazon's Virtual Private Cloud (VPC) service, which walls off resources from the public Internet. He said that it was a good choice for compute-heavy workloads, as long as it remained firmly under Pfizer's thumb."My first observation is that Mr. Miller does not appear part of Pfizer's IT organization. I am reasonably sure that Pfizer's IT organization was not thrilled by the use of AWS. Indeed, it's quite likely that a group of R+D types went outside the organization and forced the situation.
Amazon's VPC offering is nothing unique: many service providers have offered the same "private network" capabilities for quite some time. It's nothing more than table stakes in any enterprise service provider discussion.Also notice the "compute-heavy workload" characterization. In the landscape of enterprise IT, these tend to be few and far between. Indeed, most enterprise workloads are I/O and network heavy if anything.
The Story Continues
"Compliance is the bugbear. Does it actually allow us to deploy with the policies we've set?" he said. The answer was yes, but only in AWS VPC, and only after Amazon took pains to get a rigorous SAS 70 Type II certification and an extensive "cultural dialogue" with internal auditors.
"One of the things that was very useful was the SAS 70 report; we've also gone through several audits on our side and their side," Miller said. He added that auditors had to get used to virtualization technology before being able to successfully clear Pfizer's AWS operations."
Lots to comment on here.
If you're not familiar with SAS 70 Type II audits, I'd recommend you check here. Like most audits, it does not guarantee an environment as secure, compliant, etc. -- it simply states that the required processes and controls are in place.
While SAS 70 audits are a reasonable starting place, they create three problems. First, if you've ever read one, their interpretation requires a great deal of expertise. Second, they only refer to results over a specific period of time, since -- by definition -- they are historical documents. Third, any process and/or controls are only as good as the people who enforce them, and that tends to be where the problems occur.
Note the wry comment on "extensive cultural dialogue". I think Mr. Miller misses the a key point: most technology auditors I have met are comfortable with virtualization up to a point: intermixing of different workloads owned by a specific organization and/or tenant, e.g. Pfizer in this case.
What they're most definitely *not* comfortable with is intermixing of different workloads from multiple organizations in a single pooled infrastructure. Amazon's AWS model doesn't allow the segregation of workloads and infrastructure -- everything is one giant pool.
"He said Pfizer had built custom Amazon Machine Images, from the kernel up, so that it could verify integrity at every level; that kind of detail took the place of verifying hardware and physical integrity.
"We had to work with [auditors] to adjust their expectations…no, you don't get 'this chip, that box, that disk,'" he said. Instead, they get code review and kernel versions. Now that the heavy lifting is done, Miller guessed that Pfizer now consumes about 80% of CPU work on new projects in AWS, which makes up about 50% of total workload.
The hand-crafting of machine images is nothing unique to organizations who take IT compliance seriously -- like Pfizer. Every hunk of code is vetted and change-controlled. Deviations are detected and reported on. EMC, in fact, sells tools that assist with this IT compliance capability.
However, this not infrequent requirement had to be painfully retrofitted into Amazon's environment.
The comment on "this chip, that box" etc. again misses the point: had Amazon been able to go back to Pfizer and say "this is your pool of physical resources for you to virtualize, no one else will use them", I would bet that the discussion would have been a lot more productive.
The workload figures are potentially misleading: Mr. Miller is most likely commenting on his particular group's use of AWS and not all of Pfizer. I read this as "half of all workloads in this group are new projects; 80% of those go to AWS", which would mean that approximately 60% of all workloads in Pfizer's Global Research and Development Group are done using more traditional means.
The Pfizer director also said there were limitations to what his company would run on Amazon. Any older applications were tied to hardware, and sensitive data would probably never leave the firm's networks. Miller did say there were significant operational savings, something other panelists agreed with.
The claim on operational savings rings true -- but that benefit is inherent in *any* service provider model, and not necessarily unique to Amazon's offering.
What I Think Happened
A few disclaimers before we get started. I think Amazon offers a fine service for its intended use -- my comments are not meant to discredit Amazon's AWS, only its appropriateness for the vast majority of enterprise-class workloads.
Similarly, I believe that Pfizer runs a fine IT organization. But I think this one got away from them.
Pfizer's primary business is developing drugs. One would think that Pfizer's global R+D function is quite privileged in many regards -- it's their work that makes money for the company.
I believe that -- many years back -- their needs for dynamic and on-demand compute/storage services weren't being met by the traditional IT organization. This is not a slam on Pfizer's IT organization -- the same situation generally exists in many organizations that have a pronounced R+D function.
I think the R+D group found AWS and ended up embracing its model and toolset. It was exceptionally easy to consume -- anyone could open an account. Applications got written that produced results. No need to go to IT budgeting and planning meetings -- just get on with it!
Usage of the service grew over time.
At the same time, I would be surprised if vocal concerns weren't being raised by other corporate functions: the compliance and security guys, the auditors, and mainstream IT. Their concerns, while probably valid, probably went unheeded by the researchers who now had "their" cloud to go use.
Time went on. Usage continued.
And eventually Pfizer was faced with a hard choice: either invest in making AWS compliant enough for the intended usage, or pull all those applications back from AWS and convert them to something else that was more controlled.
I'm sure this boiled down to two numbers: plan A and plan B. The implied level of effort stated in this article (just think about how long and hard this effort must have been for all involved) gives much credence to the "stickiness" of applications developed using the AWS toolset, and the difficulty associated with moving them to other environments.
Simply put, I think Pfizer got pregnant.
What Should Have Happened
I believe that Pfizer's R+D group has enough scale to matter: they could consume their own cloud, or at least justify something special be built to their liking.
Were we doing things over, I'd recommend that they build their environment in virtual machines (preferably VMware), using tools of their choosing that were associated with the VM, rather than a public cloud operator's unique service.
Now they'd have some interesting options.
First, they'd be able to stand up a critical mass of internal infrastructure behind the firewall -- should they choose to. I am no expert, but I would bet that there would be less extended grinding of gears with their compliance teams using this scenario.
Second, they'd be able to go to a compatible service provider and use a hosting or co-lo model to run their environment -- with no intermixing between Pfizer's workloads and other users. If building data centers or carrying assets on the balance sheet were a concern, I'm sure there are dozens of great SPs who would love to help them out with this.
Third, they'd get some options as to what to run internally or externally. It'd be basically the same application environment; the only decision would be where to run it. Compare this with Pfizer's AWS implementation: there's one and only one place that workload can run, and that's in an Amazon data center.
And no IT professional wants to be put in that sort of position.
What Does This Mean?
I see the Pfizer / Amazon story as essentially a one-off: very unlikely to be repeated by Amazon.
Far better options are available in the market today: private cloud architectures that are very easy to deploy internally, as well as many dozens of proficient service providers who can use the exact same technology to create a compatible and controlled environment that will undoubtedly be far easier to pass compliance concerns.
Not only that, the ability of SPs to combine more traditional hosted and co-lo approaches with pooled virtualization is a big win: the economics of a cloud model without the compliance headaches.
I think this sort of story also will result in a new term being popularized: "cloud portability" -- the ability for a tenant of any external service to pack up and move to a new location without spending tens of millions of dollars for the privilege of doing so.
And finally, I can't imagine a more cautionary tale for any IT organization that falls behind the needs of its most valuable knowledge workers.
Chuck is VP, Global Marketing CTO for EMC Corporation.
• Carl Brooks (@eekygeeky) reported Amazon touts Pfizer and other wins in the enterprise on 6/25/2010 on SearchCloudComputing.com’s Cloud Computing News:
NEW YORK CITY -- Amazon.com trotted out some big customer wins to a packed, invitation-only event in Manhattan this week to show off its readiness for the enterprise. Yet users remain cautious, with many preferring to set their own pace into the public cloud.
Michael Miller, senior director at Pfizer Global Research and Development, said that his firm had embraced Amazon Web Services (AWS) only after the advent of Amazon's Virtual Private Cloud (VPC) service, which walls off resources from the public Internet. He said that it was a good choice for compute-heavy workloads, as long as it remained firmly under Pfizer's thumb.
"Compliance is the bugbear. Does it actually allow us to deploy with the policies we've set?" he said. The answer was yes, but only in AWS VPC, and only after Amazon took pains to get a rigorous SAS 70 Type II certification and an extensive "cultural dialogue" with internal auditors.
"One of the things that was very useful was the SAS 70 report; we've also gone through several audits on our side and their side," Miller said. He added that auditors had to get used to virtualization technology before being able to successfully clear Pfizer's AWS operations.
He said Pfizer had built custom Amazon Machine Images, from the kernel up, so that it could verify integrity at every level; that kind of detail took the place of verifying hardware and physical integrity.
"We had to work with [auditors] to adjust their expectations…no, you don't get 'this chip, that box, that disk,'" he said. Instead, they get code review and kernel versions. Now that the heavy lifting is done, Miller guessed that Pfizer now consumes about 80% of CPU work on new projects in AWS, which makes up about 50% of total workload.
The Pfizer director also said there were limitations to what his company would run on Amazon. Any older applications were tied to hardware, and sensitive data would probably never leave the firm's networks. Miller did say there were significant operational savings, something other panelists agreed with.
"It's a 20% to 30% decrease in our cost over building a data center and depreciating it over time," said Joseph Galarneau, CIO and SVP of Operations for Newsweek magazine. He said the move into Amazon was a fresh start for Newsweek's online operations, which had been channeling media content through various partners and outlets like MSNBC.com and Newsweek owner The Washington Post.
A decision had been made to launch Newsweek on its own platform, so Galarneau got started with a entirely new, cloud-friendly content management application and ditched his mess of old systems. That was a primary reason he was able to move into AWS, he said, because he could leave so much behind and start over fresh. Galarneau also said he had to get used to using AWS; it wasn't a cut-and-dried co-lo server or an outsourced service.
"They kind of give you a box of parts," he said, pieces that his team had to put together.
Amazon's enterprise dealings
Amazon still does not have industry-standard enterprise agreements with its customers, but apparently it's a hot topic.
"They understand they're [now] dealing with enterprises, it's just that they don't have the years of experience in dealing with them, like IBM or somebody," Galarneau said. He also said Newsweek and Amazon were in negotiations with AWS, but couldn't hazard a guess as to what the agreement might look like or when it might be finished.
"When are we going to see core enterprise applications moving out on to the cloud? Answer me that," said Hossur R. Srikantan, CIO at IT management consultancy Rav Tek. He said the answer was "probably never" and said he's tired of hearing the same story. Srikantan said the basic market for cloud computing infrastructure services was well established and unlikely to change. "It's been 'day one' in the cloud for two, three years now," he said.
Srikantan said major enterprises would take cloud services for non-critical areas, but to him the panel was a perfect example of what was already known about cloud.
"Take Newsweek -- they are in a transitional model," he said, and a Web-based, digital media product, so they were very good candidates for using cloud computing. Pfizer was doing R&D on Amazon, but that was a well-established use case, and another presenter, 650-employee QlikTech, was a Web application company more than anything else.
"Are they a Web service? Or cloud service?" Srikantan asked. "Does it matter?"
• Dmitriy Samovskiy’s CohesiveFT VPN-Cubed as Networking Fabric of the Intercloud post of 6/23/2010 observes:
It has been about 6 months since I last blogged about work, so I figured an update may be in order, especially since today CohesiveFT announced availability of VPN-Cubed on Flexiant’s cloud offerings.
We’ve been very busy on VPN-Cubed engineering side. Along with features already on the roadmap, we delivered several enhancements that were directly driven or requested by VPN-Cubed users. On the product support side, we continued to expand a range of devices with which VPN-Cubed can do IPsec interop, which now include even ones I personally have never heard about before. We grew our experience and expertise in the area of troubleshooting intra-cloud and cloud-to-datacenter connectivity issues (there are many!). We’ve also worked on a few projects that required non-trivial topologies or interconnects, successfully mapping customer requirements to VPN-Cubed capabilities.
One theme that I have had in my head for some time now, is VPN-Cubed as the networking fabric of the Intercloud. Let me explain.
VPS was a predecessor of modern IaaS clouds. In VPS land, boxes are usually provisioned individually, one by one. Typical setups in VPS consisted of 1, 2 or 3 boxes. Networking 3 independent boxes together is relatively straightforward.
At the beginning of IaaS era, I imagine most setups were also 1 or 2 boxes. But as IaaS is gaining ground, topologies headed to the cloud are getting bigger, more complex and more dependent on access to external resources. Setting up networking consistently is becoming a bigger deal. But it’s not the end.
One of the roles of Intercloud is providing customers with an alternative (competition, in other words) – if one doesn’t like cloud A, she may take entire topology to cloud B. I’d say 99 of 100 public cloud justification documents being submitted to CIOs worldwide today include a statement saying something like this: “If this cloud provider fails to deliver what we need at a level we need it, we will switch to another provider.” This is actually not as easy in practice as it may sound.
Each cloud’s networking has unique aspects, no two are alike. Public IPs, private IPs, dynamic or not, customer assignable or not, eth0 private or public, cloud-provided firewall exists or not, peculiarities of firewall – these are some of the differences (as of today, I have set up boxes in 6 IaaS clouds with admin endpoints facing public Internet – I have seen many network setups). Taking images of N boxes from one cloud and dropping them in another cloud is well understood, recreating one cloud’s networking in another cloud is where the challenge is.
It is here where I think VPN-Cubed shines as a customer-controlled network abstraction – it’s an overlay built on top of service provider’s network, which allows it to be identical no matter what the underlying infrastructure looks like.
Same story plays out when an application is hyper-distributed and runs in multiple clouds or multiple regions of one cloud (where regions are connected via public Internet). And here as well VPN-Cubed provides an abstraction that allows one to treat all compute resources as being on the same network, regardless where they are actually located at the moment.
At the same time, VPN-Cubed can be appealing to topologies that don’t care about Intercloud. Networking and network security are areas that don’t get enough attention from cloud developers today, because developers are used to working within a perimeter. Excessively wide-open security group setups, using public IPs instead of private for communications, disabled local firewalls – these are all time bombs. They don’t affect the app right now (”look, it works!”) but they can be catastrophic over time when they could become an attack vector. For such topologies, VPN-Cubed provides a virtual perimeter that confines authorized communications to a mutually-authenticated tunnel encrypted end-to-end (are you sure you want to continue forcing your incoming web traffic to HTTPS but not encrypting writes and reads from app servers to database? or do you think application-level encryption could be better, faster or easier to maintain than transport-level?)
To get started with VPN-Cubed, visit http://cohesiveft.com/vpncubed. If you have a question about how VPN-Cubed can help in your particular use case, you can ask here.
• Lori MacVittie (@lmacvittie) gave CohesiveFT props in this 6/28/2010 tweet:
RT @Route_B: CohesiveFT VPN-Cubed as Networking Fabric of the Intercloud http://tinyurl.com/26nsowp << CohesiveFT has some cool stuff!
<Return to section navigation list>
Atom 1.0 site feed
