Friday, August 26, 2011

Windows Azure and Cloud Computing Posts for 8/26/2011+

A compendium of Windows Azure, SQL Azure Database, AppFabric, Windows Azure Platform Appliance and other cloud-computing articles. image222

image433

Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:

Azure Blob, Drive, Table and Queue Services

Brent Stineman (@BrentCodeMonkey) continued his “Year of Azure” series with Page, after page, after page (Year of Azure Week 8) on 8/25/2011:

To view the live demo of this solution, check out Episode 57 of Microsoft’s Channel 9 Cloud Cover Show [see article below.]

imageLast week, I blogged on writing page blobs one page at a time. I also said that I’d talk more this week about why I was doing this. So here we are with the second half of the demo, the receiver. Its going to download the page blob as its being uploaded.

imageWe’ll skip over setting up the CloudStorageAcount, CloudBlobClient, an CloudBlobContainer (btw, I really need to write a reusable method that streamlines all this for me). This works exactly as it did for the transmitter part of our solution.

The first thing we need to do is pull a list of blobs and iterate through them.. To do this we create a foreach loop using the following line of code:

foreach (CloudPageBlob pageBlob in container.ListBlobs().OfType<CloudPageBlob>())

Note the “linqy” OfType part. My buddy Neil Mackenzie shared this tip with me via his new Azure Development Cookbook. It allows me to make sure I’m only retrieving page blobs from storage. A nice trick to help ensure I don’t accidently throw an exception by trying to treat a block blob like a page blob.

Quick product plug… I highly recommend Neil’s book. Not because I helped edit it, but because Neil did an excellent job writing it. There’s a SLEW of great tips and tricks contained in its pages.

Ok, moving on…

Now I need to get the size metadata tag I added to the blob in the transmitter. While the line above does get me a reference to the page blob, I didn’t populate the metadata property. To get those values, I need to call pageBlob.FetchAttibute. I follow this up by creating a save name for the file and associating it with a file stream.

pageBlob.FetchAttributes(); // have to get the metadata
long totalBytesToWrite = int.Parse(pageBlob.Metadata["size"].ToString()); 

//string fileName = string.Format(@"D:\Personal Projects\{0}", Path.GetFileName(pageBlob.Attributes.Uri.LocalPath));
string fileName = Path.GetFileName(pageBlob.Attributes.Uri.LocalPath);
FileStream theFileStream = new FileStream(fileName, FileMode.OpenOrCreate);
Now we’re ready to start receiving the data from the blobs populated pages. We use GetPageRanges to see where the blob has data, we check that against the last endpoint we read, and we sleep for 1 sec if we’re already read all the available information (waiting for more pages to be written). And we’ll keep doing that until we’ve written the total size of the blob. 
long lastread = 0; // last byte read
while (totalBytesToWrite > 0)
{ 
foreach (PageRange pageRange in pageBlob.GetPageRanges())
{
// if we have more to write…
if (pageRange.EndOffset > lastread)
{
// hidden region to write pages to file
}
}
Thread.Sleep(1000); // wait for more stuff to writ
}

Ok,there’s a couple things I need to call out here. My sample assumes that the pages in the blob will be written in succession. It also assumes that we’re only going to write the blobs that exist when my application started (I only list the blobs in the container once). So if blobs get added after we have retrieved our list, or we restart the receiver, we will see some odd results. So what I’m doing is STRICTLY for demonstration purposes. We’ll talk more about that later in this post.

The last big chunk of code is associating the blob with a BlobStream and writing it to a file. We do this again, one page at a time…

BlobStream blobStream = pageBlob.OpenRead(); 

// Seek to the correct starting offset in the page blob stream
blobStream.Seek(lastread, SeekOrigin.Begin);

byte[] streambuffer = new byte[512];

long numBytesToRead = (pageRange.EndOffset + 1 – lastread);
while (numBytesToRead > 0)
{
int n = blobStream.Read(streambuffer, 0, streambuffer.Length);
if (n == 0)
break;

numBytesToRead -= n;
int bytesToWrite = (int)(totalBytesToWrite – n > 0 ? n : totalBytesToWrite);
lastread += n;
totalBytesToWrite -= bytesToWrite;

theFileStream.Write(streambuffer, 0, bytesToWrite);
theFileStream.Flush(); // just for demo purposes
}
You’ll notice that I’m using a stream to read the blob and not retrieving the individual pages. If I wanted to do that, I’d need to go to the Azure Storage REST API which allows me to get a specific range of bytes from a blob using the Get Blob function. And while that’s fine, I can also demonstrate what I’m after using the stream. And since we’ve already established that I’m a bit of a lazy coder, we’ll just use the managed client.

The rest of this code snippet consists of some fairly ugly counter/position management code that handles the writing of the blob to the file. The most important part of this is that we use bytesToWrite to decide if we write the entire 512 byte buffer, or only just as much data as remains in our blob. This is where my “size” attribute comes in. I’ve used that to determine when the file stored in the series of 512 byte blocks actually has ended. Some files may be forgiving of the extra bytes, but some aren’t. So if you’re using page blobs, you may need to make sure you manage this.

So why are we doing all this?

So if you put a breakpoint on the transmitter app, and write 3-4 pages, then put a breakpoint in the receiver app, you’ll see that it will read those pages, then keep hitting the Sleep command until we go back to the transmitter and write a few more pages. What we’re illustrating here is that unlike a block blob, I can actually read a page blob while it is being written.

You can imagine that this could come in handy if you need to push large files around, basically using page blobs as an intermediary buffer for streaming of files between two endpoints. And after a bit more work and we can start adding restart semantics to this demo.

Now my demo just shows us going in a sequential order through the blob (this is the “STRICLY for demonstration” thing I mentioned above). If we start thinking that our buffers don’t have to be 512 bytes but can instead be up to 4mb, and a 4mb operation against Azure storage may take a few seconds to process, we start thinking about maybe multi-threading the upload/download of the file, potentially realizing a huge increase in throughput while also avoiding delays that would result in me having to wait until the upload completes before starting the download.

So the end result here is that my demo has little practical application. But I hope what it has done is made you think bit about the oft overlooked page blob. I’m just as guilty as you for this oversight. So in closing, I want to thank Sushma, one of my Azure trainees this past two weeks. Shushma, if you read this, know that your simple question helped teach me new respect for page blobs. And for that… thank you!

BTW, the complete code for this example is available here for anyone that wants it. Just remember to clear the blobs between runs.

Steve Marx (@smarx, pictured below) reported the availability of Episode 57 - Windows Azure Page Blobs with Brent Stineman on 8/26/2011:

Join Wade and Steve each week as they cover the Windows Azure Platform. You can follow and interact with the show at @CloudCoverShow.

imageIn this episode, Steve and Wade are joined by Brent Stineman, a Windows Azure MVP, who shows us how to use page blobs in Windows Azure to do random access reads and writes. He even shows a demo of downloading a blob while it's being created.

In the news:

Brent's blog posts about page blobs:

*Correction: Smarx had it wrong (and convinced Wade and Brent)! Page blobs have a maximum size of one terabyte, not two terabytes as stated in the show.

Is Brent standing on a box or are Steve and Wade standing in a trench?

<Return to section navigation list>

SQL Azure Database and Reporting

Grant Fritchey described Exploring SQL Azure with emphasis on performance tuning in an 8/23/2011 post to Red Gate Software’s Simple Talk blog:

imageI'm spending a little bit of time each week trying out a few things in SQL Azure. I'm convinced that we're going to be spending time tuning our SQL Azure databases much the same way as we tune our regular SQL Server databases. That is to say, we won't tune the databases at all until there's a major issue; the CIOs favorite report runs too long, developer's code slows to a stand still, or you hit Microsoft's resource threshold and they kill the connection. What? That's not one you thought of? Well, come on, if the CIO starts complaining, you know it's going to come down on your head pretty quick. oh, you mean that threshold thing? Yeah, there's a whole list of constraints on SQL Azure. Many of them make perfect sense. A few are probably going to change in the future, but all the way down at the very bottom of the sheet there's a line that reads:

imageIn order to provide a good experience to all SQL Azure Database customers, your connection to the service may be closed due to the following conditions:

  • Excessive resource usage
  • Connections that have been idle for 30 minutes or longer
  • Failover because of server failures

And a note:

Maximum allowable durations are subject to change depending on the resource usage.

imageMicrosoft is aware that they are going to hear from people regarding performance. After all, you can't call the server room and suggest something is up with the server. Your information is in the hands of Microsoft. So, they've wisely published a performance tuning guide. It makes for a very interesting read. Seriously, pick it up and read it. I'm not being nasty. I think it's useful. It's also informative about where MS sees problems coming up. According to this document, app dev in SQL Azure is going to be almost as much work as it was in SQL Server. They've got recommendations on reducing round trips to the system to avoid latency, they're recommending against the use of cursors (shock), and they have another white paper just on query performance optimization (more on that in a minute). They want you to make sure your stats are up to date and suggest running sp_updatestats regularly. Now that one is interesting. How? There is no SQL Agent, so you need to build something, somewhere, that's going to go out to your SQL Azure instance and run this maintenance routine.

The best part of the document though is at the bottom (what is it with the bottom of the documents?) where they outline some of the methods and principles being employed in the throttling. Even more important, they actually have return codes that will tell you why your process was killed. This is great information, invaluable, and something anyone that is moving databases into the Microsoft cloud has to know. For example, currently (remember, all this is subject to change), there is what MS is called a "Throttling Cycle" that lasts 10 seconds. That's how long your system could be throttled, but, if your system was throttled in the preceding Throttling Cycle, it makes it more likely that it will be throttled again (not absolute, just more likely). They even provide you with a list what is being monitored. Although the thresholds are described, the values are not supplied. You'll have to figure those out yourself, the hard way. The throttling you receive (and that's a fun sentence to type) will depend on what MS thinks it has to do. It can stop updates & inserted, all writes of any kind and finally, all reads and writes. When it rejects your connection, it will include a reason code stating why. I'd strongly suggest you talk to your developers so that they know about this and write their code to take it into account.

The guide on query performance optimization mentioned above, is just a blog post that shows how to use DMOs to retrieve performance data. It's useful, but I think you're going to want more complete guides.

I've already put up a blog post about the common performance tuning DMOs that are all still available in SQL Azure. What I haven't spent much time talking about is execution plans and query performance tuning. I'm setting up more data into my test database and I'll by playing around with SQL Azure, execution plans and queries, just to see if I find anything interesting. In the mean time, I strongly recommend taking advantage of the information MS is providing.

<Return to section navigation list>

MarketPlace DataMarket and OData

The OpenData.org site reported OData Service Validation Tool Available on Codeplex in an 8/26/2011 wiki article:

imageWe are happy to announce that the OData Service Validation Tool is now an Outercurve project and is available on Codeplex (http://bit.ly/nqEVlH). We released it in such a way that the tool is fully open source and we will be able to accept contributions. This is something we are very excited about since we saw this project as a community project from the very beginning. Please feel free to blog/tweet about this and let the OData world know.

You can immediately fork it or download the source code, play with it, deploy in your environment and experiment. We are going to be improving the Codeplex content and documentation (e.g. how to write new rules) over the next weeks. We will also populate the issue list with issues currently in our internal issue database so that you can start tackling some issues if you are itching for contributing to the project.

We will continue to have a hosted version updated with the same 2 week cadence from the Codeplex branch at http://validator.odata.org.

Thank you for your patience while we worked through all the legal issues and for your continued support for the project. We are looking forward to improving the interoperability of OData as a community through the OData Service Validation Tool.

Please let us know if you have any questions and jump in the discussion if you haven't already done so either on this mailing list or on the discussion list on project page.

<Return to section navigation list>

Windows Azure AppFabric: Apps, Access Control, WIF and Service Bus

The News Editor of MSDynamicsWorld.com posted Microsoft Dynamics CRM and Azure AppFabric: Weighing the Data Integration Options on 8/25/2011:

imageCustomer relationship management (CRM) does not exist in a world unto itself. Companies want to make use of their CRM data in other business applications. But where to start with Dynamics CRM 2011 or Dynamics CRM Online?

"Today, the way we work with CRM is that whenever data changes there are 3 ways to access that data for an integration," says Girish Raja, technical evangelist for Dynamics CRM at Microsoft. "One is workflow built through Workflow Foundation; there's also the plug-in model, which developers can build to pretty much do everything; and the third would be [access to] the AppFabric Service Bus, which is technically a plug-in that's already pre-packaged into CRM."

In terms of tools, Dynamics CRM customers can take a few different approaches to bringing an integration to life. There is BizTalk, with improved integration to CRM 2011, there are commercial middleware solutions featuring pre-built integration capabilities and simplified management tools, and there is still the custom route using things like Azure AppFabric or custom plug-ins. The choice will depend on lots of factors including your network architecture, budget, IT resources, and business and technical requirements.

Of the integration options available today, the newest and probably the most rapidly evolving is Azure AppFabric, a part of Windows Azure that offers a platform-as-a-service (PaaS) for workflow and middleware services. In the context of Microsoft Dynamics CRM, it pays to understand whether AppFabric is a strong candidate for your company's integration needs.

In the rest of this article we'll look at the case for choosing (or rejecting) an AppFabric solution over other options. Then we'll follow up with part two, where we'll hear some reasons why a commercial middleware option can often win out over rolling your own.

Integrate with Dynamics CRM using Azure AppFabric

Azure AppFabric is an evolving product that offers a good new option for certain business and IT needs, like if you need updates from Dynamics CRM to reach a line of business application across a network. AppFabric is not the right solution for every situation - it provides more value when it is connecting systems in different domains (like CRM Online and an on-premise app) than any other scenario. And most IT managers will want to rule out a commercial integration option (which we'll discuss in part two) before choosing to build their own application to listen to an AppFabric endpoint.

"AppFabric is evolving with Azure - new features and services are rolled out in a rapid timeline," says Raja.

Azure AppFabric has a pre-built plugin for the Dynamics CRM 2011/Online platform that allows you to create a notification service in Dynamics CRM, hook it to an AppFabric Service Bus in your Azure instance, and connect AppFabric to your (very basic) on-premise custom listener in a matter of minutes.

image

Experts point to the firewall-friendliness of an Azure AppFabric solution. The on-premise listener connects out to Azure for messages rather than receiving incoming network communication, so there are usually no extra firewall holes needed. Common scenarios might include connecting Dynamics CRM on-premise to cloud applications, or alternatively, connecting Dynamics CRM Online to on-premise applications. A third likely scenario might be connecting applications across divisions or disconnected organizations who need limited data sharing.

As for the CRM notification capabilities, Raja explained that there is a range of standard notification activities related to the creation and update of CRM entities. "All kinds of events that Dynamics CRM can expose it does already, and I haven't heard of any case of limitations where customers have asked about," he said. And additional notifications could be customized through the use of workflows, the plug-in framework, or through customization of the AppFabric service bus plug-in for Dynamics CRM.

"In my opinion AppFabric technologically is ready for production use, says Sebastian Waksmundzki , senior Dynamics architect at AlfaPeople and author of the Mind The Cloud blog covering Azure, .NET, and XRM. "[My company] has done a lot of tests and R&D, and we do suggest this approach mainly for integration between online systems and existing on premise systems."

AlfaPeople has developed its own AppFabric-based integration between Dynamics CRM and Dynamics NAV 2009, as well as an integration to its IT Service Management (ITSM) product, AlfaPeople ITSM 2011.

While partners say the interest appears to be growing, use of Azure AppFabric for Dynamics CRM integration situations is still in its infancy. According to Andrew Zimmer of Avtex, clients looking for new integration solutions love the idea and the approach but have yet to pull the trigger and implement it. "The reason we haven't see more movement towards Azure AppFabric is likely because customers tend to stay with technology that they are familiar with," Zimmer says. "If they currently use Scribe or SSIS in-house then they are not likely to switch technologies, [nor should they necessarily]." But, he adds, given the response from demonstrations, Avtex expects to see more clients signing on to this approach in the future.

AppFabric pricing is based on connections with both "pay as you go" and bundled pricing. A connection to Dynamics CRM will require at least two connections. Charges for transactions and outbound bandwidth also apply.

Given the importance of Windows Azure in Microsoft's cloud strategy for Dynamics products, expect to see continued commitment to AppFabric. Zimmer and Waksmundzki both give Microsoft high marks so far in supporting the product as a technically viable PaaS option for Dynamics CRM.

"I feel like Microsoft has shown a great deal of commitment to the Azure platform within CRM 2011," says Zimmer. "They have done a lot of work to make it easy to connect to Azure. It is evident by just how easy it is to setup an integration between CRM Online and Azure AppFabric. You are able to send a message to Azure with a few simple clicks within the plugin registration, a place that developers have grown to feel very comfortable. "

Coming up, we'll look at some reasons to forgo the shiny new Azure technology in favor of the more established options out there, including the benefits of a commercial middleware options.

Thiago Almeida (@thiago_bagua) and Emil Velinov (@emilvel) presented a Guidelines and Best Practices for Migrating .NET Web Services to the Azure Platform breakout session to TechEd New Zealand 2011 on 8/25/2011:

image72232222222This session will briefly discuss the different types of services supported in WCF 4, followed by detailed guidance on migrating a service implementation from an on-premises deployment to the Windows Azure platform. The session will address the Azure hosting options and considerations, the steps for securing the service using the Azure Access Control Service (ACS), and finally - enabling a hybrid solution by connecting to existing on-premises services via the Azure AppFabric Service Bus.

<Return to section navigation list>

Windows Azure VM Role, Virtual Network, Connect, RDP and CDN

imageNo significant articles today.

<Return to section navigation list>

Live Windows Azure Apps, APIs, Tools and Test Harnesses

Bryan Swan (@brian_swan) posted Pie in the Sky (August 26, 2011) to the Window Azure’s Silver Lining blog:

imageAs Larry and I investigate OSS and device development on the Windows Azure platform, we come across lots of interesting links and do lots of reading. And, since we can’t write about everything we read, we thought we’d simply share the best of what we come across on a weekly basis. The links below are what we came across this week.

Why “pie in the sky”? Well, I liked the way it sounded for one (you can blame me if you find the name to be cheesy). But, mostly I like the irony in it. By one definition, “pie in the sky” is “an idea or plan that seems good but is not likely to be achieved.” I like to think that the links we find show that OSS and device development on the Azure platform are not only a good ideas, but are ideas that are being achieved. So, look for our “pie in the sky” posts on Fridays. Here’s our first installment…

“There’ll be pie in the sky when you die” is a phrase from “The Preacher and the Slave” tune by Joe Hill, an early member of the International Workers of the World (IWW), a.k.a. the “Wobblies.” Joan Baez made Joe Hill famous among my generation with a performance of the tune of the same name (“I dreamed I saw Joe Hill last night …) at Woodstock in 1969. My interest in the radical side of the US labor movement led me to a part-time job as a studio and transmitter engineer at KPFA, Berkeley, CA, the first listener-sponsored FM radio station in the US, while I was going to school at Berkeley High and UC Berkeley.

Nathan Totten (@ntotten) announced Windows Azure Toolkit for Social Games & Tankster Version 1.0 in an 8/26/2011 blog:

imageToday we released the first stable version of the Windows Azure Toolkit for Social Games and the Tankster game. This is the third drop of the code we have done since we first introduced this project about 6 weeks ago. This release adds several new features, improves the performance and stability of the server APIs, and contains many user interface improvements to the sample game. These tools will help make it easier for developers to build great game experiences across devices in less time and lower cost. While this release represents a significant milestone in this project, we have much more planned and will continue to keep the updates coming.

SNAGHTMLeb8cd3a

imageThis release of the Social Game Toolkit and Tankster contain the following features.

Server APIs
  • Authentication (Uses Windows Azure Access Control Servers)
  • Game Management
  • Eventing
  • User Profiles
  • Leaderboards
  • Inventory
  • Real-Time Communication (Chat, etc.)
  • Notifications
  • In-App Purchases
HTML5 Features
  • Game Play
  • Turn Management
  • Animations
  • Inventory
  • Leaderboard
  • Events
  • Chat (Real-Time Communication)
  • Social Sharing (Like, Tweet, etc.)
  • Audio

Please download the toolkit and give game a try. I will continue blogging about the architecture of the toolkit and game so keep an eye on this blog. And, as always, any feedback is welcome.

Nate is presenting a session at the //BUILD Conference in September. What do you bet he’ll talk about the Social Games Toolkit?

Eric Nelson (@ericnel) reported Autoscaling Windows Azure applications coming this autumn in an 8/26/2011 post to his IUpdateable blog:

imageTechnically there are lots of way to do this already (including AzureWatch which I blogged on back in June) but it was great to see this week details on what we are working on as part of the new Windows Azure Integration Pack for Enterprise Library (Also check out details of what else may appear in this pack).

imageGrigori has shared details of the thinking and the scenarios being addressed in a upcoming Autoscaling Application Block.

In brief, the block will pull implement rules that look at data to decide on appropriate actions. Simple:

  • Rules
    • Constraint – min/max, associated with when
    • Reactive – based on a KPI
  • Actions
    • Instance Scaling: varies number of instances
    • Throttling: limit or disable expensive ops when thresholds are hit
    • Notifying: alert rather than act
    • Custom: hooks to add more
  • Triggers (data to react to)
    • Windows Azure Diagnostics tables (e.g. CPU utilization)
    • Windows Azure Storage API (e.g. # unprocessed orders on a queue)
    • Windows Azure Storage Analytics (such as transaction statistics and capacity data)
    • application data store (your custom metric)
    • real-time data sources instrumented in the app (e.g. # active users, # tenants, # documents submitted)

Check out the full post.

Related Links:

<Return to section navigation list>

Visual Studio LightSwitch and Entity Framework 4.1+

Jenni Ripley asked is LightSwitch 2011: Truly "Coding Optional"? in an 8/25/2011 interview with Steve Goguen for the infoQ blog:

imageVisual Studio LightSwitch 2011, the development tool intended to help non-developers produce business applications, was released earlier this month. Ever since the beta release last year, it has been the subject of some debate over the intended audience for the product — specifically, whether it is actually meant for people with no programming background, or if it would be better targeted toward developers with junior or entry-level experience.

imageInfoQ recently spoke to Steve Goguen, a developer who has been working with Visual Studio Lightswitch, to get his opinions on the product and its usage.

image222422222222InfoQ: How long have you been working with the product, and how many LightSwitch applications do you have currently in use?

We started using LightSwitch a few months back and have since pushed out two applications. One is an early phase of our new product database and the other is a back-end interface for some customer facing applications.

InfoQ: What are your impressions of LightSwitch so far?

As a developer, I have a mixed opinions. I love how easy it is to build a new database application and I love how designing forms doesn't mean you're counting pixels and laying out labels and textboxes by hand. Most of all, I love how quickly you can start with a concept and push out a Silverlight Application on the intranet quickly. I like getting feedback early. On the other hand, I wish the support for using existing databases was better. I would love to build some front-ends to some poorly designed databases but LightSwitch has made it a little too time consuming at this point. I like the Business Rules / Validation model they use because it's designed to be straightforward to a junior developer, not an abstraction geek like me. While the rule model might not be ideal for very large applications, it's flexible enough for many small applications.

InfoQ: Microsoft advertises that LightSwitch is meant for users of all skill levels, and that applications are "coding optional". Do you think it’s possible for someone with no development expertise to create a fully-functioning application this way?

How many applications are really "coding optional"? As much as the marketing guys love playing up "the development tool for the business guy" angle, the fact of that matter is LightSwitch will force the end-user-developer to use C# or VB.NET at some point. Even if you're just using VB or C# for calculated fields and validation, end-user-developers will struggle with it sooner than you think, because they've always struggled with imperative programming languages. End users like declarative languages which mimic the Excel model, because Excel solves the problem of when and how things are calculated for you. LightSwitch solves these problems by giving you events to handle and class methods to implement and you have to be careful how you handle these events or else you can make a mess of things.

InfoQ: We’ve seen a few articles positing that LightSwitch is a pretty good Rapid Application Development platform, but that it’s being targeted to the wrong people. Do you agree?

While I agree there is always an opportunity for an end-user LOB app builder, it would be a mistake to portray LightSwitch as being that solution. Besides, I think [some] of these articles fail to recognize a real opportunity to make LOB app development easier for average developers, particularly Microsoft developers who are typically attached to their tools.

While it remains debatable if LightSwitch is truly "coding optional," Goguen does see it as a useful application in some cases. As he sums up, "Adding a presto-instant-app tool to their belt makes sense, especially if that tool can be extended with custom controls, data providers, and other .NET libraries."

Have you worked with Visual Studio LightSwitch? If so, what has your experience been?

Return to section navigation list>

Windows Azure Infrastructure and DevOps

imageNo significant articles today.

<Return to section navigation list>

Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds

image

No significant articles today.

<Return to section navigation list>

Cloud Security and Governance

Steve Riley appeared in an Addressing and improving cloud performance issues video segment on SearchCloudComputing.com’s CloudCover TV on 8/26/2011:

imageApparently there are a lot of people out there who still don't understand what cloud computing is, or at least that's what a recent study by CompTIA has found. If you're one of those people, clearly you've come to the right place. In other news, Steve Riley, the technical lead for Riverbed, discusses the issues with cloud performance and how it needs to improve, as well as an explanation about why he left AWS. He also shares his opinion on how close IT providers are to solving security issues in the cloud in this week's episode of Cloud Cover TV.

Watch the video.

imageThis week we discuss:

  • Verizon bought CloudSwitch, which makes software that moves internal apps and data to the cloud
  • Will Verizon maintain CloudSwitch's support of mutli-clouds, specifically Amazon's?
  • Eucalyptus released version 3 of its software, with high-availability features
  • OpenStack is offering courses to teach users how to install, run and operate an OpenStack cloud
  • According to CompTIA, people still don't understand what cloud computing is
  • Steve Riley, technical lead for Riverbed, discusses his departure from AWS and why he chose Riverbed
  • The different uses of AWS, and how information is stored in the public cloud
  • How cloud performance needs to improve
  • How users can take advantage of cloud diversity
  • Why people are picking Amazon over other cloud providers and vice versa
  • Where the action lies for cloud performance
  • How to optimize codes to improve cloud performance
  • Cloud providers are reluctant to create customized instances of what they're doing
  • Problems with multi-tenancy are somewhat improved, but not necessarily solved
  • The technical aspects of solving security issues in the cloud is largely complete
  • High quality clouds are safe places to store data
More on Cloud development and testing
Full disclosure: I’m a paid contributor to SearchCloudComputing.com and writing an article about cloud app performance instrumentation for the site at the moment.

<Return to section navigation list>

Cloud Computing Events

Eric Nelson (@ericnel) reported Last couple of spaces for the Windows Azure Discovery Workshop Sep 12th at the Microsoft UK offices:

imageI just checked and we are down to our last couple of spaces for this workshop.

Check out the full details.

image

P.S. We will be repeating the workshop monthly but we haven’t yet advertised the dates.

Related Links:

<Return to section navigation list>

Other Cloud Computing Platforms and Services

Matthew Weinbert (@MattNLM) posted Verizon Acquires CloudSwitch for Cloud Migration Technology to the ReadWriteCloud blog on 8/26/2011:

imageVerizon announced the acquisition of cloud software developer CloudSwitch in a move designed to simplify cloud migrations and boost adoption. Intriguingly, the telecommunications giant plans to integrate CloudSwitch with Terremark, the cloud services provider whose acquisition kicked off Verizon’s 2011.

imageCloudSwitch’s value to Verizon is in its “breakthrough software that enables enterprises to more easily and securely move applications, or workloads, between company data centers and the cloud without changing the application or the infrastructure layer,” according to the press release. In other words, Verizon is looking to boost its hybrid cloud and cloud-to-cloud play.

A key part of the value proposition, Verizon said, is enterprises get tighter control around their cloud migrations, while also extending security to cloud applications and data. Moreover, administrators retain the same level of control over cloud applications as they would for on-premises.

John McEleney, CEO of CloudSwitch, praised the merger in a prepared statement:

“By joining Verizon, we will be able to deliver a solution that combines our software with the market-leading infrastructure cloud play. Our founding vision has always been to create a seamless and secure federation of cloud environments across enterprise data centers and global cloud services. Together, we will be able to provide enterprises with an unmatched level of flexibility, scalability and control in the cloud with point-and-click simplicity. This will go a long way in helping achieve widespread adoption of the cloud, especially when managing complex workloads.”

Actual details on the acquisition are scant: We don’t know how much Verizon is paying for CloudSwitch, or when the deal officially closes (if it hasn’t already). Nor do we know whether Verizon will be keeping CloudSwitch’s existing employees and management team.

But what we do know is that with this acquisition, Verizon is investing in Terremark’s future, and by extension, the future of its cloud play. Will it give Verizon a competitive edge against its fellow A-list telecom competitors such as Time Warner, which acquired NaviSite for the same reasons? Stay tuned to TalkinCloud for more updates.

Read More About This Topic

David Strom reported Two Cloud-Based Test Tools This Week in an 8/26/2011 post to the ReadWriteCloud blog:

With all the new products coming out in the next week at VMworld, here are two announcements that might be worth a closer look: test tools that are based in the cloud. The idea is that you run your test suites from the cloud, so you can share your test scripts and results easily, and get your test beds setup quickly. It is a nifty idea.

imageThe first is from Automation Consultants Ltd. They announced the launch of TestWave, a full-service, cloud-based test management tool. Priced at $150 per user per month, you can sign up for a free 30-day trial. TestWave manages the testing of an IT system by enabling teams of testers to store test scripts, analyze results, and record and track defects.

ixia150.pngNext up is Ixia, a company who has been around the testing space for more than a decade and offers a wide variety of tools to examine the most complex of data center infrastructures. They have a new product called IxLoad-VM, which measures the quality and capacity of cloud data center components, applications, and networks to deliver effective cloud-based services.

IxLoad-VM works with IxNetwork-VM, which assesses layer 2/3 network performance in virtual environments. IxLoad-VM uses "virtual test ports," which are virtualized software implementations to measure true quality of experience and capacity loads. No pricing information is available, but typically these tools are five-figure purchases or can be rented for a $1,000 minimum 90-day period.

Chris Czarnecki reported Euclayptus Releases Eucalyptus 3 in an 8/26/2011 post to the Learning Tree blog:

imageThis week Euclayptus systems have announced Eucalyptus 3, their next generation private and hybrid cloud software. It is claimed that Eucalyptus is the most widely deployed private cloud currently available. This new release has been architected for high availability with no single point of failure.

imageEucalyptus is Infrastructure as a Service (IaaS) cloud software designed to run on on-premise and provide organisations with many benefits including:

  • Easily managed infrastructure
  • High availability
  • High reliability
  • Feature rich
  • Integration with public clouds

The major changes in Eucalyptus 3 are that should there be a failure for example in a disk drive, Eucalyptus 3 will automatically switch to a different resource running on a different machine. Such a switch is undertaken automatically with end users unaware it has occurred. Another enhancement is resource access control including a welcome implementation of the Amazon AWS Identity and Access Mnagement (IAM) service. Many other features have been added and are details here.

The next generation of Eucalyptus addresses, after security, one of the main concerns cloud computing adopters have, high availability. These concerns have gathered some momentum with the recent highly publicised failures at Amazon and Microsoft. By explicitly addressing these concerns in their architecture, Eucalyptus have made a major step forward in making the decision to adopt and implement a cloud solution much easier with reduced risk. The fact that they integrate seamlessly with public cloud providers such as Amazon is an added bonus. Eucalyptus 3 will be available in the fourth quarter. I eagerly await its release.

SD Times Newswire announced eXo Cloud IDE Gives Developers an On-Ramp to VMware Cloud Foundry PaaS in a 8/25/2011 press release post:

imageeXo, the enterprise Java portal and cloud user experience platform (UXP) company, yesterday announced that its eXo Cloud IDE, the industry’s only cloud-based integrated development environment (IDE) for Java applications, has added Cloud Foundry to its roster of supported platform-as-a-service (PaaS) offerings. Developers deploying Java, Spring, Ruby and other types of applications to Cloud Foundry can now take advantage of the increased agility and accessibility delivered by cloud computing. View the related video at http://budurl.com/aevz.

imageCloud computing and PaaS offerings have been evolving for some time, but until now developers have not actually been able to build Java applications in the cloud. With eXo Cloud IDE, developing and deploying Java apps becomes much more streamlined and makes cloud platforms more accessible to developers.

“Cloud IDE makes it possible for developers to collaborate on building Java applications in the cloud, apps that they can deploy directly to Cloud Foundry in minutes,” said Benjamin Mestrallet, founder and CEO of eXo. “The code now lives in the cloud, accessible from virtually anywhere with a browser and Internet access—so creating an app and moving it into Cloud Foundry is now very easy.”

eXo is showcasing Cloud IDE in booth #171W at VMworld 2011, being held at The Venetian and The Wynn hotels in Las Vegas, August 29–September 1, 2011. A webinar introducing eXo Cloud IDE is scheduled for September 8, 2011; participants can register at http://budurl.com/ubm3.

Expanding Opportunities
eXo Cloud IDE expands the options for developers. It is the only development-as-a-service (DaaS) offering to support Java application development, and the first to support Java Spring applications. It’s the only offering that developers can use to build Java applications in a cloud-based IDE and deploy them directly to a PaaS. With the announcement of Cloud IDE as an on-ramp to Cloud Foundry, eXo Cloud IDE now supports a total of four PaaS environments. eXo has already announced eXo Cloud IDE support for CloudBees, Heroku and Red Hat OpenShift.

Today, developers around the world are using eXo Cloud IDE to collaborate on the creation of HTML5/JavaScript applications and OpenSocial gadgets as well as Java-, PHP-, and Ruby-based web apps. Support for additional languages, frameworks and PaaS environments is planned for the future.

eXo Cloud IDE Webinar
On September 8, eXo developer evangelist Jeremi Joslin will host a webinar on eXo Cloud IDE for developers and IT managers. Starting at 8:00 a.m. Pacific time, Jeremi will cover topics ranging from the advantages of developing in the cloud to deploying applications to PaaS offerings such as Cloud Foundry, Heroku, OpenShift and others. For more information and to register, go to http://budurl.com/ubm3.

Alex Popescu (@al3xandru, pictured below) posted Paper: Graph Based Statistical Analysis of Network Traffic to his myNoSQL blog on 8/25/2011:

imagePublished by a group from Los Alamos National Lab (Hristo Djidjev, Gary Sandine, Curtis Storlie, Scott Vander Wiel):

We propose a method for analyzing traffic data in large computer networks such as big enterprise networks or the Internet. Our approach combines graph theoretical representation of the data and graph analysis with novel statistical methods for discovering pattern and timerelated anomalies. We model the traffic as a graph and use temporal characteristics of the data in order to decompose it into subgraphs corresponding to individual sessions, whose characteristics are then analyzed using statistical methods. The goal of that analysis is to discover patterns in the network traffic data that might indicate intrusion activity or other malicious behavior.

The embedded PDF and download link after the break.

While Hadoop excels at handling large amounts of data, it is not optimized for graph processing. Google’s Pregel and recently released GoldenOrb were created to tackle the large scale graph processing problems—see how GoldenOrb compares to Google Pregel and the major differences between Pregel and MapReduce.

Download the PDF from here.

Microsoft Research’s Dryad (now part of the beta for Microsoft HPC Pack 2008 R2 SP2) is a graph database for big-data and high-performance computing:

image 

For more information, see Don Pattee’s Dryad Beta Program Update post of 5/6/2011 to the Windows HPC Team blog and my (@rogerjenn) Microsoft's, Google's big data [analytics] plans give IT an edge article of 8/2011 for SearchCloudComputing.com.

Keith Hudgins (@keithhudgins) continued his Crowbar series with part II, Cloud Foundry, Crowbar, and You, on 8/25/2011: 

imageThis is part 2 in our ongoing series on Crowbar and the new Cloud Foundry barclamp. If you haven't read part 1, go here. We'll wait for you. I've got some nice sweet tea when you get back.

If you've followed the "cloud world," you'll know that in CloudLand, there's three kinds of aaS. Crowbar was built as an *ahem* lever to get your IaaS in place. With the CloudFoundry barclamp, you can now begin to build a PaaS installation (Okay, this one's brand new and has some limitations, we'll talk about that in just a bit...) that can host web applications built on a wide variety of frameworks. (Seriously... with the latest announcement, just about any Linux/open-source based web application can be adapted to fit.)

Cloud Foundry in a Nutshell

Before we dive into running the tutorial, let's get acquainted with Cloud Foundry and it's architecture.

imageCloud Foundry is a hosting platform for web applications. It provides runtime environments for Java, Ruby, Python, and PHP, along with a handful of dependent services like MySQL, RabbitMQ, and Redis. So you can build, say, a lightweight Sinatra app that is only an API broker to other services, or a full-blown Spring-based website with a Redis cache and MySQL back end and host them in the same hands-off (from the developer's perspective, anyway) platform.

How does that work? Magic, my friend, magic... well, really it's an asynchronous Rails app handling the API, talking via a NATS messaging queue to spin up the various parts for you as needed. There's a gem you can install that'll do all the API work for you, called vmc. You can always use the code inside the gem for some further automation if you need it, or just wrap it into your continuous integration loop to automatically deploy the latest version of your app on a successful build. (My team at DTO did it, roughly, a few weeks ago as a proof of concept)

I've got some links for you with much, much more in-depth information. Please wade through them at your leisure:

Enough already, on with the worky bits!

imageI've put together a tutorial on how to get our own Cloud Foundry instance running on your Crowbar environment.

Now, this barclamp is in its early stages: it's a port of the chef-solo cookbooks VMWare put together in their development environment install script... so it only supports a self-contained single box install at the moment. Work will be proceeding on shoring up the installation so that we can break out the components across multiple servers so you can create a truly dynamic environment to host just about everything. Have questions? (Yup, their cert's bad. They know.) Want to help? Get involved!

Marcia Savage reported Verizon targets hybrid cloud security with CloudSwitch acquisition in an 8/25/2011 post to the SearchCloudSecurity.com blog:

imageVerizon Communications Inc. announced Thursday that it acquired CloudSwitch Inc., a supplier of software designed to make it easier for companies to move their applications securely to the cloud.

New York-based Verizon said it will combine privately held CloudSwitch with its Terremark IT services subsidiary. Burlington, Mass.-based CloudSwitch makes software that allows companies to move applications from the enterprise data center to cloud environments while keeping the applications integrated with existing enterprise management and security policies.

imageChris Gesell, chief innovation and strategy officer for Terremark, said CloudSwitch’s focus on ease of use, security and workload portability for the enterprise is a natural fit for Verizon. The deal positions Verizon as leader in enabling hybrid clouds, he said in an interview.

image“This makes the customer’s data center interoperable with the cloud,” Gesell said. “We believe in the future enterprises are going to adopt a multi-cloud environment and will have different clouds for different workloads. CloudSwitch enables us to help them manage those multi cloud environments.”

Ellen Rubin, CloudSwitch cofounder and vice president of products, said customers download the CloudSwitch software as a virtual machine, which installs into the customer’s VMware or Zen environment in their data center. It comes with a Web-based interface that allows them to point and click in order to move applications into the cloud. Behind the scenes, CloudSwitch enables hybrid cloud security by launching an encrypted tunnel to the target cloud and provides a bridge from the customer’s data center to the cloud over Layer 2, she said. Enterprise security policies are maintained throughout the process.

“Our feeling is that customers don’t want to have to rebuild their whole network security strategy for the cloud,” Rubin said. …

Marcia continued with more from Rubin and a related quote from Amy DeCarlo, principal analyst at Current Analysis.

Full disclosure: I’m a paid contributor to SearchCloudComputing.com, a sister publication to TechTarget’s SearchCloudSecurity.com.

Derrick Harris (@derrickharris) asserted Eucalyptus refreshes IaaS platform, isn't dead yet in an 8/24/2011 post to Giga Om’s Structure blog:

imageEucalyptus Systems released the third generation of its pioneering private cloud computing software on Wednesday, complete with high-availability capabilities to ensure maximum uptime. Rumors of Eucalyptus’s demise have been circulating since OpenStack (s rax) launched its open-source cloud project last summer, but the company hasn’t shown any signs of slowing down.

Because OpenStack is based on the Nimbula Nebula software that NASA built in order to address scalability concerns with Eucalyptus, and because OpenStack has attracted so much industry and developer support, many have speculated that Eucalyptus and its open-source software were goners. However, Eucalyptus just keeps on adding users and expanding its business.

imageThe new version of its flagship product, Eucalyptus 3, is just the latest major undertaking for the company. According to CEO Marten Mickos, large customers have been asking for high availability for a long time, and that broad interest in the capability really picked up during this year, perhaps because of the high-profile public cloud outages over the past few months.

image“HA is a known science, but it’s difficult to implement in any given product,” said Mickos, which is why it took a while to build and why Eucalyptus is particularly proud of the feature. An oversimplified explanation of how Eucalyptus’s HA works is that it runs all the software’s components in multiple places, with active components handling requests and passive components monitoring system activity. If something goes wrong with an active component, a passive one steps in to fill the void.

Eucalyptus, which is unique because of its tight integration with the Amazon Web Services API and AWS-like architecture, also added the ability to boot storage images directly from AWS Elastic Block Storage. Given the performance and reliability concerns swirling around EBS — the cause of AWS’s four-day outage in April — Eucalyptus’s new capability might prove particularly popular because it will make it easier to bring data stored in AWS back behind the firewall.

Mickos thinks strong AWS compatibility is critical because it has such strong command of public cloud market share. Companies are increasingly pulling workloads from public clouds back in-house, he noted, and most of that is coming from AWS, if only because it has so many users to begin with. Plinga, a European social-gaming startup, moved 400 AWS servers onto a Eucalyptus’s cloud, and now runs a hybrid cloud infrastructure, Mickos said.

Eucalyptus upgraded its Resource Access Control capabilities in Eucalyptus 3 to support AWS’s new Identity Access Management feature, as well as to be able to map identities from LDAP and Active Directory servers.

So, Eucalyptus isn’t dead?

Hardly, says Mickos. He said Eucalyptus has been growing nicely, particularly in emerging economies such as China that don’t have a lot of legacy applications and infrastructure in place. Eucalyptus is a great fit in greenfield situations, he explained, because it targets enterprise applications rather than service providers trying to launch their own clouds. Thus far, service providers have been driving most private-cloud software sales.

As U.S. companies begin looking to cloud computing for maximum resource utilization instead of just elasticity, Mickos thinks Eucalyptus’s business will grow even more. Not only do many U.S. companies already have legacy software in place, he said, but load for legacy applications is fairly static, which means they don’t have to worry too much using cloud computing to tackle demand spikes. As the discussion matures, though, so will the use cases.

OpenStack: Competition or complement?

Regarding OpenStack, Mickos doesn’t seem too worried because there aren’t many (or any) production OpenStack private clouds running at this point. That could change, of course, as companies such as Citrix (s ctxs), Nebula and Piston Cloud Computing started rolling out their OpenStack distributions.

For now, though, Eucalyptus appears to view OpenStack primarily as an AWS alternative for public clouds, and Mickos said Eucalyptus is ready to support the OpenStack API if demand arises. Whereas OpenStack views itself as an AWS alternative, Mickos explained, Eucalyptus views itself as an AWS complement. And it could play that role with OpenStack, too.

Where Eucalyptus does see competition, though, is from other private cloud IaaS startups such as Cloud.com (now part of Citrix) and Abiquo, as well as from established vendors such as VMware (s vmw) with vCloud Director. That market is shaping up to be very competitive, with VMware having the advantage in terms of installed base and salespeople, Cloud.com winning high-profile customers, and Abiquo and Platform Computing getting analyst props recently.

<Return to section navigation list>

Posted by Roger Jennings (--rj) at 1:55 PM

Labels: , , , , , , , , , , , , , , , ,

0 comments:

Post a Comment

Subscribe to: Post Comments (Atom)