Windows Azure and Cloud Computing Posts for 3/8/2012
| A compendium of Windows Azure, Service Bus, EAI & EDI Access Control, Connect, SQL Azure Database, and other cloud-computing articles. |  |
Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:
- Windows Azure Blob, Drive, Table, Queue and Hadoop Services
- SQL Azure Database, Federations and Reporting
- Marketplace DataMarket, Social Analytics and OData
- Windows Azure Access Control, Service Bus, and Workflow
- Windows Azure VM Role, Virtual Network, Connect, RDP and CDN
- Live Windows Azure Apps, APIs, Tools and Test Harnesses
- Visual Studio LightSwitch and Entity Framework v4+
- Windows Azure Infrastructure and DevOps
- Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds
- Cloud Security and Governance
- Cloud Computing Events
- Other Cloud Computing Platforms and Services
Azure Blob, Drive, Table, Queue and Hadoop Service
Saptak Sen presented a 20-minute Big Data Analytics with Apache Hadoop and SQL Server 2012 video for the SQL Server 2012 launch party on 3/7/2012 (requires registration):
In this session we will explore Apache Hadoop on Windows Server and Azure and powerful insights we can generate integrating with SQL Server.
<Return to section navigation list>
SQL Azure Database, Federations and Reporting
Cihan Biyikoglu (@cihangirb) reported Countdown is over for SQL Server 2012… That means Scaling on-demand with Federations gets easier! in a 3/7/2012 post:
Today SQL Server 2012 is released to web and now we have a new set of bits that provide great enterprise class data management capabilities that stretches the boundaries of SQL Server to the public and private cloud. With SQL Server 2012, moving back and forth between cloud and on premise existence is much easier.
SQL Server 2012 tooling is one of the stories that is easy to miss within the 100s of additions and improvements we have in this release. SQL Azure obviously has a set of online tools you can use – SQL Azure Management Portal simplifies life if you are purely in the cloud but for most of us the story is a hybrid deployment. Some parts on premise and others in the cloud in those scenarios, SQL Server Management Studio is the key tool! This post can help you get up to speed on how scale-out gets easier with Management Studio capabilities around Federations.
If you’d like to get a fresh overview of federations; the why and the how, here is a quick link to the video that is part of the SQL Server 2012 Launch.
Enjoy the new set of bits and make sure to play with federations using [SQL Server M]anagement [S]tudio.
For my recent take on SQL Azure Federations, see:
Adding Missing Rows to a SQL Azure Federation with the SQL Azure Federation Data Migration Wizard v1 (1/18/2012)
- Loading Big Data into Federated SQL Azure Tables with the SQL Azure Federation Data Migration Wizard v1.2 (1/17/2012)
- Creating a SQL Azure Federation in the Windows Azure Platform Portal (1/11/2012)
- Generating Big Data for Use with SQL Azure Federations and Apache Hadoop on Windows Azure Clusters (1/8/2012)
The SQL Server and SQL Azure Teams produced videos for SQL Server 2012’ launch party on 3/7/2012. Following are cloud-related video session archives (require registration):
Quentin Clark, Corporate Vice President, Database Systems Group
Keynote
Join this launch session to get a high-level look at what’s new in SQL Server 2012, the cloud-ready information platform. Through SQL Server 2012, Quentin will address some of the tough issues facing IT organizations today; mission critical SLAs that are simply expected, the explosion of data volumes and the need to get insights from this data, and the evolution into the virtualized world—all backed by product demos from SQL Server product experts.
Duration : 15 Minutes
Tiffany Wissner, Director of Product Marketing
Building the modern enterprise with Hybrid IT
In today’s modern enterprise you will likely have applications across multiple deployment environments from the Traditional non-virtualized, to Private and Public Clouds. This session will outline Microsoft’s Hybrid IT strategy and associated solutions to help you optimize your IT environment and your application portfolio. The session will focus on the database intensive application scenarios to illustrate the value a Hybrid IT environment can bring to your enterprise.
Duration : 20 Minutes
Xin Jin, Program Manager
Scale on Demand: Optimizing SQL Server 2012 for Private Cloud
SQL Server 2012 delivers a wealth of features that allow IT teams to more efficiently set up and control the database side of their private cloud environments. Some of the key enhancements include; added investments in Resource Governor for multi-tenancy environments, improvements to SysPrep for faster creation of VMs for use in the private cloud environment, and dynamic memory to better control memory on-the-fly without taking databases offline. SQL Server 2012 also aligns with System Center 2012 to best enable management through a single pane of glass for IT. Watch this demo to see how these features can help with your private cloud initiatives.
Duration : 20 Minutes
Tobias Ternstrom, Senior Program Manager Lead
Optimizing SQL Server 2012 for Private Cloud
SQL Server 2012 delivers a wealth of features that allow IT teams to more efficiently set up and control the database side of their private cloud environments. SQL Server 2012 also aligns with System Center 2012 to best enable management through a single pane of glass for IT. Watch this demo to see how these features can help with your private cloud initiatives.
Duration : 20 Minutes
Saptak Sen, Senior Product Technical Manager
Big Data Analytics with Apache Hadoop and SQL Server 2012
Duration : 20 Minutes
Mark Scurrell, Senior Program Manager Lead and Sudhesh Suresh, Sr. Program Manager
Synchronizing Data “From On-Prem to Cloud” or “Cloud to Cloud” Using SQL Azure Data Sync
Duration : 20 Minutes
Greg Leake, Director of Technical Marketing
Fast Time to Solution: Introduction to SQL Azure and Cloud Database Solutions
Duration : 20 Minutes
Cihan Biyikoglu, Senior Program Manager Lead
Scale on Demand: Enable On-demand Scale with SQL Azure Federation
Duration : 20 Minutes
<Return to section navigation list>
MarketPlace DataMarket, Social Analytics, Big Data and OData
The StreamInsight team (@streaminsight) reported StreamInsight V2.0 Released! on 3/8/2012:
The StreamInsight Team is proud to announce the release of StreamInsight V2.0! This is the version that ships with SQL 2012, and as such it has been available through Connect to SQL CTP customers already since December. As part of the SQL 2012 launch activities, we are now making V2.0 available to everyone, following our tradition of providing a separate download page.
StreamInsight V2.0 includes a number of stability and performance fixes over its predecessor V1.2. Moreover it introduces a dependency on the .NET Framework 4.0, as well as on SQL 2012 license keys. For these reasons, we decided to bump the major version number, even though V2.0 does not add new features or API surface. It can be regarded a stepping stone to the upcoming release 2.1 which will contain significantly new APIs (that will depend on .NET 4.0).
Head over here to download StreamInsight V2.0. The updated Books Online can be found here.
See also the StreamInsight team (a.k.a., The Austin Team) reported StreamInsight in Windows Azure: Austin February CTP below.
Julie Lerman (@julielerman) explained How I see Web API in a 3/8/2012 post to her Don’t Be Iffy blog:
Using OData in apps is so easy thanks to REST & JSON. But OData works directly against a data model with a limited set of operations and capabilities. You can do some customization but at some point you get beyond the point of data services.
So then it’s back to WCF Services where you can have oh so much control over the service but boy can it get complex, especially when you want to leverage various protocols in WS-* e.g., security, etc. And you have to speak the language of the service which is different from one service to the next.
I see Web API as potentially the best of both worlds. I get to have control of the logic in my service, but I can consume it very easily with REST.
Am I oversimplifying? Am I doing it wrong?
The StreamInsight team (@streaminsight, a.k.a., The Austin Team) reported StreamInsight in Windows Azure: Austin February CTP on 2/23/2012 (missed when posted):
A few days ago, we updated the CTP (Community Technology Preview) of Project Codename Austin, the Windows Azure service that provides StreamInsight-based event processing in the Cloud. Apart from the service itself, we also updated the client bits and the sample.
The new features in this version are:
- Service and query level restart upon Azure node reboot
- Monitoring of Austin instances, automatic restart of unresponsive instances
- Built-in HTTP-based ingress endpoint
- StreamInsight v1.2 feature parity: LINQ improvements, UDSO (excluding checkpointing)
- Various bugfixes
CTP participants can download the package including new client bits, samples, and documentation from the Connect download page—look for the Austin February CTP download. If you cannot see the Austin material, let us know, so that we add you to the StreamInsight Advisory Group. If you are already in the Advisory Group and can access the material, but haven’t participated in the CTP yet, we need to enable your Azure subscription before you can provision Austin. Again, let us know and we will take care of it.
<Return to section navigation list>
Windows Azure Access Control, Service Bus and Workflow
Bart X. Tubalinal (@bart_tubalinal) described Using Windows Azure AppFabric Access Control Service in an iPhone App to Integrate with Facebook in a 3/8/2012 post:
Intro
This is the third post in a series that shows how to use Windows Azure as a platform that provides key services for native mobile applications. In the first two parts of the series, I walked through the process of creating my SpeakEasy app, a fictitious iPhone app that keeps track of speakers and their events.
In this post, I will walk through how to use the Azure AppFabric Access Control Service (ACS). ACS is a service that provides federated authentication using claims. ACS provides support for different identity providers, including Facebook, Windows Live, Google, Yahoo!, and ADFS 2.0 (and other WS-Federation identity providers). In this scenario, I will be using Facebook as an identity provider. Rather than force the user to log in using their Facebook account right away, I will only ask the user to log in when they want to share the details of an event on their Facebook wall. When they attempt to share the event, they will first have to log in using their Facebook account and then, by using the token received from Facebook (which will be provided as a claim), I will show how to post a message on the user's wall.
As a reminder, this series includes:
- Using Azure Storage Services from an iPhone App
- Adding Facebook Authentication and Authorization through Access Control Services
- Push Notifications with Queues and Workers
- Offline Synching using the Sync Framework (tentative)
Facebook Application Setup
To start with, we need to create a Facebook application:
- Go to http://www.facebook.com/developers and log in.
- Click the Create New App button.
- In the Create New App dialog, enter an appropriate name and the optional namespace. I have configured my application with the following settings:
- App Name: SpeakEasy iPhone App from DeviantPoint
- App Namespace: speakeasy-dp.
- This is optional but if you want to use Facebook’s Open Graph to post different Actions and define custom Object Types, you’ll need a namespace (I presume to prevent naming collisions).
After you have agreed to the Facebook Platform Policies agreement and clicked the Continue button, you will be taken to your app’s settings page. On this page are two very important pieces of information: the App ID and App Secret. Make a copy of both values as this will be used later on in the ACS setup. The rest of the application configuration is as follows:
- Contact Email: <my email>
- Website Site URL: https://<ACS-Service-Namespace>.accesscontrol.windows.net.
- This is the URL that Facebook will redirect to after the user has successfully logged in. While you’re debugging your application, it’s ok to configure this section and set the URL to your Azure ACS service namespace URL (details in following section). However, when you’ve published your app to the App Store, you’ll want to delete this section and then configure the Native iOS App section correctly to point to your app in the app store.
ACS Setup
Configuring a Service Namespace
- Log in to the Azure Management Portal and go to the Service Bus, Access Control & Caching section.
- Select Services > Access Control from the left pane and click on the New menu item from the ribbon. My Service Namespace configuration looks like this:
The end result of this will be an ACS URL for my namespace: https://speakeasy.accesscontrol.windows.net.
Once the Service Namespace has been created, you will be redirected to the Access Control Service management portal for your new namespace.
Configuring Facebook as an Identity Provider
The next step is configuring your Identity Providers (idP). In this example, we’ll only be using Facebook as our idP. However, you can configure multiple idPs to allow your users to log in using different idP services, like Facebook, Google, Yahoo, Windows Live, and ADFS 2.0.
- Click on the Identity Providers link under Trust Relationships in the left pane.
- Click on the Add link.
- Select Facebook Application from the custom identity provider section and click Next.
- Configure the following settings:
- Display name: Facebook
- Application ID: this is one of the values that you should have copied from the Facebook Application Setup. Enter that value here.
- Application secret: this is one of the values that you should have copied from the Facebook Application Setup. Enter that value here.
- Application permissions:email, publish_stream
- The application permissions field is used to tell Facebook what permissions your app will need. The email is given by default. The publish_stream permission will allow our app to post on the user’s behalf on his/her newsfeed. Here is a list of all application permissions you can request.
- Click Save.
Configuring a Relying Party Application
The next step is to configure a relying party application. A relying party application is an application that trusts the identity provider and consumes the claims made by that identity provider about the user.
- Click on the Relying party applications under Trust Relationships in the left pane.
- Click on the Add link.
- The following are the configuration settings I used for my relying party application:
- Name: speakeasy.deviantpoint.com
- Mode: manual
- Realm: uri:speakeasy.deviantpoint.com
- Return URL:empty
- Normally, this is where ACS will redirect your application to after the user has successfully logged in. Since we’re building a native iPhone app, this setting isn’t needed. However, it is sometimes still a good idea to set this to a web page that you’ve built that can take the incoming claims and do something with them. I like to set this to a page on my site where I have some code to read the incoming claims so that I am able to look at them in case I need to debug something.
- Error URL: empty
- Token Format:SWT
- This setting is important. Originally, I had this set to SAML 2.0 but I realized after some debugging efforts that the Windows Azure Toolkit for iOS is expecting the format to be SWT (Simple Web Token).
- Token Encryption Policy: None
- Token Lifetime: 600 (secs)
- Identity Providers: Facebook
- Rule Groups: Create New Rule Group
- Token Signing: Use Service Namespace Certificate
- Click Save.
Configuring Rule Groups
Rule Groups determine how incoming claims from the identity provider are mapped to output claims delivered to the relying party application.
- Click on Rule groups under Trust Relationships in the left pane.
- Click the Generate link and select the Facebook identity provider.
Clicking the Generate link will create a default set of rules for the identity provider. ACS is smart enough to tell a standard set of claims that most of the idPs provide. If the idP provided more claims than what was generated, then you can add those claims as well. Similarly, if there are claims that your relying party application doesn’t need, you can remove them as well.
Assuming you’ve set up the Facebook application and ACS (identity provider, relying party application, and rule groups) correctly, then you should be able to use the link to an ACS-hosted login page, found in the Application Integration section under Development in the left pane, to test the process. Note that unless you configured a Return URL for the Relying Party Application, you will get an ACS50011: The RP ReplyTo address is missing.’ error. It’s probably best to configure a Return URL at first for testing purposes to make sure your setup is correct.
Using ACS in the iPhone App
Modifying the Windows Azure iOS Toolkit
Now that the set up is complete, we can start to use ACS in the SpeakEasy app – except for one little detail: if you downloaded the iOS toolkit from the Master branch on Github, the toolkit currently filters out any claims that don’t start with a prefix of ‘http://schemas.xmlsoap.org/ws/2005/05/identity/claims/’. Fortunately, you have two options. The first option is to download the toolkit from the Develop branch. This branch removes the check for this prefix and just passes any claims through. The second option is to just modify the source code yourself. This is the option I took since I wasn’t sure exactly what other changes have been made to the Develop branch and I didn’t want to introduce any unknowns.
If you’re modifying the source code, find the file WACloudAccessToken.m. Around line 80, you will see the following code:
1: 2: NSString* claimsPrefix = @"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/"; 3: 4: for(NSString* part in [_securityToken componentsSeparatedByString:@"&"]) 5: { 6: NSRange split = [part rangeOfString:@"="]; 7: if(!split.length) 8: { 9: continue; // weird 10: } 11: 12: NSString* key = [[part substringToIndex:split.location] URLDecode]; 13: NSString* value = [[part substringFromIndex:split.location + 1] URLDecode]; 14: 15: if([key hasPrefix:claimsPrefix]) 16: { 17: key = [key substringFromIndex:claimsPrefix.length]; 18: [claims setObject:value forKey:key]; 19: } 20: } 21: 22: _claims = [claims copy];Change this code to comment out the offending lines:
1: //NSString* claimsPrefix = @"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/"; 2: 3: for(NSString* part in [_securityToken componentsSeparatedByString:@"&"]) 4: { 5: NSRange split = [part rangeOfString:@"="]; 6: if(!split.length) 7: { 8: continue; // weird 9: } 10: 11: NSString* key = [[part substringToIndex:split.location] URLDecode]; 12: NSString* value = [[part substringFromIndex:split.location + 1] URLDecode]; 13: 14: // if([key hasPrefix:claimsPrefix]) 15: // { 16: // key = [key substringFromIndex:claimsPrefix.length]; 17: [claims setObject:value forKey:key]; 18: // } 19: } 20: 21: _claims = [claims copy];Rebuild the toolkit and re-add a reference to the SpeakEasy project.
UI Changes
Now that the Toolkit is ready, we want to add the ability for the user to post an event’s details to his Facebook newsfeed. The easiest place to start is to make changes to the user interface,
Storyboard Changes
- Open up the MainStoryboard.storyboard file and find the Event Details view.
- From the Object library, drag a Bar Button Item to the navigation bar.
- Select the Bar Button Item and change the Identifier to Action in the Attributes Inspector.
- Add a View Controller from the Object library to the Storyboard.
- Select the View Controller and change the Top Bar to Navigation Bar and the Bottom Bar to Tab Bar.
- Change the Navigation Item’s title to Post to Facebook.
- Add a Text View and a Round Rect Button from the Object library to the new view.
- Resize the Text View to be about 1/3 of the height of the view and almost the full width.
- Change the text of the button to Post!.
- CTRL+Drag from the Event Details View Controller to the Post to Facebook View Controller.
- Select Push for the Segue type.
- Set the segue identifier to PostToFacebookSegue.
The new part of the storyboard should now look like this:
PostToFacebookViewController
Now we need a custom view controller that will interact with the new Post To Facebook view.
- From the Project navigator, right-click the ViewControllers group and select New File.
- Select the UIViewController subclass template and click Next.
- Name the class PostToFacebookViewController as a subclass of UIViewController.
- Click Next a few times until the .h and .m files are created.
- Open the .h file and add #import statements for SEEvent.h and WACloudAccessToken.h.
- Add the following code to the interface definition:
1: @property(nonatomic, retain) WACloudAccessToken *acsToken; 2: @property(nonatomic, retain) SEEvent *event; 3: @property(nonatomic, retain) IBOutlet UITextView *messageTextView; 4: 5: -(IBAction)postMessage:(id)sender;The WACloudAccessToken is a class from the Windows Azure iOS Toolkit that represents an ACS access token.
The IBAction postMessage: is the action that will be called when the Post! button is pushed by the user. Synthesize the properties in the .m file and connect the two outlets to their interface counterparts in the storyboard. Add the postMessage: method to PostToFacebookViewController.m but just leave the implementation blank for now.
EventDetailsViewController
Now we need to add code to handle transitioning from the Event Details View to the Post To Facebook View. To do this, we need to modify the EventDetailsViewController.
- Open EventDetailsViewController.hand add the following field:
- WACloudAccessControlClient *_acsClient;
- Add the following method:
This method is the event handler for when the user clicks on the Action button we added in the storyboard for this view.
- -(IBAction)shareButtonPressed:(id)sender;
- Open the EventDetailsViewController.m.
- Modify viewDidLoad so it looks like the following:
1: - (void)viewDidLoad 2: { 3: [super viewDidLoad]; 4: 5: _acsClient = [WACloudAccessControlClient accessControlClientForNamespace:@"speakeasy" 6: realm:@"uri:speakeasy.deviantpoint.com"]; 7: }The above code is where we use the WACloudAccessControlClient class provided by the iOS Azure toolkit to create a client we can use to hit the ACS service. The namespace and realm passed in are the namespace/realm that was configured as part of the ACS setup.
Modify the prepareForSegue:sender: method so it looks like the following:
1: - (void)prepareForSegue:(UIStoryboardSegue *)segue sender:(id)sender 2: { 3: if ([segue.identifier isEqualToString:@"EventPresenterSegue"]) 4: { 5: PresenterDetailsViewController *presenterDetailsViewController = segue.destinationViewController; 6: presenterDetailsViewController.speaker = self.presenter; 7: } 8: else if ([segue.identifier isEqualToString:@"PostToFacebookSegue"]) 9: { 10: PostToFacebookViewController *postToFacebookViewController = segue.destinationViewController; 11: postToFacebookViewController.event = self.event; 12: postToFacebookViewController.acsToken = [WACloudAccessControlClient sharedToken]; 13: } 14: }In the above code, we just add a new condition for the new segue we added. If the segue we’re trying to perform is the segue to move to the Facebook view, then we pass the event and the access token (which is available from the WACloudAccessControlClient after the user has successfully signed in) to that view.
Finally, to handle the Action button being clicked, add this method to the bottom of the file:
1: -(IBAction)shareButtonPressed:(id)sender 2: { 3: [_acsClient showInViewController:self allowsClose:YES withCompletionHandler:^(BOOL authenticated) 4: { 5: if(authenticated) 6: { 7: [self performSegueWithIdentifier:@"PostToFacebookSegue" sender:self]; 8: } 9: }]; 10: }This method calls the showInViewController:allowsClose:withCompletionHandler method on the WACloudAccessControlClient instance. When the method runs after the button is clicked, the facebook login screen will be shown to the client:
After the user has successfully logged in, the completion handler code will run which basically checks to see if the user is logged in and if so, performs the segue to the Post To Facebook view.
Handling the segue to PostToFacebookViewController
In PostToFacebookViewController.m, add the following to viewDidLoad to add a default message to the Text View in the view:
1: - (void)viewDidLoad 2: { 3: [super viewDidLoad]; 4: 5: messageTextView.text = [NSString stringWithFormat:@"I'll be attending %@ presented by %@ on %@ in %@", 6: _event.eventName, _event.speakerName, _event.eventDateAsString, _event.eventLocation]; 7: }This is a sample of what the view looks like when it is first loaded from the Event Details View:
Finally, to post the actual message to Facebook when the Post button is clicked, modify postMessage to look like the following:
1: -(IBAction)postMessage:(id)sender 2: { 3: NSString *accessToken = [acsToken.claims objectForKey:@"http://www.facebook.com/claims/AccessToken"]; 4: NSURL *url = [[NSURL alloc] initWithString:[[NSString alloc] initWithFormat:@"https://graph.facebook.com/me/feed?access_token=%@&message=%@", 5: accessToken, [messageTextView.text stringByAddingPercentEscapesUsingEncoding:NSUTF8StringEncoding]]]; 6: 7: NSMutableURLRequest *request = [NSMutableURLRequest requestWithURL:url]; 8: [request setHTTPMethod:@"POST"]; 9: 10: NSURLConnection *conn = [[NSURLConnection alloc] initWithRequest:request delegate:self startImmediately:YES]; 11: 12: if (conn) 13: { 14: [[self navigationController] popViewControllerAnimated:YES]; 15: } 16: else 17: { 18: //todo: add error message 19: } 20: }In the first line of this method, I retrieve the Facebook access token that is returned to me as a claim from Facebook after a successful login. The claims property of the WACloudAccessToken is a collection of all the claims that were passed through from the idP to the relying party application.
The rest of the lines are just used to create a POST request to Facebook. Any requests made to Facebook needs to include the access token (passed in as the access_token parameter). The message parameter is the actual text that will be posted to the feed, which must be URL-escaped.
If the post is successful, then I just navigate back to the details view (I didn’t any any error-handling in this example). A successful post to Facebook looks like this:
Conclusion
That’s really all to using ACS from an iPhone app using the Windows Azure Toolkit for iOS. Like I mentioned earlier, you can use ACS for your app’s authorization and authentication needs with other, multiple idPs. This is an especially low barrier-of-entry for apps that need to use Active Directory by taking advantage of ADFS through ACS.
I hope you enjoyed this post. If you have any questions, ask them in the comments section and I’ll try to answer them as best as I can. For the next part of the series, I will cover using Azure to send push notifications.
<Return to section navigation list>
Windows Azure VM Role, Virtual Network, Connect, RDP and CDN
<Return to section navigation list>
Live Windows Azure Apps, APIs, Tools and Test Harnesses
Roope Astala posted “Cloud Numerics” Example: Analyzing Air Traffic “On-Time” Data to the Microsoft Codename “Cloud Numerics” blog on 3/8/2012:
You sit at the airport only to witness your departure time get delayed. You wait. Your flight gets delayed again, and you wonder “what’s happening?” Can you predict how long it will take to arrive at your destination? Are there many short delays in front of you or just a few long delays? This example demonstrates how you can use “Cloud Numerics” to sift though and calculate a big enough cross section of air traffic data needed to answer these questions. We will use on-time performance data from the U.S. Department of Transportation to analyze the distribution of delays. The data is available at:
http://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=236&DB_Short_Name=On-Time
This data set holds data for every scheduled flight in the U.S. from 1987 to 2011 and is —as one would expect— huge! For your convenience, we have uploaded a sample of 32 months —one file per month with about 500,000 flights in each— to Windows Azure Blob Storage at: http://cloudnumericslab.blob.core.windows.net/flightdata.
Before You Begin
- You will need to have Microsoft Codename “Cloud Numerics” installed.
- Create a Windows Azure subscription for deploying and running the application (if do not have one already). You also need a Windows Azure storage account set up for saving results.
- Install “Cloud Numerics” on your local computer where you develop your C# applications.
- Run the example application as detailed in the “Cloud Numerics” Lab Getting Started wiki page to validate your installation.
You should use two-to-four compute nodes when deploying the application to Windows Azure. One node might not have enough memory, and for larger-sized deployments there are not enough files in the sample data set to assign to all distributed I/O processes. You should not attempt to run the application on a local system because of data transfer and memory requirements.
Note!
You specify how many compute nodes are allocated when you use the Cloud Numerics Deployment Utility to configure your Windows Azure cluster. For details, see this section in the Getting Started guide.
Step 1: Create Cloud Numerics project
First, let’s create a Cloud Numerics application using the Cloud Numerics Visual Studio project template.
1. Start Microsoft Visual Studio and create a solution from the Cloud Numerics project template. Let’s call the solution “OnTimeStats.”
2. Create a new application, and delete the sample code from within the MSCloudNumericsApp subproject’s Program.cs file. Replace it with the following skeleton of an application.
using System; using System.Collections.Generic; using System.Linq; using System.Text; using msnl = Microsoft.Numerics.Local; using msnd = Microsoft.Numerics.Distributed; using Microsoft.Numerics.Statistics; using Microsoft.Numerics.Mathematics; using Microsoft.Numerics.Distributed.IO; using Microsoft.WindowsAzure; using Microsoft.WindowsAzure.StorageClient; namespace FlightOnTime { [Serializable] public class FlightInfoReader : IParallelReader<double> { } class Program { static void WriteOutput(string output) { } static void Main() { // Initialize runtime Microsoft.Numerics.NumericsRuntime.Initialize(); // Shut down runtime Microsoft.Numerics.NumericsRuntime.Shutdown(); } } }3. Add a .NET reference to Microsoft.WindowsAzure.StorageClient. This assembly is used for reading in data from blobs, and writing results.
Step 2: Add Methods for Reading Data
We’ll use the IParallelReader interface to read in the data from blob storage in the following manner:
- In the ComputeAssignment method, we must first list the blobs in the container that holds the flight data (http://cloudnumericslab.blob.core.windows.net/flightdata). Then, we assign blobs to each worker node, in round-robin fashion.
- In the ReadWorker method, we instantiate a list “arrivalDelays”. Each reader will then open the first blob on their list, download the text, break it to lines, break the line to columns, and select column 44 that holds the arrival delays in minutes. Note that if a flight was canceled or diverted, this value is blank and we must skip the row. We convert the result to a double precision type and append it to the arrivalDelays list. Then, we either move to the next blob or –if done– convert the list into NumericDenseArray and return it.
For Step 2, add the following code to the FlightInforReader class in your skeleton application.
[Serializable] public class FlightInfoReader : IParallelReader<double> { string _containerAddress; public FlightInfoReader(string containerAddress) { _containerAddress = containerAddress; } public int DistributedDimension { get {return 0;} set {} } public Object[] ComputeAssignment(int ranks) { // Get list of flight info files (blobs) from container var container = new CloudBlobContainer(_containerAddress); var blobs = container.ListBlobs().ToArray(); // Allocate blobs to workers in round-robin fashion List<Uri> [] assignments = new List<Uri> [ranks]; for (int i = 0; i < ranks; i++) { assignments[i] = new List<Uri>(); } for (int i = 0; i < blobs.Count(); i++) { int currentRank = i % ranks; assignments[currentRank].Add(blobs[i].Uri); } return (Object[]) assignments; } public msnl.NumericDenseArray<double> ReadWorker(Object assignment) { List<Uri> assignmentUris = (List<Uri>) assignment; // If there are no blobs to read, return empty array if (assignmentUris.Count == 0) { return msnl.NumericDenseArrayFactory.Create<double>(new long[] { 0 }); } List<double> arrivalDelays = new List<double>(); for (int blobCount = 0; blobCount < assignmentUris.Count; blobCount++) { // Open blob and read text lines var blob = new CloudBlob(assignmentUris[blobCount].AbsoluteUri); var rows = blob.DownloadText().Split(new char[] {'\n'}); int nrows = rows.Count(); // Offset by one row because of header file, also, note that last row is empty for (int i = 1; i < nrows - 1; i++) { // Remove quotation marks and split row var thisRow = rows[i].Replace("\"", String.Empty).Split(new char[] { ',' }); // Filter out canceled and diverted flights if (!thisRow[49].Contains("1") && !thisRow[51].Contains("1")) { // Add arrival delay from column 44 to list arrivalDelays.Add(System.Convert.ToDouble(thisRow[44])); } } } // Convert list to numeric dense array and return it from reader return msnl.NumericDenseArrayFactory.CreateFromSystemArray<double>(arrivalDelays.ToArray()); } }Note!
See the blog post titled “Cloud Numerics” Example: Using the IParallelReader Interface for more details on how to use the IParallelReader interface.
Step 3: Implement the Algorithm
After reading in the delays, we compute mean, we center the data to the mean (to make the worse-than-average delays positive and better-than-average ones negative), and then compute sample standard deviation. Then, to analyze the distribution of data, we count how many flights, on average, are more than k standard deviations away from the mean. Also, we keep track how many flights are above or below to mean, so as to detect any skew in data.
For example, if the data were normal distributed one would expect it to be symmetric around the mean, and obey the three-sigma rule —or, equivalently, about 16% of flights would be delayed by more than 1 standard deviation, 2.2 % by more than 2 standard deviations and 0.13% by more than 3.
To implement the algorithm, we add following code as the Main method to the application.
static void Main() { // Initialize runtime Microsoft.Numerics.NumericsRuntime.Initialize(); // Instantiate StringBuilder for writing output StringBuilder output = new StringBuilder(); // Read flight info string containerAddress = @"http://cloudnumericslab.blob.core.windows.net/flightdata/"; var flightInfoReader = new FlightInfoReader(containerAddress); var flightData = Loader.LoadData<double>(flightInfoReader); // Compute mean and standard deviation var nSamples = flightData.Shape[0]; var mean = Descriptive.Mean(flightData); flightData = flightData - mean; var stDev = BasicMath.Sqrt(Descriptive.Mean(flightData * flightData) * ((double)nSamples / (double)(nSamples - 1))); output.AppendLine("Mean (minutes), " + mean); output.AppendLine("Standard deviation (minutes), " + stDev); // Compute how much of the data is below or above 0, 1,...,5 standard deviations long nStDev = 6; for (long k = 0; k < nStDev; k++) { double aboveKStDev = 100d * Descriptive.Mean((flightData > k * stDev).ConvertTo<double>()); double belowKStDev = 100d * Descriptive.Mean((flightData < -k * stDev).ConvertTo<double>()); output.AppendLine("Samples below and above k standard deviations (percent), " + k + ", " + belowKStDev + ", " + aboveKStDev); } // Write output to a blob WriteOutput(output.ToString()); // Shut down runtime Microsoft.Numerics.NumericsRuntime.Shutdown(); }Step 4: Write Results to a Blob
We add the WriteOutput method that writes results to binary large object (blob) storage as .csv-formatted text. The WriteOutput method will create a container “flightdataresult” and a blob “flightdataresult.csv.” You can then view this blob by opening your file in Excel. For example: http://<myAccountName>.blob.core.windows.net/flightdataresult/flightdataresult.csv
--Where <myAccountName> is the name of your Windows Azure account.
Let’s fill in the last missing piece from the application, the WriteOutput method, with following code.
Note that you’ll have to change "myAccountKey" and "myAccountName" to reflect your own storage account key and name.
static void WriteOutput(string output) { // Write to blob storage // Replace "myAccountKey" and "myAccountName" by your own storage account key and name string accountKey = "myAccountKey"; string accountName = "myAccountName"; // Result blob and container name string containerName = "flightdataresult"; string blobName = "flightdataresult.csv"; // Create result container and blob var storageAccountCredential = new StorageCredentialsAccountAndKey(accountName, accountKey); var storageAccount = new CloudStorageAccount(storageAccountCredential, true); var blobClient = storageAccount.CreateCloudBlobClient(); var resultContainer = blobClient.GetContainerReference(containerName); resultContainer.CreateIfNotExist(); var resultBlob = resultContainer.GetBlobReference(blobName); // Make result blob publicly readable, var resultPermissions = new BlobContainerPermissions(); resultPermissions.PublicAccess = BlobContainerPublicAccessType.Blob; resultContainer.SetPermissions(resultPermissions); // Upload result to blob resultBlob.UploadText(output); }Step 5: Deploy the Application and Analyze Results
Now, you are ready to run the application. Set AppConfigure as the startup project, build the application, and use the Cloud Numerics Deployment Utility to deploy and run the application.
Let’s take a look at the results. We can immediately see they’re not normal-distributed at all. First, there’s skew —about 70% of the flight delays are better than average of 5 minute. Second, the number of delays tails off much more gradually than a normal distribution would as one moves away from the mean towards longer delays. A step of one standard deviation (about 35 minutes) roughly halves the number of delays, as we can see in the sequence 8.5 %, 4.0 %, 2.1%, 1.1 %, 0.6 %. These findings suggests that the tail could be modeled by an exponential distribution.
This result is both good news and bad news for you as a passenger. There is a good 70% chance you’ll arrive no more than five minutes late. However, the exponential nature of the tail means —based on conditional probability— that if you have already had to wait for 35 minutes there’s about 50-50 chance you will have to wait for another 35 minutes.
Brian Hitney posted Launching “Learn the Cloud, Make a Difference” DC Effort on 3/8/2012:
Two years ago, Jim O’neil and I developed a quick Azure training program called “@home with Windows Azure” – a way to learn Windows Azure and have some fun contributing to a well known distributed computing effort, Folding@home.
A few months later, Peter Laudati joined the cloud team and we developed the game RockPaperAzure. RockPaperAzure was a lot of fun and is still active, but we decided to re-launch the @home with Windows Azure project because of all of the changes in the cloud since that effort in 2010.
Stanford University’s Pande Lab has been sponsoring Folding@home for nearly 12 years, during which they’ve used the results of their protein folding simulations (running on thousands of machines worldwide) to provide insight into the causes of diseases such as Alzheimer’s, Mad Cow disease, ALS, and some cancer-related syndromes.
When you participate in @home with Windows Azure, you’ll leverage a free, 3-month Windows Azure Trial (or your MSDN benefits) to deploy Stanford’s Folding@home application to Windows Azure, where it will execute the protein folding simulations in the cloud, thus contributing to the research effort. Additionally, Microsoft is donating $10 (up to a maximum of $5000) to Stanford’s Pande Lab for everyone that participates.
We’ve provided a lot of information to get you started, including four short screencasts that will lead you through the process of getting an Azure account, downloading the @home with Windows Azure software, and deploying it to the cloud. And we won’t stop there! We have a series of webcasts also planned to go into more detail about the application and other aspects of Windows Azure that we leveraged to make this effort possible.
Here is the schedule for webcasts, and of course, you can jump in before at any time:
- 3/15/2012 12pm EDT @home with Azure Overview
- 3/22/2012 12pm EDT Windows Azure Roles
- 3/29/2012 12pm EDT Azure Storage Options
- 4/05/2012 12pm EDT Debugging in the Cloud
- 4/12/2012 12pm EDT Async Cloud Patterns
<Return to section navigation list>
Visual Studio LightSwitch and Entity Framework 4.1+
Return to section navigation list>
Windows Azure Infrastructure and DevOps
Steven Martin posted Announcing Reduced Pricing on Windows Azure Storage and Compute to the Windows Azure blog on 3/8/2012:
- Windows Azure Storage Pay-As-You-Go pricing has been reduced by 12% ($0.14 to $0.125)
- 6 Month Plans for Windows Azure Storage have been reduced across all tiers by up to 14%
- Windows Azure Extra Small Compute has been reduced by 50% ($.04 to .$02)
With these changes, a 24x7 Extra Small Compute instance with a 100MB SQL Azure DB costs less than $20 per month. The price of compute and storage continues to be consistent across all datacenters and Pay-As-You-Go customers will automatically see the reduced prices reflected (as of today) on their next bill; there is no action required. [Emphasis added.]
Customers tell us that cost savings is only part of the story. While we are delighted to pass along savings (60% reduction in XSmall compute and 17% reduction in Storage in the last 6 months), we are just as focused on improving the developer experience and adding new capabilities to ensure the best overall value in the industry. Storage is a great example. While the reduced price of $.125/GB provides cost benefits, Geo-Replication differentiates Windows Azure Storage from other services in market. SQL Azure is another great example; $5.00/month for a 100 MB Database is great, but it’s much more appealing to developers when it comes with high availability, fault tolerance, self-management, elastic scale-out, on-premises connectivity, and a full Service Level Agreement.
A few additional details:
- Windows Azure 3 Month Free Trial and MSDN customers will receive twice as many extra small compute instances per month vs. previous allocations [Emphasis added.]
- Additional details on pricing and pricing calculator can be found at:
Steven Martin, General Manager, Windows Azure Business Strategy & Planning
A response to Amazon Web Services’ recent price cuts?
David Linthicum (@DavidLinthicum) asserted “Software quality assurance for the cloud is not the same as for traditional applications, and the industry needs to adjust” in a deck for his Microsoft Azure outage shows clear need for different cloud testing article of 3/8/2012 for InfoWorld’s Cloud Computing blog:
Last week, Microsoft's Windows Azure cloud service suffered an outage that was later blamed on a leap year issue, a bug that is now fixed. Although this seems like par for the course in the world of software, cloud computing providers need to be much better than this.
The reality is that most of these outages can be traced back to human error, such as the Azure leap year bug. Bugs are errors in programming, and although people are bound to make mistakes, their impact is minimized if you perform quality assurance.
It's not my intent to pick on Microsoft, but it's a fact that more and more cloud computing systems are quickly being built and placed in production. As more users adopt cloud computing, glitches and bugs that lead to outages or other breaks will become commonplace.
Testing is the solution. As clouds move toward production, they need to undergo a unique set of trials. These tests are not yet well understood in the work of software testing and quality assurance. The unique needs of the cloud include the ability to tenant management, resource management, security, auditing, and yes, the ability to make it past leap day, to name just a few components.
The larger issue is that most quality assurance teams don't understand how to test cloud computing systems. Thus, they fall back on traditional software testing models, which are outdated for what they're trying to accomplish.
Cloud providers need to figure this problem out soon. Enterprises want cloud computing platforms they can trust. With a bit more work by providers, they could get them.
Nuno Filipe Godinho (@NunoGodinho) described the Importance of Affinity Groups in Windows Azure in a 3/4/2012 post (missed when published):
During this last week some friends asked me more about what are the Affinity Groups in Windows Azure, and their benefits since for some people this is nothing more than a way to logically group both Compute and Storage.
In order to explain this we need to dig a little deep in terms of how Windows Azure Data Centers are created. Basically Windows Azure Data Centers are built using “Containers” that inside are full of clusters and racks. Each of those Containers have specific services, like for example, Compute and Storage, SQL Azure, Service Bus, Access Control Service, and so on. Those containers are spread across the data center and each time we subscribe/deploy a service the Fabric Controller (which chooses based on our solution configuration where the services should be deployed) can place our services spread across the data center.
Basically Affinity Groups is a way to tell the Fabric Controller that those two elements, Compute and Storage, should always be together and close to one another, and what this does is when the Fabric Controller is searching for the best suited Container to deploy those services will be looking for one where it can deploy both in the same Cluster, making them as close as possible, and reducing the latency, and increasing the performance.
So in summary, Affinity Groups provide us:
- Aggregation, since it aggregates our Compute and Storage services and provide the Fabric Controller the information needed for them to be kept in the same Data Center, and even more, in the same Cluster.
- Reducing the Latency, because by providing information to the Fabric Controller that they should be kept together, allow us to get a lot better latency when accessing the Storage from the Compute Nodes, which makes difference in a highly available environment.
- Lowering costs, as by using them we don’t have the possibility of getting one service in one Data Center and the other in another Data Center if for some reason we choose the wrong way, or even because we choose for both one of the “Anywhere” options in the.
Based on this, don’t forget to use Affinity Groups right from the start, since it’s not possible after having deployed both the Compute or Storage to change them into an Affinity Group.
To finalize, and since now you can be thinking that this would be very interesting for other services also, no other services are able to take advantage of this Affinity, since neither of them share the same Container.
<Return to section navigation list>
Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Cloud
Bruce Kyle asserted Hybrid Cloud Combine Windows Azure, Private Clouds in a 3/7/2012 post:
Hybrid applications and environments combine software running on public cloud platforms, such as Windows Azure, with software running on premises. Developers and customers using this model get both the cloud benefits of low cost, ubiquity, scale, etc. along with the control, flexibility, security, etc. of their on-premises platforms.
Windows Azure Service Bus Demo – Brian Loesgen
“a system that listens for events, and when critical events occur, they are multicast to listeners/subscribers through the Azure ServiceBus. The listeners use the Service Bus relay bindings, the subscribers use the topical pub/sub mechanism of the ServiceBus” – This demo is useful for anyone looking to use the Service Bus for hybrid, mobile, or loosely coupled apps. Demo includes code and video walkthroughs.
Building Hybrid Applications in the Cloud on Windows Azure – Microsoft Patterns and Practices
This free downloadable book is the third in a series starting with Volume 1, Moving Applications to the Cloud on Windows Azure and Volume 2, Developing Applications for the Cloud on Windows Azure. This third volume covers all the things a developers needs to consider when building hybrid apps that join on-premises services with services in the cloud. The book is organized around the scenario of a company that wants to extend an existing order application to the cloud.
Managing a hybrid cloud with System Center 2012 – Andrew Fryer
A blog post that covers the basics of using System Center to manage applications running in a hybrid cloud.
.NET On-Premises/Cloud Hybrid Application Using Service Bus Relay – Windows Azure Team
You will learn:
- How to create or adapt an existing web service for consumption by a web solution.
- How to use the Windows Azure Service Bus relay to share data between a Windows Azure application and a web service hosted elsewhere.
Windows Azure Queues and Windows Azure Service Bus Queues - Compared and Contrasted -- Valery Mizonov and Seth Manheim
This article analyzes the differences and similarities between the two types of queues offered by Windows Azure today: Windows Azure Queues and Windows Azure Service Bus Queues. By using this information, you can compare and contrast the respective technologies and be able to make a more informed decision about which solution best meets your needs.
<Return to section navigation list>
Cloud Security and Governance
Bruce Kyle continued his series with Windows Azure Security Best Practices -- Part 2: What Azure Provides Out-of-the-Box on 3/8/2012:
In Windows Azure Security Best Practices -- Part 1: The Challenges, Defense in Depth, I described the threat landscape and introduces the plan for your application to employ defense in depth.
In this part, I explain that security with Windows Azure is a shared responsibility, and Windows Azure provides your application with security features than you may have employed in your on premises application. But then, it also exposes other vulnerabilities that you should consider. And in the end, you should be proactive in your application development to secure your application.
This section is meant to provide an overview of what Windows Azure provides. For more in depth information, see Global Foundation Services Online Security. The Global Foundation Services team delivers trustworthy, available online services that create a competitive advantage for you and for Microsoft’s Windows Azure.
The intent of this series is to provide a context for you to learn more and empower you to write great applications for the public cloud.
How Does Windows Azure Help Secure the Host?
You may well be saying, “Not so fast Bruce. How does Windows Azure help secure the host?”
Windows Azure Security Overview provides a comprehensive look at the security available with Windows Azure. Written by Charlie Kaufman and Ramanathan Venkatapathy. The paper examines the security functionality available from the perspectives of the customer and Microsoft operations, discusses the people and processes that help make Windows Azure more secure, and provides a brief discussion about compliance. I’ll summarize the features, and recommend you read and understand the Overview for an in-depth understanding.
Windows Azure is designed to abstract much of the infrastructure that typically underlies applications (servers, operating systems, web & database software, and so on) so you can focus on building applications.
From the outside looking in, you see cloud-based compute and storage, for you to build and manage applications and their associated configurations. Each of the ways into Windows Azure can require some level of authorization that you are in control of.
And I’ll describe some best practices that you should do for the authentication mechanisms in a later post.
Windows Azure must provide confidentiality, integrity, and availability of customer data, just like any other application hosting platform. It must also provide transparent accountability to allow you and their agents to track administration of applications and infrastructure, by yourself, and by Microsoft.
Based on the number of role instances specified by customers, Windows Azure creates a virtual machine (VM) for each role instance, then runs the role in those VMs. These VMs in turn run on a hypervisor that’s specifically designed for use in the cloud (the Windows Azure Hypervisor).
One VM is special: it runs a hardened operating system called the root OS that hosts a fabric agent (FA). FAs are used in turn to manage your application and storage nodes. FAs are managed by a fabric controller (FC), which exists outside of compute and storage nodes (compute and storage clusters are managed by separate FCs).
Ten Things to Know About Azure Security
Windows Azure must provide confidentiality, integrity, and availability of customer data, just like any other application hosting platform. It must also provide transparent accountability to allow customers and their agents to track administration of applications and infrastructure, by themselves and by Microsoft.
The blog post 10 Things to know about Azure Security provides a good summary of many of these features:
- SSL Mutual Authentication for Internal Control Traffic. All communications between Windows Azure internal components are protected with SSL.
- Certificate and Private Key Management. To lower the risk of exposing certificates and private keys to developers and administrators, they are installed via a separate mechanism than the code that uses them.
- Least Privilege Customer Software. You and other customers do not have administrative access to their VMs, and customer software in Windows Azure is restricted to running under a low-privilege account by default (in future versions, customers may select different privilege models at their option).
- Access Control in Windows Azure Storage. Each Storage Account has a single secret key that is used to control access to all data in that Storage Account. This supports the typical scenario where storage is associated with applications and those applications have full control over their associated data.
- Isolation of Hypervisor, Root OS, and Guest VMs. A critical boundary is the isolation of the root VM from the guest VMs and the guest VMs from one another, managed by the hypervisor and the root OS.
- Isolation of Fabric Controllers. As the central orchestrator of much the Windows Azure Fabric, significant controls are in place to mitigate threats to fabric controllers, especially from potentially compromised FAs within customer applications.
- Packet Filtering. The hypervisor and the root OS provide network packet filters that assure that the untrusted VMs cannot generate spoofed traffic, cannot receive traffic not addressed to them, cannot direct traffic to protected infrastructure endpoints, and cannot send or receive inappropriate broadcast traffic.
- VLAN Isolation. VLANs are used to isolate the FCs and other devices.
- Isolation of Customer Access. The systems managing access to customer environments (the Windows Azure Portal, SMAPI, and so on) are isolated within a Windows Azure application operated by Microsoft.
- Deletion of Data. Where appropriate, confidentiality should persist beyond the useful lifecycle of data. Windows Azure's Storage subsystem makes customer data unavailable once delete operations are called.
IMPORTANT: The strongest security controls available are no protection against an attacker who gains unauthorized access to credentials or keys. Thus, credential and key management are critical components of the security design and implementation of Windows Azure.
Data Integrity
Customers seeking to outsource their data compute and storage workloads to Windows Azure obviously expect it to be protected from unauthorized changes. The details are explained in the Windows Azure Security Overview. But I wanted to highlight some of the key points here.
The primary mechanism of integrity protection for customer data lies within the Fabric VM design itself. Each VM is connected to three local Virtual Hard Drives (VHDs).
• The D: drive contains one of several versions of the Guest OS, kept up-to-date with relevant patches, selectable by the customer.
• The E: drive contains an image constructed by the FC based on the package provided by the customer.
• The C: drive contains configuration information, paging files, and other storage.OS and Application Integrity. The D: and E: virtual drives are effectively read-only because their ACLs are set to disallow write
access from customer processes. This design strictly preserves the integrity of the underlying operating system and customer applications.Configuration Integrity. Only authorized customers accessing their Hosted Services via the Windows Azure Portal or SMAPI can change the configuration file. Internal processes explained in the Security Overview integrity of the customer configuration is protected, maintained, and persisted constantly during an application’s lifetime.
Storage. Each Storage Account has two storage account keys that are used to control access to all data in that Storage Account, and thus access to the storage keys provide full control over the associated data.
Fabric. The integrity of the Fabric itself is carefully managed from bootstrap through operation.
Availability
One of the main advantages provided by cloud platforms is robust availability based on extensive redundancy achieved with virtualization technology.
Replicated data. Data is replicated within Windows Azure to three separate nodes within the Fabric to minimize the impact of hardware failures.
Geographically distributed data. Customers can leverage the geographically distributed nature of the Windows Azure infrastructure by creating a second Storage Account to provide hot-failover capability. You can replicate and synchronize data between Microsoft facilities. Customers may also write customized roles to extract data from storage for offsite private backups.
The guest agents (GAs) on every VM monitor the health of the VM.
Service Operations
Because cloud computing platforms are effectively an outsourced computing environment, they have to be able to demonstrate safe operation to customers and their designated agents on a regular basis. Windows Azure implements multiple levels of monitoring, logging, and reporting to provide this visibility to customers. Primarily, the monitoring agent (MA) gathers monitoring and diagnostic log information from many places including the FC and the root OS and writes it to log files. It eventually pushes a digested subset of the information into a pre-configured Windows Azure Storage Account. In addition, the Monitoring Data analysis Service (MDS) is a
freestanding service that reads various monitoring and diagnostic log data and summarizes/digests the information, writing it to an integrated log.Service Operations. Windows Azure developers and administrators have, by design, been given sufficient privileges to carry out their assigned duties to operate and evolve the service. As noted throughout this document, Microsoft deploys combinations of preventive, detective and reactive controls including the following mechanisms to help protect against unauthorized developer and/or
administrative activity:
- Tight access control to sensitive data
- Combinations of controls that greatly enhance independent detection of malicious
activity- Multiple levels of monitoring, logging, and reporting
Security Response. Microsoft security vulnerabilities can be reported to the Microsoft Security Response Center or via email to secure@microsoft.com. Microsoft follows a consistent process to assess and respond to vulnerabilities and incidents reported via the standard facilities.
Network. Windows Azure internal network is isolated by strong filtering of traffic to and from other networks. This provides a “backplane” for internal network traffic that is high speed and at low risk from malicious activity generally.
Physical Security. A system cannot be more secure than the physical platform on which it runs. Windows Azure runs in geographically distributed Microsoft facilities, sharing space and utilities with other Microsoft Online Services. Each facility is designed to run 24 x 7 and employs various measures to help protect operations from power failure, physical intrusion, and network outages. These data centers comply with industry standards for physical security and reliability and they are managed, monitored, and administered by Microsoft operations personnel. They are designed for “lights out” operation.
Compliance
Trusted third-party certification provides a well-established way to demonstrate how we protect customer data without giving excessive access to teams of independent auditors that may threaten the integrity of the overall platform.
ISO 27001
The Online Services Security and Compliance (OSSC) Information Security Management System (ISMS) uses the ISO/IEC 27001:2005 Information technology — Security techniques — Information security management systems — Requirements (ISO/IEC 27001:2005) as its basis because it provided a well-established framework for integrating risk evaluation into the daily operations of running a cloud infrastructure. (For a copy of this standard, see http://www.iso.org.)
Windows Azure has obtained ISO 27001 certification for its core services following a successful audit by the British Standards Institute (BSI). You can view details of the ISO certificate here, which lists the scope as:
The Information Security Management System for Microsoft Windows Azure including development, operations and support for the compute, storage (XStore), virtual network and virtual machine services, in accordance with Windows Azure ISMS statement of applicability dated September 28, 2011. The ISMS meets the criteria of ISO/IEC 27001:2005 ISMS requirements Standard.
The ISO certification covers the following Windows Azure features:
- Compute (includes Web and Worker roles)
- Storage (includes Blobs, Queues, and Tables)
- Virtual Machine (includes the VM role)
- Virtual Network (includes Traffic Manager and Connect)
Included in the above are Windows Azure service management features and the Windows Azure Management Portal, as well as the information management systems used to monitor, operate, and update these services.
Microsoft’s Global Foundation Services division has a separate ISO 27001 certification for the data centers in which Windows Azure is hosted.
Learn more about ISO 27001 certification. View the certificate for Windows Azure.
Compliance Framework
The Microsoft Compliance Framework for Online Services (Compliance Framework) was developed by OSSC to address this need. The Compliance Framework includes a standard methodology for defining compliance domains, determining which objectives apply to a given team or asset, and capturing how domain control objectives are addressed in sufficient detail as they apply to a given set of regulations or requirements.
Microsoft’s is committed to Microsoft’s Compliance Framework for Online Services. The link explains the Compliance Framework in more detail, and provides examples of how to develop compliance domains and to apply control objectives to them in the context of specific industry standards or regulatory requirements. The OSSC team within the GFS division builds on the same security principles and processes Microsoft has developed through years of experience managing security risks.
Safe Harbor
Microsoft Corporation is a signatory to Safe Harbor and is committed to fulfill all of its obligations under the Safe Harbor
Framework.Geo-Location Selection
You choose where their data is stored. You can segment customer data and processing across multiple systems, geographies, and regulatory jurisdictions.
Data in Microsoft datacenters around the world based on the geo-location properties that you specify when you create the storage in Azure Portal. You minimize compliance risk by actively selecting the geographic locations in which regulated data will reside.
References
- Windows Azure Achieves IS0 27001 Certification from the British Standards Institute
- Information Security Management System for Microsoft Cloud Infrastructure
- Windows Azure™ Security Overview
- Security Best Practices for Developing Windows Azure Applications
- Microsoft’s Compliance Framework for Online Services
- Webcast: Online Services Security and Compliance: Microsoft’s Compliance Framework for Online Services
- Whiteboard Session: Online Services Security and Compliance: Microsoft’s Compliance Framework for Online Services
Next Up
Windows Azure Security Best Practices – Part 3: Identifying Your Security Frame. This post explores how you can examine your application and identify attack surfaces. The idea of a Security Frame is a way to look at your application to determine treats and your responses, before you even begin coding. I point you to checklists that you can use when you are architecting your application.
Ariel posted Cloud Encryption – a Fundamental First Step in Cloud Security on 3/7/2012 to the Portico Cloud Security blog:
Exposing a virtualization weakness for data theft, Snapshotting your data, and the internal threat, are new cloud risks that didn’t exist when the data was stored between the four walls of your datacenter. Data encryption is a critical first step for any organization considering the cloud. In Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) clouds, protecting data at rest is your responsibility. To meet privacy obligations to your customers and employees, and to comply with regulatory standards such as PCI DSS, HIPAA and SOX, businesses must securely encrypt cloud-based data, while keeping operational overhead to a minimum.
New Cloud Security risks to your data:
Gaining access from a different virtual server within the same physical hardware:
Cloud computing is all about virtualization. Multiple customers will share a single physical server and will be logically separated from each other. In theory, one can share the same physical server with his competitor. Gaining access to sensitive data from a different virtual server inside the same physical server can be achieved by an attacker exploiting a virtualization operating system vulnerability, or by one of your other cloud system administrators (a “malicious insider” from a different project in your own organization) using his credentials or exploiting one of many known web application vulnerabilities to launch an attack on the virtual server in order to access and steal sensitive data. Encrypting your data will not enable the attacker to view it, even if he did gain access to your virtual OS.
Snapshotting your virtual storage:
Here’s an interesting infrastructure as a service scenario: A malicious user gains access to your cloud console by stealing your credentials (or by exploiting vulnerabilities in the cloud access control infrastructure), allowing him access to your cloud servers. Once in, a simple snapshot will move your data to a deferent location of his choice. This risk is in our opinion the most obvious reason for cloud encryption, but surprisingly enough, not all cloud customers are aware of the threat, hence exposing their cloud residing data to a significant risk.
The Insider Threat
Back in March 2011, Health Net had publically announced it had lost 1.9 Million customer records as a result of its IT vendor misplacing nine server drives following a move to a new data center. The point is that an insider threat is not necessarily a malicious user, but the impact is almost similar to a malicious action. Data encryption in this case would have saved a significant amount of money to the provider but more importantly the private records would not have been exposed.
Providing an effective encryption solution in a cloud environment is not an easy task. An effective cloud encryption solution must cover the entire data layer (including virtual disks, distributed storage, databases, etc…), while not exposing your encryption keys to anyone but yourself. Porticor infuses trust into the cloud with secure, easy to use, and scalable solutions for data encryption and key management. Porticor enables companies of all sizes to safeguard their data, comply with regulatory standards, and streamline operations.
The next best choice after Transparent Data Encryption (TDE, which isn’t available for SQL Azure) is Microsoft Codename “Trust Services.” Stay tuned for an OakLeaf tutorial on same.
<Return to section navigation list>
Cloud Computing Events
Sebastian W (@qmiswax) reported March CRM Technical Community Call: Windows Azure on 3/7/2012:
UK CRM Technical Community. The next live meeting is scheduled for the 20th of March at 5pm and you can register at the usual link http://aka.ms/ukcrmtech . This time we will have session about Windows Azure together with MS CRM 2011 , sounds really exciting .
Erlang Factory Bay Area 2012 will present Erlang in the Cloud hosted by Ram C. Singh on 3/20/2012 in downtown San Francisco at the Marines' Memorial Club and Hotel in Union Square:
Host: Ram C Singh
Date: 30-03-2012
Room: The Crystal LoungeIt has been said that Erlang if a perfect fit for cloud computing. Not only for developing applications, but also for powering cloud infrastructure. It has been used for cloud computing like applications and infrastructure a decade before the term was coined. The distributed and scalable nature of the cloud, both when implementing the underlying architecture and for applications running on them, fits Erlang like a glove. This track will at look at Erlang in the clouds from both sides.
Time Talk 10:35 - 11:20 Erlang as a Cloud Citizen
Paolo Negri11:25 - 12:10 Accelerated Erlang Development with e2
Garrett Smith13:25 - 14:10 Erlang manages supercomputers: above the clouds.
Viktor Sovietov14:15 - 15:00 Deploying Erlang applications in the Heroku cloud
Blake Gentry Geoff Cant15:20 - 16:05 Handling a Polyglotten WAN
Karl Anderson16:10 - 16:55 How Cloud Foundry, the first Open PaaS, can help you deploy apps faster and experiment with new technologies.
Paul Jones Monica Wilkinson17:15 - 18:00 Erlang and PaaS, panel debate
Paul Jones
All tracks:
29-03-2012, Thursday
DevOps, host: Christian WestbrookDevOps" is a new track for the 2012 Erlang factories looking at the emerging set of principles, methods and practices for communication, collaboration and integration between Software development and operations staff. Erlang old timers will say they have been doing DevOps for years, only calling it something different. This track looks at setting that record straight. It will cover tools, procedures and infrastructure needed in the management and deployment of Erlang systems.
29-03-2012, Thursday
Erlang In Business, host: Charles McKnightAs Erlang grows into new niches, more and more companies are betting on Erlang as a strategic, game-changing technology. In this track, you will hear non technical talks from people who are using Erlang in business-critical settings or supplanting legacy technologies. They will share the benefits and advantages of using Erlang as a tool to build massively scalable, soft real time systems.
29-03-2012, Thursday
Big Data, host: Steve VinoskiEver since Mnesia entered the scene in 1995, Erlang has been a language for the most discriminating database hackers. With CouchDB, Scalaris and Riak, Erlang is now on the forefront of the NoSQL wave - the biggest revolution in the database world since E.F. Codd invented the Relational Model. Alongside massive non-relational storage mediums, Erlang’s ability to distribute load across massive clusters has resulted in a wide range of frameworks, applications and uses around big data. This tracks investigates them all.
29-03-2012, Thursday
Cool Tools and Gadgets, host: Jesper Louis AndersenThe Tools and Gadgets Track explores the latest tools, academic research projects and library applications for the Erlang Developers and Testers.
30-03-2012, Friday
Erlang, Libraries & the VM, host: Michal PtaszekIn this track, you will learn from the leading experts and Erlang committers about new language constructs, virtual machine implementations and powerful libraries. Esoteric VM implementations are presented, alongside improvements and enhancements to the existing VM. You will learn how many of its features work and how to best use them to write fast and efficient code. You will also find language specific talks, covering subjects from NIFs to extension proposals.
30-03-2012, Friday
The Polyglot Programmer, host: Dave SmithUse the right tool for the right job! Erlang has often been used together with other languages or systems. This track looks at examples of how Erlang can be integrated efficiently with other languages for optimal usage. It also lets you widen your horizons and delve into things non-Erlang.
30-03-2012, Friday
Case Studies & Architecture, host: James AimonettiAs Erlang's popularity grows, it keeps breaking into new niches and adopted companies not previously associated with the language. In this track we get to listen to the experiences of users better known for their association with other technologies rather than Erlang, looking at architectures and benefits of the language.
30-03-2012, Friday
Erlang in the Cloud, host: Ram C SinghIt has been said that Erlang if a perfect fit for cloud computing. Not only for developing applications, but also for powering cloud infrastructure. It has been used for cloud computing like applications and infrastructure a decade before the term was coined. The distributed and scalable nature of the cloud, both when implementing the underlying architecture and for applications running on them, fits Erlang like a glove. This track will at look at Erlang in the clouds from both sides.
- 31-03-2012, Saturday Hackathon
- 31-03-2012, Saturday Community Day
- 31-03-2012, Saturday UnConference
- 31-03-2012, Saturday DevOps Bootcamp
- 31-03-2012, Saturday Tutorials
<Return to section navigation list>
Other Cloud Computing Platforms and Services
Alex Popescu (@al3xandru) reported a Major Riak Release Includes Tons of Improvements, Plus a Riak Admin UI and Riaknostic in a 3/8/2012 post to his myNoSQL blog:
One of the major releases that happened around the end of February (and I’ve missed due to some personal problems), is Riak 1.1. I assume that by now everyone using Riak already knows all the goodies packaged by the Basho team in this new release, but for those that are not yet onboard here is a summary:
From the Release notes:
- Numerous changes to Riak Core which address issues with cluster scalability, and enable Riak to better handle large clusters and large rings
- New Ownership Claim Algorithm: The new ring ownership claim algorithm introduced as an optional setting in the 1.0 release has been set as the default for 1.1. The new claim algorithm significantly reduces the amount of ownerhip shuffling for clusters with more than N+2 nodes in them.
- Riak KV improvements:
- Liskeys backpressure: Backpressure has been added to listkeys to prevent the node listing keys from being overwhelemed.
- Don’t drop post-commit errors on floor
- MapReduce Improvements
- The MapReduce interface now supports requests with empty queries. This allows the 2i, list-keys, and search inputs to return matching keys to clients without needing to include a reduce_identity query phase.
- MapReduce error messages have been improved. Most error cases should now return helpful information all the way to the client, while also producing less spam in Riak’s logs.
- Bitcask and LevelDB improvements
Then there’s also Riaknostic and the new Riak admin tool: Riak Control.
What is Riaknostic?
From the initial Riaknostic announcement:
Riaknostic is an Erlang script (escript) that runs a series of “diagnostics” or “checks”, inspecting your operating system and Riak installation for known potential problems and then printing suggestions for how to fix those problems. Riaknostic will NOT fix those problems for you, it’s only a tool for diagnostics. Some of the things it checks are:
- How much memory does the Riak process currently use?
- Do Riak’s data directories have the correct permissions?
- Did the Riak node crash in the past and leave a dump file?
Riaknostic project page is here.
What is Riak Control?
From Riak Control GitHub page:
Riak Control is a set of webmachine resources, all accessible via the /admin/* paths, allow you to inspect your running cluster, and manipulate it in various ways.
Now that description doesn’t make Riak Control any justice. What Riak Control is a very fancy REST-driven admin interface for Riak. You don’t have to take my word for it, so check this screenshot:
Riak Control covers different details of a Riak cluster:
- general cluster status
- details about the cluster
- details about the ring
This blog post gives more details about Riak Control and a couple more sexy screenshots. If you’d like to dive a bit deeper into Riak Control, you can also watch after the break a 25min video of Mark Phillips talking about it.
Since the Riak 1.1.0 release, there has been a bug fix release 1.1.1 addressing some MapReduce bugs described on the mailing list but also on the Riak 1.1.1 release notes.
Riak and WebMachine are the two systems for which I wished I knew Erlang so I could dive into and learn more about. I’m already (slowly) working to change this.
See the Erlang Factory Bay Area 2012 will present Erlang in the Cloud hosted by Ram C. Singh on 3/20/2012 in downtown San Francisco at the Marines' Memorial Club and Hotel in Union Square post in the Cloud Computing Events section above.
Jeff Barr (@jeffbarr) reported EC2 Updates: New Medium Instance, 64-bit Ubiquity, SSH Client on 3/7/2012:
Big News
I have three important announcements for EC2 users:
- We have introduced a new instance type, the Medium (m1.medium).
- You can now launch 64-bit operating systems on the m1.small and c1.medium instances.
- You can now log in to an EC2 instance from the AWS Management Console using an integrated SSH client.
New Instance Type
The new Medium instance type fills a gap in the m1 family of instance types, splitting the difference, price and performance-wise, between the existing Small and Large types and bringing our instance count to thirteen (other EC2 instance types). Here are the specs:
- 3.75 GB of RAM
- 1 virtual core running at 2 ECU (EC2 Compute Unit)
- 410 GB of instance storage
- 32 and 64-bit
- Moderate I/O performance
The Medium instance type is available now in all 8 of the AWS Regions. See the EC2 Pricing page for more information on the On Demand and Reserved Instance pricing (you can also acquire Medium instances in Spot form).
64-bit Ubiquity
You can now launch 64-bit operating systems on the Small and Medium instance types. This means that you can now create a single Amazon Machine Image (AMI) and run it on an extremely wide range of instance types, from the Micro all the way up to the High-CPU Extra Large and and the High-Memory Quadruple Extra Large, as you can see from the console menu:
This will make it easier for you to scale vertically (to larger and smaller instances) without having to maintain parallel (32 and 64-bit) AMIs.
SSH Client
We've integrated the MindTerm SSH client into the AWS Management console to simplify the process of connecting to an EC2 instance. There's now second option on the Connect window:
And there you have it! What do you think?
<Return to section navigation list>
Atom 1.0 site feed
0 comments:
Post a Comment