OakLeaf Systems

Windows Azure and Cloud Computing Posts for 2/1/2012+

2012-02-01T16:19:00.001-08:00

A compendium of Windows Azure, Service Bus, EAI & EDI Access Control, Connect, SQL Azure Database, and other cloud-computing articles.

Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:

Windows Azure Blob, Drive, Table, Queue and Hadoop Services
SQL Azure Database, Federations and Reporting
Marketplace DataMarket, Social Analytics and OData
Windows Azure Access Control, Service Bus, and Workflow
Windows Azure VM Role, Virtual Network, Connect, RDP and CDN
Live Windows Azure Apps, APIs, Tools and Test Harnesses
Visual Studio LightSwitch and Entity Framework v4+
Windows Azure Infrastructure and DevOps
Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds
Cloud Security and Governance
Cloud Computing Events
Other Cloud Computing Platforms and Services

Azure Blob, Drive, Table, Queue and Hadoop Services

Andrew Brust (@andrewbrust) asserted “By responding to potential threats with thoughtfulness, and a zeal to add value, SQL and Big Data could be big business for Redmond” as a deck for his Big Data and SQL Server: Disruption or Harmony? article of 2/1/2012 for the Redmond Developer News site:

The SQL Server relational engine matured a long time ago. There have been advances, of course, in performance, fault tolerance and high availability, not mention encryption, compression and file-system integration. And, yes, there's been support for XML, geospatial data and even a service broker. But these have been improvements on the margins. All these features are new flavors of icing; the cake has stayed the same.

And yet, over the 20-year lifetime of SQL Server, Microsoft has continued to add value to the product, even as the core features have gone into maintenance mode. The big story here has been business intelligence: BI capabilities started in SQL Server 7 with OLAP Services, and have been expanded in meaningful ways with every subsequent release. It's been a great strategy and it still is. But will it keep working?

Core capabilities can't stay static forever; eventually disruption comes along. Today that disruption is here, coming from Big Data, Hadoop and its MapReduce distributed-computing approach. As strong as SQL Server BI capabilities are (and they're getting even stronger with SQL Server 2012), if Microsoft can't embrace Big Data technology, SQL Server could find itself in a position of desperation, going from underdog to contender, to retiree. What's a poor enterprise software company to do?

Build Bridges, Don't Burn Them
Microsoft could ignore Hadoop, but that would be foolish. It could try to build a competitor, which it almost did with the Microsoft Research Dryad project, but I fear not too many people would have come to that party. Microsoft could just adopt Hadoop, plain vanilla, but that would most likely be a race to the bottom, and it wouldn't even win. Really, Microsoft must mix Hadoop into its bag of tricks and do what it has always done best: take a raw technology and make it approachable to the enterprise. I can't be sure yet, but I think that's what Microsoft has done, and it has enhanced the value of Windows Azure in the process.

With code name "Project Isotope," Microsoft has taken the step of implementing Hadoop on Windows. It's a no-slouch effort, too: Microsoft's distribution of Hadoop is being developed in concert with Hortonworks, a startup company founded and staffed by many former Hadoop team members at Yahoo! (where the open source project began). But what Microsoft has also done is integrate Hadoop into its BI stack, and that may be one of the smartest moves it's made in quite some time.

Microsoft has created an Excel add-in for Hive, which provides a SQL-like abstraction over Hadoop and MapReduce. The add-in is based on an ODBC driver, which in turn is compatible with PowerPivot, so business users can do meaningful analysis on Big Data, on their own terms. And because the same engine that drives PowerPivot has been implemented inside Analysis Services in SQL Server 2012, that product has access to Hadoop now, too. With that, Microsoft has joined the Big Data and Enterprise BI worlds. It has also tied together SQL Server and Hadoop.

Simplify and Succeed
Hadoop runs on Windows Server and on Windows Azure, with an installer that makes setup really easy. In addition to those configurations, Microsoft allows Windows Azure users to provision an entire Hadoop cluster from a Web portal, without any discrete installation steps at all. Once the cluster's up, customers can connect to its head node via Remote Desktop. The full Java-based command-line personality of Hadoop is there if you want it, but there are also Hive and JavaScript consoles in the browser. And then it only takes a minute or two to build out a connection from Microsoft BI tools or Excel, putting Hadoop to work for users fitting a number of profiles. It's classic Microsoft, mixed with equal parts open source and Java.

I've said before that Microsoft does some of its best work when it embraces standards from outside the company. That's what happened with jQuery and ASP.NET, and it may well be what happens with HTML5 and JavaScript in Windows 8. In the case of implementing Hadoop, Microsoft makes Windows Azure more valuable and more agnostic. It brings continued relevance to Windows Server and SQL Server. And it widens the reach and utility of Hadoop. By responding to a potential threat with thoughtfulness -- and a zeal to add value -- Big Data could be big business for Redmond.

Full disclosure: I’m a contributing editor for Visual Studio Magazine, which is published by 1105 Media who also own the Redmond Developer News site.

SQL Azure Database, Federations and Reporting

Benjamin Guinebertière explained how to Use SSIS to push data to SQL Azure | Utiliser SSIS pour pousser des données vers SQL Azure in a 1/31/2012 post. From the English version:

The goal of this blog post is to show that it is as easy with SQL Azure as with SQL Server to push data with SSIS.

In this sample, let’s use SQL Server 2008 R2, which is the latest SQL Server version in production (1).

The scenario is to push some data from SQL Server 2008 R2 on premises to SQL Azure in the cloud. This is done with SQL Server 2008 R2 Integration Services (SSIS).

If and when starting from an empty SQL Azure database, in order to create the schema, one would typically use SQL Azure Migration Wizard which has an option to generate schema only.

Here are the steps to create a very simple SSIS package and execute it (in debug mode):

NB: the SSIS package needs an outbound access to port 1433 in order to connect to SQL Azure. The SQL Azure firewall must also have been opened for the public IP address the SSIS package will use to access the Internet.

Here is a synopsis of what we just did. Data was copied from SQL Server 2008 R2 to SQL Azure thru SSIS exactly as it would have been from SQL Server to SQL Server.

This sample SSIS package was very simple, because its goal was to show connectivity is seamless. Of course, in real life this package would contain much more shapes in order to copy several tables, transform data and so on.

Benjamin

(1) SQL Server 2012 is in Release Candidate 0 for now so I expect companies to have SQL Server 2008 R2 deployed

In my opinion, George Huey’s SQL Azure Migration Wizard is a simpler and more straightforward method of uploading data to SQL Azure. See my Using the SQL Azure Migration Wizard v3.3.3 with the AdventureWorksLT2008R2 Sample Database of 7/10/2010 for more details.

MarketPlace DataMarket, Social Analytics and OData

The Codename “Data Explorer” Team (@DataExplorer) described Consuming “Data Explorer” published data feeds in Visual Studio primer in a 2/1/2012 post:

Part 1 – Consuming a Basic Open Feed

In a previous post we discussed the Publishing capabilities of Microsoft Codename “Data Explorer”. After mashing up your data and generating interesting results, you can share your findings literally with the world at large. One of the formats available to publish data is as an OData feed, which can be used to allow programmatic access to the results.

In this post we cover a basic example of consuming a published feed in Visual Studio (VS) using a simple console application along with some pointers to set it up. We are actively working on improving the ease of access to published results in “Data Explorer” from VS and other tools, so bear with us as we iron out the wrinkles and please reach out to us if you run into any issues. Our goal is to reduce the need for function calls by adding extra user interface support, but in the meantime, for those of you who like to tinker, here are some roll-back-your-sleeves-and-get-your-hands-dirty instructions.

1. Ensuring the published mashup is set up to be consumable

Currently, in order to consume a published OData feed using VS, resources need to have primary keys and include only supported data types (more on this below). For the purposes of this example, the mashup also needs to be published as visible to everyone, including unauthenticated users (meaning, that no authentication mechanism is requested when trying to access the feed). We will cover the explanation for how to access feeds secured via a feed key in a subsequent blog post.

1.a. Verifying the mashup resources can be externally consumed as a feed

First and foremost, all published resources that are to be consumed by VS need to have a primary key. You can ensure that a given table has a primary key by using the Table.AddKey() function. We are working on an “add key” button to the user interface to facilitate this, but in the meantime you can manually add a primary key using the formula bar.

For example, let’s start with a very simple table of IDs, First and Last names, created by directly inputting the text into “Data Explorer” and converting to a table:

Create a new mashup (call it, say, “Basic publish to VS”) and add data by inputting text.

Use “To Table” after pressing “Done” to convert the free-form text into a table.

Take the default options (using spaces as delimiters and the first row as headers) to create the table.

With a simple table in place, we can now start the process to make it consumable.

The first step is to add a primary key. To invoke the Table.AddKey() function using the formula bar, click on the fx icon next to the Edit button and select “New Task”:

In the new task, call the function on the TableFromText reference by typing it in as seen in the figure. This function call makes the “ID” column the primary key (since primary keys must be unique, remember that multiple columns can be used to create a key, thus ensuring uniqueness). Note that “TableFromText” happens to be the name of the previous task.

Once sure that the table has a primary key, there are a couple of caveats regarding the data that will be exposed in the feed:

Currently, the set of supported data types between VS and “Data Explorer” is different, so in order to ensure VS consumption, the data published in “Data Explorer” (which supports a larger set of data types) needs to be adjusted to be consumed in VS. Among the data types not yet recognized in VS are Time, Duration, DateTimeOffset, and Geospatial. Attempting to import a feed with these data types will cause the entire VS import operation to fail, so make sure that the feed you wish to import into VS does not have any of these data types exposed.

Similarly, an exposed service operation that returns a keyed-table type will import correctly into VS, but the service operations will be ignored.

Exposed service operations that return tables without keys will fail the VS import, though, as will service operations that have parameters.

Lastly, for now tables with navigation properties will also fail the VS import, but we are actively working on a fix that should address this in the near future.

With the above in mind, and with a mashup that can be consumed by VS via feed, we are ready to publish and consume the data.

1.b. Publishing the mashup

When publishing the mashup, for this example we will make the published results visible to everyone, including unauthenticated users, to allow programmatic access (we will cover the more complex case of consuming authenticated feeds in a future post).

Click on Publish and follow the link to access the publish page. From there, copy the full web address under “Data Feed” for use in programmatic access.

This link to the feed is what will allow VS to access the data.

2. Consuming a feed in Visual Studio

Once the feed has been created in a consumable format, there are a few things to be aware of when programmatically accessing the feed via VS.

2.a. Installing the Microsoft WCF Data Services October 2011 CTP

In order to successfully add a service reference using the steps outlined below, be sure to install the community technical preview that included the updated version of the WCF Data Services library; click here for the Download Center page.

2.b. Adding a service reference in Visual Studio

In your VS project, right-click on References or on Service References to “Add Service Reference”.

Enter the feed URL from the Data Explorer publish page into the Address box and click Go. Make sure the URL you enter matches the published feed URL exactly (for example, verify there are no trailing spaces). You should be able to see the published resources and click OK to add the service.

2.c. Using the service reference in a console application

Here is a sample console program to consume the above feed:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text; 
namespace ConsumingDEpublishedDataInVSPart1
{
    class Program
    {
        static void Main(string[] args)
        {
            // Note: Make sure you replace the URI 
            // with the feed URL used when adding a service reference.
            EntityCatalog catalog = new EntityCatalog(new 
            Uri("https://ws32842467.dataexplorer.sqlazurelabs.com/
 Published/Basic%20publish%20to%20VS%20test/Feed"));
            foreach (EntityType0 element in catalog.Text1)
            {   //Note: ‘Text1’ is the name of your mashup resource
                Console.WriteLine(element.ID + " " + 
element.FirstName 
                                                 + " " 
element.LastName + " ");
            }
            Console.WriteLine("Press any key to continue...");
            Console.ReadKey();
        }
    }
}
As you can see in the figure below, the two “EntityCatalog” references are unresolved (and there are a couple of other interesting service-specific items as well). Note that the resource name in our mashup (“Text1”) is used when accessing the elements in the feed. Note also that the column names (“ID”, “FirstName”, and “LastName”) are also used when accessing each element.

Before this code can compile, we need to resolve the “EntityCatalog” references via a “using” statement.

This resolves both the “EntityCatalog” and “EntityType0” references because they are names generated by the “Data Explorer” feed.

As a quick aside, those names and their relationship to the mashup resources can be seen in the $metadata description of the “Data Explorer” feed.

The above code, when compiled and run, will output the data in the table from our simple mashup.

This pattern can of course be extended to extract data from more meaningful or complex tables. The metadata for the feed (which is accessible by adding “/$metadata” to the feed URL and opening it up in a browser) can be used to tailor the queries over the exposed feed, accessing data from different resources.

We are still working hard to improve the experience of consuming “Data Explorer” feeds using VS but hopefully the above example can unlock some interesting new ways to work with your data.

What do you think? Let us know your thoughts and stay tuned as we further integrate and simplify the user interface to more easily publish and consume your data with Microsoft Codename “Data Explorer”.

3. Some troubleshooting FAQs

While support for programmatic consumption using VS is still a work in progress, following the above steps should allow you to create a service reference and pull data from your feed. Here are some common issues with setting the reference up; please have a look to see if they help, or ask a question in our forum with more specific issues you find.

3.a. I encounter an Add Service Reference Error. What’s up with that?

Depending on the specific details, this Add Service Reference Error may mean that an incorrect feed URL was used. For instance, one common mistake may be to use the link to the “Data Explorer” publish page rather than the full feed URL (which is the publish page link plus “/Feed”). Double check that the address you use is the full feed link that is presented in the publish page, and not the publish page’s URL.

3.b. Why does the service require me to enter a user name and password?

This is most likely due to having selected “Make visible to workspace consumers only” instead of “Make visible to everyone, including unauthenticated users” when publishing the mashup. For this example, try going back to “Data Explorer”, and republish by making visible to everyone. We will explain how to consume feeds protected by a feed key in a future blog post.

3.c. Why do I get an “Unrecognized version 3.0” error?

This error crops up if the WCF Data Services October 2011 CTP is not installed. Please see above and verify that this CTP is installed.

4. Resources used in this post

The publicly available publish page for the basic feed used in this post can be found at https://ws32842467.dataexplorer.sqlazurelabs.com/Published/Basic%20publish%20to%20VS%20test

The feed URL used for this post is available at https://ws32842467.dataexplorer.sqlazurelabs.com/Published/Basic%20publish%20to%20VS%20test/Feed

The sample VS project used for this post can be downloaded here.

The OData Team reported OData Service Validation Tool Update: 7 new rules on 1/30/2012:

OData Service Validation Tool was updated with 4 more new rules and couple other changes:

2 new common rules

2 new entry rules

Added new rule targets for Link, Property and Value payload type (targets that we used to call Other)

Added Not Applicable test result category to all code rules

This rule update brings the total number of rules in the validation tool to 140. You can see the list of the active rules here and the list of rules that are under development here.

Windows Azure Access Control, Service Bus and Workflow

No significant articles today.

Windows Azure VM Role, Virtual Network, Connect, RDP and CDN

No significant articles today.

Live Windows Azure Apps, APIs, Tools and Test Harnesses

BusinessWire (@BusinessWire) reported CopperEgg™ Expands Server Monitoring into Windows Azure Environments in a 2/1/2012 press release via the Azure Cloud on Ulitzer:

CopperEgg, Corp., a real-time monitoring company, today announced the support of Windows Azure™ server monitoring with RevealCloud Pro™.

RevealCloud Pro is a SaaS-based server monitoring service that provides a detailed, common view of all of your servers, running any OS, across public and private clouds, and virtual and physical environments. It provides real-time, in-depth insight into all the components participating in application and service delivery, which is critical for monitoring changing loads as well as identifying, isolating and addressing performance issues before service quality is impacted. RevealCloud Pro now supports server monitoring for Windows, Windows Azure, Hyper-V, Linux, Mac OSX and FreeBSD operating systems.

Today users expect applications and services to perform seamlessly. If an application stops responding – even for a few seconds – customers get easily frustrated and often abandon their efforts, resulting in lost revenue and/or productivity. Server monitoring using RevealCloud Pro provides split-second monitoring of the operating and performance metrics for servers that support critical application services, no matter where they are located. This enables IT and development teams to accelerate time-to-market during the development phase, optimize deployments as they learn customer usage patterns, and identify early warning signs of service degradation.

“Our customers upload large volumes of data in batches regularly and when that occurs, we need to scale up quickly so we can process the data efficiently. Using RevealCloud Pro we now have instant insight as to when our servers are reaching capacity,” said Carl Ryden, chief technology officer of PrecisionLender. “Then using their WebHooks combined with the Azure Management API, we’ve implemented automatic scaling of our servers so we can address load spikes without any intervention on our part. For 15 minutes worth of work and a couple dollars a month, we couldn’t have asked for a better server monitoring solution.”

“RevealCloud Pro is quickly gaining traction with our customers deploying and managing services in the cloud, virtual environments and data centers,” said Scott Johnson, CEO of CopperEgg. “The addition of server monitoring for Windows Azure addresses the needs of developers building services on the Azure platform as well as hybrid environments, who are looking for a robust server monitoring solution that is quickly deployed, easy to use and provides instant insight into server health.”

CopperEgg’s Server Monitoring – Key Benefits

Monitor servers across environments: including clouds, virtual and physical servers.

Install in 10 seconds: package for automatic deployment on new instances, so you never lose an instance.

Get real-time updates: get instant visibility into the performance and health of your systems and detect trends as they develop.

Customize alerts: set thresholds for alerts and direct them to individuals or groups as applicable for instant insight into the health of a system.

Tailor views: see your whole infrastructure from a sortable top-level systems view; or tag and group systems as needed for optimal server monitoring.

Enable server monitoring from any browser: PC, Mac, iPad, iPhone, Android or other mobile device.

Server monitoring with CopperEgg’s RevealCloud Pro is available here with a $15 instant credit on a ‘pay for consumption’ basis.

About CopperEgg

CopperEgg provides real-time insight into the speed and availability of applications and services deployed on cloud, virtual and physical servers. Our SaaS-based server monitoring tools are designed to install in seconds with no configuration and immediately deliver fine-grained visibility into critical performance metrics, insight into developing trends, and split-second decision support for organizations of all sizes. CopperEgg is backed by Silverton Partners and based in Austin, Texas. For more information, visit: www.copperegg.com.

Himanshu Singh posted IFS 360 Scheduling – Always Optimal on Windows Azure on 2/1/2012:

IFS has announced the immediate availability of IFS 360 Scheduling on Windows Azure as part of its strategy to leverage cloud computing, citing Windows Azure as a cost-efficient platform for innovative business solutions.

IFS 360 Scheduling empowers organizations to deliver on-site customer service excellence. Optimizing resource plans to minimize travel and maximize productive work time for hundreds, or even thousands, of technicians is a complex and compute-intensive task, and IFS 360 Scheduling’s ‘always optimizing’ approach means the schedule is constantly up-to-date no matter what is happening out in the field – for organizations with tight deadlines to meet, minutes matter.

Meeting tough customer expectations and providing consistent service demands a reliable and robust computing infrastructure. The speed of deployment on Windows Azure and on-demand scaling of resources echo the needs of these dynamic environments, for which resource scheduling is mission critical.

You can read more on the IFS announcement on their website, more about the work of IFS 360 Scheduling here, and a recent blog posting by IFS’ CTO Dan Matthews on cloud computing.

Himanshu Singh (pictured below) posted Real-World Windows Azure: Interview with Michael Ross, VP of Delivery at Aidmatrix on 2/1/2012:

As part of the Real World Windows Azure interview series, I talked to Michael Ross, vice president for Delivery at Aidmatrix, about using Windows Azure to help power its humanitarian aid delivery solutions. Here’s what he had to say.

Himanshu Kumar Singh: Tell me about the Aidmatrix Foundation.

Michael Ross: Well, we create web-based supply-chain management (SCM) solutions that help our partners optimize distributing humanitarian relief. We started in 2001 and are headquartered in Dallas, with offices in Wisconsin, Washington D.C., Germany and India. More than 40,000 leading nonprofit, business, and government partners leverage our solutions to mobilize more than $1.5 billion in aid annually, worldwide. The donated goods, money and services impact the lives of more than 65 million people.

HKS: How does Aidmatrix leverage technology to deliver humanitarian aid?

MR: Increasingly, technology plays a vital role in enabling the rapid, targeted distribution of relief supplies to those in need. Our solutions help NGOs procure, manage and deliver humanitarian relief more efficiently by having real-time access to know what inventory is on hand and what unmet needs still exist. In this way, NGOs can save money in their purchasing, be more efficient in their distribution, and more responsive and transparent with their donors. .

HKS: Are there also technology challenges you face?

MR: To perform optimally both in daily humanitarian relief and unplanned disasters, we need to ensure that our applications can consistently deliver the highest levels of stability and throughput performance. And because we work with so many different kinds of partner organizations, our solutions need to synchronize with business systems that run on diverse operating system platforms. They also need to flexibly scale to handle massive bursts in demand. Following several recent natural disasters, our applications have experienced usage spikes on the order of 1,000 times the standard rate.

HKS: How much demand is there for technology for humanitarian aid from Aidmatrix?

MR: In 2010, we noted a 20 percent year-over-year increase in the number of people who used our applications. And as more NGOs adopt technology as a strategic part of their global operational success, the demand will continue to grow.

HKS: How has this increase in demand impacted your ability to meet global needs?

MR: This rapid growth has only compounded the challenges that we face in deploying, configuring, and scaling online relief management solutions. For example, for several years, we hosted our applications in data centers in the United States but in response to recent requests for services from humanitarian organizations in Europe, we’ve needed to locate additional server resources overseas. Setting up and maintaining data centers around the globe can be expensive and time-consuming. In a few cases, we’ve needed to pre-position assets and leave them turned off. Instead of paying for what we use, we’ve ended up investing up front for resources that we may or may not eventually need. But that’s all part of being ready for unplanned disasters.

We also found that the time required to source and deploy new hardware impacted our agility. In mid-2010, we began a partnership with a large food bank network that has locations spread across the United Kingdom. After a few weeks, we noticed that the distance between our servers and end users caused the application to run slowly and even time out before people could complete their donations.

HKS: What was the solution?

MR: To empower more partners to help more people around the world, we needed the ability to scale applications at a moment’s notice while maintaining reliable performance. With an eye on keeping operating costs as low as possible, we began to investigate the advantages of moving to the cloud. In considering this change, we wanted to minimize the time needed to migrate existing applications and placed a priority on a cloud technology platform that would support the agile development of new functionality, along with the creation of new solutions to meet the unforeseen demands of future humanitarian crises.

HKS: What led you to choose Windows Azure?

MR: After evaluating several cloud services technologies, including Amazon Elastic Compute Cloud (EC2), we decided to adopt Windows Azure because, simply put, Windows Azure gives us all the tools we need to be more agile. It offers platform-as-a-service capabilities, so we don’t have to push out updates or worry about building out our own redundancy system. Plus, it incorporates a familiar development environment, so we can maximize productivity.

HKS: Which of your solutions did you move to Windows Azure first?

MR: The first solution we moved to Windows Azure was the Aidmatrix Program Metrics and Evaluation, which one of our U.S. partners uses to track the services delivered to clients at more than 1,000 locations. Moving the database component to Microsoft SQL Azure took five minutes, and we instantly gained high availability, along with reliable fault tolerance and security, at a fraction of what it would cost to build out those capabilities ourselves.

In early 2011, working with Accenture and Avanade, we also moved our Online Warehouse solution, which provides end-to-end inventory management tools for tracking relief supplies, to Windows Azure. The conversion process for each of these applications took approximately six weeks to complete.

We also recently used Windows Azure to configure and deploy a web portal application built by Microsoft to assist Second Harvest Japan, the Japanese food banking network. I deployed the application to the Microsoft data center in Hong Kong from home following the tsunami and I didn’t need to worry about how many instances to deploy because I could quickly scale out server resources if needed.

HKS: Are you using any other Windows Azure technologies?

MR: We take advantage of several Windows Azure technologies to ensure reliable, scalable performance. For example, we will use Windows Azure Connect to enable data sharing between our Online Warehouse solution and the on-premises operational systems used by our partners, including Oracle and SAP enterprise resource planning systems. And we rely on Windows Azure storage services to collect data on session state management so our staff can monitor application performance and troubleshoot issues in near real time.

HKS: What are some of the benefits you’ve seen from your move to Windows Azure?

MR: One of the key benefits has been noticeable cost savings. Because we no longer need to purchase, set up, and maintain database management and web servers, we expect to save 20 percent on data center costs, which could mean up to $100,000 in savings. We will be able to redirect the time and money we save toward creating and enhancing applications that help save lives. And instead of spending valuable resources on server upkeep, we can redirect time and cost savings to higher-value tasks, like developing a new module for one of our applications.

Another benefit is the global availability of Microsoft data centers and the pay-as-you-go model, which enables us to ensure cost-efficient dynamic scalability for our solutions. Our applications need to handle usages spikes of 1,000 times the normal load. With Windows Azure, we can scale up or down in a very agile and efficient way, which is essential for the kind of work we do.

Finally, the growing ecosystem of ISVs that have adopted Windows Azure allows us to accelerate our own development lifecycle. Taking advantage of functionality from other ISVs speeds our development and ultimately increases the value of our solutions. This is a major benefit of working with a cloud provider like Microsoft that has a large and growing network of partners.

Read the full case study. Learn how others are using Windows Azure.

Avkash Chauhan (@avkashchauhan) described Handling two known issues with Windows Azure node.js SDK 0.5.2 in a 1/31/2012 post:

My recent development work with Windows Azure Node.js SDK 0.5.2 helped me to found two issues. Until next SDK update is available please use the given workaround to solve your problem.

Issues 1: Exception while testing your node.js application in Compute Emulator

After you installed the latest available Node.js SDK 0.5.2 and launch your node application in Azure Emulator as below:

>> Start-AzureEmulator -Launch

You might see an exception as below:

>> Windows Azure Web Role Entry Point host has stopped working…

Root Cause:

The issue appears to be with server.js.logs, when the web role is started, IISConfigurator changes the ACL for the web role directory (and all contained directories). This process if server.js.logs is present and cause IISConfigurator process to crash which ultimately cause your role to show above error.

Solution:

Please delete the server.js.logs folder before launching the emulator and you will not see the problem. I have tested this workaround and it does work fine with SDK 0.5.2.

Issues 2: Unable to publish MongoDB enabled Node.js app to Azure

When you try to publish a MongoDB enabled node.js app using “Publish-AzureService” powershell command, you might get an error and publish process interrupted.

PS>> Publish-AzureService

Publish-AzureService : Object reference not set to an instance of an object.

NullReferenceException

Root Cause:

When you try to publish the application, the code look for certificate node in the CSCFG file and it this node is not present it generate an exception as seen by you. As certificate node ins CSCFG file is optional so the code must not dependent on it. PG could not repro this issue because they had Certificate node I their CSCFG and did not look for dependencies. We sure can overcome this problem by defining empty Certificates nodes in Role sections of cscfg.

Solution:

So to work around this issue please add the empty Certificate node in your CSCFG file in the appropriate section.

<Certificates/>

Avkash Chauhan (@avkashchauhan) described NougakuDoCompanion: A “Ruby on Rails” companion for Windows Azure in a 1/31/2012 post:

I was recently informed about a great “Ruby on Rails” companion package to run your Ruby on Rails application on Windows Azure.

With this package you can:

Run multiple instance of Ruby Application on Windows Azure

Deploy your application virtually from any machine

Connect to SQL Azure

The application architectures is as below:

Companion Package information:

http://nougakudo.codeplex.com/

Documentation:

http://nougakudo.codeplex.com/documentation

Test results:

NougakuDo 1.0.5 tested with Rails 3.0.9

NougakuDo 1.1.7 tested with Rails 3.1.0

NougakuDo 1.1.9 tested with Rails 3.1.1

Visual Studio LightSwitch and Entity Framework 4.1+

Jan Van der Haegen (@janvanderhaegen) described LightSwitch achievements: Chapter two (Events) – Classic events: “The observer” on 2/1/2012:

In this blog post series, we’re going to create a fun extension that will stimulate the users of our LightSwitch applications by rewarding them with points and achievements when they use the application…

Also, in this particular post, I wrote my first VB.NET code ever, which seems to be the .NET language of choice of about 1/3rd of my readers… All comments on the VB.NET code will be considered positive feedback, no matter what you write or how you write it!

Functional POV

Conceptually, when talking about events in .NET, where are talking about a message that one object sends to zero-to-many other objects to let them know that something happened. There’s four important elements here:

the sender (aka the source): the object that sends the message

the message: what happened

the timing: when did it happen

the event handlers (aka the listeners): the objects that want to take an action for the event

Most events will find their use in the presentation layer of our applications, where we want to execute code when the user clicks a button, hover the mouse cursor over an image, …

Semantics POV

It’s hard to talk code without a small example..
using System;

namespace ConsoleApplicationCSharp
{

    //1. Define the event "signature", usually (but NOT required) has two arguments: sender & event arguments.
    public delegate void ChangedEventHandler(object sender, EventArgs e);

    public class ClassWithEvent
    {
        //2. Define the event.
        public event ChangedEventHandler Changed;

        public void InvokeTheEvent()
        {
            Console.WriteLine("Firing the event");
            // 3. Invoking the event (can only be done from inside the class that defined the event)
            if (Changed != null)
                Changed(this, EventArgs.Empty);
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            var classWithEvent = new ClassWithEvent();
            //4. Adding an event handler to the event
            classWithEvent.Changed += new ChangedEventHandler(classWithEvent_Changed);
            classWithEvent.InvokeTheEvent();

            Console.ReadKey();
        }

        static void classWithEvent_Changed(object sender, EventArgs e)
        {
            Console.WriteLine("The event has fired");
        }
    }
}
And the same example in VB.NET (I think…)
Imports System

Namespace ConsoleApplicationVB

    '1. Define the event "signature", usually (but NOT required) has two arguments: sender & event arguments.
    Public Delegate Sub ChangedEventHandler(sender As Object, e As EventArgs)

    Public Class ClassWithEvent
        '2. Define the event.
        Public Event Changed As ChangedEventHandler

        Public Sub InvokeTheEvent()
            Console.WriteLine("Firing the event")
            ' 3. Invoking the event (can only be done from inside the class that defined the event)
            RaiseEvent Changed(Me, EventArgs.Empty)
        End Sub
    End Class

    Module Program

        Sub Main()
            Dim classWithEvent As New ClassWithEvent
            ' 4. Adding an event handler
            AddHandler classWithEvent.Changed, AddressOf classWithEvent_Changed
            classWithEvent.InvokeTheEvent()
            Console.ReadKey()
        End Sub

        Private Sub classWithEvent_Changed(sender As Object, e As EventArgs)
            Console.WriteLine("The event has fired")
        End Sub
    End Module

End Namespace
Just as the functional POV, the code samples reveal that when dealing with events, there are four important elements:

What happened: the first step is to define the event “signature”, which happens in the form of a delegate. There’s a lot of “standard” event signatures provided by the .NET framework and I would advise you to use them over custom events, but as the samples show, it’s really easy to roll your own. Event signatures will often have two arguments: sender (a reference to the source, the object that raised the event), and (a subclass of) EventArgs (additional information about what happened).

The sender: we add a property to the class that exposes the event, and add the .NET keyword “event”. Conceptually, it’s ok to think of this property as a “collection of method references” that all match the signature from step1. From a technical POV, what we actually do is add a property which is an empty delegate, aka method pointer that has no value yet. On a traditional property, the compiler will generate two “subroutines”: “get” and “set”. When we add the keyword “event” to the property declaration however, the compiler will add two “subroutines” called “add” and “remove”. In the generated “add” subroutine, the value (new delegate) will be combined with the existing value, as opposed to a normal property, where the “set” will replace the existing value, and ofcourse the opposite happens in the “remove” “subroutine”.

When it happened: raising the event is quite simple: do a null-reference check then call the delegate (C#, more technical) or use the (VB.Net, more descriptive) keyword RaiseEvent. Conceptually, each method in our “collection of method references” from step 2 will be called. Raising the event can only be done from inside the class that defined it, although often you will see classes that expose this functionality to anyone ( depending on the access modifier: private/public/…).

The event handlers: these are the classes that subscribed to the event, in other words: told the sender they want to be notified when the event occurs, in the form of a method being called. To do this, use the VB.Net keyword AddHandler, or the C# operator +=. Both ways are just semantics that when compiled, result in the “add” subroutine from step 2 being called.

There’s a lot more technical aspects of events that I could discuss, the most important ones would be Garbage Collection (or more specifically: the prevention of GC on the event handlers) and thread safety, but let’s not forget that this is a LightSwitch blog, so let’s find out why this information is important for the Achievements extension that we are building in this blog post series…

LightSwitch POV

I wanted to take you in this shallow dive in events in .NET, because for our achievements extension, where most of the code resides in the model layer, not in the presentation layer, we’ll choose not to use this classic observer pattern, and use the event mediator pattern instead. To justify exactly why we’ll go with a different pattern, it’s important to understand the conceptual and technical differences between the two…

However, the achievements extension will have a little bit of graphical artwork to it, and for this code (/XAML) there’s one event in particular that could use some special attention: INotifyPropertyChanged.

What’s INotifyPropertyChanged?

INotifyPropertyChanged is an interface that exposes just one property: the PropertyChanged event. This event is extremely important in the MVVM pattern, because the bindings that we define in XAML recognize this event and will subscribe to it. A Binding instance will update its value (“refresh the view”) if the source raises the event.

The viewmodel that implements this interface can use the PropertyChangedEventArgs in two ways:

Fill in the PropertyName: by doing this, the sender signifies to the listeners that one particular property has changed, and that all bindings on this property should update.

Leave the PropertyName blank: by doing this, the sender signifies to the listeners that the object changed so radically, that all bindings to any property on this object should update. It’s basically requesting a general “refresh” of the view.

INotifyPropertyChanged has a brother called INotifyCollectionChanged, also recognized by bindings, useful for “ItemsSource” properties, …

An easy base class…

If you are using EME, (if you’re not: what the hell?) then there’s an easy base class included in the framework for your LightSwitch development convenience…

The ExtensionsMadeEasy.ClientAPI.Utilities.Base.NotifyPropertyChangedBase (source code here, it has been included for a while but never got around to blogging about it) is a base class (based on the beautiful work by Rob Eisenberg) for your (/my) ViewModels. It exposes three (undocumented, sorry) methods:
namespace LightSwitchApplication
{
    public class SomeViewModel : ExtensionsMadeEasy.ClientAPI.Utilities.Base.NotifyPropertyChangedBase
    {
        private int myProperty;

        public int MyPublicProperty
        {
            get { return myProperty; }
            set {
                myProperty = value;
                base.Refresh(); //Refresh all bindings
                base.OnPropertyChanged("MyPublicProperty"); //Classic approach
                base.OnPropertyChanged(() => MyPublicProperty); //Approach with lambda expression
            }
        }        

    }
}
Or, in VB equivalent (I think…)
Public Class SomeViewModel
    Inherits ExtensionsMadeEasy.ClientAPI.Utilities.Base.NotifyPropertyChangedBase

    Private myProperty As Integer
    Public Property MyPublicProperty() As Integer
        Get
            Return MyProperty
        End Get
        Set(ByVal value As Integer)
            myProperty = value
            MyBase.Refresh() 'Refresh all bindings
            MyBase.OnPropertyChanged("MyPublicProperty") 'Classic approach
            MyBase.OnPropertyChanged(Function() MyPublicProperty) 'Approach with lambda expression
        End Set
    End Property

End Class
The first method (Refresh) raises the PropertyChanged event without a PropertyName on the EventArgs, all bindings on “SomeViewModel” will be refreshed.

The second method raises the PropertyChanged event with a hardcoded PropertyName (“MyPublicPropertyName”) on the EventArgs, all bindings on this property will be refreshed. The string must match the actual property name exactly…

The third method does exactly the same as the second, but uses a lambda expression. The major advantage is that the property is not hardcoded (ie: not a string), which gives you compile-time safety: you can not misspell the property name by accident or it won’t compile (happened to me many times before), but will also be updated automagically if you decide to refactor your class a bit, and rename the property.

To each his own of course, but I definitely prefer the latter. Anyways, let’s see what this event mediator pattern is all about, and why it’s more suited for the model layer of our extension…

Return to section navigation list>

Windows Azure Infrastructure and DevOps

Tiffany Trader reported Gartner Says Platform as a Service is on the Cusp of Several Years of Strategic Growth in a 2/1/2012 post to the HPC in the Cloud blog:

Platform as a service (PaaS) is a core layer of the cloud computing architecture, and its evolution will affect the future of most users and vendors in enterprise software markets, according to Gartner, Inc.

"With large and growing vendor investment in PaaS, the market is on the cusp of several years of strategic growth, leading to innovation and likely breakthroughs in technology and business use of all of cloud computing," said Yefim Natis, vice president and distinguished analyst at Gartner. "Users and vendors of enterprise IT software solutions that are not yet engaged with PaaS must begin building expertise in PaaS or face tough challenges from competitors in the coming years."

PaaS is a common reference to the layer of cloud technology architecture that contains all application infrastructure services, which are also known as "middleware" in other contexts. PaaS is the middle layer of the end-to-end software stack in the cloud. It is the technology that intermediates between the underlying system infrastructure (operating systems, networks, virtualization, storage, etc.) and overlaying application software. The technology services that are part of a full-scope comprehensive PaaS include functionality of application containers (servers), application development tools, database management systems, integration middleware, portal products, business process management suites and others — all offered as a service.

In the Gartner Special Report, "PaaS 2012: Tactical Risks and Strategic Rewards" (http://www.gartner.com/technology/research/cloud-computing/report/paas-cloud.jsp), Gartner analysts said 2011 was a pivotal year for the PaaS market. As Gartner predicted last year in the report "PaaS Road Map: A Continent Emerging" (http://www.gartner.com/resId=1521622), the broad vendor adoption in 2011 amounted to a sound industry endorsement of PaaS as an alternative to the traditional middleware deployment models.

In 2012, the PaaS market is at its early stage of growth and does not yet have well-established leaders, best use or business practices or dedicated standards. The adoption of PaaS offerings is still associated with some degree of uncertainty and risk.

"However, PaaS products are likely to evolve into a major component of the overall cloud computing market, just as the middleware products — including application servers, database management systems (DBMSs), integration middleware and portal platforms — are the core foundation of the traditional software industry," Mr. Natis said. "The tension between the short-term risk and the long-term strategic imperative of PaaS will define the key developments in the PaaS market during the next two to three years."

Some of the newly announced PaaS offerings will reach general availability late in 2012, and by the end of 2013, all major software vendors will have competitive production offerings in the PaaS market. By 2016, competition among the PaaS vendors will produce new programming models, new standards and new software market leaders. However, until then, users will continue to experience architectural changes to technologies, business models and vendor alignments in the PaaS market.

As vendors continue to invest in PaaS services, and the major software vendors look to deliver comprehensive PaaS service portfolios, activity in all segments of PaaS will accelerate and the fast pace of growth and change in the PaaS market will create confusion, making user adoption decisions more difficult.

"While there are clear risks associated with the use of services in the new and largely immature PaaS market, the risk of avoiding the PaaS market is equally high," said Mr. Natis. "The right strategy for most mainstream IT organizations and software vendors is to begin building familiarity with the new cloud computing opportunities by adopting some PaaS services now, albeit with the understanding of their limitations and with the expectation of ongoing change in the market offerings and use patterns."

Additional information is available in the Gartner Special Report "PaaS 2012 — Tactical Risks and Strategic Rewards." The special report includes video commentary from Mr. Natis, as well as links to more than 30 related reports about the PaaS market.

Lori MacVittie (@lmacvittie) asserted “It’s about operational efficiency and consistency, emulated in the cloud by an API to create the appearance of a converged platform” in an introduction to her The Cloud API is Pseudo-Consolidation of Infrastructure post of 2/1/2011 to F5’s DevCentral blog:

In most cases, the use of the term “consolidation” implies the aggregation (and subsequently elimination) of like devices. Application delivery consolidation, for example, is used to describe a process of scaling up infrastructure that often occurs during upgrade cycles.

Many little boxes are exchanged for a few larger ones as a means to simplify the architecture and reduce the overall costs (hard and soft) associated with delivering applications. Consolidation.

But cloud has opened (or should have opened) our eyes to a type of consolidation in which like services are aggregated; a consolidation strategy in which we layer a thin veneer over a set of adjacent functionalities in order to provide a scalable and ultimately operationally consistent experience: an API. A cloud API consolidates infrastructure from an operational perspective. It is the bringing together of adjacent functionalities into a single “entity.” Through a single API, many infrastructure functions and services can be controlled – provisioning, monitoring, security, and load balancing (one part of application delivery) are all available through the same API. Certainly the organization of an API’s documentation segments services into similar containers of functionality, but if you’ve looked at a cloud API you’ll note that it’s all the same API; only the organization of the documentation makes it appear otherwise.

This service-oriented approach allows for many of the same benefits as consolidation, without actually physically consolidating the infrastructure. Operational consistency is one of the biggest benefits.

OPERATIONAL CONSISTENCY

The ability to consistently manage and monitor infrastructure through the same interface – whether API or GUI or script – is an important factor in data center efficiency. One of the reasons enterprises demand overarching data center-level monitoring and management systems like HP OpenView and CA and IBM Tivoli is consistency and an aggregated view of the entire data center.

It is no different in the consumer world, where the consistency of the same interface greatly enhances the ability of the consumer to take advantage of underlying services. Convenience, too, plays a role here, as a single device (or API) is ultimately more manageable than the requirement to use several devices to accomplish the same thing. Back in the day I carried a Blackberry, a mobile phone, and a PDA – each had a specific function and there was very little overlap between the two. Today, a single “smart”phone provides the functions of all three – and then some. The consistency of a single interface, a single foundation, is paramount to the success of such consumer devices. It is the platform, whether consumers realize it or not, that enables their highly integrated and operationally consistent experience.

The same is true in the cloud, and ultimately in the data center. Cloud (pseudo) consolidates infrastructure the only way it can – through an API that ultimately becomes the platform analogous to an iPhone or Android-based device.

Cloud does not eliminate infrastructure, it merely abstracts it into a consolidated API such that the costs to manage it are greatly reduced due to the multi-tenant nature of the platform. Infrastructure is still managed, it’s just managed through an API that simplifies and unifies the processes to provide a more consistent approach that is beneficial to the organization in terms of hard (hardware, software) and soft (time, administration) costs.

The cloud and its requisite API provide the consolidation of infrastructure necessary to achieve greater cost savings and higher levels of consistency, both of which are necessary to scale operations in a way that makes IT able to meet the growing demand on its limited resources.

Mary Jo Foley (@maryjofoley) “… speculates on the future of Microsoft's public cloud play” while asking Can Microsoft Save Windows Azure? in a 2/1/2012 article for RedmondMag.com:

Microsoft is slowly but surely working to make its Windows Azure cloud platform more palatable to the masses -- though without the benefit of roadmap leaks, it would be hard for most customers to know this.

When Microsoft began cobbling together its Windows Azure cloud plans back in 2007, there was a grand architectural plan. In a nutshell, Microsoft wanted to recreate Windows so that Redmond could run users' applications and store their data across multiple Windows Server machines located in Microsoft's (plus a few partners') own datacenters. In the last five years, Microsoft has honed that vision but has never really deviated too far from its original roadmap.

For Platform as a service (PaaS) purists -- and Microsoft-centric shops -- Windows Azure looked like a distributed-systems engineer's dream come true. For those unwilling or unable to rewrite existing apps or develop new ones that were locked into the Microsoft System Center- and .NET-centric worlds, it was far less appealing.

How many external, paying customers are on Windows Azure? Microsoft officials won't say -- and that's typically a sign that there aren't many. My contacts tell me that even some of the big Azure wins that Microsoft trumpeted ended up trying Windows Azure for one project and then quietly slinking away from the platform. However, Windows Azure is no Windows Vista. Nor is it about to go the way of the Kin. But without some pretty substantial changes, it's not on track to grow the way Microsoft needs it to.

This fact hasn't been lost on the Microsoft management. Starting last year, Microsoft began making a few customer- and partner-requested tweaks to Windows Azure around pricing. Then the 'Softies started getting a bit more serious about providing support for non-Microsoft development tools and frameworks for Windows Azure. Developer champion and .NET Corporate Vice President Scott Guthrie traded his red shirt for an Azure-blue one (figuratively -- still not yet literally) and moved to work on the Windows Azure application platform.

Starting around March this year, Microsoft is slated to make some very noticeable changes to Windows Azure. That's when the company will begin testing with customers its persistent virtual machine that will allow users to run Windows Server, Linux(!), SharePoint and SQL Server on Windows Azure -- functionality for which many customers have been clamoring. This means that Microsoft will be, effectively, following in rival Amazon's footsteps and adding more Infrastructure as a Service components to a platform that Microsoft has been touting as pure PaaS.

The first quarterly update to Windows Azure this year -- if Microsoft doesn't deviate from its late 2011 roadmap -- will include a number of other goodies, as well, such as the realization of some of its private-public cloud migration and integration promises. If you liked Microsoft's increased support for PHP, Java, Eclipse, Node.js, MongoDB and Hadoop from last year, take heart that the Windows Azure team isn't done improving its support for non-Microsoft technologies. Also on the Q1 2012 deliverables list is support for more easily developing Windows Azure apps not just on Windows, but also on Macs and Linux systems.

Microsoft's new focus with Windows Azure is to allow users to start where they are rather than making them start over. That may sound like rhetoric, but it's actually a huge change, both positioning- and support-wise for Microsoft's public cloud platform. Not everyone -- inside or outside the company -- agrees that this is a positive. Hosting existing apps in the cloud isn't the same as re-architecting them so they take advantage of the cloud. It will be interesting to see whether users who are tempted by the "new" Windows Azure are happy with the functionality for which they've been clamoring.

Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds

No significant articles today.

Cloud Security and Governance

David Navetta (@DavidNavetta) asked Cyber Insurance: An Efficient Way to Manage Security and Privacy Risk in the Cloud? in a 2/1/2012 post to the InfoLaw Group blog:

As organizations of all stripes increasingly rely on cloud computing services to conduct their business, (with many organizations entering into cloud computing arrangements with multiple cloud providers), the need to balance the benefits and risks of cloud computing is more important than ever. This is especially true when it comes to data security and privacy risks. Cloud providers are sitting on reams of data from thousands of customers, including sensitive information such as personal information, trade secrets, and confidential and proprietary information. To criminals Cloud providers are prime targets.

At the same time, based in large part on the amount of risk aggregated by Cloud providers, most Cloud customers are unable to secure favorable contract terms when it comes to data security and privacy. While customers may enjoy some short term cost-benefits by going into the Cloud, they may be retaining more risk then they want (especially where Cloud providers refuse to accept that risk contractually). In short, the players in this industry are at an impasse. Cyber insurance may be a solution to help solve the problem.

A Short History of Cyber Insurance Coverage*

*This section ended up longer than I anticipated. If you already have a base knowledge of cyber coverage or don’t want to bother with some historical background, please skip ahead to this section: "Where Privacy and Security Risk Breaks Down in Cloud Computing Contracts"

In the early 2000s, just around the “DotCom Bust”, some insurers began developing a product designed to address the financial loss that might arise out of a data breach. This was a time where most “brick and mortar” companies were just beginning to leverage the economic potential of the Internet. At that time insurers wanted to target the big “dotcom” companies like Amazon, Yahoo, eBay, Google, etc., and other companies pioneering e-commerce and online retailing. At some point, somebody dubbed this type of insurance “cyber insurance.”

The early cyber policies included liability and property components. The liability coverages addressed claim expenses and liability arising out of a security breach of the insured’s computer systems (some early policies only covered “technical” security breaches, as opposed to policy violation-based security breaches). The property-related components covered business interruption and data asset loss/damage arising out of a data breach (during the holiday season many online retailers suddenly developed a tasted for business interruption coverage after realizing just how negatively their business would be impacted by a denial of service attack). Additional first party coverages included cyber-extortion coverage and crisis management/PR coverage.

Unfortunately for the carriers, it was not easy to get people to understand the need for this coverage (and that is still a challenge today, but certainly a lesser challenge with all of the security and privacy news constantly streaming). Early on there were very few lawsuits and regulators were just beginning to consider enforcement of relatively new statutes like GLB and HIPAA.

Two things changed that made cyber insurance much more relevant. One was a rather sudden event, and the other more gradual.

First, in 2003, California passed SB1386, the world’s first breach notification law. The reality then (as now) is that companies suffer security breaches each and every day. Prior to SB1386, however, breaches of personal information simply went unreported. With SB1386 and the subsequent passage of breach notice laws in 45 other states (and now coming internationally), the risk profile changed for data breaches. Instead of burying the breaches, companies were required to incur significant direct expenses to investigate security breaches and comply with applicable breach notice laws, including the offering of credit monitoring to affected individuals (which is not legally required by existing breach notice laws, but is optionally provided by many companies or "suggested" by state regulators). As a result, the plaintiffs’ bar now had notice of security breaches and began filing class action lawsuits after big breaches (usually involving high-profile brand name organizations). As such, cyber insurance coverage went from coverage addressing a hypothetical risk of future lawsuits, to a coverage addressing real-life risk (and now we have lawsuits getting deeper into litigation and public settlements of these types of cases). Moreover, shortly after the passage of SB 1386 many cyber insurance policies began covering the direct costs associated with complying with breach notification laws, including attorney fees, forensic investigation expenses, printing and mailing costs, credit monitoring expenses and call center expenses. Breach notification costs are direct and almost unavoidable after a personal information breach. Regardless of lawsuit activity, a direct financial rationale for cyber insurance coverage now existed.

The other change that occurred more gradually over time, but which has had a significant impact concerning the frequency and magnitude of data breaches was organized crime. In the early 2000s hacking was more of an exercise in annoyance or a used for bragging purposes. Hackers at that time wanted their exploits talked about and know. They wanted credit for hacking into or bringing down a sophisticated company (or better yet a division of the Federal Government or military). As such, when an attack happened it was discovered and remediated, and that would be the end of it.

True criminals, of course, are less interested in such notoriety. In fact, when trying to steal thousands/millions of records to commit identity theft or credit card fraud it is much better to NOT be detected. Lingering on a company’s network taking information for months or years is a much more profitable endeavor. Recognizing that this type of crime is low risk (it can be performed from thousands of miles away in Eastern Europe with almost not chance of getting caught) and high reward, organized crime flooded into the space. And in this context the word “organized” is truly appropriate – these enterprises retain very smart IT-oriented people that use every tool possible to scale and automate their crimes. They leverage the communication tools on the Internet to fence their “goods” creating, for example, wholesale and retail markets for credit cards, or “eBay”-like auction sites to hawk their illicit wares (e.g. valuable information). The change in orientation described above has essentially resulted in a 24/7/365 relentless crime machine constantly attacking and looking for new ways to attack, and always seeming to be one step ahead of those seeking to stop them. That is why we read about security and privacy breaches practically every day in the newspaper.

Fast-forward to present time. Cyber insurance is a much more established market with more carriers entering on a regular basis. There are primary and excess markets available for big risks, and companies of all sizes are looking at cyber more as a mandatory purchase rather than discretionary. As the world continues to change at seemingly light-speed and cyber risks increase (with the advent of hacktivism, social media and the consumerization of IT/BYOD ) the need for cyber is also growing. With competition pushing cyber insurance prices down, and significant security and privacy risk being retained by organizations, risk transfer is becoming very attractive (and from an overall big picture systemic point of view, spreading is risk is also attractive). Another area where cyber may help smooth out security and privacy risk is with cloud computing.

Where Privacy and Security Risk Breaks Down in Cloud Computing Contracts

As we have written extensively of in the past, Cloud computing raises significant privacy and security risks that are often difficult to hammer out in a Cloud computing negotiation (to the extent a Cloud customer gets a chance to negotiate at all). The net result of these contract negotiation difficulties and Cloud provider unwillingness in many cases to take on meaningful risk contractually, is that the risk is retained solely by the Cloud customer. The following examples outline the privacy and security-related Cloud issues that impact the Cloud customer's risk:

a Cloud provider failing to maintain reasonable security to prevent data breaches;

a Cloud provider failing to comply with privacy and security laws applicable to the Cloud customer;

a Cloud provider refusing to allow a Cloud customer to conduct its own independent forensic investigation of a data breach suffered by a Cloud provider;

potential conflict of interests with respect a Cloud provider’s handling a data breach that may have been the fault of the Cloud provider, including failing to cooperate with its Cloud customers if that cooperation could adversely impact the Cloud provider;

the Cloud customer’s potential obligation to comply with breach notice laws, including absorbing expenses for legal fees, forensic investigators, printing and mailing, credit monitoring and maintain a call center;

lawsuits and regulatory actions against the Cloud customer because of Cloud provider security and privacy breaches, and the legal fees, judgments, fines, penalties and settlement costs associated with them; and

Cloud providers seeking to leverage and data mine Cloud customer information being processed in the Cloud.

The justification used by Cloud providers to avoid responsibilities for these risks and the costs associated with them is essentially risk aggregation. Cloud providers maintain that, because they serve hundreds or thousands of customers on shared computing resources, a single attack could expose Cloud providers to liability from all of those customers at the same time. In fact, we already have one example involving a business interruption of a Cloud provider that demonstrates how multiple customers can be affected by a security breach. They also claim that independent forensic investigations by customers in the wake of a data breach are not possible because they cannot accommodate multiple customers at one time, and even if they could a forensic assessment would essentially expose each Cloud customer’s data to every Cloud customer conducting such an investigation.

Cyber Insurance: Addressing Retained Risk in the Cloud

So how does cyber insurance fit into this picture? As it currently stands, cyber insurance can be a very valuable tool for Cloud customers who are not able to get their providers to contractually take financial responsibility for security and privacy risk. Most cyber insurance policies cover data security and privacy breaches of not only the computer networks directly under the control of the insured, but also those computer networks operated by third parties for or on behalf of the insured. What this means in the Cloud context is that most cyber insurance policies may cover data breaches of the Cloud provider’s systems where the Cloud customer's/insured's data is stored and processed on those systems. This coverage will typically include most of the expenses listed above, including those direct expenses to comply with breach notice laws and costs to defend lawsuits and regulatory actions arising out of Cloud provider data breaches. As such, in the event a Cloud customer cannot get reasonable contract terms, assuming it has purchased the correct cyber coverage, it will have a fallback risk transfer and will not be retaining that risk solely on its own.

Is there a catch? Not really currently, except of course the premium that must be paid and the fact that most cyber insurance policies have a self-insured retention that must be satisfied by the insured before the carriers is required to pay. However, there may be longer term problems that arise for the carriers.

At this point, whether they like it or not, carriers whose cyber insurance policies cover security and privacy breaches of third party service providers are already beginning to aggregate their risk when it comes to Cloud providers. Imagine a world with a relatively small number of Cloud providers serving a much larger customer base (to some degree we may already live in such a world considering the dominance of Google, Amazon, Rackspace and other big cloud players). Many insureds/Cloud customers are going to be dealing with this relatively small number of Cloud providers. For example, I am sure that for most cyber insurance companies, if they were to check their books, would find that many of their insureds already use the same Cloud providers and/or other third party service providers to store and process the insureds’ data. Further consolidation of Cloud provider, should that occur, will only increase the aggregation of risk.

However, as long as cyber insurance is more widely adopted, the aggregation risk may be manageable. The entire purpose of insurance is to spread the risk across a wide community of insureds, and by doing so hopefully individual insureds that experience a breach are not catastrophically impacted. At the same time carriers can build reserves and achieve reasonable profits. The long term question is whether there are enough insureds purchasing cyber insurance to spread the risk and allow for the building of reserves to cover a breach of a major cloud provider that impacts a wide audience of insureds.

We probably are not there yet, and unless demand increases, we may not get there. One thing that may happen, perhaps, is a push from the Cloud provider/customer community to somehow make cyber insurance more of a mandatory condition of doing business in the Cloud. Time will tell as to whether the cyber insurers view this aggregation issue as serious, and whether they will take steps to mitigate it (hopefully those steps will not involve narrowing the coverage). In the meantime, companies that are going deep into the Cloud should quantify the risk they are retaining and seriously consider Cyber insurance coverage. The price may be right, and the peace of mind priceless.

Cloud Computing Events

David Pallman described Upcoming February Sessions on Web and Cloud at Portland Cloud Intelligence and CloudFest Denver in a 2/1/2012 post:

Here are some talks I and my Neudesic colleagues are giving on web, cloud, and programming at Portland Cloud Intelligence (Feb 2) and CloudFest Denver (Feb 9):

Development in the Cloud
Stuart Celarier
Portland Cloud Intelligence Thu Feb 2 1:00p
View Walk-through
What’s it like to develop for the cloud? In this session, you’ll see what the development experience is like for Windows Azure. We’ll start with “Hello, Cloud” in Visual Studio, running it first in the local simulation environment and then deploying to a data center in the cloud. We’ll progressively add more features to the application to illustrate the use of different cloud services. Both the local simulation environment and deployment to a cloud data center will be shown. We’ll also discuss the software development lifecycle and share best practices for cloud development. You’ll leave with an understanding of the differences and similarities between cloud development and enterprise development.

Keeping an Eye on the Cloud: Azure Diagnostics
Mike Erickson (@mgerickson)
CloudFest Denver Thu Feb 9 4:10p
When an application is running in Windows Azure it can be difficult to know what is happening. In this session we will cover the techniques that you can implement to gather diagnostic information from the instances running in Azure. We will discuss how you can instrument your code to help you understand what is happening while it is running. We will also look at how you can connect to instances that are running in Azure.

When Worlds Collide: HTML5 Meets the Cloud
David Pallmann (@davidpallmann)
Portland Cloud Intelligence Thu Feb 3:30p
CloudFest Denver Thu Feb 9 1:20p
View Presentation
What does HTML5 mean for cloud computing, and vice-versa? When two revolutions are happening simultaneously you can expect interesting synergies and interactions. Even as cloud computing and social networks are profoundly transforming the back end, HTML5 and mobile devices are profoundly transforming the front end. When these worlds collide we can achieve something truly remarkable: rich, take-anywhere immersive experiences backed by on-demand, elastic services running at global scale. Both worlds are driving changes in the way we design software, and good solution architecture demands they be considered jointly. In this session you'll see how HTML5 and the cloud combine functionally and architecturally, illustrated by example.

Other Cloud Computing Platforms and Services

Jeff Barr (@jeffbarr) posted New Elastic MapReduce Features: Metrics, Updates, VPC, and Cluster Compute Support (Guest Post) by Adam Gray on 1/31/2012:

Today's guest blogger is Adam Gray. Adam is a Product Manager on the Elastic MapReduce Team.

We’re always excited when we can bring features to our customers that make it easier for them to derive value from their data—so it’s been a fun month for the EMR team. Here is a sampling of the things we’ve been working on.

Free CloudWatch Metrics
Starting today customers can view graphs of 23 job flow metrics within the EMR Console by selecting the Monitoring tab in the Job Flow Details page. These metrics are pushed CloudWatch every five minutes at no cost to you and include information on:

Job flow progress including metrics on the number of map and reduce tasks running and remaining in your job flow and the number of bytes read and written to S3 and HDFS.

Job flow contention including metrics on HDFS utilization, map and reduce slots open, jobs running, and the ratio between map tasks remaining and map slots.

Job flow health including metrics on whether your job flow is idle, if there are missing data blocks, and if there are any dead nodes.

Please watch this video to see how to view CloudWatch graphs in the EMR Console:

You can also learn more from the Viewing CloudWatch Metrics section of the EMR Developer Guide.

You can view the new metrics in the AWS Management Console:

Further, through the CloudWatch Console, API, or SDK you can set alarms to be notified via SNS if any of these metrics go outside of specified thresholds. For example, you can receive an email notification whenever a job flow is idle for more than 30 minutes, HDFS Utilization goes above 80%, or there are five times as many remaining map tasks as there are map slots, indicating that you may want to expand your cluster size.

Please watch this video to see how to set EMR alarms through the CloudWatch Console:

Hadoop 0.20.205, Pig 0.9.1, and AMI Versioning
EMR now supports running your job flows using Hadoop 0.20.205 and Pig 0.9.1. To simplify the upgrade process, we have also introduced the concept of AMI versions. You can now provide a specific AMI version to use at job flow launch or specify that you would like to use our “latest” AMI, ensuring that you are always using our most up-to-date features. The following AMI versions are now available:

Version 2.0.x: Hadoop 0.20.205, Hive 0.7.1, Pig 0.9.1, Debian 6.0.2 (Squeeze)

Version 1.0.x: Hadoop 0.18.3 and 0.20.2, Hive 0.5 and 0.7.1, Pig 0.3 and 0.6, Debian 5.0 (Lenny)

You can specify an AMI version when launching a job flow in the Ruby CLI using the --ami-version argument (note that you will have to download the latest version of the Ruby CLI):

$ ./elastic-mapreduce --create --alive --name "Test AMI Versioning" --ami-version latest --num-instances 5 --instance-type m1.small

Please visit the AMI Versioning section of the Elastic MapReduce Developer Guide for more information.

S3DistCp for Efficient Copy between S3 and HDFS
We have also made available S3DistCp, an extension of the open source Apache DistCp tool for distributed data copy, that has been optimized to work with Amazon S3. Using S3DistCp, you can efficiently copy large amounts of data between Amazon S3 and HDFS on your Amazon EMR job flow or copy files between Amazon S3 buckets. During data copy you can also optimize your files for Hadoop processing. This includes modifying compression schemes, concatenating small files, and creating partitions.
For example, you can load Amazon CloudFront logs from S3 into HDFS for processing while simultaneously modifying the compression format from Gzip (the Amazon CloudFront default) to LZO and combining all the logs for a given hour into a single file. As Hadoop jobs are more efficient processing a few, large, LZO-compressed files than processing many, small, Gzip-compressed files, this can improve performance significantly.

Please see Distributed Copy Using S3DistCp in the Amazon Elastic MapReduce documentation for more details and code examples.

cc2.8xlarge Support
Amazon Elastic MapReduce also now supports the new Amazon EC2 Cluster Compute instance, Cluster Compute Eight Extra Large (cc2.8xlarge). Like other Cluster Compute instances, cc2.8xlarge instances are optimized for high performance computing, giving customers very high CPU capabilities and the ability to launch instances within a high bandwidth, low latency, full bisection bandwidth network. cc2.8xlarge instances provide customers with more than 2.5 times the CPU performance of the first Cluster Compute instance (cc1.4xlarge) instance, more memory, and more local storage at a very compelling cost. Please visit the Instance Types section of the Amazon Elastic MapReduce detail page for more details.

In addition, we are pleased to announce an 18% reduction in Amazon Elastic MapReduce pricing for cc1.4xlarge instances, dropping the total per hour cost to $1.57. Please visit the Amazon Elastic MapReduce Pricing Page for more details.

VPC Support
Finally, we are excited to announce support for running job flows in an Amazon Virtual Private Cloud (Amazon VPC), making it easier for customers to:

Process sensitive data - Launching a job flow on Amazon VPC is similar to launching the job flow on a private network and provides additional tools, such as routing tables and Network ACLs, for defining who has access to the network. If you are processing sensitive data in your job flow, you may find these additional access control tools useful.

Access resources on an internal network - If your data is located on a private network, it may be impractical or undesirable to regularly upload that data into AWS for import into Amazon Elastic MapReduce, either because of the volume of data or because of its sensitive nature. Now you can launch your job flow on an Amazon VPC and connect to your data center directly through a VPN connection.

You can launch Amazon Elastic MapReduce job flows into your VPC through the Ruby CLI by using the --subnet argument and specifying the subnet address (note that you will have to download the latest version of the Ruby CLI):

$ ./elastic-mapreduce --create --alive --subnet "subnet-identifier"

Please visit the Running Job Flows on an Amazon VPC section in the Elastic MapReduce Developer Guide for more information.

Windows Azure and Cloud Computing Posts for 1/30/2012+

2012-01-31T17:11:00.000-08:00

A compendium of Windows Azure, Service Bus, EAI & EDI Access Control, Connect, SQL Azure Database, and other cloud-computing articles.

Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:

Windows Azure Blob, Drive, Table, Queue and Hadoop Services
SQL Azure Database, Federations and Reporting
Marketplace DataMarket, Social Analytics and OData
Windows Azure Access Control, Service Bus, and Workflow
Windows Azure VM Role, Virtual Network, Connect, RDP and CDN
Live Windows Azure Apps, APIs, Tools and Test Harnesses
Visual Studio LightSwitch and Entity Framework v4+
Windows Azure Infrastructure and DevOps
Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds
Cloud Security and Governance
Cloud Computing Events
Other Cloud Computing Platforms and Services

Azure Blob, Drive, Table, Queue and Hadoop Services

Added TeraSort and TeraValidate jobs on 1/30 and 1/31/2012 to my (@rogerjenn) Introducing Apache Hadoop Services for Windows Azure post. From the preface:

The SQL Server Team (@SQLServer) announced Apache Hadoop Services for Windows Azure, a.k.a. Apache Hadoop on Windows Azure or Hadooop on Azure, at the Profesional Association for SQL Server (PASS) Summit in October 2011.

Update 1/31/2011: Added steps 15 and 16 with Job History and Cluster Management for the TeraSort job.

Update 1/30/2011: Added TeraSort job, steps 10 through 14, and TeraValidate job, steps 17 through 19. Waiting for response regarding interpretation of TeraValidate results.

Table of Contents

Introduction

Tutorial: Running the 10GB GraySort Sample’s TeraGen Job

Tutorial: Running the 10GB GraySort Sample’s TeraSort Job

Tutorial: Running the 10GB GraySort Sample’s TeraValidate Job

Apache Hadoop on Windows Azure Resources

Denny Lee (@dennylee) asked Moving data to compute or compute to data? That is the Big Data question in a 1/31/2012 post:

… As noted in the previous post Scale Up or Scale Out your Data Problems? A Space Analogy, the decision to scaling up or scaling out your data problem is a key facet in your Big Data problem. But just as important as the ability to distribute the data across commoditized hardware, another key facet is the movement of data.

Latencies (i.e. slower performance) are introduced when you need to move data from one location to another. To solve this problem within the data world, you can solve this by making it easier to move the data faster (e.g. compression, delta transfer, faster connectivity, etc.) or you design a system that reduces the need to move the data in the first place (i.e. moving data to compute or compute to data).

Scaling Up the Problem / Moving Data to Compute

To help describe the problem, the diagram below is a representation of a scale up traditional RDBMS. The silver database boxes on the left represent the database servers (each with blue platters representing local disks), the box with 9 blue platters represents a disk array (e.g. SAN, DAS, etc.), the blue arrows represent fiber channel connections (between the server and disk array), and the green arrows represent the network connectivity.

In an optimized scale up RDBMS, we often will setup DAS or SANs to quickly transfer data from the disk array to the RDBMS server or compute node (often allocating the local disk for the compute node to hold temp/backup/cache files). This scenario works great under the specific scenario that you can ensure low latencies.

And this is where things can get complicated, because if you were to lose disks on the array and/or fiber channel connectivity to the disk array – the RDBMS would go offline. But as described in the above diagram, perhaps you setup active clustering so the secondary RDBMS can take over.

Yet, if you were to lose network connectivity (e.g. the secondary RDBMS is not aware the primary is offline) or lose fiber channel connectivity, you would also lose the secondary.

The Importance of ACID

It is important to note that many RDBMS systems have features or designs that work around these problems. But to ensure availability and redundancy, if often requires more expensive hardware to work around the problematic network and disk failure points.

As well, this is not to say that RDBMS are based design – they are designed with ACID in mind – atomicity, consistency, isolation, and durability – to guarantee the reliability and robustness of database transactions (for more info, check out the Wikipedia entry: ACID).

Scaling Out the Problem / Moving Compute to Data

In a scale out or distributed solution, the idea is to have many commodity servers; they are many points of failure but there are also many paths for success.

Key to a distributed system is that as data comes in (the blue file icon on the right represent data such as web logs), the data is distributed and replicated in chunks to many nodes within the cluster. In the case of Hadoop, files are broken into 64K chunks and each of these chunks are placed into three different locations (if you set the replication factor to 3).

While you are using more disk space to replicate the data, now that you have placed the data into the system, you have ensured redundancy by replicating the data within it.

What is great about these types of distributed systems, they are designed right from the beginning to handle latency issues whether they be disk or network connectivity problems to out right losing a node. In the above diagram, a user is requesting data, but there is a loss to some disks and some network connections.

Nevertheless, there are other nodes that do have network connectivity and the data has been replicated so it is available. Systems that are designed to scale out and distribute like Hadoop can ensure availability of the data and will complete the query just as long as the data exists (it may take longer if nodes are lost, but the query will be completed).

The importance of BASE

By using many commodity boxes, you distribute and replicate your data to multiple systems. But as there are many moving parts, distributed systems like these cannot ensure the reliability and robustness of database transactions. Instead, they fall under the domain of eventual consistency where over a period of time (i.e. eventually) the data within the entire system will be consistent (e.g. all data modifications will be replicated throughout the cluster). This concept is also known as BASE (as opposed to ACID) – Basically Available, Soft State, Eventually Consistent. For more information, check out the Wikipedia reference: Eventual Consistency.

Discussion

Similar to the post Scale Up or Scale Out your Data Problems? A Space Analogy, choosing whether ACID or BASE works for you is not a matter of which one to use – but which one to use when. For example, as noted in the post What’s so BIG about “Big Data”?, the Yahoo! Analysis Services cube is 24 TB (certainly a case of moving data to compute with my obsession on random IO with SSAS) and the source of this cube is a 2PB of data from a huge Hadoop cluster (moving compute to data).

Each one has its own set of issues – scaling out increases the complexity of maintaining so many nodes, scaling up becomes more expensive to ensure availability and reliability, etc. It will be important to understand the pros/cons of each type – often it will be a combination of these two. Another great example can be seen in Dave Mariani (@mariani)’s post: Big Data, Bigger Brains at Klout’s blog.

ACID and BASE each have their own set of problems, the good news is that mixing them together often neutralizes the problems.

…

SQL Azure Database, Federations and Reporting

The SQL Azure Labs Team (@SQLAzureLabs) announced Microsoft Codename "SQL Azure Security Services" on 1/30/2012:

Scan your databases for security vulnerabilities

Microsoft Codename "SQL Azure Security Services" is an easy-to-use Web application that enables you to assess the security state of one or all of the databases on your SQL Azure server. The service scans databases for security vulnerabilities and provides you a report. You can use the recommendations and best practices provided in the report to improve the security state of your databases. The steps are simple:

Provide your SQL Azure account credentials to the service.

Choose the target for the report: Display in your browser or send to Windows Azure Blob storage.

View the report and get details from supporting documentation.

Take the corrective steps recommended in the report to secure your data.

Provide feedback on the data protection services you want in SQL Azure.

"SQL Azure Security Services" is an early prototype for solving the problem of securing your data in the cloud: no matter where it is, whatever the capacity and scale. Your feedback is very valuable. We look forward to hearing from you!

You will need a Windows Azure and SQL Azure subscription to use the service. If you don’t have one, click here to sign up for a free trial.

Click here to get started with this lab.

The SQL Server Team (@SQLServer) announced Microsoft SQL Server Migration Assistant 5.2 is Now Available in a 1/30/2012 post:

SQL Server Migration Assistant (SSMA) v5.2 is now available. SSMA simplifies database migration process from Oracle/Sybase/MySQL and Microsoft Access to SQL Server and SQL Azure. SSMA automates all aspects of migration including migration assessment analysis, schema and SQL statement conversion, data migration as well as migration testing to reduce cost and reduce risk of your database migration project.

The new version of SSMA - SSMA 5.2 provides the following major enhancements:

Support conversion of Oracle %ROWTYPE parameters with NULL default

Support conversion of Sybase’s Rollback Trigger

Better user credential security to support Microsoft Access Linked Tables

Reduce Cost and Risk of Competitive Database Migration
Does your customer have Oracle, Sybase, MySQL or Access databases that you like to migrate to SQL Server or SQL Azure? SQL Server Migration Assistant (SSMA) automates all aspects of migration including migration assessment analysis, schema and SQL statement conversion, data migration as well as migration testing. The tool provides the following functionalities:

Database Migration Analyzer: Assess and report complexity of the source database for migration to SQL Server. The generated report include detailed information on the database schema, percentage of the schema objects that can be converted by the tool, and hours estimate for manually migrating those schema objects that cannot be converted automatically. The information can be used for your to decide and to plan for migration. Visit SSMA team site for video demonstration of this feature.

Schema Converter: Automate conversion of schema objects (including programming code inside package, procedure, function) into equivalent SQL Server objects and T-SQL dialect. The tool provides ability to customize the conversion from hundreds of project setting options according to your specific business requirements. A report will be generated at the end of conversion for any object statement not supported for automated conversion. You can drill down to each of the migration issues and obtain side-by-side comparison between the original source code and the converted source code as well as make necessary modification directly from SSMA user interface. Visit SSMA team site for video demonstration of this feature.

Data Migrator: Migrate data from the source database to the SQL Server using the same conversion logic and type mapping specified in the project setting during schema conversion. Visit SSMA team site for video demonstration of this feature.

Migration Tester: Facilitate unit testing of converted program in SQL Server.

Simplify Migration to SQL Azure
Customers can simplify their move to the cloud with the SSMA. You can migrate from competitive database directly to SQL Azure. The tool reports possible migration issue to SQL Azure, convert schema, and migrate data to SQL Azure database. SSMA supports SQL Azure migration from Microsoft Access, MySQL, and Sybase.

Receive FREE technical support and migration resources.
Microsoft Customer Service and Support (CSS) provides free email technical support for SSMA. SSMA product web site and SSMA team blog site provides many resources to help customer to reduce cost and risk for database migration.

Download SQL Server Migration Assistant (SSMA) v.5.2
Launch the download of the SSMA for Oracle.
Launch the download of the SSMA for Sybase.
Launch the download of the SSMA for MySQL.
Launch the download of the SSMA for Access.

Cihan Biyikoglu (@cihangirb) posted a brief PHP and Federations in SQL Azure - Sample Code from Brian article on 1/29/2012:

Brian has a ton[n]e of samples on PHP and Federations in this post right here... Another great one!

http://blogs.msdn.com/b/silverlining/archive/2012/01/18/using-sql-azure-federations-via-php.aspx

MarketPlace DataMarket, Social Analytics and OData

The Social Analytics Team announced Enhanced Analytics and Sentiment Analysis Arrive in Social Analytics in a 1/31/2012 post to the Microsoft Codename “Social Analytics” blog:

We are proud to announce an update to the Microsoft Codename “Social Analytics” Lab. This update builds on the services we announced in October by providing the following new capabilities:

Enhanced analytics in our API

Sample usage of the enhanced analytics in the Engagement Client - 5 new analytic widgets

Improved sentiment analysis for Tweets

1. Enhanced Analytics:

Our existing API now provides richer analytics views of social data. This makes it easier to find top conversations, contributors, keywords and sources related to any filter with a simple ODATA query and returns conversations related to these calculations to your application.

Here’s a sample* ODATA query returning the top 5 keywords found in a filter for the past 7 days:

https://api.datamarket.azure.com/Vancouver/VancouverBillGates/GetTopKeywords?TimeIntervalSize='day'&TimeIntervalCount=7&TimeIntervalEnd=&FilterId=4&ResultsLimitTo=5

You can start investigating the details of the API by going to this link. If you are new to Social Analytics, check out our API overview here.

* Note: You will need to use your existing credentials for the lab to use this sample code. (Look at the API section here for details)

2. New Analytics Widgets in the Engagement Client:

See the new enhanced analytics in use in our sample Silverlight UI – the Engagement Client. We built these 5 analytic widgets using the new analytic enhancements to our API. These widgets provide the following analytic views for any filter:

Top Conversations

Top Contributors

Top Keywords

Volume

Sources

Here's an illustration of the new analytics in the sample Silverlight client:

3. Improved Sentiment Analysis for Tweets

In this release, we snuck in a little bonus! We updated the sentiment engine with minor enhancements to improve the analysis of sentiment in Tweets.

That's what's new in this release! In a few short months, we've made major progress on this experimental cloud service.

Now it's time for you to try the updated lab and let us know what you think. We're already working on the top feature request from our fall release - giving you the ability to define your own topics. We plan on making that functionality part of a future release.

For those new to this Lab, please check out our initial blog post for an overview of Microsoft Codename "Social Analytics". Here are some additional links from that post for your convenience:

Get started today

Visit our official homepage and Connect Site for more information

Connect with us in our forums

Stay tuned to our product blog for future updates

Update 2/1/2012: The Social Analytics Team promised in a comment “more details for the upgrade to the sentiment engine, as well as a description of the technology/algorithm(s) behind” its upgrade to the Microsoft Research sentiment code in the next post to the team’s blog.

Here’s my Microsoft Codename “Social Analytics” Windows Form Client about halfway through a test of 200,000 of the latest Tweets, posts, and other items:

Miguel Llopis (@mllopis) of the Microsoft Codename “Data Explorer” team posted Crunching Big Data with “Data Explorer” and “Social Analytics” on 1/30/2012:

Last week an interesting article was published in SearchCloudComputing which shows how to consume feeds from the “Social Analytics” Lab in “Data Explorer”, then perform some filter, transform and grouping operations on this data using Data Explorer’s intuitive UI and finally publish the resulting dataset as a snapshot so the contents at a given point in time can be accessed later on. You can take a look at the article following this link.

The author of this article, Roger Jennings, has also published a few other blog posts about Data Explorer on his blog. We particularly recommend you to follow the “41-step illustrated tutorial on creating a mashup”.

If you still haven’t done, you can sign up to try Data Explorer cloud service today!

Thanks for the kind words, Miguel.

Glenn Gailey (@ggailey777) posted More on the New OData T4 Template: Service Operations on 1/30/2012:

I’ve long missed support for calling service operations by using the proxy client code-generated by WCF Data Services, and I’ve described some workarounds in the post Calling Service Operations from the WCF Data Services Client. This is why I was excited to discover that the new T4 template, which I introduced in my previous post New and Improved T4 Template for OData Client and Local Database, now supports calling service operations as a first class behavior.

For example, the following T4-generated method on the NorthwindEntities context calls a GetOrdersByCity GET service operation:
public global::System.Data.Services.Client.DataServiceQuery<Order> 
    GetOrdersByCity(global::System.String city)
{ 
    return this.CreateQuery<Order>("GetOrdersByCity")
        .AddQueryOption("city" ,"'"+city+"'");
}
And, here’s another generated method that returns a collection of strings:
public  System.Collections.Generic.IEnumerable<string> GetCustomerNames()
{ 
    return (System.Collections.Generic.IEnumerable<string>)
        this.Execute<string>(new global::System.Uri("GetCustomerNames",
            global::System.UriKind.Relative),
            Microsoft.Data.OData.HttpMethod.Get,false);
}
Note that these methods use essentially the same techniques described in Calling Service Operations from the WCF Data Services Client.

Before Getting Started with the T4 Template

Before we get started, I should point out that there is currently a bug ( which I reported) in line 868, which can be fixed by changing this line of code as follows:
parameters=string.Concat(parameters,"global::System." + GetNameFromFullName(p.Type.FullName()));
(You might also want to do a global replace of "refrence" with "reference"--if you care about such things.)

Also, remember that this T4 template requires the current Microsoft WCF Data Services October 2011 CTP release for the upcoming OData release. To install this new T4 template into your project:

Make sure that you have NuGet installed. You can install it from here: https://nuget.org/.

If you haven’t already done so, use the Add Service Reference tool Visual Studio to add a reference to the OData service.
(The template needs the service.edmx file generated by the tool).

In your project, use the NuGet Package Manager Console to download and install the ODataT4-CS package:
PM> Install-Package ODataT4-CS

Open the Reference.tt template file and edit the line 868 to fix the bug described above.

In the Reference.tt template file, change the value of the MetadataFilepath property in the TransformContext constructor to the location of the .edmx file generated by the service reference and update the Namespace property to a namespace that doesn’t collide with the one generated by the service reference.

Now let’s compare the ease of using the client proxy generated by this new T4 template against the examples from the topic Calling Service Operations (WCF Data Services).

Calling a Service Operation that Returns a Collection of Entities

For example, here’s the previously difficult to compose URI-based query to call the GetOrdersByCity operation:
string queryString = string.Format("GetOrdersByCity?city='{0}'", city)
    + "&$orderby=ShippedDate desc"
    + "&$expand=Order_Details";

var results = context.Execute<Order>(
    new Uri(queryString, UriKind.Relative));
With the new template, this becomes a much nicer, LINQ query:
var results = from o in context.GetOrdersByCity(city)
                   .Expand("Order_Details")
              orderby o.ShippedDate descending
              select o;
Because this service operation returns an IQueryable<T> collection of entities, it can be further composed against and you get support for all the nice LINQ operations. And, as you would expect with DataServiceQuery<T>, the request to the service operation is made when the result is assigned or enumerated.

Calling a Service Operation that Returns a Single Entity

The benefits of this new client support for service operations are even more evident when calling a service operation that returns a single entity, which used to look like this:
string queryString = "GetNewestOrder";
Order order = (context.Execute<Order>(
              new Uri(queryString, UriKind.Relative)))
              .FirstOrDefault();
Which can now be simplified to this one clean line of code:
Order order = context.GetNewestOrder();
Note that in this case, the request is sent when the method is called.

Calling a Service Operation by using POST

It was a pet peeve of many folks that you couldn’t call a POST service operation from the client, even though it’s perfectly legal in OData. In this new version, you can now call DataServiceContext.Execute() and select the request type (GET or POST). This means that the new T4 template enables you to call POST service operations directly from methods:
// Call a POST service operation that returns a collection of customer names.
IEnumerable<string> customerNames = context.GetCustomerNamesPost();
(Please don’t give me grief over having a POST operation named Get…it’s just for demo purposes.) As before, you can only upload data by using parameters, which is still an OData requirement.

The Remaining Examples

Here’s the new versions of the remaining service operation examples (called synchronously) by using the new service operation methods on the client. In a way, they become almost trivial.

Calling a Service Operation that Returns a Collection of Primitive Types
var customerNames = context.GetCustomerNames();
Calling a Service Operation that Returns a Single Primitive Type
int numOrders = context.CountOpenOrders();
Calling a Service Operation that Returns Void
context.ReturnsNoData();
Calling a Service Operation Asynchronously

OK, so it’s now super easy to synchronously call service operations using this new template-generated proxy client, but what about asynchronous calls? Well, there is a way to do this with the generated proxy, but it only works for service operations that return a collection of entities. In this case, it works because the service operation execution is represented as a DataServiceQuery<T> instance, which has its own async execution methods. For this kind of service operation, you can make the execution work asynchronously, like this:
public static void GetOrdersByCityAsync()
{
    // Define the service operation query parameter.
    string city = "London";            

    // Create the DataServiceContext using the service URI.
    NorthwindEntities context = new NorthwindEntities(svcUri);

    // Define a query for orders based on calling GetOrdersByCity.
    var query = (from o in context.GetOrdersByCity(city)
                          .Expand("Order_Details")
                 orderby o.ShippedDate descending
                 select o) as DataServiceQuery<Order>;

     // Asynchronously execute the service operation that returns 
     // all orders for the specified city. 
     query.BeginExecute(OnAsyncExecutionComplete, query);
 }

private static void OnAsyncExecutionComplete(IAsyncResult result)
{
    NorthwindEntities context = new NorthwindEntities(svcUri);
    
    // Get the context back from the stored state.
    var query = result.AsyncState as DataServiceQuery<Order>;            
    
    try
    {
        // Complete the exection and write out the results.
        foreach (Order o in query.EndExecute(result))
        {
            Console.WriteLine(string.Format("Order ID: {0}", o.OrderID));

            foreach (Order_Detail item in o.Order_Details)
            {
                Console.WriteLine(String.Format("\tItem: {0}, quantity: {1}",
                    item.ProductID, item.Quantity));
            }
       }
   }
   catch (DataServiceQueryException ex)
   {
        QueryOperationResponse response = ex.Response;
        Console.WriteLine(response.Error.Message);
    }
}
Since the other types of execution rely on calling the synchronous DataServiceContext.Execute() method, I’m not sure how I would make this work with the existing APIs—I need to think more about it.

Service Operations Won’t Work on Asynchronous Clients

And don’t even bother trying to make any of this work with the Windows Phone client. First, there isn’t currently a Windows Phone client available that supports OData v3. (In fact, I removed the service operations code altogether from my hybrid T4 template, just to prevent any errors.) While the October CTP did include a Silverlight client, there is a bigger problem. Under the covers, most of the generated methods call DataServiceContext.Execute(), which is a synchronous method that doesn’t exist in the asynchronous clients. This means lots of errors should you encounter a service operation when using the template in a Silverlight app. The problem is that if you leave the T4 template code in place for a Silverlight client, when it comes across a service operation that requires the use of DataServiceContext.Execute(), which doesn’t exist in the Silverlight client, you will get a compiler error.

Just to be clear…on these async clients, only generated service operations methods are broken. The rest of the T4 template code works fine.

I’ve been providing feedback to the author of this excellent new template, so I have high hopes that much of this can be addressed.

Windows Azure Access Control, Service Bus and Workflow

Mick Badran (@mickba) explained Azure ServiceBus: Fixing the dreaded ‘The X.509 certificate CN=servicebus.windows.net chain building failed’ error in a 1/30/2012 post:

Scotty & myself have had this error going for over 2 weeks now, and have tried many options, settings, registry keys, reboots and so on.
(we have had this on 2 boxes now, that are *not* directly connected to the internet. They are locked down servers with only required services accessible through the firewall)

Generally you’ll encounter this error is you install Azure SDK v1.6 – there has been people that have revert back to Azure v1.5 SDK when this error has been encountered and this seems to fix most of their problems.

Here I’m using netTcpRelayBinding, BizTalk 2010 but this could just have easily have been IIS or your own app.

Finding the outbound ports and Azure datacenter address space is always the challenge. Ports 80,443,9351 and 9352 are the main ones with the remote addresses being the network segments of your Azure Datacenter.

The problem: “Oh it’s a chain validation thing, I’ll just go and turn off Certificate checking…” let me see the options.
(this is what we thought 2+ weeks ago)

Here I have a BizTalk shot of the transportClientEndpointBehaviour with Authentication node set to NoCheck and None (you would set these from code or a config file outside of biztalk)

We found that these currently have NO BEARING whatsoever…2 weeks we’ll never get back.

Don’t be drawn into here, it’s a long windy path and you’ll most likely end up short.

I am currently waiting to hear back from the folks on the product team to see what the answer is on this – BUT for now as a workaround we sat down with a network sniffer to see the characteristics.

Work around:

1. Add some Host Entries

2. Create a dummy site so the checker is fooled into grabbing local CRLs.

Add these Entries to your HOSTs file.

127.0.0.1 www.public-trust.com
127.0.0.1 mscrl.microsoft.com
127.0.0.1 crl.microsoft.com
127.0.0.1 corppki

Download and extract these directories to your DEFAULT WEB SITE (i.e. the one that answers to http://127.0.0.1/…..)
This is usually under C:\inetpub\wwwroot (even if you have SharePoint installed)

-------------------- The nasty error -------------------

The Messaging Engine failed to add a receive location "<receive location>" with URL "sb://<rec url>" to the adapter "WCF-Custom". Reason: "System.ServiceModel.Security.SecurityNegotiationException: The X.509 certificate CN=servicebus.windows.net chain building failed. The certificate that was used has a trust chain that cannot be verified. Replace the certificate or change the certificateValidationMode. The revocation function was unable to check revocation because the revocation server was offline.

---> System.IdentityModel.Tokens.SecurityTokenValidationException: The X.509 certificate CN=servicebus.windows.net chain building failed. The certificate that was used has a trust chain that cannot be verified. Replace the certificate or change the certificateValidationMode. The revocation function was unable to check revocation because the revocation server was offline.

at Microsoft.ServiceBus.Channels.Security.RetriableCertificateValidator.Validate(X509Certificate2 certificate)
at System.IdentityModel.Selectors.X509SecurityTokenAuthenticator.ValidateTokenCore(SecurityToken token)
at System.IdentityModel.Selectors.SecurityTokenAuthenticator.ValidateToken(SecurityToken token)
at System.ServiceModel.Channels.SslStreamSecurityUpgradeInitiator.ValidateRemoteCertificate(Object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
at System.Net.Security.SecureChannel.VerifyRemoteCertificate(RemoteCertValidationCallback remoteCertValidationCallback)
at System.Net.Security.SslState.CompleteHandshake()
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.ServiceModel.Channels.SslStreamSecurityUpgradeInitiator.OnInitiateUpgrade(Stream stream, SecurityMessageProperty& remoteSecurity)
--- End of inner exception stack trace ---
at System.ServiceModel.Channels.SslStreamSecurityUpgradeInitiator.OnInitiateUpgrade(Stream stream, SecurityMessageProperty& remoteSecurity)
at System.ServiceModel.Channels.StreamSecurityUpgradeInitiatorBase.InitiateUpgrade(Stream stream)
at System.ServiceModel.Channels.ConnectionUpgradeHelper.InitiateUpgrade(StreamUpgradeInitiator upgradeInitiator, IConnection& connection, ClientFramingDecoder decoder, IDefaultCommunicationTimeouts defaultTimeouts, TimeoutHelper& timeoutHelper)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.SendPreamble(IConnection connection, ArraySegment`1 preamble, TimeoutHelper& timeoutHelper)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.DuplexConnectionPoolHelper.AcceptPooledConnection(IConnection connection, TimeoutHelper& timeoutHelper)
at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at Microsoft.ServiceBus.RelayedOnewayTcpClient.RelayedOnewayChannel.Open(TimeSpan timeout)
at Microsoft.ServiceBus.RelayedOnewayTcpClient.GetChannel(Uri via, TimeSpan timeout)
at Microsoft.ServiceBus.RelayedOnewayTcpClient.ConnectRequestReplyContext.Send(Message message, TimeSpan timeout, IDuplexChannel& channel)
at Microsoft.ServiceBus.RelayedOnewayTcpListener.RelayedOnewayTcpListenerClient.Connect(TimeSpan timeout)
at Microsoft.ServiceBus.RelayedOnewayTcpClient.EnsureConnected(TimeSpan timeout)
at Microsoft.ServiceBus.Channels.CommunicationObject.Open(TimeSpan timeout)
at Microsoft.ServiceBus.Channels.RefcountedCommunicationObject.Open(TimeSpan timeout)
at Microsoft.ServiceBus.RelayedOnewayChannelListener.OnOpen(TimeSpan timeout)
at Microsoft.ServiceBus.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at Microsoft.ServiceBus.SocketConnectionTransportManager.OnOpen(TimeSpan timeout)
at Microsoft.ServiceBus.Channels.TransportManager.Open(TimeSpan timeout, TransportChannelListener channelListener)
at Microsoft.ServiceBus.Channels.TransportManagerContainer.Open(TimeSpan timeout, SelectTransportManagersCallback selectTransportManagerCallback)
at Microsoft.ServiceBus.SocketConnectionChannelListener`2.OnOpen(TimeSpan timeout)
at Microsoft.ServiceBus.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint.Enable()
at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint..ctor(BizTalkEndpointContext endpointContext, IBTTransportProxy transportProxy, ControlledTermination control)
at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiver`2.AddReceiveEndpoint(String url, IPropertyBag adapterConfig, IPropertyBag bizTalkConfig)".

Scott Densmore (@scottdensmore) announced a Fluent API for Windows Azure ACS Management in a 1/29/2012 post:

The guys at SouthWorks have been hard at work again. We have been using the same setup program for our Windows Azure Projects to configure ACS for a while now. It is a bit rough around the edges but we have not had time to invest a little upgrade. Well, leave it to these guys to go above and beyond: Jorge Rowies has create a fluent API for setting things up. Looks great. Check it out and start forking in GitHub.

Windows Azure VM Role, Virtual Network, Connect, RDP and CDN

No significant articles today.

Live Windows Azure Apps, APIs, Tools and Test Harnesses

MarketWire (@Marketwire) asserted “Further Investment in Microsoft's Cloud Platform Brings Greater Agility to Redknee's Cloud-Based Converged Billing Solutions” in an introduction to a Redknee Expands Cloud Offering With Windows Azure press release of 1/31/2012:

Redknee Solutions, Inc., a leading provider of business-critical billing, charging and customer care software and solutions for communications service providers, is pleased to announce that it is expanding its strategic alliance with Microsoft Corp. to support Redknee's strategy to deliver on-premise, private cloud, and public cloud-based converged billing solutions. The integration of Redknee's real-time converged billing platform to Windows Azure will reinforce the commitment by Redknee and Microsoft to develop solutions that deliver greater scalability and agility while reducing capital expenditure for communications service providers across the world.

Redknee's expanded cloud offering will enable service providers in competitive and fast growing markets to invest in a real-time converged billing and customer care solution based on the open and scalable Windows Azure public cloud. It will bring additional flexibility to Redknee's multi-tenant converged billing solution, which enables service providers to scale as required with a 'pay as your grow' model and minimizes risk. Redknee's cloud solution is supporting the growth of group operators, Tier 1 sub-brands, MVNOs, and MVNEs by enabling them to reduce CAPEX, standardize their billing operations, and launch into new markets quickly and effectively.

Lucas Skoczkowski, Redknee's CEO, commented: "Redknee's commitment to invest in Windows Azure reiterates our common goal with Microsoft to deliver solutions for communication service providers that empower them with greater business agility. This joint solution provides exceptional flexibility for service providers that lower the barriers to entry for next generation converged billing solutions by taking advantage of the benefits of the cloud. At Redknee, we continue to invest in developing cloud-based and on-premise monetization solutions to enable service providers to increase revenues, improve the customer experience and grow profitability."

The pre-integration of Windows Azure follows earlier investment by Redknee to pre-integrate its converged billing solution with Microsoft SQL Server 2012, formerly code-named "Denali," and Microsoft Dynamics CRM 2011.

Walid Abu-Hadba, Corporate Vice President of the Developer & Platform Evangelism Group at Microsoft, commented: "We are delighted to see Redknee launching a cloud strategy for the communications service provider community, with Windows Azure at its core. Microsoft's cloud platform delivers greater business agility to service providers, while significantly reducing capital expenditure so that they can focus on their core business. With this commitment to Windows Azure, Redknee advances its vision of creating a suite of agile, flexible solutions for communications service providers. Redknee's adoption of Windows Azure highlights the scalability, the reliability and the performance of the Microsoft cloud platform. Together we are ready to meet the most demanding needs of telecommunications service providers around the world."

For more information about Redknee and its solutions, please go to www.redknee.com .

About Redknee Redknee is a leading global provider of innovative communication software products, solutions and services. Redknee's award-winning solutions enable operators to monetize the value of each subscriber transaction while personalizing the subscriber experience to meet mainstream, niche and individual market segment requirements. Redknee's revenue generating solutions provide advanced converged billing, rating, charging and policy for voice, messaging and new generation data services to over 90 network operators in over 50 countries. Established in 1999, Redknee Solutions Inc. /quotes/zigman/38292 CA:RKN +2.00% is the parent of the wholly-owned operating subsidiary Redknee Inc. and its various subsidiaries. References to Redknee refer to the combined operations of those entities. For more information about Redknee and its solutions, please go to www.redknee.com .

Roope Astala explained “Cloud Numerics” Example: Statistics Operations to Azure Data in a 1/30/2012 post to the Codename “Cloud Numeric” blog:

This post demonstrates how to use Microsoft.Numerics C# API to perform statistical operations on data in Windows Azure blob storage. We go through the steps of loading data using IParallelReader interface, performing distributed statistics operations, and saving results to blob storage. As we sequence through the steps, we highlight the code samples from the application.

Note!

You will need to download and install the “Cloud Numerics” lab in order to run this example. To begin that process, click here.

Before You Run the Sample Application

Before you run the sample “Cloud Numerics” statistics application, complete the instructions in the “Cloud Numerics” Getting Started wiki post to:

Create a Windows Azure account (if you do not have one already).

Install “Cloud Numerics” on your local computer where you build and develop applications with Visual Studio.

Configure and deploy a cluster in Azure (only if you have not done so already).

Submit the sample C# “Cloud Numerics” program to Windows Azure as a test that your cluster is running properly.

Download the project file and source code for the sample “Cloud Numerics” statistics application.

Blob Locations and Sample Application Download

You can download the sample application from the Microsoft Connect site (connect.microsoft.com). If you have not already registered for the lab, you can do that here. Registering for the lab provides you access to the “Cloud Numerics” lab materials (installation package, reference documentation, and sample applications).

Note!

If you are signed into Microsoft Connect, and you have already registered for your invitation to the “Cloud Numerics” lab, you can access the various sample applications using this link.

For your convenience, we have staged sample datasets of pseudorandom numbers in Windows Azure Blob Storage. You can access the small and medium datasets at their respective links:

http://cloudnumericslab.blob.core.windows.net/smalldata

http://cloudnumericslab.blob.core.windows.net/mediumdata

These datasets are intended merely as examples to get you started. Also, feel free to customize the sample application code to suit your own datasets.

Choosing the Mode: Run on Local Development Machine or on Windows Azure

To run the application on your local workstation:

Set StatisticsCloudApplication as your StartUp project within Visual Studio. (From Solution Explorer in the Visual Studio IDE, right click the StatisticsCloudApplication subproject and select Set as Startup Project). Although, your application will run on your local workstation, the application will continue to use Windows Azure storage for data input and output.

Change Start Option paths for the project properties to reflect your local machine.
a. Right click the StatisticsCloudApplication subproject, and select Properties.
b. Click the Debug tab
c. In the Start Options section of the Debug tab, edit the following fields to reflect the paths on your local development machine:
- For the Command line arguments field, change:
c:\users\roastala\documents\visual studio 2010\Projects… to
c:\users\<YourUsername>\documents\visual studio 2010\Projects… - For the Working directory field, change:
c:\users\roastala\documents\… to
c:\users\<YourUsername>\documents\…

--Where c:\users\<YourUsername>\ reflects the home directory of the user on the local development machine where you installed the “Cloud Numerics” software.

To submit the application to Windows Azure (run on Windows Azure rather than locally):

Set AppConfigure as the StartUp project. (From Solution Explorer in the Visual Studio IDE, right click the AppConfigure subproject and select Set as Startup Project).

Note!

If you have already deployed your cluster or if it was pre-deployed by your site administrator, do not deploy it again. Instead, you only need to build the application and submit the main executable as a job.

Step 1: Supply Windows Azure Storage Account Information for Output

To build the application you must have a Windows Azure storage account for storing the output. Replace the string values “myAccountKey” and “myAccountName” with your own account key and name.
static string outputAccountKey = "myAccountKey";
static string outputAccountName = "myAccountName";
The application creates a public blob for the output under this storage account. See Step 4 for details.

Step 2: Read in Data from Blob Storage Using IParallelReader Interface

Let us take a look at code in AzureArrayReader.cs file.

The input array in this example is in Azure blob storage, where each blob contains a subset of columns of the full array. By using the Microsoft.Numerics.Distributed.IO.IParallelReader interface we can read the blobs in distributed fashion and concatenate the slabs of columns into a single large distributed array.

First, we implement the ComputeAssignment method, which assigns blobs to the MPI ranks of our distributed computation.
public object[] ComputeAssignment(int nranks)
{
    Object[] blobs = new Object[nranks];

    var blobClient = new CloudBlobClient(accountName);
    var matrixContainer = blobClient.GetContainerReference(containerName);
    var blobCount = matrixContainer.ListBlobs().Count();
    int maxBlobsPerRank = (int)Math.Ceiling((double)blobCount / (double)nranks);
    int currentBlob = 0;
    for (int i = 0; i < nranks; i++)
    {
        int step = Math.Max(0, Math.Min(maxBlobsPerRank, blobCount - currentBlob));
        blobs[i] = new int[] { currentBlob, step };
        currentBlob = currentBlob + step;
    }
    return (object[])blobs;
}
Next, we implement the property DistributedDimension, which in this case is initialized to 1 so that slabs will be concatenated along the column dimension.
public int DistributedDimension
{
    get { return 1; }
    set { }
}
The ReadWorker method:

Reads the blob metadata that describes the number of rows and columns in a given slab.

Checks that the slabs have an equal number of rows so they can be concatenated columnwise.

Reads the binary data from blobs.

Constructs a local NumericDenseArray.
public msnl.NumericDenseArray<double> ReadWorker(Object assignment)
{
    var blobClient = new CloudBlobClient(accountName);
    var matrixContainer = blobClient.GetContainerReference(containerName);
    int[] blobs = (int[])assignment;
    long i, j, k;
    msnl.NumericDenseArray<double> outArray;
    var firstBlob = matrixContainer.GetBlockBlobReference("slab0");
    firstBlob.FetchAttributes();
    long rows = Convert.ToInt64(firstBlob.Metadata["dimension0"]);
    long[] columnsPerSlab = new long[blobs[1]];
    if (blobs[1] > 0)
    {
        // Get blob metadata, validate that each piece has equal number of rows
        for (i = 0; i < blobs[1]; i++)
        {
            var matrixBlob = matrixContainer.GetBlockBlobReference("slab" + (blobs[0] + i).ToString());
            matrixBlob.FetchAttributes();
            if (Convert.ToInt64(matrixBlob.Metadata["dimension0"]) != rows)
            {
                throw new System.IO.InvalidDataException("Invalid slab shape");
            }
            columnsPerSlab[i] = Convert.ToInt64(matrixBlob.Metadata["dimension1"]);
        }

        // Construct output array
        outArray = msnl.NumericDenseArrayFactory.Create<double>(new long[] { rows, columnsPerSlab.Sum() });

        // Read data
        long columnCounter = 0;
        for (i = 0; i < blobs[1]; i++)
        {
            var matrixBlob = matrixContainer.GetBlobReference("slab" + (blobs[0] + i).ToString());
            var blobData = matrixBlob.DownloadByteArray();
            for (j = 0; j < columnsPerSlab[i]; j++)
            {
                for (k = 0; k < rows; k++)
                {
                    outArray[k, columnCounter] = BitConverter.ToDouble(blobData, (int)(j * rows + k) * 8);
                }
                columnCounter = columnCounter + 1;
            }
        }
    }
    else
    {
        // If a rank was assigned zero blobs, return empty array
        outArray = msnl.NumericDenseArrayFactory.Create<double>(new long[] { rows, 0 });
    }
    return outArray;
}
When an instance of reader is invoked by the Microsoft.Numerics.Distributed.IO.Loader.LoadData method, the ReadWorker instances are executed in parallel on each rank, and the LoadData method automatically takes care of concatenating the local pieces produced by the ReadWorkers.

Step 3: Compute Statistics Operations on Distributed Data

The source code in the Statistics.cs file implements the statistics operations performed on distributed data.

The sample data is stored at:
static string inputAccountName = @"http://cloudnumericslab.blob.core.windows.net";
This is a storage account for our data. It contains the samples of random numbers in publicly readable containers named “smalldata” and “mediumdata.”

In the beginning of the main entry point of the application, we initialize the Microsoft.Numerics distributed runtime. This allows us to execute distributed operations by calling Microsoft.Numerics library methods.
Microsoft.Numerics.NumericsRuntime.Initialize();
Next, we instantiate the array reader described earlier, and read data from blob storage.
var dataReader = new AzureArrayReader.AzureArrayReader(inputAccountName, arraySize);

var x = msnd.IO.Loader.LoadData<double>(dataReader);
The output x is a columnwise distributed array loaded with the sample data. We then compute the statistics of the data: min, max, mean, median and percentiles, and write the results to an output string.
// Compute summary statistics: max, min, mean, median
output.AppendLine("Summary statistics\n");
var xMin = ArrayMath.Min(x);
output.AppendLine("Minimum, " + xMin);
var xMax = ArrayMath.Max(x);
output.AppendLine("Maximum, " + xMax);
var xMean = Descriptive.Mean(x);
output.AppendLine("Mean, " + xMean);
var xMedian = Descriptive.Median(x);
output.AppendLine("Median, " + xMedian);

// Compute 10% quantiles
var tenPercentQuantiles = Descriptive.QuantilesExclusive(x, 10, 0).ToLocalArray();
As x is a distributed array, the overloaded variant of the method (QuantilesExclusive) that distributes processing over nodes of the Azure cluster is used. Note that the result of the quantiles operation is a distributed array. We copy it to a local array in order to write the result to an output string.

Step 4: Write Results to Blob Storage as a .csv File

The application, by default, writes the result to the file system of the virtual cluster. This storage is not permanent; the file will be removed when you delete the cluster. The application creates a public blob on the named Azure account you supplied in the beginning of the application.
// Write output to blob storage
var storageAccountCredential = new StorageCredentialsAccountAndKey(outputAccountName, outputAccountKey);
var storageAccount = new CloudStorageAccount(storageAccountCredential, true);
var blobClient = storageAccount.CreateCloudBlobClient();
var resultContainer = blobClient.GetContainerReference(outputContainerName);
resultContainer.CreateIfNotExist();
var resultBlob = resultContainer.GetBlobReference(outputBlobName);

// Make result blob publicly readable,
// so it can be accessed using URI
// https://<accountName>.blob.core.windows.net/statisticsresult/statisticsresult
var resultPermissions = new BlobContainerPermissions();
resultPermissions.PublicAccess = BlobContainerPublicAccessType.Blob;
resultContainer.SetPermissions(resultPermissions);

resultBlob.UploadText(output.ToString());
You can then view and download the results by using a web browser to open the blob. For example, the syntax for the URI would be:

https://<accountName>.blob.core.windows.net/statisticsresult/statisticsresult

--Where <accountName> is the name of the cluster account you deployed to Windows Azure.

For more background on Codename “Cloud Numerics,” see my Introducing Microsoft Codename “Cloud Numerics” from SQL Azure Labs and Deploying “Cloud Numerics” Sample Applications to Windows Azure HPC Clusters posts of 1/28/2012.

Scott Densmore (@scottdensmore) described Creating a SSL Certificate for the Cloud Ready Packages for the iOS Windows Azure Toolkit

With the iOS Windows Azure Toolkit, you can use the ready made Windows Azure Packages that use ACS or Membership to manage users to Windows Azure Storage. These packages require you to have a SSL certificate. More than likely you are running on a Mac.

This is pretty straight forward. You will need to create a certificate and then create a PKCS12 (.pfx) file for it. You can generate the certificate from terminal using openssl with the following steps:

open a terminal window

enter the command: openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout iOSWAToolkit.pem -out iOSWAToolkit.pem

iOSWAToolkit is the name of the file you want and can be anything you like.

This will create a certificate that you can use to create the PKCS12 certificate. This command will ask for all the information for your cert that you enter.

enter the command: openssl pkcs12 -export -out iOSWAToolkit.pfx -in iOSWAToolkit.pem -name "iOSWAToolkit"

iOSWAToolkit is the name of the file you want and can be anything you like.

This will ask you for a password that you need to remember so you can enter it when uploading to Windows Azure.

enter the command: openssl x509 -outform der -in iOSWAToolkit.pem -out iOSWAToolkit.cer

This will export a certificate with the public key which is what you will need for the service config file.

You will have two certificates that you will need to get the Cloud Ready Package(s) deployed. This is really easy when you use the Cloud Configuration Utility.

Visual Studio LightSwitch and Entity Framework 4.1+

Beth Massi (@bethmassi) described Calling Web Services to Validate Data in Visual Studio LightSwitch in a 1/30/2012 post:

Very often in business applications we need to validate data through another service. I’m not talking about validating the format of data entered – this is very simple to do in LightSwitch -- I’m talking about validating the meaning of the data. For instance, you may need to validate not just the format of an email address (which LightSwitch handles automatically for you) but you also want to verify that the email address is real. Another common example is physical Address validation in order to make sure that a postal address is real before you send packages to it.

In this post I’m going to show you how you can call web services when validating LightSwitch data. I’m going to use the Address Book sample and implement an Address validator that calls a service to verify the data.

Where Do We Call the Service?

In Visual Studio LightSwitch there are a few places where you can place code to validate entities. There are Property_Validate methods and there are Entity_Validate methods. Property_Validate methods run first on the client and then on the server and are good for checking the format of data entered, doing any comparisons to other properties, or manipulating the data based on conditions stored in the entity itself or its related entities. Usually you want to put your validation code here so that users get immediate feedback of any errors before the data is submitted to the server. These methods are contained on the entity classes themselves. (For more detailed information on the LightSwitch Validation Framework see: Overview of Data Validation in LightSwitch Applications)

The Entity_Validate methods only run on the server and are contained in the ApplicationDataService class. This is the perfect place to call an external validation service because it avoids having clients calling external services directly -- instead the LightSwitch middle-tier makes the call. This gives you finer control over your network traffic. Client applications may only be allowed to connect to your intranet internally but you can allow external traffic to the server managing the external connection in one place.

Calling Web Services

There are a lot of services out there for validating all sorts of data and each service has a different set of requirements. Typically I prefer REST-ful services so that you can make a simple http request (GET) and get some data back. However, you can also add service references like ASMX and WCF services as well. It’s all going to depend on the service you use so you’ll need to refer to their specific documentation.

To add a service reference to a LightSwitch application, first flip to File View in the Solution Explorer, right-click on the Server project and then select Add Service Reference…

Enter the service URL and the service proxy classes will be generated for you. You can then call these from server code you write on the ApplicationDataService just like you would in any other application that has a service reference. In the case of calling REST-ful services that return XML feeds, you can simply construct the URL to call and examine the results. Let’s see how to do that.

Address Book Example

In this sample we have an Address table where we want to validate the physical address when the data is saved. There are a few address validator services out there to choose from that I could find, but for this example I chose to sign up for a free trial of an address validation service from ServiceObjects. They’ve got some nice, simple APIs and support REST web requests. Once you sign up they give you a License Key that you need to pass into the service.

A sample request looks like this:

http://trial.serviceobjects.com/av/AddressValidate.asmx/ValidateAddress?Address=One+Microsoft+Way&Address2=&City=Redmond&State=WA&PostalCode=98052&LicenseKey=12345

Which gives you back the result:
<?xml version="1.0" encoding="UTF-8"?>
<Address xmlns="http://www.serviceobjects.com/"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Address>1 Microsoft Way</Address>
    <City>Redmond</City>
    <State>WA</State>
    <Zip>98052-8300</Zip>
    <Address2/>
    <BarcodeDigits>980528300997</BarcodeDigits>
    <CarrierRoute>C012</CarrierRoute>
    <CongressCode>08</CongressCode>
    <CountyCode>033</CountyCode>
    <CountyName>King</CountyName>
    <Fragment/>
  </Address>
If you enter a bogus address or forget to specify the City+State or PostalCode then you will get an error result:
<?xml version="1.0" encoding="UTF-8"?>
<Address xmlns="http://www.serviceobjects.com/"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Error>
    <Desc>Please input either zip code or both city and state.</Desc>
    <Number>2</Number>
    <Location/>
  </Error>
</Address>
So in order to interact with this service we’ll first need to add some assembly references to the Server project. Right-click on the Server project (like shown above) and select “Add Reference” and import System.Web and System.Xml.Linq.

Next, flip back to Logical View and open the Address entity in the Data Designer. Drop down the Write Code button to access the Addresses_Validate method. (You could also just open the Server\UserCode\ApplicationDataService code file if you are in File View).

First we need to import some namespaces as well as the default XML namespace that is returned in the response. (For more information on XML in Visual Basic please see: Overview of LINQ to XML in Visual Basic and articles here on my blog.) Then we can construct the URL based on the entity’s Address properties and query the result XML for either errors or the corrected address. If we find an error, we tell LightSwitch to display the validation result to the user on the screen.
Imports System.Xml.Linq
Imports System.Web.HttpUtility
Imports <xmlns="http://www.serviceobjects.com/">

Namespace LightSwitchApplication
  Public Class ApplicationDataService

    Private Sub Addresses_Validate(entity As Address, results As EntitySetValidationResultsBuilder)
      Dim isValid = False
      Dim errorDesc = ""

      'Construct the URL to call the web service
      Dim url = String.Format("http://trial.serviceobjects.com/av/AddressValidate.asmx/ValidateAddress?" &
                              "Address={0}&Address2={1}&City={2}&State={3}&PostalCode={4}&LicenseKey={5}",
                              UrlEncode(entity.Address1),
                              UrlEncode(entity.Address2),
                              UrlEncode(entity.City),
                              UrlEncode(entity.State),
                              UrlEncode(entity.ZIP),
                              "12345")

      Try
          'Call the service and load the XML result 
          Dim addressData = XElement.Load(url)

          'Check for errors first
          Dim err = addressData...<Error>
          If err.Any Then
              errorDesc = err.<Desc>.Value
          Else
              'Fill in corrected address values returned from service
              entity.Address1 = addressData.<Address>.Value
              entity.Address2 = addressData.<Address2>.Value
              entity.City = addressData.<City>.Value
              entity.State = addressData.<State>.Value
              entity.ZIP = addressData.<Zip>.Value
              isValid = True
          End If

      Catch ex As Exception
          Trace.TraceError(ex)
      End Try

      If Not (isValid) Then
          results.AddEntityError("This is not a valid US address. " & errorDesc)
      End If
    End Sub
  End Class
End Namespace
Run it!

Now that I’ve got this code implemented let’s enter some addresses on our contact screen. Here I’ve entered three addresses, the first two are legal and the last one is not. Also notice that I’ve only specified partial addresses.

If I try to save this screen, an error will be returned from the service on the last row. LightSwitch won’t let us save until the address is fixed.

If I delete the bogus address and save again, you will see that the other addresses were verified and all the fields are updated with complete address information.

Wrap Up

I hope this gives you a good idea on how to implement web service calls into the LightSwitch validation pipeline. Even though each service you use will have different requirements on how to call them and what they return, the LightSwitch validation pipeline gives you the necessary hooks to implement complex entity validation easily.

Jan Van der Haegen (@janvanderhaegen) asked Dude, where’s my VB.Net code? on 1/30/2012:

It came to my attention that a large part of the LightSwitch community prefers VB.NET over C#. I always thought that C# and VB.NET are close enough together in semantics that any VB.NET developer can read C# and vica versa. However, I must admit that I looked at the VB.NET version of my own samples, and once there’s generics & optional parameters involved, the VB.NET version becomes significantly different compared to its C# brother. To overcome this language barrier, thank god there’s tools online like “Google Translate”, which for .NET code, is provided to us for free by Telerik.

A special thanks to Telerik, Todd Anglin in particular, for providing us with this awesome translating power!

I’ll promise to do my best to post my samples in both C# and VB.Net in the future.

Return to section navigation list>

Windows Azure Infrastructure and DevOps

InformationWeek::Reports posted Wall Street & Technology: February 2012 on 1/31/2012:

Download

EMBRACING CLOUD: Facing declining revenues and increasing pressure on capital expenditures, many Wall Street organizations are warming up to semi-public cloud offerings.

Table of Contents

Cloud Watching: As Wall Street continues its wild ride, capital markets firms are looking to cut spending in order to prop up the bottom line. Targeting the enormous costs of building and maintaining data centers and server farms, more and more Wall Street organizations are looking to outsource parts of their infrastructures to the cloud. And increasingly, financial services firms are warming up to private clouds run outside their firewalls.

What Went Wrong? Breaking Down Thomson Reuters Eikon: Tabbed as the future of the company, Thomson Reuters' next-generation market data platform, Eikon, has been off to a sluggish start, and several executives have paid the price. But Philip Brittan, the man in charge of the Eikon group, says the company's commitment to the product has never waned.

Anatomy of A Data Center (Photo Gallery): DFT's NJ1 facility in Piscataway, N.J., comprises 360,000 square feet of LEEDcertified, energy-efficient data center space. Home to a growing number of financial industry clients, including the NYSE, the data center is designed to lower costs to tenants. WS&T takes you along for a behind-the-scenes look at the facility.

Securing Advent's Legacy: Founder Stephanie DiMarco will step down at the end of June as Advent's CEO. In an exclusive interview, she explains why she's stepping down, reflects on the company's impact on the investment management industry over the past 30 years and discusses her plans for the future.

PLUS:
Wall Street IT Pros In Strong Demand
How to Measure IT Productivity
Larry Tabb's Suggestions for Fixing the Futures Market
The Cloud's Time Is Now
And more ...

Lori MacVittie (@lmacvittie) asserted While web applications aren’t sensitive to jitter, business processes are in an introduction to her Performance in the Cloud: Business Jitter is Bad post of 1/30/2012 to F5’s DevCentral blog:

One of the benefits of web applications is that they are generally transported via TCP, which is a connection-oriented protocol designed to assure delivery. TCP has a variety of native mechanisms through which delivery issues can be addressed – from window sizes to selective acks to idle time specification to ramp up parameters. All these technical knobs and buttons serve as a way for operators and administrators to tweak the protocol, often at run time, to ensure the exchange of requests and responses upon which web applications rely. This is unlike UDP, which is more of a “fire and forget” protocol in which the server doesn’t really care if you receive the data or not.

Now, voice and streaming video and audio over the web has always leveraged UDP and thus it has always been highly sensitive to jitter. Jitter is, without getting into layer one (physical) jargon, an undesirable delay in the otherwise consistent delivery of packets. It causes the delay of and sometimes outright loss of packets that are experienced by users as pauses, skips, or jumps in multi-media content.

While the same root causes of delay – network congestion, routing changes, time out intervals – have an impact on TCP, it generally only delays the communication and other than an uncomfortable wait for the user, does not negatively impact the content itself. The content is eventually delivered because TCP guarantees that, UDP does not.

However, this does not mean that there are no negative impacts (other than trying the patience of users) from the performance issues that may plague web applications and particularly those that are more and more often out there, in the nebulous “cloud”. Delays are effectively business jitter and have a real impact on the ability of the business to perform its critical functions – and that includes generating revenue.

BUSINESS JITTER and the CLOUD

David Linthicum summed up the issue with performance of cloud-based applications well and actually used the terminology “jitter” to describe the unpredictable pattern of delay:

Are cloud services slow? Or fast? Both, it turns out -- and that reality could cause unexpected problems if you rely on public clouds for part of your IT services and infrastructure.

When I log performance on cloud-based processes -- some that are I/O intensive, some that are not -- I get results that vary randomly throughout the day. In fact, they appear to have the pattern of a very jittery process. Clearly, the program or system is struggling to obtain virtual resources that, in turn, struggle to obtain physical resources. Also, I suspect this "jitter" is not at all random, but based on the number of other processes or users sharing the same resources at that time.

-- David Linthicum, “Face the facts: Cloud performance isn't always stable”

But what the multitude of articles coming out over the past year or so with respect to performance of cloud services has largely ignored is the very real and often measurable impact on business processes. That jitter that occurs at the protocol and application layers trickles up to become jitter in the business process; a process that may be critical to servicing customers (and thus impacts satisfaction and brand) as well as on the bottom line. Unhappy customers forced to wait for “slow computers”, as it is so often called by the technically less adept customer service representatives employed by many organizations, may take to the social media airwaves to express displeasure, or cancel an order, or simply refuse to do business in the future with the organization based on delays experienced because of unpredictable cloud performance.

Business jitter can also manifest as decreased business productivity measures, which it turns out can be measured mathematically if you put your mind to it.

Understanding the variability of cloud performance is important for two reasons:

You need to understand the impact on the business and quantify it before embarking on any cloud initiative so it can be factored in to the overall cost-benefit analysis. It may be that the cost savings from public cloud are much greater than the potential loss of revenue and/or productivity, and thus the benefits of a cloud-based solution outweigh the risks.

Understanding the variability and from where it comes will have an impact and help guide you to choosing not only the right provider, but the right solutions that may be able to normalize or mitigate the variability. If the primary source of business jitter is your WAN, for example, then it may be that choosing a provider that supports your ability to deploy WAN optimization solutions would be an appropriate strategy. Similarly, if the variability in performance stems from capacity issues, then choosing a provider that allows greater latitude in load balancing algorithms or the deployment of a virtual (soft) ADC would likely be the best strategy.

It seems clear from testing and empirical (as well as anecdotal) evidence that cloud performance is highly variable and, as David puts it, unstable. This should not necessarily be seen as a deterrent to adopting cloud services – unless your business is so highly sensitive to latency that even milliseconds can be financially damaging – but rather it should be a reality that factors into your decision making process with respect to your choice of provider and the architecture of the solution you’ll be deploying (or subscribing to, in the case of SaaS) in the cloud.

Knowing is half the battle to leveraging cloud successfully. The other half is strategy and architecture.

I’ll be at CloudConnect 2012 and we’ll discuss the subject of cloud and performance a whole lot more at the show!

Sessions

Abel B. Cruz reported Cloud Computing Fueling Global Economic Growth: London School of Economics study on 1/30/2012:

The development of cloud computing will promote economic growth, increase productivity and shift the type of jobs and skills required by businesses, according to a new study by the London School of Economics and Political Science.

http://www.forbes.com/sites/joemckendrick/2012/01/27/cloud-computing-fueling-global-economic-growth-london-school-of-economics-study/

Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds

David Linthicum (@DavidLinthicum) asserted “Amazon Storage Gateway is a prime example of using an old-fashioned data center approach in the cloud -- which seems to miss the point” in a deck for his Cloud appliances: The cloud that isn't the cloud post of 1/31/2012 to InfoWorld’s Cloud Computing blog:

Last week Amazon.com let it be known that it has launched a public beta test of Amazon Storage Gateway -- my colleague Matt Prigge has done an in-depth, hands-on look of it. This software appliance stores data on local hardware and uploads backup instances to Amazon Web Services' S3 (Simple Storage Service). The idea is to provide low-latency access to local data while keeping snapshots of that data in the AWS cloud.

Cloud appliances are nothing new -- we've been using them for years to maintain software control within the firewall. Their applications include caching data as it moved to and from the cloud to enhance performance, local storage that is replicated to the cloud (as is the case with AWS gateway), and providing protocol mediation services to systems that are not port 80-compliant.

However, the larger question here is obvious: If cloud computing is really around eliminating the cost of local hardware and software, why are cloud computing providers selling hardware and software?

There are practical reasons for using cloud appliances, such as those I just cited. However, the larger advantage may be more perceptual than technical. IT managers want their data in some box in their data center that they can see and touch. They also want to tell people that they are moving to cloud computing. The cloud appliance provides the best of both worlds, and I suspect that we'll see other IaaS providers follow if Amazon.com is successful.

However, I can't help thinking that we're just trading an internal path to complexity and increasing costs for a cloud path to complexity and increasing costs. That's antithetical to the cloud's notions of reducing costs through efficiency and scale and of reducing complexity through abstraction. But yet here we are.

The Windows Server and Cloud Platform Team reported System Center Cloud Services Management Pack RC Now Available! on 1/30/2012:

Check out Travis Wright’s blog and learn more about the System Center Cloud Services Management Pack Release Candidate that is now available for download.

The System Center Cloud Services Process Pack offers a self-service experience to facilitate private cloud capacity requests from your business unit IT application owners and end users, including the flexibility to request additional capacity as business demands increase. Download the Management Pack today!

Cloud Security and Governance

Chris Hoff (@Beaker) posted Building/Bolting Security In/On – A Pox On the Audit Paradox! on 1/31/2012:

My friend and skilled raconteur Chris Swan (@cpswan) wrote an excellent piece a few days ago titled “Building security in – the audit paradox.”

This thoughtful piece was constructed in order to point out the challenges involved in providing auditability, visibility, and transparency in service — specifically cloud computing — in which the notion of building in or bolting on security is debated.

I think this is timely. I have thought about this a couple of times with one piece aligned heavily with Chris’ thoughts:

Revisiting virtualization and cloud stack security — Back to the future (baked in or bolted on)

Chris’ discussion really contrasted the delivery/deployment models against the availability and operationalization of controls:

If we’re building security in, then how do we audit the controls?

Will platform as a service (PaaS) give us a way to build security in such that it can be evaluated independently of the custom code running on it?

Further, as part of some good examples, he points out the notion that with separation of duties, the ability to apply “defense in depth” (hate that term,) and the ability to respond to new threats, the “bolt-on” approach is useful — if not siloed:

There lies the issue – bolt on security is easy to audit. There’s a separate thing, with a separate bit of config (administered by a separate bunch of people) that stands alone from the application code.

…versus building secure applications:

Code security is hard. We know that from the constant stream of vulnerabilities that get found in the tools we use every day. Auditing that specific controls implemented in code are present and effective is a big problem, and that is why I think we’re still seeing so much bolting on rather than building in.

I don’t disagree with this at all. Code security is hard. People look for gap-fillers. The notion that Chris finds limited options for bolting security on versus integrating security (building it in) programmatically as part of the security development lifecycle leaves me a bit puzzled.

This identifies both the skills and cultural gap between where we are with security and how cloud changes our process, technology, and operational approaches but there are many options we should discuss.

Thus what was interesting (read: I disagree with) is what came next wherein Chris maintained that one “can’t bolt on in the cloud”:

One of the challenges that cloud services present is an inability to bolt on extra functionality, including security, beyond that offered by the service provider. Amazon, Google etc. aren’t going to let me or you show up to their data centre and install an XML gateway, so if I want something like schema validation then I’m obliged to build it in rather than bolt it on, and I must confront the audit issue that goes with that.

While it’s true that CSP’s may not enable/allow you to show up to their DC and “…install and XML gateway,” they are pushing the security deployment model toward the virtual networking hooks, the guest based approach within the VMs and leveraging both the security and service models of cloud itself to solve these challenges.

I allude to this below, but as an example, there are now cloud services which can sit “in-line” or in conjunction with your cloud application deployments and deliver security as a service…application, information (and even XML) security as a service are here today and ramping!

While immature and emerging in some areas, I offer the following suggestions that the “bolt-on” approach is very much alive and kicking. Given that the “code security” is hard, this means that the cloud providers harden/secure their platforms, but the app stacks that get deployed by the customers…that’s the customers’ concerns and here are some options:

Introspection APIs (VMsafe)

Security as a Service (Cloudflare, Dome9, CloudPassage)

Auditing frameworks (CloudAudit, STAR, etc)

Virtual networking overlays & virtual appliances (vGW, VSG, Embrane)

Software defined networking (Nicira, BigSwitch, etc.)

Yes, some of them are platform specific and I think Chris was mostly speaking about “Public Cloud,” but “bolt-on” options are most certainly available an are aggressively evolving.

I totally agree that from the PaaS/SaaS perspective, we are poised for many wins that can eliminate entire classes of vulnerabilities as the platforms themselves enforce better security hygiene and assurance BUILT IN. This is just as emerging as the BOLT ON solutions I listed above.

In a prior post “Silent Lucidity: IaaS – Already a Dinosaur. Rise of PaaSasarus Rex”

As I mention in my Cloudifornication presentation, I think that from a security perspective, PaaS offers the potential of eliminating entire classes of vulnerabilities in the application development lifecycle by enforcing sanitary programmatic practices across the derivate works built upon them. I look forward also to APIs and standards that allow for consistency across providers. I think PaaS has the greatest potential to deliver this.

There are clearly trade-offs here, but as we start to move toward the two key differentiators (at least for public clouds) — management and security — I think the value of PaaS will really start to shine.

My opinion is that given the wide model of integration between various delivery and deployment models, we’re gonna need both for quite some time.

Back to Chris’ original point, the notion that auditors will in any way be able to easily audit code-based (built-in) security at the APPLICATION layer or the PLATFORM layer versus the bolt-on layer is really at the whim on the skillset of the auditors themselves and the checklists they use which call out how one is audited:

Infrastructure as a service shows us that this can be done e.g. the AWS firewall is very straightforward to configure and audit (without needing to reveal any details of how it’s actually implemented). What can we do with PaaS, and how quickly?

This is a very simplistic example (more infrastructure versus applistructure perspective) but represents the very interesting battleground we’ll be entrenched in for years to come.

In the related posts below, you’ll see I’ve written a bunch about this and am working toward ensuring that as really smart folks work to build it in, the ecosystem is encouraged to provide bolt-ons to fill those gaps.

Cloud Computing Events

Adron Hall (@adron) reported on 1/30/2012 Aaron Stannard Presenting “Building Node.JS Applications on Windows Azure” @ #NodePDX:

Aaron Stannard

Aaron is coming at us from southern California, but don’t hold that against him. He’s going to shed some light on the Windows Azure support around Node.js. Yup, you read that right, Microsoft is all googly over open source these days. Aaron describes his presentation thus:

Microsoft is moving towards making Windows Azure a PaaS cloud capable of hosting applications of any shape, size, framework, and programming language, and Node.JS is one of the first non-.NET technologies we’ve made a first-class citizen on Windows Azure.

In this talk you’ll get a brief overview of Windows Azure itself, iisnode, and what the dev / test / production lifecycle looks like for Node.JS applications built for Windows Azure.

If you’d like to come and check out this presentation and the other kick ass presentations lined up, get involved in some coding, hear what Node.js is all about, or just hang out please RSVP and get the event on your calendar! Besides, what better reason to come visit the amazing city of Portland, Oregon than to come hack some node.js and chill for the weekend!

Jo Maitland (@JoMaitlandSF) asserted There is more to Node.js than buzz in a 1/27/2012 post to the GigaOm Pro Blog (trial or paid subscription required):

Judging by the attendance and conversation at this week’s Node Summit in San Francisco, the relatively new open-source programming language Node.js is catching on among developers and notably among enterprise developers.

Several hundred people showed up at the event, most of them developers from startups around Silicon Valley as well as business and architect leads from Microsoft, eBay, Wal-Mart, Sprint, Google, Rackspace, VMware and the usual VC crowd. Everyone onstage had enormous enthusiasm for Node and their increasing use of the technology.

The summit was organized by Joyent, the company that invented Node.js and is shepherding the open-source community around it. Joyent’s SmartOS and SmartDataCenter products run on Node.js and are the backbone of its public cloud, JoyentCloud. Joyent also licenses the software to service providers to run their own public clouds.

…

Jo goes on to describe Node.js, list users (such as Twitter, LinkedIn and eBay), and concludes:

The most interesting thing to me about Node is Joyent’s decision to keep it open and cross platform. Microsoft, for example, is porting Node to Windows Azure so that its developer base can use it. And with 40 percent of the developer market working on Windows, it is a smart way to ensure the technology reaches the mainstream. Amazon Web Services, take note!

Other Cloud Computing Platforms and Services

Chris Czarnecki posted Amazon Announces Trusted Advisor to the Learning Tree blog on 1/31/2012:

When considering using cloud computing services, one of the major factors is the level of support that can be provided by the cloud vendor. In particular, when using Infrastructure as a Service, it is vital to ensure that cloud resources are not only appropriately secured but are also running at their most cost effective. Who better to provide this information than the cloud provider.

In the case of Amazon Web Services, where the range of services is so wide (consider storage: SimpleDB, Simple Storage Service, Relational Data Service, Elastic Block Storage, DynamoDb) with many alternatives, it is not always easy to make the correct choice. When buying infrastructure, the purchase model is no longer straightforward either, with options for on-demand, reserved and spot instances. When deploying high availability systems, instances are typically spread across availability zones. Are these balanced and evenly distributed ?

To help better support deployed systems making use of Amazon infrastructure, Amazon have today announced AWS Trusted Advisor. This service, which uses the experience of supporting hundreds of thousands of Amazon customers, will inspect a customer’s AWS environment and, if appropriate, will make recommendations on improving security, performance and cost reduction. All of this is automated and makes use of a large knowledge base built up by Amazon over the last few years.

The Trusted Advisor is an example of the way in which Amazon continues to innovate in providing services that not only provide great new functionality, but also in the monitoring and control of these services. If you would like to know more about Amazon Web Services and how they can benefit your organisation, consider attending Learning Tree’s comprehensive course, Cloud Computing with Amazon Web Services™, in which you’ll learn from an expert instructor about the major services and how they can be integrated to form a coherent scalable reliable infrastructure.

Adalberto Foresti reported Beta of Amazon Web Services SDK for Windows Phone released on 1/30/2012:

I am pleased to announce the beta release of the Amazon Web Services (AWS) SDK for Windows Phone. Built by Microsoft as an open source project, this SDK provides developers with a speed dial that lets them quickly connect and integrate Windows Phone applications with AWS (S3, SimpleDB, and SQS Cloud Services)

To create cloud-connected mobile applications, developers want to have choice and be able to reuse their assets and skills. For developers familiar with AWS, whether they’ve been developing for Android, iOS or any other technology, this SDK will allow them to comfortably port their applications to the Windows Phone Platform.

Terry Wise, Director of Business Development for Amazon Web Services, welcomes the release of the Amazon Web Services SDK for Windows Phone to the Developer community.

“Our approach with AWS is to provide developers with choice and flexibility to build applications the way they want and give them unlimited storage, bandwidth and computing resources, while paying only for what they use. We welcome Windows Phone developers to the AWS community and look forward to providing customers with new ways to build and deploy Windows Phone applications,” he says.

Jean Paoli, General Manager of Interoperability Strategy at Microsoft, adds that Windows Phone was engineered from the get-go to be a Cloud-friendly phone.

“The release of the AWS SDK for Windows Phone Beta proves that Microsoft’s goal of building a Cloud-friendly phone is true across vendor boundaries. It literally takes minutes to create a Cloud-ready application in C# with this SDK. We look forward to this SDK eventually resulting in many more great apps in the rapidly growing Windows Phone marketplace,” he said.

Developers can download the SDK , along with the complete source code under the Apache license. A Getting Started guide can be found on the Windows Phone Interoperability Bridges site along with other resources.

And as always your feedback on how to improve this beta is welcome!

Jeff Barr (@jeffbarr) reported Amazon S3 Growth for 2011 - Now 762 Billion Objects in a 1/30/2012 post:

As of the end of 2011, there are 762 billion (762,000,000,000) objects in Amazon S3. We process over 500,000 requests per second for these objects at peak times.

Here's the annual growth chart:

This represents year-over-year growth of 192%; S3 grew faster last year than it did in any year since it launched in 2006.

Where are all of these objects coming from? Although we definitely made it easier for you to delete objects using Multi-Object Deletion and Object Expiration, we also gave you plenty of ways to upload new objects using Multipart upload, AWS Direct Connect, and AWS Import/Export.

Introducing Apache Hadoop Services for Windows Azure

2012-01-31T12:48:00.000-08:00

The SQL Server Team (@SQLServer) announced Apache Hadoop Services for Windows Azure, a.k.a. Apache Hadoop on Windows Azure or Hadooop on Azure, at the Profesional Association for SQL Server (PASS) Summit in October 2011.

Update 1/31/2011: Added steps 15 and 16 with Job History and Cluster Management for the TeraSort job.

Update 1/30/2011: Added TeraSort job, steps 10 through 14, and TeraValidate job, steps 17 through 19. Waiting for response regarding interpretation of TeraValidate results.

Table of Contents

Introduction
Tutorial: Running the 10GB GraySort Sample’s TeraGen Job
Tutorial: Running the 10GB GraySort Sample’s TeraSort Job
Tutorial: Running the 10GB GraySort Sample’s TeraValidate Job
Apache Hadoop on Windows Azure Resources

Introduction

Val Fontama’s Availability of Community Technology Preview (CTP) of Hadoop based Service on Windows Azure post of 12/14/2011 described Apache Hadoop on Windows Azure and how to obtain an invitation to the CTP:

In October at the PASS Summit 2011, Microsoft announced expanded investments in “Big Data”, including a new Apache Hadoop™ based distribution for Windows Server and service for Windows Azure. In doing so, we extended Microsoft’s leadership in BI and Data Warehousing, enabling our customers to glean and manage insights for any data, any size, anywhere. We delivered on our promise this past Monday, when we announced the release of the Community Technology Preview (CTP) of our Hadoop based service for Windows Azure.

Today this preview is available to an initial set of customers. Those interested in joining the preview may request to do so by filling out this survey. Microsoft will issue a code that will be used by the selected customers to access the Hadoop based Service. We look forward to making it available to the general public in early 2012. Customers will gain the following benefits from this preview:

Broader access to Hadoop through simplified deployment and programmability. Microsoft has simplified setup and deployment of Hadoop, making it possible to setup and configure Hadoop on Windows Azure in a few hours instead of days. Since the service is hosted on Windows Azure, customers only download a package that includes the Hive Add-in and Hive ODBC Driver. In addition, Microsoft has introduced new JavaScript libraries to make JavaScript a first class programming language in Hadoop. Through this library JavaScript programmers can easily write MapReduce programs in JavaScript, and run these jobs from simple web browsers. These improvements reduce the barrier to entry, by enabling customers to easily deploy and explore Hadoop on Windows.

Breakthrough insights through integration Microsoft Excel and BI tools. This preview ships with a new Hive Add-in for Excel that enables users to interact with data in Hadoop from Excel. With the Hive Add-in customers can issue Hive queries to pull and analyze unstructured data from Hadoop in the familiar Excel. Second, the preview includes a Hive ODBC Driver that integrates Hadoop with Microsoft BI tools. This driver enables customers to integrate and analyze unstructured data from Hadoop using award winning Microsoft BI tools such as PowerPivot and PowerView. As a result customers can gain insight on all their data, including unstructured data stored in Hadoop.

Elasticity, thanks to Windows Azure. This preview of the Hadoop based service runs on Windows Azure, offering an elastic and scalable platform for distributed storage and compute.

We look forward to your feedback! Learn more at www.microsoft.com/bigdata.

Val Fontama
Senior Product Manager
SQL Server Product Management

Mary Jo Foley (@maryjofoley) ranked Apache Hadoop on Windows Azure #9 of her 10 sexiest Microsoft business teases for 2012 on 12/29/2011:

…

9. AzureHadoop (or is it HadoopAzure?): Microsoft made available the preview bits for the Hadoop distribution for Windows Azure in December 2011. The final release is slated for March 2012. (Microsoft and partner Hortonworks are also working on an on-premises Hadoop on Windows Server distribution.) Hadoop on Windows Azure is interesting because it combines Microsoft’s big-data plans and products with its cloud platform. The idea Microsoft will be pushing in 2012 is that Hadoop on Azure will give users of Microsoft’s analytics tools, including plain-old Excel, a way to make use of the growing number of data sets stored on Windows Azure.

…

Tutorial: Running the 10GB GraySort Sample’s TeraGen Job

Following is a step-by-step tutorial for running the first process of the 10GB GraySort sample project:

1. After you receive your invitation code, navigate to https://www.hadooponazure.com/ and log-in with your Windows Live ID and invitation code to open the Account page with the Request a New Cluster content active. Type a globally unique DNS Name for your cluster, hadoop1 for this example, select a Cluster Size (Large for this example), and type a administrative Username, Password and password confirmation:

Click screen captures for a full-size (1024x768-px) image.

Note: There is no charge for Windows Azure resources used during the CTP, so you don’t need to provide a credit card to create your cluster.

2. When your cluster is provisioned, the Account page’s content changes to include tiles to create a new job as well as access your cluster by different methods:

Note: You must renew your cluster every three days. It’s clear that that faux-Metro UIs are de rigueur at Microsoft these days.

3. Click the Samples tile to open the Account/Samples page, which describes the currently available samples.

4. The GraySort MapReduce sample is a useful starting point because it runs in a reasonably short time (about 4 minutes with a Large cluster), so click the 10GB GraySort tile to open its Account/SampleDetails page, which describes the sample:

5. Click the Deploy to Your Cluster button to automatically populate text boxes with values for the TeraGen program, which generates 10 GB of data:

Note: If you have tried SQL Azure Labs’ Microsoft Codename “Data Numerics” CTP, you’ll notice that the process for creating the Hadoop cluster and executing the first MapReduce job is much more automated that that described in my Deploying “Cloud Numerics” Sample Applications to ... post of 1/28/2012 (updated 1/30/2012).

5. Click the Execute Job button to run the TeraGen program, which initially displays this Job Info page:

6. After a few seconds, the program begins adding lines of Debut Output for the 50 maps in increments close to 1 percent:

Note: Hadoop automatically repairs the failures reported above, but it’s surprising that lines for 78 and 79 percent are missing.

7. When processing completes, click the left-arrow button to return to the Account page with a tile for the TeraGen process added:

8. Click the Job History tile to display a summary of the preceding operation, which confirms successful completion with an Exit Code = Ok cell:

9. Click the left-arrow button to return to the main Accounts page and click the Manage Cluster tile to display total storage used (30 GB) and other data source options (Data Market, Windows Azure blob storage, and Amazon S3):

Tutorial: Running the 10GB GraySort Sample’s TeraSort Job

10. Return to the Account page, click the Samples tile to open the Account/Samples page (see step 3), click the 10GB GraySort tile to open its Account/SampleDetails page (see step 4), and click the Deploy to Your Cluster button to open the Create Job page.

11. Replace teragen with terasort in the Parameter 1 text box, add a space and =Dmapred.reduce.tasks=25 as a suffix to the existing Parameter 2 text box value, and change the Parameter 3 text box’s value to /example/data/10GB-sort-input /example/data/10GB-sort-out:

12. Click the top, middle, and bottom arrow symbols adjacent to the three text boxes to validate the three parameter values:

Note: If you receive Invalid instead of OK for any of the parameter values and you’ve verified the content is identical to the above, click the garbage can symbols next to the offending text boxes to remove them, click Add Parameter button three times to recreate them, retype the parameter values shown above, and click the validation arrows again.

The Final Command should read:

Hadoop jar hadoop-examples-0.20.203.1-SNAPSHOT.jar terasort "-Dmapred.map.tasks=50 -Dmapred.reduce.tasks=25" /example/data/10GB-sort-input /example/data/10GB-sort-out

13. Click the Execute Job button to start the sorting process:

14. About 45 minutes after you start the job, the StdOutput and StdError results appear:

Notice that Reduce operations don’t occur until Mapping is ~80% complete.

15. Click the Job History tile to display the summary for the TeraSort option:

16. Return to the Account page and click the Manage Cluster tile to determine the additional storage space used (10 GB) by the TeraSort operation:

Tutorial: Running the 10GB GraySort Sample’s TeraSort Job

17. Return to the Account page, click the Samples tile to open the Account/Samples page (see step 3), click the 10GB GraySort tile to open its Account/SampleDetails page (see step 4), and click the Deploy to Your Cluster button to open the Create Job page.

18. Delete the parameters and add three new empty parameters. Type teravalidate in the Parameter 1 text box, click the arrow to validate the parameter, type "-Dmapred.map.tasks=50 -Dmapred.reduce.tasks=25" in the Parameter 2 text box value and validate it, and type /example/data/10GB-sort-out /example/data/10GB-sort-validate in the Parameter 3 text box’s value and validate it:

The final command should read:

Hadoop jar hadoop-examples-0.20.203.1-SNAPSHOT.jar teravalidate "-Dmapred.map.tasks=50 -Dmapred.reduce.tasks=25" /example/data/10GB-sort-out /example/data/10GB-sort-validate

19. Click the Execute Job button to start the task:

20. After about five minutes, the task completes with the following (partial) Debug Output:

Note: It’s not clear from the debug output above how to determine the result of the validation task.

21. Return to the Account page and click the Job history tile to verify the outcome:

Note: Job #3’s failure was due to a mismatch in the input file name, as emphasized above.

Apache Hadoop on Windows Azure Resources

Download the Apache Hadoop-based Services for Windows Azure How-To and FAQ whitepaper in PDF or *.docx format.

Wesley McSwain posted a Apache Hadoop Based Services for Windows Azure How To Guide, which is similar (but not identical) to the above document, to the TechNet wiki on 12/13/2011. The latest update when this post was written was 1/18/2012. Here’s its content:

This content is a work in progress for the benefit of the Hadoop Community. Please feel free to contribute to this wiki page based on your expertise and experience with Hadoop.

If you have any questions, please use the groups DL http://tech.groups.yahoo.com/group/hadooponazurectp/

Table of Contents

How-Tos

FAQs

More Information

Blogs / Twitter to Follow

How-Tos

Setup your Hadoop cluster

Provision a temporary Hadoop cluster on Microsoft's Elastic Map Reduce Portal

Provision a new Hadoop cluster on your Windows Azure subscription.

Provision a new Hadoop Cluster on your on-premise hardware.

Running Sample Jobs

How to run Sample Pi Estimator Job

Running Sample WordCount Hadoop Job with a few twists

Running 10GB Sort Hadoop Job with Teragen, TeraSort and TeraValidate Options *

Writing your own Job and running on Cluster

Writing your very own WordCount Hadoop Job in Java and deploying to Windows Azure Cluster

Running a JavaScript Map/Reduce Job

Job Administration

Understanding MapReduce Job administration by running 10GB Sort Hadoop Job with TeraSort Option

Interactive Console:

Tasks with the Interactive JavaScript Console

How to create and run a JavaScript Map Reduce Job

Tasks with Hive on the Interactive Console

How to run Hive Queries from the Interactive Console

Remote Desktop

How to remote login to Hadoop Cluster

Using The Hadoop Command Shell

View the Job Tracker

View Hdfs

Connecting Windows Azure Blob Storage from Hadoop Cluster

Configuring Hadoop Cluster to connect with Azure Storage

Running a Hadoop job using Azure Storage as Input and Output paramteres

Open Ports

How to connect Excel to Hadoop on Azure via HiveODBC

How to FTP data to Hadoop on Azure

How to SFTP data to Hadoop on Azure

Manage Data

Import Data from Data Market

Setup ASV - Use your Windows Azure Blob Storage Account

Setup S3 - Use your Amazon S3 account

Apache Hadoop on Windows Azure:

Tips and Tricks to manage your Hadoop Cluster

Running Apache Pig (Pig Latin) at Apache Hadoop on Windows Azure

Scenarios

Querying a Web Log File via HiveQL

* Does not include details for TeraSort or TeraValidate options

FAQs

Frequently asked questions with Hadoop on Windows Azure

More Information

See Apache Hadoop On Windows.

Blogs / Twitter to Follow

Below are some blogs to follow on Hadoop on Azure [links added]

Alexander Stojanovic (Founder and [General Manager]) of Hadoop on Azure and Windows), @stojanovic, http://conceptualorigami.blogspot.com/

Dave Vronay, @davevr, http://dvronay.blogspot.com/2011/12/design-of-portal-for-hadooponazurecom.html

Brad Sarsfield, @bradoop

Denny Lee, @dennylee, http://dennyglee.com/

Avkash Chauhan, @avkashchauhan, http://blogs.msdn.com/b/avkashchauhan/

Technorati Tags: Windows Azure,Apache Hadoop on Windows Azure,Apache Hadoop,Hadoop,MapReduce,Cloud Computing,Big Data,Big Data Analytics,Windows Azure Marketplace DataMarketDataMarket,Windows Azure Blobs,Amazon S3