OakLeaf Systems

Windows Azure and Cloud Computing Posts for 2/1/2012+

2012-02-02T16:27:00.000-08:00

A compendium of Windows Azure, Service Bus, EAI & EDI Access Control, Connect, SQL Azure Database, and other cloud-computing articles.

• Updated 2/2/2012 with new articles marked •.

Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:

Windows Azure Blob, Drive, Table, Queue and Hadoop Services
SQL Azure Database, Federations and Reporting
Marketplace DataMarket, Social Analytics and OData
Windows Azure Access Control, Service Bus, and Workflow
Windows Azure VM Role, Virtual Network, Connect, RDP and CDN
Live Windows Azure Apps, APIs, Tools and Test Harnesses
Visual Studio LightSwitch and Entity Framework v4+
Windows Azure Infrastructure and DevOps
Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds
Cloud Security and Governance
Cloud Computing Events
Other Cloud Computing Platforms and Services

Azure Blob, Drive, Table, Queue and Hadoop Services

• Cory Fowler (@SyntaxC4) described getting the Windows Azure Storage Emulator Account Name and Key in a 2/1/2012 post:

It’s almost funny to say, but I’m working on a legacy Windows Azure Application in which the need of the Windows Azure Storage Account Name and Account Key need to be given instead of a Storage Account Connection String.

What’s the Difference? You need the Name & Key for a Connection String

While this is true in some scenarios, there is one distinct difference especially when it comes down to running in the local Storage Emulator. The following are considered Connection Strings:
<Setting name="DataConnectionString" value="usedevelopmentstorage=true" />
or
<appSettings>
     <add key="DataConnectionString" value="DefaultEndpointsProtocol=https;AccountName=myuniqueaccountname;AccountKey=superlongsecurestoragekey" />
</appSettings>
The second example of a connection string could be easily parsed to extract the AccountName and AccountKey, unlike the first example which is absolutely impossible to provide any valuable information. If you’re using the original Microsoft Sample code for Windows Azure Storage Session State Management which doesn’t use the convention of the connection string, you’d be sadly out of luck. This is what you’ll find in the Web.Config file:
<appSettings>
     <add key="AccountName" value="myuniqueaccountname"/>
     <add key="AccountKey" value="superlongsecurestoragekey"/>
</appSettings>
Oh No, We’re doomed right?

Not at all. Microsoft has provided the storage credentials for the local Storage Emulator on MSDN. Here’s a quick place to copy the default name and key:

AccountName
devstoreaccount1
AccountKey
Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw== 

Mariano Vazquez (@marianodvazquez) explained Browsing Blobs within a container using the Windows Azure Node SDK in a 1/30/2012 post to Southworks’ Node on Azure blog:

The last couple of weeks we were working on a new, exciting project: a collaborative, real-time markdown editor that runs on a NodeJS server, hosted on Windows Azure (you'll hear more about this soon). One of the features that this app will have is the ability to store the .markdown files in either your local disk or Azure Blob Storage. To achieve this, we investigated the best way to list the containers and blobs of a specific account, and navigate them like we were dealing with an hierarchical structure. We found out that this can be done, but is not as easy and it sounds (you have to use a combination of the prefix and delimiter options of the listBlobs operation, along with inspecting the BlobPrefix element returned in the Blobs REST API response).

In the next lines, we are going to explain how you can implement this functionality in your application. We start from showing the blobs in a simple, flat-list to demonstrate how to organize them in a more complex structure, like if you were navigating your local file system directories.

To access blob storage, we used the Windows Azure SDK for Node.js (you can install this module in your node application by doing npm install azure). Also, don't forget to install the Windows Azure SDK for Node.js to emulate the Azure environment locally.

Basic scenario

This is all the code you need to list the containers in your storage account, and show the blobs inside. Because all blobs are stored inside a container, you need to perform two separated calls: one to retrieve all containers in the account and other to retrieve the blobs inside a particular container.
var azure = require('azure');

...

function Home () {
    this.blobService = azure.createBlobService();
};

Home.prototype = {
    showContainers: function (req, res) {
        var self = this;
        self.blobService.listContainers(function(err, result){
            // some code here to show the results
        });
    },
    showBlobs: function(req, res){
        var self = this;
        var containerName = req.query['containerName'];
        if (!containerName)
            self.showContainers(req, res);
        else
            self.blobService.listBlobs(containerName, function(err, result){
                // some code here to show the results
            });
    },

    ...
};
Below is a sample result of what you may get. In this case, we listed only the blob names. Notice that the listBlobs operation is returning every blob within the container, using flat blob listing (more info about flat blob listing here).

There's nothing wrong with the code above, and it might be sufficient for you and your business needs (actually, it will work great if all your blobs are children of the container). But what happens if your containers have a considerable amount of blobs, organized in a logic way, and you want to retrieve them in a lazy manner? If that's the case you're facing, keep reading.

Using directories approach

You can filter the results of the listBlob operation by setting the prefix and delimiter parameters. The first one is used, as its name claims, to return only the blobs whose names begin with the value specified. The delimiter has two purposes: to skip from the result those blobs whose names contains the delimiter, and to include a BlobPrefix element in the REST API response body. This element will act as a placeholder for all blobs whose names begin with the same substring up to the appearance of the delimiter, and will be used to simulate a directory hierarchy (the folders will be listed there).
var azure  = require('azure');

...

function Home () {
    this.blobService = azure.createBlobService();
};

function getFiles(collection){
    var items = [];
    for(var key in collection){
        var item = collection[key];
        var itemName = item.name.split('/')[item.name.split('/').length - 1];
        items.push({ 'text': itemName, 'classes': 'file'});
    }
    return items;
}

function getFolders(containerName, collection){
    var items = [];
    //if BlobPrefix contains one folder is a simple JSON. Otherwise is an array of JSONs
    if (collection && !collection.length){
        temp = collection;
        collection = [];
        collection.push(temp);
    }
    for(var key in collection){
        var item = collection[key];
        var itemName = item.Name.replace(/\/$/, '').split('/')[item.Name.replace(/\/$/, '').split('/').length - 1];
        items.push({ 'text': itemName, 'classes': 'folder' });
    }
    return items;
}

Home.prototype = {

    ...

    listBlobs: function(containerName, prefix, delimiter, callback){
        var self = this;
        self.blobService.listBlobs(containerName,{ 'prefix': prefix, 'delimiter': delimiter}, function(err, result, resultCont, response){
            if(!err){
                var files = getFiles(result);
                var folders = getFolders(containerName, response.body.Blobs.BlobPrefix);
                var childs = folders.concat(files);
                // return the childs
            }
        });
    },

    ...
};
This is what we're doing in the lines above:

First, we parse the result to access the blobs returned (by calling getFiles).

To generate the folders, we parse the BlobPrefix element (located inside the Response Body).

Lastly, we join these two collections into one.

We created this sample to demonstrate how you can use all of this in a real-world scenario. It shows you how to correctly parse the JSON returned by the listBlobs operation, and how to use this information to populate a control in the View, like a jQuery Treeview Plugin.

Enjoy coding!

• Josh Wills (@josh_wills) described the “practice of data science” in a Seismic Data Science: Reflection Seismology and Hadoop post of 1/25/2012 to Cloudera’s Developer Center blog:

When most people first hear about data science, it’s usually in the context of how prominent web companies work with very large data sets in order to predict clickthrough rates, make personalized recommendations, or analyze UI experiments. The solutions to these problems require expertise with statistics and machine learning, and so there is a general perception that data science is intimately tied to these fields. However, in my conversations at academic conferences and with Cloudera customers, I have found that many kinds of scientists– such as astronomers, geneticists, and geophysicists– are working with very large data sets in order to build models that do not involve statistics or machine learning, and that these scientists encounter data challenges that would be familiar to data scientists at Facebook, Twitter, and LinkedIn.

The Practice of Data Science

The term “data science” has been subject to criticism on the grounds that it doesn’t mean anything, e.g., “What science doesn’t involve data?” or “Isn’t data science a rebranding of statistics?” The source of this criticism could be that data science is not a solitary discipline, but rather a set of techniques used by many scientists to solve problems across a wide array of scientific fields. As DJ Patil wrote in his excellent overview of building data science teams, the key trait of all data scientists is the understanding “that the heavy lifting of [data] cleanup and preparation isn’t something that gets in the way of solving the problem: it is the problem.”

I have found a few more characteristics that apply to the work of data scientists, regardless of their field of research:

Inverse problems. Not every data scientist is a statistician, but all data scientists are interested in extracting information about complex systems from observed data, and so we can say that data science is related to the study of inverse problems. Inverse problems arise in almost every branch of science, including medical imaging, remote sensing, and astronomy. We can also think of DNA sequencing as an inverse problem, in which the genome is the underlying model that we wish to reconstruct from a collection of observed DNA fragments. Real-world inverse problems are often ill-posed or ill-conditioned, which means that scientists need substantive expertise in the field in order to apply reasonable regularization conditions in order to solve the problem.

Data sets that have a rich set of relationships between observations. We might think of this as a kind of Metcalfe’s Law for data sets, where the value of a data set increases nonlinearly with each additional observation. For example, a single web page doesn’t have very much value, but 128 billion web pages can be used to build a search engine. A DNA fragment in isolation isn’t very useful, but millions of them can be combined to sequence a genome. A single adverse drug event could have any number of explanations, but millions of them can be processed to detect suspicious drug interactions. In each of these examples, the individual records have rich relationships that enhance the value of the data set as a whole.

Open-source software tools with an emphasis on data visualization. One indicator that a research area is full of data scientists is an active community of open source developers. The R Project is a widely known and used toolset that cuts across a variety of disciplines, and has even been used as a basis for specialized projects like Bioconductor. Astronomers have been using tools like AIPS for processing data from radio telescopes and IRAF for data from optical telescopes for more than 30 years. Bowtie is an open source project for performing very fast DNA sequence alignment, and the Crossbow Project combines Bowtie with Apache Hadoop for distributed sequence alignment processing.

We can use the term “data scientist” as a specialization of “scientist” in the same way that we use the term “theoretical physicist” as a specialization of “physicist.” Just as there are theoretical physicists that work within the various subdomains of physics, such as cosmology, optics, or particle physics, there are data scientists at work within every branch of science.

Data Scientists Who Find Oil: Reflection Seismology

Reflection seismology is a set of techniques for solving a classic inverse problem: given a collection of seismograms and associated metadata, generate an image of the subsurface of the Earth that generated those seismograms. These techniques are primarily used by exploration and production companies in order to locate oil and natural gas deposits, although they were also used to identify the location of the Chicxulub Crater that has been linked to the extinction of the dinosaurs.

Marine Seismic Survey

Seismic data is collected by surveying an area that is suspected to contain oil or gas deposits. Seismic waves are generated from a source, which is usually an air gun in marine surveys or a machine called a Vibroseis for land-based surveys. The seismic waves reflect back to the surface at the interfaces between rock layers, where an array of receivers record the amplitude and arrival times of the reflected waves as a time series, which is called a trace. The data that is generated from a single source is called a shot or shot record, and a modern seismic survey may consist of tens of thousands of shots and multiple terabytes of trace data.

Common Depth Point (CDP) Gather

In order to solve the inversion problem, we take advantage of the geometric relationships between traces that have different source and receiver locations but a common depth point (also known as a common midpoint). By comparing the time it took for the seismic waves to travel from the different source and receiver locations and experimenting with different velocity models for the waves moving through the rock, we can estimate the depth of the common subsurface point that the waves reflected off of. By aggregating a large number of these estimates, we can construct a complete image of the subsurface. As we increase the density and the number of traces, we can create higher quality images that improve our understanding of the subsurface geology.

A 3D seismic image of Japan's southeastern margin

Additionally, seismic data processing has a long history of using open-source software tools that were initially developed in academia and were then adopted and enhanced by private companies. Both the Seismic Unix project, from the Colorado School of Mines, and SEPlib, from Stanford University, have their roots in tools created by graduate students in the late 1970s and early 1980s. Even the most popular commercial toolkit for seismic data processing, SeisSpace, is built on top of an open source foundation, the JavaSeis project.

Hadoop and Seismic Data Processing

Geophysicists have been pushing the limits of high-performance computing for more than three decades; they were early adopters of the first Cray supercomputers as well as the massively parallel Connection Machine. Today, the most challenging seismic data processing tasks are performed on custom compute clusters that take advantage of multiple GPUs per node, high-performance networking and storage systems for fast data access.

The data volume of modern seismic surveys and the performance requirements of the compute clusters means that data from seismic surveys that are not undergoing active processing are often stored offsite on tape. If a geophysicist wants to re-examine an older survey, or study the effectiveness of a new processing technique, he must file a request to move the data into active storage and then consume precious cluster resources in order to process the data.

Fortunately, Apache Hadoop has emerged as a cheap and reliable online storage system for petabyte-scale data sets. Even better, we can export many of the most I/O intensive steps in the seismic data processing into the Hadoop cluster itself, thus freeing precious resources in the supercomputer cluster for the most difficult and urgent processing tasks.

Seismic Hadoop is a project that we developed at Cloudera to demonstrate how to store and process seismic data in a Hadoop cluster. It combines Seismic Unix with Crunch, the Java library we developed for creating MapReduce pipelines. Seismic Unix gets its name from the fact that it makes extensive use of Unix pipes in order to construct complex data processing tasks from a set of simple procedures. For example, we might build a pipeline in Seismic Unix that first applies a filter to the trace data, then edits some metadata associated with each trace, and finally sorts the traces by the metadata that we just edited:
sufilter f=10,20,30,40 | suchw key1=gx,cdp key2=offset,gx key3=sx,sx b=1,1 c=1,1 d=1,2 | susort cdp gx
Seismic Hadoop takes this same command and builds a Crunch pipeline that performs the same operations on a data set stored in a Hadoop cluster, replacing the local susort command with a distributed sort across the cluster using MapReduce. Crunch takes care of figuring out how many MapReduce jobs to run and which processing steps are assigned to the map phase and which are assigned to the reduce phase. Seismic Hadoop also takes advantage of Crunch’s support for streaming the output of a MapReduce pipeline back to the client in order to run the utilities for visualizing data that come with Seismic Unix.

Challenges to Solve Together

Talking to a geophysicist is a little bit like seeing into the future: the challenges they face today are the challenges that data scientists in other fields will be facing five years from now. There are two challenges in particular that I would like the broader community of data scientists and Hadoop developers to be thinking about:

Reproducibility. Geophysicists have developed tools that make it easy to understand and reproduce the entire history of analyses performed on a particular data set. One of the most popular open source seismic processing toolkits, Madagascar, even chose reproducibility.org as its home page. Reproducible research has enormous benefits in terms of data quality, transparency, and education, and all of the tools we develop should be built with reproducibility in mind.

Dynamic and resource-aware scheduling of jobs on heterogeneous clusters. MR2 and YARN will unleash a Cambrian explosion of data-intensive processing jobs on Hadoop clusters. What was once only MapReduce jobs will now include MPI jobs, Spark queries, and BSP-style computations. Different jobs will have radically different resource requirements in terms of CPU, memory, disk, and network utilization, and we will need fine-grained resource controls, intelligent defaults, and robust mechanisms for recovering from task failures across all job types.

It is an unbelievably exciting time to be working on these big data problems. Join us and be part of the solution!

Josh is a data scientist for Cloudera and the creator of Crunch.

Andrew Brust (@andrewbrust) asserted “By responding to potential threats with thoughtfulness, and a zeal to add value, SQL and Big Data could be big business for Redmond” as a deck for his Big Data and SQL Server: Disruption or Harmony? article of 2/1/2012 for the Redmond Developer News site:

The SQL Server relational engine matured a long time ago. There have been advances, of course, in performance, fault tolerance and high availability, not mention encryption, compression and file-system integration. And, yes, there's been support for XML, geospatial data and even a service broker. But these have been improvements on the margins. All these features are new flavors of icing; the cake has stayed the same.

And yet, over the 20-year lifetime of SQL Server, Microsoft has continued to add value to the product, even as the core features have gone into maintenance mode. The big story here has been business intelligence: BI capabilities started in SQL Server 7 with OLAP Services, and have been expanded in meaningful ways with every subsequent release. It's been a great strategy and it still is. But will it keep working?

Core capabilities can't stay static forever; eventually disruption comes along. Today that disruption is here, coming from Big Data, Hadoop and its MapReduce distributed-computing approach. As strong as SQL Server BI capabilities are (and they're getting even stronger with SQL Server 2012), if Microsoft can't embrace Big Data technology, SQL Server could find itself in a position of desperation, going from underdog to contender, to retiree. What's a poor enterprise software company to do?

Build Bridges, Don't Burn Them
Microsoft could ignore Hadoop, but that would be foolish. It could try to build a competitor, which it almost did with the Microsoft Research Dryad project, but I fear not too many people would have come to that party. Microsoft could just adopt Hadoop, plain vanilla, but that would most likely be a race to the bottom, and it wouldn't even win. Really, Microsoft must mix Hadoop into its bag of tricks and do what it has always done best: take a raw technology and make it approachable to the enterprise. I can't be sure yet, but I think that's what Microsoft has done, and it has enhanced the value of Windows Azure in the process.

With code name "Project Isotope," Microsoft has taken the step of implementing Hadoop on Windows. It's a no-slouch effort, too: Microsoft's distribution of Hadoop is being developed in concert with Hortonworks, a startup company founded and staffed by many former Hadoop team members at Yahoo! (where the open source project began). But what Microsoft has also done is integrate Hadoop into its BI stack, and that may be one of the smartest moves it's made in quite some time.

Microsoft has created an Excel add-in for Hive, which provides a SQL-like abstraction over Hadoop and MapReduce. The add-in is based on an ODBC driver, which in turn is compatible with PowerPivot, so business users can do meaningful analysis on Big Data, on their own terms. And because the same engine that drives PowerPivot has been implemented inside Analysis Services in SQL Server 2012, that product has access to Hadoop now, too. With that, Microsoft has joined the Big Data and Enterprise BI worlds. It has also tied together SQL Server and Hadoop.

Simplify and Succeed
Hadoop runs on Windows Server and on Windows Azure, with an installer that makes setup really easy. In addition to those configurations, Microsoft allows Windows Azure users to provision an entire Hadoop cluster from a Web portal, without any discrete installation steps at all. Once the cluster's up, customers can connect to its head node via Remote Desktop. The full Java-based command-line personality of Hadoop is there if you want it, but there are also Hive and JavaScript consoles in the browser. And then it only takes a minute or two to build out a connection from Microsoft BI tools or Excel, putting Hadoop to work for users fitting a number of profiles. It's classic Microsoft, mixed with equal parts open source and Java.

I've said before that Microsoft does some of its best work when it embraces standards from outside the company. That's what happened with jQuery and ASP.NET, and it may well be what happens with HTML5 and JavaScript in Windows 8. In the case of implementing Hadoop, Microsoft makes Windows Azure more valuable and more agnostic. It brings continued relevance to Windows Server and SQL Server. And it widens the reach and utility of Hadoop. By responding to a potential threat with thoughtfulness -- and a zeal to add value -- Big Data could be big business for Redmond.

Full disclosure: I’m a contributing editor for Visual Studio Magazine, which is published by 1105 Media who also own the Redmond Developer News site.

OakLeaf Systems

Windows Azure and Cloud Computing Posts for 2/1/2012+

What’s the Difference? You need the Name & Key for a Connection String

Oh No, We’re doomed right?

AccountName

AccountKey

Basic scenario

Using directories approach

The Practice of Data Science

Data Scientists Who Find Oil: Reflection Seismology

Hadoop and Seismic Data Processing

Challenges to Solve Together

How the classifier works:

Training time:

Runtime:

How to configure your node application with everyauth and Windows Azure ACS

Sample application

Developer Center Updates

How Do I Videos – Learning Made Easier

Notable Content this Month

Extensions released in January (see all 73 of them here!):

Team Articles:

Community Articles:

Presentations:

Samples (see all of them here):

LightSwitch Team Community Sites

Join Us!

Functional POV

Semantics POV

LightSwitch POV

What’s INotifyPropertyChanged?

An easy base class…

OPERATIONAL CONSISTENCY

Table of Contents

Windows Azure and Cloud Computing Posts for 1/30/2012+

Scaling Up the Problem / Moving Data to Compute

Scaling Out the Problem / Moving Compute to Data

Discussion

Scan your databases for security vulnerabilities

Before Getting Started with the T4 Template

Calling a Service Operation that Returns a Collection of Entities

Calling a Service Operation that Returns a Single Entity

Calling a Service Operation by using POST

The Remaining Examples

Calling a Service Operation that Returns a Collection of Primitive Types

Calling a Service Operation that Returns a Single Primitive Type

Calling a Service Operation that Returns Void

Calling a Service Operation Asynchronously

Service Operations Won’t Work on Asynchronous Clients

Before You Run the Sample Application

Blob Locations and Sample Application Download

Choosing the Mode: Run on Local Development Machine or on Windows Azure

To run the application on your local workstation:

To submit the application to Windows Azure (run on Windows Azure rather than locally):

Step 1: Supply Windows Azure Storage Account Information for Output

Step 2: Read in Data from Blob Storage Using IParallelReader Interface

Step 3: Compute Statistics Operations on Distributed Data

Step 4: Write Results to Blob Storage as a .csv File

Where Do We Call the Service?

Calling Web Services

Address Book Example

Run it!

Wrap Up

Table of Contents

BUSINESS JITTER and the CLOUD