Windows Azure and Cloud Computing Posts for 8/12/2010+
 | A compendium of Windows Azure, SQL Azure, Azure AppFabric, Visual Studio LightSwitch and other cloud-computing articles. |
Update 8/12/2010: Windows Azure Platform Appliance (WAPA) topic replaced with Visual Studio LightSwitch section. Added Google Blog Search and Real-Time Updates for “LightSwitch” to left frame. WAPA section will be reinstated when Microsoft releases it.
Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:
- Azure Blob, Drive, Table and Queue Services
- SQL Azure Database, Codename “Dallas” and OData
- AppFabric: Access Control and Service Bus
- Live Windows Azure Apps, APIs, Tools and Test Harnesses
- Visual Studio LightSwitch
- Windows Azure Infrastructure
- Cloud Security and Governance
- Cloud Computing Events
- Other Cloud Computing Platforms and Services
To use the above links, first click the post’s title to display the single article you want to navigate.
Cloud Computing with the Windows Azure Platform published 9/21/2009. Order today from Amazon or Barnes & Noble (in stock.)
Read the detailed TOC here (PDF) and download the sample code here.
Discuss the book on its WROX P2P Forum.
See a short-form TOC, get links to live Azure sample projects, and read a detailed TOC of electronic-only chapters 12 and 13 here.
Wrox’s Web site manager posted on 9/29/2009 a lengthy excerpt from Chapter 4, “Scaling Azure Table and Blob Storage” here.
You can now freely download by FTP and save the following two online-only PDF chapters of Cloud Computing with the Windows Azure Platform, which have been updated for SQL Azure’s January 4, 2010 commercial release:
- Chapter 12: “Managing SQL Azure Accounts and Databases”
- Chapter 13: “Exploiting SQL Azure Database's Relational Features”
HTTP downloads of the two chapters are available for download at no charge from the book's Code Download page.
Tip: If you encounter articles from MSDN or TechNet blogs that are missing screen shots or other images, click the empty frame to generate an HTTP 404 (Not Found) error, and then click the back button to load the image.
Azure Blob, Drive, Table and Queue Services
No significant articles today.
<Return to section navigation list>
SQL Azure Database, Codename “Dallas” and OData
Russell Sinclair announced on 8/12/2010 that you can now Easily migrate Access data to SQL Azure with a new SQL Server Migration Assistant (SSMA) for Access:
A couple of months ago, we posted a blog entry on how to use Access 2010 with SQL Azure. At that time, the easiest way to migrate your Access data to SQL Azure was to first migrate the tables and data to a non-Azure SQL Server then to use SQL Server tools to migrate these items to a SQL Azure database.
Today, we are pleased to let you know that the SQL Server team has released a new version of the SQL Server Migration Assistant (SSMA) for Access that will allow you to migrate your data and schema directly from Access to SQL Azure, as well as SQL Server 2005, 2008 and 2008 R2. Just like with SQL Server, you can use your Access 2010 application as the front-end to your SQL Azure database.
More details on the release are available on the SQL Azure Team Blog [see below] and Zane Adam’s Blog , or you can download SSMA for Access directly from the Microsoft Download Center.
Wayne Walter Berry (@WayneBerry) explains Leveraging SQL Azure with Microsoft Access with the SQL Server Migration Assistant (SSMA) for Access v4.2:
SQL Server Migration Assistant (SSMA for Access v4.2) has been updated to support migrating your schema and data from Microsoft Access to SQL Azure. This release makes it possible to move data directly and easily from local Microsoft Access databases. The SQL Server Migration Assistant toolkits were designed to tackle the complex manual process customers deal with when migrating databases. By using the SQL Server Migration Assistants, customers and partners reduce the manual effort; as a result the time, cost and risks associated with migrating are significantly reduced.
For more information on how to use Access 2010 with SQL Azure, please also see the detailed blog that was posted at TechEd which walks through how to get started. This blog is slightly out of date with the new release of SQL Server Migration Assistant; at TechEd it was not possible to move data directly to SQL Azure. We will be following up with updated blog post so stay tuned.
Download for free SSMA for Access v4.2.
Zane Adam posted Announcing the Third Community Technology Preview of Codename “Dallas” on 8/12/2010:
During his keynote speech at the Worldwide Partner Conference in July, Microsoft President Bob Muglia announced that the third Community Technology Preview of “Dallas” would be available within 30 days. Today, we are thrilled to announce the release of “Dallas” CTP3 at www.sqlazureservices.com.
This new community technology preview brings with it a number of exciting improvements, rounding out the “Dallas” experience and making it easier to develop applications using data from “Dallas” subscriptions. CTP3 brings with it a number of small incremental changes as well as larger, highly-requested new features such as:
- Basic Authentication to authenticate users of Dallas data offerings
- Flexible querying of OData content, as announced earlier this year at MIX, beyond simple web services
- Support for Add Service Reference in Visual Studio
- New content providers and data
For more information and to experience “Dallas” CTP3 for yourself, visit www.sqlazureservices.com.
Raoul Illyés wrote SQL Azure and read-consistency (English version) on 8/12/2010:
Since René Løhde who is an Architecture Evangelist at Microsoft Denmark posted one of my articles regarding SQL Azure in version2.dk, I’ve received a lot of requests from the English speaking audience of my blog to do the same for them. So here you are, SQL Azure and read-consistency English version.
There is a lot of thing that you can’t do in SQL Azure compared to your “regular” flavor of SQL Server, and because of this a lot of focus lately have been on its missing features.
Microsoft standard reply to this criticism seems to be that we need to think in new ways when it comes to SQL Azure, something that is quite easy to say but might not always be as easy to implement for its customers.
But is that really all there is to it, an ultra slim version of SQL Server with a bunch of marketing people trying to push it as the latest thing?
Actually all you have to do is to scratch the surface and some quite interesting differences between SQL Azure and all the other versions of SQL Server will emerge.
So where do we begin?
To be able to tell this story I have to go back a couple of years ago at PASS at a very secret meeting, actually so secret that only afterwards have I heard rumors about me ever attending. SQL Server 2005 was the flagship back then and what were later to become SQL Server 2008 was still in CTP.
In this very secret meeting there was a discussion going on between the pros and cons for the latest version of SQL Server and believe me, everybody had some story to tell.
As the discussion was moving forward and everybody had their turn there were two persons standing out, my former director and myself. Every time we got the chance to say anything the answer was always the same, read-consistency.
And as you probably already have guessed the whole scene repeated its self when it was time to list the cons.
How come we thought it were both a pro and a con?
It is true that read-consistency as a option has been available since SQL Server 2005 and this makes it a pro, it has not yet to date been implemented a default setting for new user databases in SQL Server, (SQL Server R2 being the latest) which in my book also makes it a con.
What is read-consistency anyway?
Well to make a long story short, read consistency is a mechanism through which it is possible for someone to read data in a consistent state while others are updating the very same data or the other way around. This has not been possible to do in previous versions of SQL Server before SS2005, and has often led to solutions where people have accepted the possibility to do non-consistent operations with the use of hints like NOLOCK.
And how does this relate to SQL Azure you might ask?Well, one of the people “supposing” to have attended to this meeting many years ago is now part of the SQL Azure team and I had the pleasure of meeting with him recently at a tree day seminar on SQL Azure at MS campus, Redmond.
Talking back and forth the discussion came to be about read-consistency and he told me not only had they implemented it as a default setting for every new database created in SQL Azure there is also no way to turn if off! (For the database that is.)So I have no doubt that the team behind SQL Azure in many aspects have had the chance to start over, not burden by the requirements of the past but instead fully utilizing that experiences and focused on the future.
It is because of things like this that I am so excited to work with this new technology and I just can’t wait to see what comes next.
Raoul
- Fact I: Actually you don’t have a real database at all in SQL Azure only a virtual one and it is also because of this we can’t do SET operations on our VDBS.
- Fact II: You can still control and change isolation levels if you want to for the session, just use the following syntax: (SET TRANSACTION ISOLATION LEVEL).
- Fact III: Both READ_COMMITTED_SNAPSHOT (statement) and ALLOW_SNAPSHOT_ISOLATION (transaction level) options have been enabled for user databases.
<Return to section navigation list>
AppFabric: Access Control and Service Bus
The Windows Azure AppFabric Team announced the availability of a New version of the Windows Azure AppFabric SDK available for download on 8/12/2010:
A new version of the Windows Azure AppFabric SDK is available for download starting 8/12. The new version introduces a fix in the Windows Azure AppFabric SDK for the following Service Bus bug:
Adding a Ws2007HttpRelayBinding endpoint that uses TransportWithMessageCredential security mode to a ServiceHost that also exposes a local MEX endpoint (not visible through the Service Bus) using the mexHttpsBinding, causes the local MEX endpoint to stop working. Invoking the MEX endpoint results in an InvalidOperationException.
The new version of the SDK is available here.
The Windows Azure AppFabric Team
Maarten Balliauw explains Simplified access control using Windows Azure AppFabric Labs in this 8/12/2010 post:
Earlier this week, Zane Adam announced the availability of the New AppFabric Access Control service in LABS. The highlights for this release (and I quote):
- Expanded Identity provider support - allowing developers to build applications and services that accept both enterprise identities (through integration with Active Directory Federation Services 2.0), and a broad range of web identities (through support of Windows Live ID, Open ID, Google, Yahoo, Facebook identities) using a single code base.
- WS-Trust and WS-Federation protocol support – Interoperable WS-* support is important to many of our enterprise customers.
- Full integration with Windows Identity Foundation (WIF) - developers can apply the familiar WIF identity programming model and tooling for cloud applications and services.
- A new management web portal - gives simple, complete control over all Access Control settings.
Wow! This just *has* to be good! Let’s see how easy it is to work with claims based authentication and the AppFabric Labs Access Control Service, which I’ll abbreviate to “ACS” throughout this post.
What are you doing?
In essence, I’ll be “outsourcing” the access control part of my application to the ACS. When a user comes to the application, he will be asked to present certain “claims”, for example a claim that tells what the user’s role is. Of course, the application will only trust claims that have been signed by a trusted party, which in this case will be the ACS.
Fun thing is: my application only has to know about the ACS. As an administrator, I can then tell the ACS to trust claims provided by Windows Live ID or Google Accounts, which will be reflected to my application automatically: users will be able to authenticate through any service I configure in the ACS, without my application having to know. Very flexible, as I can tell the ACS to trust for example my company’s Active Directory and perhaps also the Active Directory of a customer who uses the application
Prerequisites
Before you start, make sure you have the latest version of Windows Identity Foundation installed. This will make things easy, I promise! Other prerequisites, of course, are Visual Studio and an account on https://portal.appfabriclabs.com. Note that, since it’s still a “preview” version, this is free to use.
In the labs account, create a project and in that project create a service namespace. This is what you should be seeing (or at least: something similar):
Getting started: setting up the application side
Before starting, we will require a certificate for signing tokens and things like that. Let’s just start with creating one so we don’t have to worry about that further down the road. Issue the following command in a Visual Studio command prompt:
MakeCert.exe -r -pe -n "CN=<your service namespace>.accesscontrol.appfabriclabs.com" -sky exchange -ss my
This will create a certificate that is valid for your ACS project. It will be installed in the local certificate store on your computer. Make sure to export both the public and private key (.cer and .pkx).
That being said and done: let’s add claims-based authentication to a new ASP.NET Website. Simply fire up Visual Studio, create a new ASP.NET application. I called it “MyExternalApp” but in fact the name is all up to you. Next, edit the Default.aspx page and paste in the following code:
1 <%@ Page Title="Home Page" Language="C#" MasterPageFile="~/Site.master" AutoEventWireup="true" 2 CodeBehind="Default.aspx.cs" Inherits="MyExternalApp._Default" %> 3 4 <asp:Content ID="HeaderContent" runat="server" ContentPlaceHolderID="HeadContent"> 5 </asp:Content> 6 <asp:Content ID="BodyContent" runat="server" ContentPlaceHolderID="MainContent"> 7 <p>Your claims:</p> 8 <asp:GridView ID="gridView" runat="server" AutoGenerateColumns="False"> 9 <Columns> 10 <asp:BoundField DataField="ClaimType" HeaderText="ClaimType" ReadOnly="True" /> 11 <asp:BoundField DataField="Value" HeaderText="Value" ReadOnly="True" /> 12 </Columns> 13 </asp:GridView> 14 </asp:Content>
Next, edit Default.aspx.cs and add the following Page_Load event handler:
1 protected void Page_Load(object sender, EventArgs e) 2 { 3 IClaimsIdentity claimsIdentity = 4 ((IClaimsPrincipal)(Thread.CurrentPrincipal)).Identities.FirstOrDefault(); 5 6 if (claimsIdentity != null) 7 { 8 gridView.DataSource = claimsIdentity.Claims; 9 gridView.DataBind(); 10 } 11 }
So far, so good. If we had everything configured, Default.aspx would simply show us the claims we received from ACS once we have everything running. Now in order to configure the application to use the ACS, there’s two steps left to do:
- Add a reference to Microsoft.IdentityModel (located somewhere at C:\Program Files\Reference Assemblies\Microsoft\Windows Identity Foundation\v3.5\Microsoft.IdentityModel.dll)
- Add an STS reference…
That first step should be easy: add a reference to Microsoft.IdentityModel in your ASP.NET application. The second step is almost equally easy: right-click the project and select “Add STS reference…”, like so:
A wizard will pop-up. Here’s a secret: this wizard will do a lot for us! On the first screen, enter the full URL to your application. I have mine hosted on IIS and enabled SSL, hence the following screenshot:
In the next step, enter the URL to the STS federation metadata. To the what where? Well, to the metadata provided by ACS. This metadata contains the types of claims offered, the certificates used for signing, … The URL to enter will be something like https://<your service namespace>.accesscontrol.appfabriclabs.com:443/FederationMetadata/2007-06/FederationMetadata.xml:
In the next step, select “Disable security chain validation”. Because we are using self-signed certificates, selecting the second option would lead us to doom because all infrastructure would require a certificate provided by a valid certificate authority.
From now on, it’s just “Next”, “Next”, “Finish”. If you now have a look at your Web.config file, you’ll see that the wizard has configured the application to use ACS as the federation authentication provider. Furthermore, a new folder called “FederationMetadata” has been created, which contains an XML file that specifies which claims this application requires. Oh, and some other details on the application, but nothing to worry about at this point.
Our application has now been configured: off to the ACS side!
Getting started: setting up the ACS side
First of all, we need to register our application with the Windows Azure AppFabric ACS. his can be done by clicking “Manage” on the management portal over at https://portal.appfabriclabs.com. Next, click “Relying Party Applications” and “Add Relying Party Application”. The following screen will be presented:
Fill out the form as follows:
- Name: a descriptive name for your application.
- Realm: the URI that the issued token will be valid for. This can be a complete domain (i.e. www.example.com) or the full path to your application. For now, enter the full URL to your application, which will be something like https://localhost/MyApp.
- Return URL: where to return after successful sign-in
- Token format: we’ll be using the defaults in WIF, so go for SAML 2.0.
- For the token encryption certificate, select X.509 certificate and upload the certificate file (.cer) we’ve been using before
- Rule groups: pick one, best is to create a new one specific to the application we are registering
Afterwards click “Save”. Your application is now registered with ACS.
The next step is to select the Identity Providers we want to use. I selected Windows Live ID and Google Accounts as shown in the next screenshot:
One thing left: since we are using Windows Identity Foundation, we have to upload a token signing certificate to the portal. Export the private key of the previously created certificate and upload that to the “Certificates and Keys” part of the management portal. Make sure to specify that the certificate is to be used for token signing.
Allright, we’re almost done. Well, in fact: we are done! An optional next step would be to edit the rule group we’ve created before. This rule group will describe the claims that will be presented to the application asking for the user’s claims. Which is very powerful, because it also supports so-called claim transformations: if an identity provider provides ACS with a claim that says “the user is part of a group named Administrators”, the rules can then transform the claim into a new claim stating “the user has administrative rights”.
Testing our setup
With all this information and configuration in place, press F5 inside Visual Studio and behold… Your application now redirects to the STS in the form of ACS’ login page.
So far so good. Now sign in using one of the identity providers listed. After a successful sign-in, you will be redirected back to ACS, which will in turn redirect you back to your application. And then: misery :-)
ASP.NET request validation kicked in since it detected unusual headers. Let’s fix that. Two possible approaches:
- Disable request validation, but I’d prefer not to do that
- Create a custom RequestValidator
Let’s go with the latter option… Here’s a class that you can copy-paste in your application:
1 public class WifRequestValidator : RequestValidator 2 { 3 protected override bool IsValidRequestString(HttpContext context, string value, RequestValidationSource requestValidationSource, string collectionKey, out int validationFailureIndex) 4 { 5 validationFailureIndex = 0; 6 7 if (requestValidationSource == RequestValidationSource.Form && collectionKey.Equals(WSFederationConstants.Parameters.Result, StringComparison.Ordinal)) 8 { 9 SignInResponseMessage message = WSFederationMessage.CreateFromFormPost(context.Request) as SignInResponseMessage; 10 11 if (message != null) 12 { 13 return true; 14 } 15 } 16 17 return base.IsValidRequestString(context, value, requestValidationSource, collectionKey, out validationFailureIndex); 18 } 19 }
Basically, it’s just validating the request and returning true to ASP.NET request validation if a SignInMesage is in the request. One thing left to do: register this provider with ASP.NET. Add the following line of code in the <system.web> section of Web.config:
<httpRuntime requestValidationType="MyExternalApp.Modules.WifRequestValidator" />
If you now try loading the application again, chances are you will actually see claims provided by ACS:
There', that’s it. We now have successfully delegated access control to ACS. Obviously the next step would be to specify which claims are required for specific actions in your application, provide the necessary claims transformations in ACS, … All of that can easily be found on Google Bing.
Conclusion
To be honest: I’ve always found claims-based authentication and Windows Azure AppFabric Access Control a good match in theory, but an ugly and cumbersome beast to work with. With this labs release, things get interesting and almost self-explaining, allowing for easier implementation of it in your own application. As an extra bonus to this blog post, I also decided to link my ADFS server to ACS: it took me literally 5 minutes to do so and works like a charm!
Final conclusion: AppFabric team, please ship this soon :-) I really like the way this labs release works and I think many users who find the step up to using ACS today may as well take the step if they can use ACS in the easy manner the labs release provides.
By the way: more information can be found on http://acs.codeplex.com.
Dominic Baier posted Moving StarterSTS to the (Azure) Cloud to his LeastPrivilege blog on 8/12/2010:
Quite some people asked me about an Azure version of StarterSTS [see item below]. While I kinda knew what I had to do to make the move, I couldn’t find the time. Until recently.
This blog post briefly documents the necessary changes and design decisions for the next version of StarterSTS which will work both on-premise and on Azure.
Provider
Fortunately StarterSTS is already based on the idea of “providers”. Authentication, roles and claims generation is based on the standard ASP.NET provider infrastructure. This makes the migration to different data stores less painful. In my case I simply moved the ASP.NET provider database to SQL Azure and still use the standard SQL Server based membership, roles and profile provider.In addition StarterSTS has its own providers to abstract resource access for certificates, relying party registration, client certificate registration and delegation. So I only had to provide new implementations. Signing and SSL keys now go in the Azure certificate store and user mappings (client certificates and delegation settings) have been moved to Azure table storage.
The one thing I didn’t anticipate when I originally wrote StarterSTS was the need to also encapsulate configuration. Currently configuration is “locked” to the standard .NET configuration system. The new version will have a pluggable SettingsProvider with versions for .NET configuration as well as Azure service configuration. If you want to externalize these settings into e.g. a database, it is now just a matter of supplying a corresponding provider.
Moving between the on-premise and Azure version will be just a matter of using different providers.
URL Handling
Another thing that’s substantially different on Azure (and load balanced scenarios in general) is the handling of URLs. In farm scenarios, the standard APIs like ASP.NET’s Request.Url return the current (internal) machine name, but you typically need the address of the external facing load balancer.There’s a hotfix for WCF 3.5 (included in v4) that fixes this for WCF metadata. This was accomplished by using the HTTP Host header to generate URLs instead of the local machine name. I now use the same approach for generating WS-Federation metadata as well as information card files.
New Features
I introduced a cache provider. Since we now have slightly more expensive lookups (e.g. relying party data from table storage), it makes sense to cache certain data in the front end. The default implementation uses the ASP.NET web cache and can be easily extended to use products like memcached or AppFabric Caching.Starting with the relying party provider, I now also provide a read/write interface. This allows building management interfaces on top of this provider. I also include a (very) simple web page that allows working with the relying party provider data. I guess I will use the same approach for other providers in the future as well.
I am also doing some work on the tracing and health monitoring area. Especially important for the Azure version.Stay tuned.
Christian Weyer and Dominic Baier are coordinators of the thinktecture Starter STS (Community Edition) project on CodePlex:
Project Description
Disclaimer
Though you could use StarterSTS directly as a production STS, be aware that this was not the design goal.StarterSTS did not go through the same testing or quality assurance process as a "real" product like ADFS2 did. StarterSTS is also lacking all kinds of enterprisy features like configuration services, proxy support or operations integration. The main goal of StarterSTS is to give you a learning tool for building non-trivial security token services. Another common scenario is to use StarterSTS in a development environment.
High level features
- Active and passive security token service
- Supports WS-Federation, WS-Trust, REST, OpenId, SAML 1.1/2.0 tokens and Information Cards
- Supports username/password and client certificate authentication
- Based on standard ASP.NET membership, roles and profile infrastructure
- Control over security policy (SSL, encryption, SOAP security) without having to touch WIF/WCF configuration directly
- Automatic generation of WS-Federation metadata for federating with relying parties and other STSes
Documentation
http://identity.thinktecture.com/stsce/docs/Screencasts
Initial Setup & Configuration
Federating your 1st Web App
Federating your 1st Web Service
Single-Sign-On
Using the REST Endpoint
Using the OpenId Bridge
Tracing
Using Client Certificates
Using Information Cards
<Return to section navigation list>
Live Windows Azure Apps, APIs, Tools and Test Harnesses
David Pallman posted Every Penny Counts in the Cloud on 8/12/2010:
There are many potential benefits of cloud computing that an organization can evaluate, but my favorite one to talk about is the efficient use of your I.T. dollars that is made possible by the elasticity of the cloud. This is easily explained through 2 simple diagrams that you can jot down on a nearby whiteboard or napkin whenever the opportunity arises. There are more elaborate versions of these diagrams out there in slide decks, but you can tell the story effectively just by drawing a couple of lines and narrating them.
The first diagram has a wavy line showing changing application load over time. To this we add a staircase which depicts cycles of hardware purchases (the vertical parts of the staircase) and time passing (the horizontal parts of the staircase). What this shows is that companies are forced to buy more capacity than they really need just to be on the safe side—in effect, spending more money than is necessary or spending it earlier than necessary. Even worse, if there is an unexpected change in load an undersupply situation can result where there is insufficient capacity. This arrangement is what we’re used to in the enterprise—but it is hardly ideal.
The second diagram has an identical wavy line showing changing application load over time. Instead of being joined by a staircase, the wavy line is paralleled by another wavy line that is nearly identical. This second line is the size of your footprint in the cloud. It is dialed larger or smaller in accordance with the load you measure. Anyone can see quite readily how more efficient this is.
Does this superior arrangement the cloud makes possible come at a price? Yes it does: you need to monitor your application regularly and expand or reduce your deployment in the cloud accordingly. Failure to do this will undermine the financial premise of using the cloud. Today this monitoring and adjusting is not something Windows Azure does for you as an out-of-box feature. However, you can achieve the monitoring and adjusting programatically via the Windows Azure Diagnostic and Service Management APIs.
David manages the Custom App Development practice at Neudesic, a Microsoft Gold Partner and National Systems Integrator.
Microsoft’s Retail and Hospitality Group wrote Feature: I want always gets for the Autumn 2010 edition of the redesigned Speak, formerly Retailspeak and posted it on 8/12/2010:
Despite their increasing demands and often fickle behaviour, customers have maintained their place at the heart of the retail operation and, in order to maintain their share of the wallet, retailers must do everything they can to keep them happy.
In today’s fast-paced environment, this means a move towards demand-driven retailing. “Today’s customers expect retailers to quickly respond to their changing preferences,” says Colin Masson, worldwide director for Microsoft’s enterprise resource planning (ERP) and supply chain solution areas. “Traditional retailing, which relies on pushing products into channels, merchandising, assortment planning and allocating inventory based on aggregated order information, and relying on market ‘insights’ from past experiences, is no longer enough.”
Lindsay Carpen, Junction Solutions’ retail director agrees: “The biggest thing that has impacted retailers is the speed of change,” he says. “Instead of nice, easily predictable patterns, we have all witnessed significant changes in consumer demand. Having a supply chain that can quickly react to these changes is a necessity.”
According to AMR’s Retail 2010 Outlook, demand-driven retailers are the ones that will gather valuable information at each point of interaction, make better use of their knowledge of shopper behaviour, and translate demand insight into effective response and shaping activities. The results are higher return on assets, more inventory turns, and greater revenue growth than their less mature demand-driven peers.
“The forward-thinking retailer is reorganising its business to become demand driven in order to get closer to the needs of the consumers,” says Alexandria Rumble, global product marketing director at TXT e-solutions. “This means using shopper insights to its full potential, connecting business processes, creating personalised offerings and giving consumers the power to browse, enquire, buy and return on any channel that most fits their needs. This allows retailers to meet the consumer need for flexibility and create brand loyalty, with a consistent brand message. Ultimately retailers are putting the consumer first, but in the most profitable way possible to the business.” …
“The completely integrated demand-driven retail supply chain will soon be a reality, providing substantial bottom-line benefits for both manufacturers and retailers, while satisfying consumer concerns about the environment and reducing waste,” concludes Masson. “Changes in stores will be instantly visible at manufacturers’ plants, and everyone in the extended supply chain will be working to a single set of numbers.
In addition to this, Masson believes that the cloud will make a significant contribution to driving supply chain efficiencies in the future. “Imagine that the compute power to achieve the vision of demand-driven ‘greentailing’ is being delivered at a tenth of current costs, taking one tenth of the time to provision, and therefore requiring just one tenth of the IT resource to maintain – that’s the vision Microsoft has for the cloud,” he explains. “RedPrairie’s Collaborative Flowcasting is a major step towards solving the maths problem, while the Microsoft’s multi-billion dollar investment in data centres and the Windows Azure platform is delivering the only end-to-end solution for ISVs to deliver infrastructure as a service, platform as a service, and software as a service, which will enable the necessary multi-enterprise integration and collaboration, and cost and productivity to fulfill this vision of demand-driven retailing.”
“Macy's strong second-quarter results were attributed to several changes, including more locally tailored merchandise and new exclusive brands” (both “I wants”) according to a Macy's profit surges in second quarter article in the Los Angeles Times’ Business section. Click here to read the full Speak article.
Joannes Vermorel announced Lokad.Snapshot released, open source app to backup / snapshot your Azure Storage account in this brief Virtualization Times article:
We have just released Lokad.Snapshot , an open source C#/.NET Windows Azure app that provide snapshotting capabilities targeting the Windows Azure Storage. Perfectly reliable storage is no substitute for snapshots as accidental deletion is not recoverable.
The project is still a bit early stage, but already in production at Lokad.
We would love to have community feedback. Don't hesitate to let us know what you think.
Best regards,
Joannes Vermorel
Lokad Sales Forecasting
Return to section navigation list>
Visual Studio LightSwitch
![image[6]](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZTU_Nf4qzVGYw6rXcnyB2wOWFT-A9tFDTz5gi82_3fxCLb9JVNJr54z15r7If6AjJmwy2eZlbW4ixzoDdTolCxg2V3vHnNehG7DJRrVRNxU2kJI758CRjTVPyXsN6eCvG8mqZYeC2/?imgmax=800 "image[6]")
Paul Patterson’s Microsoft LightSwitch – A Data Driven Approach post of 8/12/2010 reviews LightSwitch’s data handling technlogy:
With LightSwitch, it’s all about the data. Data can be anything from a simple list of customers to a complex organization of products, suppliers, and sales. LightSwitch provides you the graphical designers to either create new data “entities” from scratch, or link into existing data from external sources. Using the LightSwitch IDE designers, you create and/or connect to the data that you want your users to use.
Create or attach the data you want to use in your application (courtesy Microsoft)
Interesting to note is that LightSwitch also includes some “custom” entity types, or “business domain” data types. These are types of data that are not typically available for storage in an underlying data store, but are available as types in LightSwitch because they are a commonly used type of data. For example, a phone number is not an entity type in an SQL database, however by using a LightSwitch defined PhoneNumber type, you can leverage the built in validation that LightSwitch uses for this custom type. A phone is still stored as a string type in the SQL database however the validation that occurs on that string is performed in LightSwitch. Using these business domain types is pretty handy and will save you a lot of validation coding.
Something that is also very interesting is the ability to customize and extend an external data source without changing the external data source. For example, let’s say that you already have an existing table of employees with email addresses. You then want to consume an external list of employee information from a Sharepoint site (yes, you can use Sharepoint lists as data sources). There is no relational references, such as an employee number or id for example, that you can use to link the two sources with. However the Sharepoint source contains email addresses of the employees. Within LightSwitch you can redefine the type of the list column containing the email address information to that of a business domain type of email address, without effecting the actual list in Sharepoint. Remember, you are updating the LightSwitch model representation of the data, not the meta model of the source it self. This makes the ability to link the two sources via the email address much easier. As well, you have the benefit of having LightSwitch validate the data.
Check out this Channel9 video for a great demonstration of this, and more!
Creating a screen based on your data entity. (courtesy Microsoft)
Check out this video on the Microsoft Showcase site to get a good overview of connecting to multiple data sources from within LightSwitch.
Connecting Multiple Datasources in LightSwitch (courtesy Microsoft)
I can’t wait to get my hands on the beta!
Jason Zander posted LightSwitch Architectural Overviews on 8/11/2010:
Last week we announced LightSwitch, a new tool for developers to make it easier to build line of business applications. You gave us a lot of great feedback, thank you for all of the comments and questions! One of the key requests was to provide more information about the architecture of the applications that are built with LightSwitch. The team has been working hard to publish a set of material on this which I wanted to share with you, including the top level architecture topology:
Hopefully you will find the following links useful:
- Channel 9 Video for Beyond the Basics
- Keynote Video
- MSDN Developer Center
- Architecture Overview Part 1
- Architecture Overview Part 2 – Presentation Tier
We will continue to post more information on the architecture and boundaries of LightSwitch. Expect to also see more tutorial information when the bits go live on August 23.
The Visual Studio LightSwitch Team (@VSLightSwitch) continued it’s LightSwitch architectural series with a lengthy The Anatomy of a LightSwitch Application Series Part 2 – The Presentation Tier post on 8/9/2010:
The client is responsible primarily for human interaction, data visualization and data entry. While these may sound simple, there really is a lot to it. The client should be easy to deploy and upgrade. It should have good looking, intuitive and functional visuals, including keyboard-accessibility. It must track data changes and updates and perform business logic such as data validation. It must handle threading and asynchrony, be responsive, and interact efficiently with the logic and data tiers. Even with all of the great .NET technologies available, there is a lot to putting it all together! The LightSwitch design philosophy aims to simplify creating great clients by providing a solid architectural foundation and letting the developer focus on the screens, data and logic that are specific to their business problem.
As Julius Caesar would say, the client est omnis divisa en partes tres. Or rather, the LightSwitch client is divided into three parts, one of which is the shell, the second is screens, and the third is the data workspace.
Even if you don't remember your Gallic War commentaries, let's divide and conquer to get a general understanding of the LightSwitch client. We'll cover the following areas.
- The LightSwitch Shell
- LightSwitch Business Objects and Business Rules (Screens and Entities)
- The LightSwitch Screen
- The LightSwitch Data Workspace
The LightSwitch Shell
The LightSwitch shell is made up of the hosting process, the shell UI, and the theming service.
Hosting Process
For a LightSwitch desktop application, we configure the application to allow out-of-browser install and we control the application startup sequence to ensure that the application is indeed installed and running in the out-of-browser host (sllauncher.exe). We configure desktop application to require elevated permissions. This gives LightSwitch desktop applications access Windows and Office automation and enables the built-in "Export to Excel" capability from within screens.
A LightSwitch browser application can be hosted within any browser and on any OS that Silverlight 4.0 supports. The application runs in the Silverlight sandbox. Access to OLE automation is not available (no "Export to Excel") and access to the Windows file system is restricted. LightSwitch handles the common problem where navigating away from the page or closing the browser would terminate the application when there are unsaved changes in a screen.
LightSwitch manages the application startup, upgrade, and shutdown sequences.
Shell UI
The application shell provides the root-level user interfaces for logging in, launching and activating screens. The shell UI is pluggable. LightSwitch comes with a default shell, but one can write a custom shell and plug it in via a LightSwitch Extension package. This design enforces consistent behavior in the application while allowing overall the look and feel to be highly customized.
An application shell is really made up of shell chrome and shell regions. Shell chrome determines what regions the shell uses and specifies their placement and layout. The standard shell includes following regions:
- screens region - the main area where all screens show up
- navigation region - for launching and activating screens
- commands region - where top-level commands are shown
- validation region - for displaying validation errors
The LightSwitch runtime does not put any restrictions on how many or what kind of regions are included in a shell, so other shells might include these and other custom regions. Each region in turn is composed from a view and view model. The view model is a non-visual component that exposes all of the data and behavior needed for the region and the view data binds to the view model and determines the region looks.
Read more about shell extensibility from a guy at Infragistics who has actually tried it out!
Regions use shell services to do their job. Shell services allow regions to obtain information about the application (i.e. provide access to application metadata), learn the current state of the run time (e.g. what screens are open, which screen is active), receive notifications about run time state changes (e.g. when user is attempting to shut down the application), as well as perform actions on behalf of the application user (such as launching new screen).
If this all sounds complicated, don't worry. The LightSwitch developer just uses the tool to pick a pre-built shell-and it just works! You can use the visual Studio Gallery to find and download shells that are published by the community and by vendors.
Theming Service
LightSwitch supports a basic level of theming to control colors, fonts, and to support the OS high-contrast themes. The intrinsic Silverlight controls support simple styling. LightSwitch goes one further by defining a set of styles as a theme and ensuring that the controls we use for the shell and in screens will select styles from the current theme. Themes are extensible and can be created and added via a LightSwitch Extension package.
Business Objects and Business Rules
LightSwitch business objects are the foundational components for defining the business logic of your application. There are two basic kinds of business objects: screens and entities. Screens represent a unit of work on the client (a client use-case) and entities represent an identifiable and durable piece of business data. When you define a business object in LightSwitch, you can also attach business rules to it. LightSwitch business rules help you to encapsulate the business logic where it belongs. No more putting all of your inventory update logic behind a button click event!
Business rules are associated with a business object or a member (a property or command) of the object. Business rules come in several categories:
- Computed Property - a read-only property on an entity or screen that is dynamically evaluated
- Validation Rule - a rule on an entity or screen property that returns a set of validation results based on the state of the object
- Read-Only Rule - a rule on an entity or screen property that determines the present UI read-only state of the property
- Can-Execute Rule - a rule on a command that determines whether it is presently available for execution
LightSwitch automatically evaluates and keeps track of these business rules. Evaluation is lazy and just-in-time: If no one is observing the state of a rule, the rule will not be evaluated. The rule is guaranteed to be up-to-date at the point you observe its value.
The result of a business rule is observable in the API from the Details property on the member to which the rule is applied. The Details property also includes other view-model state such as display name and description. Here are some example Details properties for a FirstName property and a PostOrder command.
customer.Details.Properties.FirstName.DisplayName
customer.Details.Properties.FirstName.IsReadOnly
customer.Details.Properties.FirstName.ValidationResults.HasErrors
screen.Details.Commands.PostOrder.CanExecuteBusiness rules are typically associated with a block of code-similar to an event handler. LightSwitch tracks the dependencies of the rule during runtime evaluation. In this way we know when a rule is "dirty" and must be re-evaluated. So long as the code refers to data elements defined on itself or another business object, we can track it properly. However, if you refer to something that is not defined on a business object, such as
DateTime.Now, we can't track that. But you can enforce re-evaluation of a rule programmatically.Some business rules are declarative and specified at design-time. There are several built-in declarative validation rules. Declarative validation rules are also extensible. You can create new ones and publish them via a LightSwitch Extension package.
Screens
A LightSwitch screen represents and independent unit of work on the client. Each screen has isolated state from other screens-there is no shared data between screens. Multiple screens can be opened simultaneously, but the shell keeps track of a single active screen.
LightSwitch screens are made up of three layers of objects that follow the MVVM pattern. The screen object is the model, encapsulating the data and business logic. The screen layout defines a tree of view-models that define the logical layout of the screen. And the visual tree is the resulting presentation view. The visual tree is made up of actual Silverlight controls that are data-bound to the view-models in the screen layout.
Screen Object
The screen object is a LightSwitch business object that contains the data and business logic for a given screen-but no UI. It defines a sort of data-type for the screen's internal state. At design-time, developer writes code against the screen object's class to provide custom business logic for the screen.
When defining a screen, the developer can add properties (scalar or entity values), parameters (scalar values), screen collections (query results), and commands. For each of these members, LightSwitch tracks detail properties such as display name, enable and read-only states, the data-loaded state, and validation results. The detail properties provide additional model state for the screen layout.
Each instance of a running screen owns its own isolated state. No two screens can share the same copy of an entity, but each can have its own copy. This is all managed by the screen's data workspace which is described in more detail below.
Screen Layout
The screen layout is a tree of content items. A content item is a view model that represents a single element on the screen. It forms a data-bindable bridge between the screen object (model) and the visual tree (view). Each content item may represent some data item, list, or command that is reachable from the screen object. It may also represent a container for child content items. Each content item specifies a control type and properties that govern the display behavior. The control type is just a name that will be mapped later to an appropriate visual control.
When a screen is created, the LightSwitch runtime builds the screen layout tree dynamically by interpreting the screen layout model that was persisted via the screen designer at design-time. Content items are accessible programmatically from screen code for advanced scenarios. In most cases, you will not need to access it to define the business logic for a screen, but it may be useful to control the presentation.
Screen Visual Tree
The visual tree is the actual tree of Silverlight controls that present the screen. The LightSwitch developer typically does not interact with the visual tree. LightSwitch builds the visual tree dynamically by walking the screen layout tree at runtime. The layout strategy specified by each content item node is used to activate the right Silverlight control. The control is then given the content item as its
DataContext. The control binds to various properties of the content item in order to display such things as the current value, display name, or to determine child elements to display.This dynamic layout strategy provides two important benefits. The first is professional 2D flow, spacing, control alignment, and automatic placement of labels. This is a huge time saver-no hours spent tweaking the layout margins, borders, alignment, etc. The second benefit is the ability to modify running screens when running from the LightSwitch tool. When the screen is in runtime design mode, the runtime editor simply modifies the screen layout and content items dynamically. The screen runtime can render the new visual tree dynamically and provide immediate live feedback to the developer.
Where's my XAML?
If you area already familiar with using the Silverlight presentation framework, you might be wondering where XAML is. The short answer is-there isn't any. LightSwitch developers aren't required to know XAML to build screens. LightSwitch saves the logical layout of the screen and dynamically creates the visual tree based on Silverlight controls that are registered for each data and layout type.
When you need more control of the UI, you can drop down to the XAML level by plugging in just about any Silverlight control as the presenter for any content item. You can even replace the entire screen. This feature allows you to wire-up an off-the-shelf control or to write your own. If you write your own, you can still inject LightSwitch data presenters and get the built-in presentation support for LightSwitch data elements.
Data Workspace
The data in a screen is managed by a data workspace. The data workspace tracks local copies of entities. An entity type represents a specific kind of data record. The entity encapsulates the business data, business logic, and durable storage for the data. The data workspace is responsible for fetching entities from a data service in the logic tier via entity queries, for tracking changes, and for sending updates back to data service for processing. The data workspace also tracks relationships between entities.
Data Service Client
When you add a data source to a LightSwitch application, we create a data service for accessing its data. The data services lives in the logic tier. The client accesses it via a data service client. A data workspace aggregates data from multiple sources by providing one data service client instance per data service.
For example, if you add a Northwind SQL data source and a MySharePoint data source, you'll see one data service client for each in the data workspace.
The data service client manages the entities and tracks changes (change set) for the corresponding data service. LightSwitch manages the granularity of updates in the following way: One screen à one screen Save command à one data service* à one change set à one update transaction.
(* In Beta 1, the screen's Save command operations over N data sources in an arbitrary order. This is not the final design. LightSwitch does not enforce correct ordering or updates for multiple data sources, nor do we handle compensating transactions for update failures. We are leaning toward selecting a single updatable data source per screen, where all other data sources represent non-updatable reference data. You can expect to see changes in after Beta 1.)
The data service client exposes entity query operations and update operations. In the following examples "Northwind" is a data service client.
var customer = this.DataWorkspace.Northwind.Customers_Single(customerId);this.DataWorkspace.Northwind.SaveChanges();Entities
Entities are the center of gravity for data and business logic in a LightSwitch application. These represent persistent business data such as a
Customer, anInvoice, aBookor aTimeCardEntry. In LightSwitch you can define new entity types and we will create the corresponding storage in SQL tables. Or you can import the schema from an existing data source and LightSwitch will define the corresponding entity types.Entities encapsulate business logic including validation rules, data update rules, and have attributes that drive defaults for visualization and data entry. Entities are also shared between the client and the logic tier, so that you write business logic such as validation once and LightSwitch can execute it on both tiers.
In the client runtime, LightSwitch uses a pair of objects to represent an entity instance. There is a hidden, low-level data-transfer-object (DTO) that integrates with the WCF RIA Services client. A there is a public business object that encapsulates the business logic and provides view-model support for data binding. The section above regarding business objects applies. The entity has business rules for data validation, computed properties, and computed readonly status. The entity object has an API for getting to the simple data values on the entity and a more advanced API for accessing the validation status and various other attributes such as the default display name and description of each member.
Entity Relationships
LightSwitch models and manages relationships between entities. We support multiplies of many-to-one, many-to-zero-or-one, and one-to-zero-or-one. We do not support one-to-one or have direct support for many-to-many. You can, however, manage many-to-many relationships by using your own join entity.
LightSwitch creates navigation properties on entities to manage the entity or entities on the other side of the relationship. An entity reference property points to a single entity and an entity collection property points to many. Navigation properties encapsulate the details of foreign keys, so you don't typically deal with those directly. LightSwitch handles loading the related entities on-demand when the navigation property is requested.
The LightSwitch team is well aware of performance concerns with loading data on demand from the client. This design provides a simple and powerful way to load data for screens and data needed with business logic. With power comes responsibility. We are currently working on ways to optimize data load that provides good client response times but also minimizes round-trips and load on the back end.
We are all familiar with relationships within a single data source. LightSwitch also supports inter-data-source relationships. These relationships cannot guarantee referential integrity, but they are very convenient for managing associated external data that shares a common value such as an ID or an email address.
Entity Queries
LightSwitch models a table in the storage tier as an entity set in the logic tier. The entity set is exposed by a data service as a query operation to the client.
Query operations support query composition. This means that when the client invokes a query operation, it can send over additional query predicates to further refine the filter and ordering of the query. LightSwitch uses query composition to compose queries from screens and from LINQ queries in client-side business logic. Under the hood LightSwitch uses WCF RIA Services to create the query request and serialize it to a
DomainServiceon the logic tier.The LightSwitch client attempts to parallelize query execution as much as possible to improve client response times. For example, when a single screen attempts to load multiple lists, the queries are executed asynchronously and in parallel. Likewise, when two different screens run, their queries (and updates) are executed in parallel. However, LightSwitch business logic code loads data from queries and navigation properties synchronously to keep the API simple to use and easy to understand.
We will look into what happens in the logic tier during a query operation in a later post.
Entity Updates
The data service client exposes a single
SaveChangesoperation. This operation packages up the change set for that data service and sends it to the logic tier for processing. LightSwitch implements aDomainService(WCF RIA Services) in the logic tier to process the update.LightSwitch ensures that the data is valid on the client before processing the SaveChanges operation. If the client data is valid, it is sent to the data service where it is validated again. If modified entities on the client have validation errors, the screen's Save command is automatically disabled to prevent the end-user from attempting to save bad data.
Just as with entity queries, there is a log going on in the logic tier to process the update. We will get into those details in a later post when we cover the logic tier.
Summary
We came, we saw, and we conquered the client. Well, actually, there is a lot more to be said about all of this. We've looked briefly at the shell, screens, and the data workspace. In our next posts we'll look at the logic tier and the storage tier. Later we can come back around and cover cross-cutting issues such as deployment, security, and for the truly curious, the client threading model. Feel free to post your questions and let us know what additional topics you'd like to learn about.
The LightSwitch Team posted The Anatomy of a LightSwitch Application Series Part 1 - Architecture Overview on 8/6/2010:
Here's an outline of topics we're thinking about:
- The Anatomy of a LightSwitch Application - Architecture Overview
- The Anatomy of a LightSwitch Application - The Presentation Tier
- The Anatomy of a LightSwitch Application - The Logic Tier
- The Anatomy of a LightSwitch Application - The Storage Tier
- The Anatomy of a LightSwitch Application - Extending a LightSwitch Application
- The Anatomy of a LightSwitch Application - Deploying the Application
With that, let's get started...
The Anatomy of a LightSwitch Application - Architecture Overview
Microsoft Visual Studio LightSwitch applications are built on a classic three-tier architecture. Each tier runs independently of the others and performs a specific role in the application. The presentation tier is responsible for human interaction with the application. Its primary concern is data visualization and editing. The logic tier processes requests from a client to fetch data, update data, or to perform other operations. This tier’s primary role is to shield direct access to the data from unwanted or invalid reads and updates. This helps to ensure the long-term integrity and security of the data. The data storage tier is responsible for durable storage of the application data.
A typical three-tier application
Designing a new three-tier application from scratch can be difficult and complex. Each tier presents a myriad of technology choices and internal system design. Each has distinct deployment and manageability concerns. Each must handle communications with the adjacent tier, and ensure secure access from trusted sources.
LightSwitch removes the complexity of building a three-tier application by making specific technology choices for you. You concentrate on the business logic and not the plumbing.
When we map the specific technologies used in LightSwitch to this architecture you find that the LightSwitch presentation tier is a Silverlight 4.0 application. It can run as a Windows desktop application or hosted in a browser. The LightSwitch logic tier exposes a set of WCF RIA DomainServices running in ASP.NET 4.0. The logic tier process can be hosted locally (on the end-user’s machine), on a IIS server, or in a Windows Azure WebRole. A LightSwitch application’s primary application storage uses SQL Server or SQL Azure and can consume data from existing SharePoint 2010 lists, databases accessible via an EDM provider and custom build WCF RIA DomainServices.
A LightSwitch 1.0 three-tier application
As you can see, a LightSwitch application is built on top of existing .NET technologies and proven architectural design patterns
Matthew Van Horn of Infragistics posted his LightSwitch Technical Breakdown analysis of LightSwitch’s Model View ViewModel (MVVM) architecture on 8/5/2010:
Those of you who haven’t had the pleasure of playing with LightSwitch for over a month might be wondering what lives under the hood. I would like to take this opportunity to let you know what you can expect when you start developing for LightSwitch. Please note that I said developing for and not developing with, as this post is aimed at developers who will write add-ins, not apps. This is just a high-level appetite wetter for several more detailed posts coming soon to a browser near you.
Those of you who like MVVM will feel right at home in the LightSwitch environment. The view models which LightSwitch provides for you should be enough to get most shell developers up and running. For shell developers, here is a quick rundown of the view models you will be working with, as well as what they will do for you:
View Models
- ActiveScreensViewModel – This view model tells you the screen with which the user is currently working.
- CommandsViewModel – This view model has a list of your various commands, complete with everything that you would need to display and invoke them, including the image for the command.
- NavigationViewModel – This view model contains a list of navigation items broken down into groups.
- ScreenValidationViewModel – This view model is used to keep you informed of validation issues on a screen.
Commands
Given how much LightSwitch does to keep the UI thread free of long-running commands (such as loading data or activating any screen) you will find that you will be using a dispatcher that will keep the UI that you build responsive, while LightSwitch’s framework does the user’s bidding in the background.
Calling a command is accomplished as follows:
var button = sender as Button;
var shellCommand = (IShellCommand)button.DataContext;IBusinessCommand businessCommand = (IBusinessCommand)shellCommand.ExecutableObject;
IDispatcherObject dispatcher = (IDispatcherObject)businessCommand;
dispatcher.Dispatcher.BeginInvoke(delegate()
{if (businessCommand.CanExecute)
businessCommand.Execute();
});Above you can see how to handle a button click from a button that has a data context of one of the commands from the collection of shell commands from the CommandsViewModel. This is how to launch screens with LightSwitch.
LightSwitch Earns points on Services
In LightSwitch, almost anything that you want to know can be figured out through one of the many services or providers offered by the SDK. LightSwitch has one interface with which you will be spending a lot of time. This interface offers up the above-mentioned view models, as well as the ModelService, NotificationService, and ScreenViewService.
Notifications
In a nutshell, LightSwitch offers an eventing system with the notification service. This is used to inform the shell and framework when a user changes the active screen, a screen changes (data entered), or a new screen is activated.
As tech lead on this project for Infragistics, I got very close to the LightSwitch framework and I have to say that while many people are calling this a reboot of Access, I can assure you that this is not the case. It could certainly be said that LightSwitch leverages many new and powerful tools from Microsoft to accomplish a lot of what Access could be stretched to do. However, Access was clearly not designed to stretch as far as many developers have stretched it in the past.
Over the next few weeks, several of my colleagues on this project will blog about their perspectives and lessons learned during development of our recently built shell.
I will be posting more details later but feel free to ask questions and I'll do my best to answer them all.
<Return to section navigation list>
Windows Azure Infrastructure
Mary Jo Foley continues her Windows Azure overview with Windows Azure one year later: Walking the cloud vs. on-premises tightrope post of 8/12/2010:
Microsoft’s Windows Azure development engine is chugging along, following closely the original path the team outlined for its cloud operating environment. But the marketing and positioning of Azure isn’t tracking as smoothly.
As I noted in part one of my ‘Azure year in review’ post yesterday, Microsoft has made a number of under-the-cover additions to Azure. Microsoft is now in the position of leading with its cloud on the Windows development front. In other words, new Windows Server features are developed for and launched in Windows Azure before they make it into a new release of Windows Server software.
Microsoft is in the process of attempting to lead with the cloud on the Azure marketing front, as well. But its messaging here is murkier. Company execs abruptly stopped touting “Software + Services” as the way Microsoft is going to market with its products. Chief Operating Officer Kevin Turner recently told Wall Street analysts that playing up customer choice between software and cloud services wasn’t a good move strategically and that Microsoft is now “course-correcting” by convincing customers to go with its cloud wares over its on-premises software.
The problem is Microsoft makes far and away most of its money from software, not services — and is likely to do so for a number of years ahead. Despite Turner’s claims “the cloud actually helps better position Microsoft to sell more on-premise products than we ever have before,” there’s little to back up that assertion. Microsoft officials have made a case for how/why leading with the cloud offers the company new opportunities to reach brand-new customers, but haven’t offered any real evidence that the cloud will generate the high margins that software sales traditionally have.
In short, Microsoft can’t — and won’t — suddenly turn the switch and stop selling software in favor of cloud services. In spite of the corporate edict to “lead with the cloud,” officials still seem conflicted about how to tell customers they still have a choice. Example: Microsoft is touting its recently announced Windows Azure Appliances as the way large enterprise customers will be able to run a private Azure cloud at their sites. But it also is continuing to offer its Dynamic Datacenter Toolkit, Windows Server AppFabric, BizTalk Server and other software components the building blocks for customers desiring their own private clouds.
In spite of the company’s “We’re All In” cloud messaging, you still find Microsoft using charts like the one below that highlight the software and services options available to users:
When I asked the Softies about the challenges of positioning Windows Azure and cloud services vs. Windows Server and on-premises enterprise software products, I received this statement from a company spokesperson:
“Ultimately, it’s up to the customer if, when and how they move to the cloud. We see it as our job to provide the choice and the right solutions, guidance and tools to enable them to use their existing skills and achieve the fastest time to business value. For those organizations and businesses that want a highly optimized, scalable environment where we prescribe the hardware and normalize the cost of operations, that’s our services platform, the Windows Azure platform. For those that want the versatility to enable environments of any scale, or that need custom hardware configurations and operating models, that’s our server platform, built on Windows Server.”
Page 1 of 2 Next Page: Customer choice vs. confusion. Click here to read Mary Jo’s take on lower pricing for developers, VM Role capability, Project “Sydney” and dogfooding by Microsoft’s IT teams.
Petri I. Salonen posted Law #8 of Bessemer’s Top 10 Cloud Computing Laws and the Business Model Canvas –Leverage and monetize the data asset on 8/11/2010:
I am now in the eight law in Bessemer’s Top 10 Computing Laws with an emphasis how to be able to monetize and utilize the data asset that the SaaS vendor is accumulating by users that are generating a wealth of information and all of this information is accessible for the SaaS vendor if the contractual agreement allows the use of this data.
When reading about this law my memories from my youth crept to my mind when studying in the Swedish School of Economics in Helsinki (Hanken) where each student in the accounting/management accounting class had to analyze public financial statements from Finnish companies as part of the advanced classes. This analysis was then used to compare different companies in the same vertical and to create some base-line standards for analyzing and comparing companies. This analysis gave me a perspective on accounting and how it really works. This learning has helped me throughout my life and especially now as an entrepreneur. The power of cash-flow statements can’t be underestimated which has been a reality for many software vendors the last couple of years.
However, during that time (1984-1989), the data was stored in Lotus Symphony spreadsheets, so most of the analysis and data entry was manual and very labor intensive. Companies at that time were still using green screen terminals and only a few companies started deploying personal desktops, but laptops were still not on the market as far as I remember. The first real PC with MS-DOS that I used was Italian Olivetti with two floppy stations. I can’t believe how far we have come since then.
This law (#8) is a reminder to me of the changes in business models and how cloud can provide new business model opportunities for organizations. Ten years ago, the notion of “data asset use” was very rare and if you had the possibility to accrue information from companies in the same vertical, you had a tremendous asset in your hands. I know a company owner in the US that sold his company for a few millions by having collected financial data for financial institutions over time and then sold the assets to a larger company as part of an exit. Let’s view how this law impacts software businesses and business model canvas overall.
Law #8: Leverage and monetize the data asset
Imagine having access to a massive amount of data that your clients are generating in your cloud application. Image to have access to performance metrics for a given vertical market segment such as banks at large and their performance. Wouldn’t it be valuable for somebody to be able to compare banks with each other from different perspectives and maybe even sell this data to external parties? Imagine being able to service this through Microsoft Dallas service that enables developers and information workers to easily discover, purchase, and manage premium data subscriptions in the Windows Azure platform. Maybe this collective information serves as benchmark information for other companies in the same vertical and enables these companies to compare how well the perform in their respective business? The Bessemer blog entry gives examples of this kind of benchmarking by referencing the leading expense management software company Concur where companies in the same peer group can compare for example expense costs such as travel with other companies.
Another successful example is the company Mint.com that used to compete with Intuit Quicken by providing a free SaaS solution for consumers to track expenses automatically. The Mint story is very interesting; it was started by Aaron Patzer, Matt Snider and Poornima Vijayashanker in Mr. Patzer’s apartment. Two years later the company was acquired by Intuit for $170 million. The story is simple and amazing at the same time. The founders felt that it was too difficult to track personal finances online and that people had to spend far too much time in setups and entering data. The founders felt that once the Mint.com is linked to the credit card statements and bank accounts, most of the daily stuff should happen automatically. I found out about this service as part of my own research and decided to test it out. It is as easy as Mint.com claims and today I get updates from the system on regular basis. The business model that Mint had initially was to use the consumer data to propose better credit card deals etc. to generate leads for credit card companies etc. It is obvious that Mint.com became a threat to Intuit so they decided to acquire them. Mint.com success has led Intuit to replace Quicken Online product in favor of Mint.com. Let’s analyze how this law can be viewed from Business Model Canvas perspective.
Petri concludes with a “Summary of our findings in respect to Business Model Canvas” topic.
R “Ray” Wang offered Tuesday’s Tip: 10 SaaS/Cloud Strategies For Legacy Apps Environments on 8/11/2010:
Legacy Apps Customers Seek Practical Advice
Organizations determining when and how to make the move to SaaS and Cloud face realistic challenges in gaining buy-in and realizing the apparent and hidden benefits of SaaS/Cloud. In a recent survey of over 300 companies, 73 respondents who were wary of SaaS/Cloud were asked to list the top 3 reasons they did not plan to deploy a SaaS/Cloud solution in the next 12 months (see Figure 1). The top 3 reasons related to legacy environments, org structure, and governance include:
- Legacy apps CIO’s. CIO’s vested in protecting the existing investments may often proceed with caution for SaaS and Cloud solutions. In some cases, sunk cost mentality takes hold and the goal of being 100% pure with a single vendor clouds the vision to meet needed business requirements.
- Burden of legacy apps. Legacy apps maintenance and upkeep represents a key barrier to SaaS and Cloud adoption. Organizations often remain complacent about maintenance and upgrades, preferring to avoid substantial changes and risk. Becuase the money and resources to support legacy apps consume most of the budget, organizations have little funds for innovation and experimentation. Eventually, business decision makers procure SaaS/Cloud solutions to by-pass IT.
- No IT team buy in. Many constrained IT teams have not taken the time to understand the requirements to support SaaS and Cloud apps in a hybrid mode. SaaS requires organizations to revisit SOA strategies, integration requirements, and master data management. Business leaders and decision makers often overlook these dependencies at the organization’s long term expense.
Figure 1. Legacy Issues Hamper SaaS/Cloud Adoption
SaaS/Cloud Strategies Must Transcend The Burden Of Legacy Apps Over Time
Next generation apps strategies must account for both a future of hybrid deployments and growing independence of legacy apps. Ten of the most common go forward strategies include:
Ray continues with narrative descriptions of the following 10 strategies:
Figure 2. Ten SaaS/Cloud Adoption Strategies Span Business Requirements And Legacy Adoption
Ray concludes with “The Bottom Line For Buyers (Users) - Proceed With Cloud/SaaS Strategies Based On Business Requirements Not Hype.” Read Ray’s entire post and comments to it here.
<Return to section navigation list>
Cloud Security and Governance
Bridget Mintz Testa reported “CloudAudit & AICPA Independently Tackle Cloud Security Controls” as a preface to her The Risks Of The Cloud post of 8/13/2010 (apparently, her time) to Processor Magazine:
Without audits and controls, it’s impossible to tell if an enterprise’s financials are in order; if an organization operates safely; or if a third-party service provider, such as a payroll company, is securing its customers’ information. Existing IT audit requirements and controls may not apply or may be insufficient and can potentially make it hard for customers to find out what controls a cloud service provider has in place. Consequently, customers are notoriously anxious about the safety and security of using the cloud.
To address the issue, the CSA (Cloud Security Alliance) created CloudAudit (www.cloudaudit.org), formerly known as A6 (Automated Audit, Assertion, Assessment, and Assurance API). Headed by Christofer Hoff, director of cloud and virtualization at Cisco and a technical advisor and founding member of CSA, CloudAudit will use the work of the Trusted Cloud Initiative—also a CSA effort—to create an automated method for service providers to show customers the controls they have in place.
Simultaneously, the AICPA (American Institute of Certified Public Accountants), aware of the misuse of its SAS 70 in cloud computing, is stepping up to the issue with new guidance for an appropriate existing standard. With all of these efforts, cloud security challenges may finally be addressed.
Existing IT Controls
Controls are essentially rules to ensure that enterprises follow proper financial, safety, security, business, and other practices. A rule that requires signatures of two execs on a check that exceeds a specific amount is an example of simple financial control. Common IT controls include rules governing access to information by employees, the powers granted to a systems administrator, or the types of storage devices allowed in sensitive areas. Audits ensure that the controls an organization claims to have are in place and work properly.
COBIT and ISO/IEC 27002:2005 are two well-known IT controls. COBIT was developed by the IT Governance Institute (www.itgi.org), a nonprofit organization created by the ISACA (Information Systems Audit and Control Association), to cover the development of controls for IT policy and practice in enterprises. ISO/IEC 27002:2005 was created by the ISO and the IEC so organizations could establish controls for good information security.Although these predate cloud technology, they “can apply to cloud audits,” says H. Peet Rapp, CISA, an independent information security consultant. They aren’t necessarily sufficient, however.
“Evaluation, certification, and auditing all tend to dwell on best practices,” says Jay Heiser, a research vice president at Gartner. “For Windows and Linux, we know their vulnerabilities. But for this new [cloud] platform, this new [cloud] computing model, we don’t know those risks. We don’t know if their creation, design, and build are secure. I don’t see any of [these existing IT controls] addressing these problems.”
The ISACA and other organizations, which could easily work independently, have instead joined the CloudAudit/A6 working group. Evaluating the appropriateness of existing IT and other relevant controls isn’t the goal, or at least not an initial goal, of CloudAudit. It aims to help cloud customers by helping service providers.“The problem we’re trying to solve is that you’re giving up direct control of your infrastructure to a third party, and you lose a lot of the transparency that security, audit, and compliance people are used to having,” says Hoff. “So [customers] send off these 600-page documents to cloud providers to get them to answer [questions about their controls]. The providers pull that together, spending a lot of money and time, and the user waits a long time.”
The Trusted Cloud Initiative is compiling a list of existing IT and other security standards and controls that may be applicable to the cloud. Known as the CSA Audit Matrix, version 1 is already available.
Cloud service providers may implement one or more of these control frameworks, and CloudAudit’s goal is to establish a common way “to automate the exchange of this information,” Hoff says. CloudAudit is creating a series of directories of the control frameworks. Using these directories, service providers can automatically check off the controls they use. Customers can automatically query the directories to find out what controls the service providers have in place. The first version of CloudAudit’s directories should be out by the time you read this.
SAS 70 is a standard used by auditors when the financial controls of their clients are outsourced, explains Chuck Landes, vice president of professional standards for the AICPA. For example, if an auditor’s client has outsourced its payroll to a third-party provider, the auditor needs to be sure that the third party has sufficient controls in place because they affect the auditor’s client.“A lot of people over time have misused SAS 70 for reasons never intended,” says Amy Pawlicki, the AICPA’s director of business reporting assurance and advisory services. “Third-party service providers are trying to say that current and prospective customers can trust that they have the controls in place for security, availability, privacy, confidentiality, and processing integrity. The standard covering that is AICPA Attestation 101. AT 101 isn’t restricted to financial controls [as SAS 70 is]. It covers controls regarding security, availability, privacy, confidentiality, and processing integrity. Third-party providers have tried to use SAS 70 instead of AT 101.”
The AICPA is developing a new guide for AT 101, which is a standard that has existed for a long time. “The new guide provides guidance to an auditor for testing security, availability, processing integrity, confidentiality, and privacy,” Pawlicki says. The guide should be available by the end of this year.
Cloud computing isn’t risk-free today. With the Trusted Cloud Initiative, CloudAudit/A6 and AICPA’s new guidance on AT 101, it’s possible that cloud computing’s risk will be at least reduced within a few years.
From a sidebar:
SAS 70 & Risk
“SAS 70 wasn’t intended to assess risk,” says Jay Heiser, research vice president at Gartner. “It isn’t even meant to be for prospective clients. It’s meant for auditors, for providers, and existing customers, not for sales and marketing. Some providers are hiding behind that provision, saying they can’t use it for sales and marketing, but we have it.” Having SAS 70 is essentially meaningless, however, because a company can choose which controls SAS 70 checks.A service provider claiming to be “SAS 70-certified” is also meaningless. “There is no such thing as ‘SAS 70-certified’ because every SAS 70 is different,” says Amy Pawlicki, director of business reporting assurance and advisory services for the AICPA. That’s because “every service organization’s controls are different,” she says. “SAS 70 is only relevant to the customer auditor and manager, not to anyone else.”
<Return to section navigation list>
Cloud Computing Events
Maarten Balliauw announced the Belgian Azure User Group’s AZUG September Event on 9/14/2010 at 6:30 PM in the Ordina (Mechelin) office:
AZUG is organizing its second event in 2010. Come join us for two sessions on one evening!
Azure pricing and SLA’s explained by Kurt Claeys
This session provides details about the pricing model and the service level agreements for Azure hosted services, Azure storage, data transfer, appfabric messaging and SQL Azure databases. We discuss the different purchasing models and consumption prices and see which metrics are used to calculate the total monthly price. We see examples of price estimations for various scenarios and their SLA’s and provide tips to minimize the total costs and avoid costly surprises. We also have a look at some TCO analyzer tools and the billing portal.
Running in the cloud: real-life Windows Azure by Maarten Balliauw
In this session, an overview of the first Windows Azure project in Belgium will be given: ChronoRace. Learn about the needs of the customer, opportunities in the cloud model and about technical pitfalls and solutions that were made to host live time tracking and streaming video for the 20km through Brussels, including automatic up- and downscaling for Windows Azure.
Location: Ordina (Mechelen office), Blarenberglaan 3B, B-2800 Mechelen
Route details can be found on the Ordina website. Make sure to print these out as most GPS systems do not navigate to this address correctly!
Dario announced that Devs4Devs is back in Johannesburg! on 7/29/2010:
It’s been a while since we ran our last Devs4Devs event, but it’s coming back!
Devs4Devs is - to turn a cliché - an event by the community for the community. Anyone can participate and presenters can volunteer to present a session on any developer topic. Attendees get to hear from their fellow developers about something new and cool in a short, 20 minute presentation by one of their peers. Presenters get to experience what it’s like to prepare for and deliver a technical presentation for an audience.
Devs4Devs will run on 4 September 2010 from 9:00 AM to
12:00 PM12:30 PM at the Microsoft offices in Johannesburg. What better way to spend your Saturday morning than in the company of your fellow developers, listening to some great technical presentations and networking with your community peers?This is a free event and we’ll be providing breakfast and tea and coffee during the break and may even have some cool giveaways…
For Attendees
Register now to let us know you’re coming so that we can arrange catering.
For Presenters
We have timeslots for up to
1214 presentations and we’ll update this post as people submit new sessions. If you’re interested in presenting on a topic (any development related topic will do) we’d love to have you! Remember, this is a relaxed environment and you’re in front of your peers. This is a great way to build up your confidence if you’re new to presenting, or, for experienced presenters to practice and polish up their skills.
If you’d like to presentcontact usand let us know what topic you’d like to talk about, provide a description and get preparing!If you’d like to present at Devs4Devs keep your eye on our blog for the next event – we’ll try and ensure you get an opportunity to show your stuff!Agenda
We’ve had an overwhelming response and our mailbox has been full of great topic suggestions. We’ve decided to include extend the day by an extra 20 minutes and now we’ll be accommodating 14 sessions! Thanks to everyone who’s submitted a presentation and, for those who missed out, try again next time we run Devs4Devs!
For everyone attending: we’ve got some excellent content for you to choose from, so be sure to register your place before you miss out.
Following are sessions on Windows Azure and ASP.NET Dynamic Data, which is still alive and well in VS 2010:
| Session Title | Session Description | Presenter |
| Getting ready for Windows Azure development | Heard about the cloud? Excited about the possibilities? In this session we have a 1000-mile introduction to Microsoft’s operating system for the cloud, Windows Azure, how it compares to the other cloud offerings that are out there and how to get your hands dirty with the skill-up process. Endless possibilities + new tech = fun stuff. | Ryno Rijnsburger |
| ASP.NET Dynamic Data |
I will briefly introduce ASP.NET Dynamic Data by showing how to build a complete data maintenance web application with almost zero code. Moving on, I will demonstrate some standard ways of customising a Dynamic Data application, and some more advanced non-standard customisation techniques. I will finish off by illustrating how Dynamic Data libraries and controls can be leveraged in other applications that don't normally use dynamic data. |
Jaco Pretorius and Kobus Brummer |
Alistair Croll on 8/12/2010 issued a Cloud Connect [2011] Call for Papers by 9/30/2010:
Welcome to the Cloud Connect 2011 Call for Papers.
Cloud computing is the biggest disruption in IT of the past decade. It represents nothing less than the democratization of IT, putting the power of the world's computers at anyone's fingertips. On-demand, pay-as-you-go computing platforms make previously unthinkable applications commonplace. Cloud are forcing enterprises to break apart IT monopolies. And they're giving birth to entirely new industries even as they tear down old ones.
And Cloud Connect is the place to see it happen.
The inaugural Cloud Connect, held in early 2010, was the industry's largest conference on cloud computing. Over 2,000 attendees and 100 speakers came together for four days of workshops, panel discussions, hands-on demonstrations, and presentations.
We were thrilled with the response to the event, which balanced provocative, fast-paced discussions with in-depth technical expertise. And now it's time to plan next year's event. Cloud Connect 2011 promises to once again be the defining event on the IT calendar. We want to hear your suggestions for presentations, demonstrations, and debates. We welcome vendors and users, pundits and practitioners.
When suggesting a topic, please make sure:
- Your content is interesting, nonpartisan, and product agnostic. There's an expo hall for sales pitches; our content is what makes or breaks the event, and we take it very seriously.
- You use real-world examples and case studies wherever possible. We're beyond the era of hypotheticals and abstract taxonomies, and our audience wants to know what's really happening.
- You provide contact information for the actual presenter within the submission. We don't like cut-and-paste submissions from agencies, and are much less likely to choose them.
- You keep the submission short and to the point. If it takes too long to explain, it's likely that attendees won't understand what it's about and won't attend that session
If you have any questions regarding speaking submissions, please contact Manuela Farrell at (415) 947-6250 or mfarrell@techweb.com. Thank you for your interest in Cloud Connect.
Alistair Croll
Conference ChairThe deadline for submissions is September 30th, 2010.
<Return to section navigation list>
Other Cloud Computing Platforms and Services
Bruce Guptil asks HP Scenarios After Hurd’s Departure: Cloud IT Master, or Enabler? in this 8/11/2010 Research Alert for Saugatuck Technologies (requires site registration):
What is Happening? Mark Hurd, Chairman and CEO of HP, resigned effective August 7 2010 after an internal investigation found evidence that Mr. Hurd falsified expense reports and “acted inappropriately” with a contractor hired to assist in HP’s marketing.
While Saugatuck sees Mr. Hurd’s departure as potentially disruptive to investors, customers and partners in the short term, we also see it as a chance to shift HP’s corporate vision and direction to embrace, profit more from, and possibly dominate Cloud IT. …
Bruce continues with an analysis of HP’s cloud-computing positioning in the usual “Why is it Happening” and “Market Impact” sections.
Max Indelicato describes Designing Web Applications for Scalability in this 8/12/2010 post:
I can’t even count the number of times that I’ve heard this phrase: “don’t worry about scaling your web application, worry about visitor (or customer) acquisition.” My response to this is always that you don’t need to choose one or the other, you can do both!
In this post, I’m going to go over some of the strategies I’ve used to architect web applications for scalability, right from the start of the design process, in such a way that I’m prepared to scale when I need to, but not forced into doing so before its necessary. Easing the transition from small scale to large scale can be made much easier by choosing the right technologies and implementing the right coding patterns up front.
Use the Right Tool for the Job
I’m going to start this off by reiterating the age-old recommendation: Use the right too for the job.
Using the right tool for the job is crucial when it comes to the success of your web application’s ability to scale linearly over time. The technology stack, server environment, and your target audience all play a role in deciding which specific software will serve you best in your quest to support an ever evolving web application.
Some examples of coming to the game as prepared as possible are by using some of the following:
- Amazon EC2, SoftLayer, or a similar cloud service offering
- Using a data storage solution with scalability features built in like MongoDB, CouchDB, Cassandra, or Redis
- Adding a caching layer to your application that can be expanded upon as the need arises by using software like Memcached.
The above examples are all pre-packaged building blocks that vastly improve your chances of success when the need to scale arises. All three have scalability at the core of their creator’s original intent.
Choosing Your Data Storage Setup
Nearly all web applications are data driven in some form or another. Choosing your data storage setup carefully will therefore prove to be quite important relative to most other components of your web application. Your options here are many, but some good rules of thumb are as follows.
Partitioning (a.k.a sharding)
Partitioning allows you to split your dataset up amongst a series of servers so that you can distribute read and write load. This is a particularly important point in that whether or not you end up choosing a solution that has this built in, as I’m suggesting here, or you go with something more traditional (MySQL) and end up writing in partitioning at the application level, you’ll be doing partitioning at some point during your web application scaling process.
Some examples of data storage solutions that include a partitioning feature are:
- MongoDB
- Cassandra
- Redis
- Tokyo Cabinet
- Project Voldemort
Update: Removed CouchDB from the above list. Thanks Alex for pointing out the error!
I’ll be using the terms “partitioning” and “sharding” interchangeably throughout this post and later “node” and “shard” interchangeably as well.
Replication
Most data storage solutions, with advanced functionality, include a replication feature of some sort. The advantage of replication in our case is not for scaling reads and writes, but for ensuring a high level of availability.
MongoDB, for example, includes the concept of replica sets as a feature separate from its auto-sharding (auto-partitioning) feature. That is to say that it does not automatically replicate its sharded nodes’ data to other nodes in case of node failure. Cassandra, on the other hand, auto-replicates data amongst its shards so that you’re free to not worry about the specifics of high availability – failover is handled automagically as well.
Elasticity
Lastly, it is preferable to pick a solution that supports the addition and removal of individually sharded or replicated nodes. Ideally, we want the ability to add nodes to our data storage solution and have data automatically moved to utilize new node resources as demand increases. In the case of a decrease in demand, say after a flash flood of traffic that resulted from hitting the front page of Digg, we want to be able to remove nodes and have the data hosted on those removed nodes transition over to other existing nodes. This feature is also very helpful when maintenance is required on a node as a result of a hardware failure or upgrade, a large scale schema change, or any number of reasons that a node might require downtime.
Design Your Schema with Scalability in Mind
Assuming we’ve chosen a data storage solution that supports linear scalability through its feature-set, we can move on to designing our schema in such a way that it can take advantage of these features.
As an example, let’s use this simple typical User schema defined in pseudo-code. This is something we’d likely see in a web application that has a User component:
Now let’s assume that our application has a User Feed feature, similar to Facebook’s News Feed, where a User can post to their feed on their profile for others to see. We’ll use the following UserFeedEntry to define the schema that will hold each user feed entry:
Now that we’ve defined a simple schema typical of most user-based web applications, let’s talk about how we might partition (or shard) this schema to distribute read and write load across a series of nodes (or shards).
First, it’s necessary to define a key that will be used to partition on. In general, we’ll want to choose a Field from our Collection that conceptually defines itself as an independent object. That is to say that it is a root Collection within our schema and most other Collections are directly related to it in some way. It’s also important to consider what the distribution will be of the chosen key. For example, if we choose a key that is small in number (1 – 10), then too many of its related Collections will be included and the density will work against us – too much data within a single shard is bad.
Let’s pretend that the User Collection’s UserId Field is unique, therefore having good distribution, and that there is a fairly good distribution of related UserFeedEntries per UserId. This being so, the UserId is a good key to partition on.
By partitioning on the UserId field, in both the User Collection and the UserFeedEntry Collection, we’ll be clumping the two related data chunks together on the same node. All UserFeedEntry entries with a UserId of xxx-xxx-xxx-xxx will be contained on the same shard as the User entry with a UserId value of xxx-xxx-xxx-xxx.
Why is this scalable? Because our requirement for this application is perfect for this distribution of data. As each visitor visits a User’s profile page, a request will be made to a single shard to retrieve a User to display that user’s details and then a second request will be made to that same shard to retrieve that user’s UserFeedEntries. Two requests, one for a single row and another for a number of rows all contained on the same shard. Assuming most user’s profile gets hit about the same amount throughout the day, we’ve designed a scalable schema that supports our web application’s requirements.
Wondering what we should do if we get hot spots on a particular User’s UserFeedEntries? Partition the User’s UserFeedEntries to avoid that and/or duplicate the data amongst multiple shards. Also keep in mind that if you predict that hot spots will occur on any given key, avoid partitioning on that key if you can. I’m contemplating writing up system architecture designs using a different storage solution every time in such a way that the architecture highlights the data storage solution being used i.e. a scalable MongoDB solution, a scalable Casssandra solution, a scalable MySQL solution, etc.
As a side note -
I find myself wanting to get as detailed as possible so as to hit on many of the fringe cases that people get hung up on when implementing scalable architectures in the real world, but then these posts turn into novels and I worry I’m losing my readership by doing so. Disagree? Let me know in the comments please!Implement Application Abstraction Layers
The next step in the process of ensuring scalability is a core feature or your web application’s architecture is to consider the architecture of the code you’re writing. Adding layers of abstraction goes a long way towards providing flexibility both for salability and for feature addition. I like to use the following three layers, inserted amongst the various other layers of your application that are specific to the technology stack or design pattern (MVC, etc).
Repository Layer
The Repository Pattern is common in the web application world, so I won’t delve into what it is. However, it’s important that we touch on some of the reasons why the Repository Pattern is particularly applicable to providing scalability.
Repositories normally wrap a single concept’s (table, collection, object, etc) persistence access. I’ve seen various implementations, all with their pros and cons, but here are some guidelines that I try to stick to:
- Don’t name methods in a manner particular to the data storage you’re abstracting. For example, if you’re abstracting relational storage, its common to see Select(), Insert(), Delete(), Update() functions defined for performing SQL queries and commands. Don’t do this. Instead, name your functions something less specific like Fetch(), Put(), Delete(), and Replace(). This will ensure your’re more closely following the Repository Patterns intent and make life easier if you need to switch out the underlying storage.
- Use Interfaces (or abstract classes, etc) if possible. Pass these interfaces into higher layers of the application so that you’re never actually directly referencing a specific concrete implementation of the Repository. This is great for building for unit testability too because you can write alternate concrete implementations that are pre-filled with data for test cases.
- Wrap all of the storage specific code in a class (or module, etc) that the actual Repositories reference or inherit from. Only put the necessary specifics of an accessor function in each function (query text, etc).
- Always remember that not all Repositories need to abstract the same data storage solution. You can always have the Users stored in MySQL and the UserFeedEntries stored in MongoDB if you wish, and Repositories should be implemented in such a way that they support needing to do this down the road without much overhead. The previous three points indirectly help with this as well.
Following the above guidelines will make refactoring easier and allow you to mix and match data storage solutions to help scale features separately as needed. Most of the above is a reminder of web application best practices, but I hope it’s helpful to think them out as design points that promote scalability as well.
Caching Layer
Caching is a technique that most web applications require at one point or another. Building in a simple caching layer from the start is prudent planning on many levels. I often start with simple page (or View, etc) level caching or Service Layer caching as these are two areas where it’s not uncommon to see state change infrequently.
At a later point in an application’s lifetime, you can evolve this simple caching into something that scales beyond the web server. Moving your cache onto a distributed solution, like Memcached, is a good way to achieve this. Ideally, minimal code will need to be changed outside of the Caching Layer abstraction if we’ve done our job well.
Service Layer
A Service Layer can take many forms. Sometimes it simply encapsulates Repository access logic, sometimes it encapsulates more complex processing logic, and sometimes it encapsulates out-of-process calls. The later form is what we’re preparing ourselves for by ensuring a Service Layer is sufficiently abstracted such that scaling a Service (perhaps specific to a feature that is seeing too much activity) from its initial web server resident implementation to an independent out-of-process call (i.e.Web Service, etc) when the need arises.
Building your Service Layer while keeping the above point in mind, will help ease the pain of scaling later on.
Closing Thoughts
I’m interested in getting feedback as to whether this kind of high-level overview is helpful in building an understanding of how web applications should be architected for scalability. Minimizing massive refactorings that hit every level of an application is the goal here and I’ve used the above guidelines to such an end successfully myself. I hope these guidelines help you prepare your applications for the inevitable traffic rush!
<Return to section navigation list>
Atom 1.0 site feed
0 comments:
Post a Comment