Windows Azure and Cloud Computing Posts for 8/19/2011+
| A compendium of Windows Azure, SQL Azure Database, AppFabric, Windows Azure Platform Appliance and other cloud-computing articles. |  |
•• Updated 8/20/2011 8:50 AM PDT with a UI screen capture of George Huey’s SQL Azure Federation Data Migration Wizard v0.1 in the SQL Azure Database and Reporting section below.
• Updated 8/19/2011 4:30 PM PDT with articles marked • by George Huey, Avkash Chauhan, Paul Patterson, Joel Foreman, Scott M. Fulton III, and Petri Salonen.
Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:
- Azure Blob, Drive, Table and Queue Services
- SQL Azure Database and Reporting
- Marketplace DataMarket and OData
- Windows Azure AppFabric: Apps, Access Control, WIF and Service Bus
- Windows Azure VM Role, Virtual Network, Connect, Traffic Manager, RDP and CDN
- Live Windows Azure Apps, APIs, Tools and Test Harnesses
- Visual Studio LightSwitch and Entity Framework v4+
- Windows Azure Infrastructure and DevOps
- Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds
- Cloud Security and Governance
- Cloud Computing Events
- Other Cloud Computing Platforms and Services
Azure Blob, Drive, Table and Queue Services
Wade Wegner (@WadeWegner, pictured below) described Using Error Logging Modules and Handlers (ELMAH) in Windows Azure with Table Storage and a live Windows Azure demo in an 8/18/2011 post:
In this week’s episode of Cloud Cover, Steve and I covered Logging, Tracing, and ELMAH in Windows Azure. Steve explored the first two topics while I looked into ELMAH in Windows Azure. You should make sure and take a look at his posts – they’re useful:
ELMAH (Error Logging Modules and Handlers) itself is extremely useful, and with a few simple modifications can provide a very effective way to handle application-wide error logging for your ASP.NET web applications. If you aren’t already familiar with ELMAH, take a look at the ELMAH project page.
In the news from Cloud Cover 56:
- Announcing Simplified Data Transfer Billing Meters and Swappable Compute Instances
- Windows Azure Storage Libraries in Many Languages
- Windows Azure Accelerator for Web Roles Version 1.1
- Debugging an Application in the Cloud
- CloudXplorer Introduces Support for $logs Container
NuGet
Before going any further, I thought I’d let you know that I’ve created a NuGet package that makes this extremely easy to try. You can take a look at ELMAH with Windows Azure Table Storage on the NuGet gallery or immediately try this out with the following command:
Install-Package WindowsAzure.ELMAH.Tables
By default this NuGet package is configured to use the local storage emulator. If you want to use your actual Windows Azure storage account you can uncomment the following line in the Web.Config file:
<!-- <errorLog type="WebRole1.TableErrorLog, WebRole1" connectionString="DefaultEndpointsProtocol=https;AccountName=YOURSTORAGEACCOUNT; AccountKey=YOURSTORAGEKEY" /> -->Incidentally, if you like NuGet, then you should check out Cory Fowler’s post on Must have NuGet packages for Windows Azure development.
Demo
For those of you unfamiliar with ELMAH, I put together a simple demo. You can try it out on http://elmahdemo.cloudapp.net/. Just enter a message (keep it clean, please!) and throw an exception.
Click the ELMAH button to then load the handler. You’ll see all the errors logged with a lot of great detail.
The great part is that these files are getting serialized into Windows Azure table storage. The benefit of this is you can read them from anywhere – in fact, you don’t have to even deploy the elmah.axd handler with your web application! You could run it locally.
Here’s what [Wade’s original] files look like in table storage:
How Does it Work?
The nice part is you can easily grab the NuGet package to view all the source code. There are two items of interest: ErrorEntity.cs and Web.Config.
In ErrorEntity.cs we first create our ErrorEntity:
public class ErrorEntity : TableServiceEntity { public string SerializedError { get; set; } public ErrorEntity() { } public ErrorEntity(Error error) : base(string.Empty, (DateTime.MaxValue.Ticks - DateTime.UtcNow.Ticks).ToString("d19")) { this.SerializedError = ErrorXml.EncodeString(error); } }Then we implement the ErrorLog abstract class from ELMAH to create a TableErrorLog class with all the implementation details.
public class TableErrorLog : ErrorLog { private string connectionString; public override ErrorLogEntry GetError(string id) { return new ErrorLogEntry(this, id, ErrorXml.DecodeString(CloudStorageAccount.Parse( connectionString).CreateCloudTableClient().GetDataServiceContext() .CreateQuery<ErrorEntity>("elmaherrors").Where(e => e.PartitionKey == string.Empty && e.RowKey == id).Single().SerializedError)); } public override int GetErrors(int pageIndex, int pageSize, IList errorEntryList) { var count = 0; foreach (var error in CloudStorageAccount.Parse(connectionString). CreateCloudTableClient().GetDataServiceContext() .CreateQuery<ErrorEntity>("elmaherrors") .Where(e => e.PartitionKey == string.Empty).AsTableServiceQuery() .Take((pageIndex + 1) * pageSize).ToList().Skip(pageIndex * pageSize)) { errorEntryList.Add(new ErrorLogEntry(this, error.RowKey, ErrorXml.DecodeString(error.SerializedError))); count += 1; } return count; } public override string Log(Error error) { var entity = new ErrorEntity(error); var context = CloudStorageAccount.Parse(connectionString) .CreateCloudTableClient().GetDataServiceContext(); context.AddObject("elmaherrors", entity); context.SaveChangesWithRetries(); return entity.RowKey; } public TableErrorLog(IDictionary config) { connectionString = (string)config["connectionString"] ?? RoleEnvironment .GetConfigurationSettingValue((string)config["connectionStringName"]); Initialize(); } public TableErrorLog(string connectionString) { this.connectionString = connectionString; Initialize(); } void Initialize() { CloudStorageAccount.Parse(connectionString).CreateCloudTableClient() .CreateTableIfNotExist("elmaherrors"); } }Now, to leverage these assets, we update the Web.Config file to include an <elmah> … </elmah> section that specifies our custom error log (and also allows remote access to the handler:
<elmah> <security allowRemoteAccess="yes" /> <errorLog type="WebRole1.TableErrorLog, WebRole1" connectionString="UseDevelopmentStorage=true" /> </elmah>That’s all there’s to it!
Of course, there are many other ways you could define your ErrorEntity and implement the TableErrorLog (i.e. you could extract more details into additional entities within your table), but this way is pretty effective.
<Return to section navigation list>
SQL Azure Database and Reporting
•• George Huey released v0.1 of his SQL Azure Federation Data Migration Wizard project to CodePlex on 8/19/2011:
From the Home Page:
…
Requirements
SQLAzureFedMW and tools requires SQL Server 2008 R2 bits to run.
SQLAzureFedMW also requires BCP version 10.50.1600.1 or greaterProject Description
SQL Azure Federation Data Migration Wizard simplifies the process of migrating data from a single database to multiple federation members in SQL Azure Federation.SQL Azure Federation Data Migration Wizard (SQLAzureFedMW) is an open source application that will help you move your data from a SQL database to (1 to many) federation members in SQL Azure Federation. SQLAzureFedMW is a user interactive wizard that walks a person through the data migration process.
The SQLAzureFedMW tool greatly simplifies the data migration process. If you don’t have an SQL Azure account and have been thinking about moving your data to the cloud (SQL Azure), but have been afraid to try because of “unknowns” like cost, compatibility, and unfamiliarity, take advantage of the Try Windows Azure Platform 30 Day Pass.
SQLAzureFedMW Project Details
The SQL Azure Federation Data Migration Wizard (SQLAzureFedMW ) allows you to select a SQL Server database and specify which tables (Data Only) to migrate. The data will be extracted (via BCP) and then uploaded to SQL Azure Federation. The BCP data upload process can be done in a sequential process or parallel process (where you specify the number of parallel threads). See Documentation for more detail.SQL Azure Migration Wizard Utils (SQLAzureMWUtils)
SQLAzureMWUtils is a class library that a developer can use to assist them in developing for SQL Azure.Note: SQLAzureFedMW expects that your database schema has already been migrated to SQL Azure Federation and that they tables on the source database match the tables in the federated member databases.
SQLAzureFedMW is built off of the SQLAzureMWUtils library found in the codeplex project {url:SQL Azure Migration Wizard|http://sqlazuremw.codeplex.com].
Further Reading: Microsoft SQL Server Migration Assistant (SSMA): The free Microsoft SQL Server Migration Assistant (SSMA) makes it easy to migrate data from Oracle, Microsoft Access, MySQL, and Sybase to SQL Server. SSMA converts the database objects to SQL Server database objects, loads those objects into SQL Server, migrates data to SQL Server, and then validates the migration of code and data.
For information on SQL Azure and the Windows Azure Platform, please see the following resources:
- Cihan Biyikoglu's blog
- SQL Azure Community Technology Preview (CTP) Import and Export
- SQL Azure
- SQL Azure Videos
- SQL Azure Community
- Microsoft Cloud
- SQL Server 2008: Solutions
- SQL Server 2008 R2: Trial Download
- SQL Server 2008: Express
- Microsoft Visual Studio 2010 Express
- SQL Server 2008 R2 Express
- Windows Azure Platform
- Windows Azure AppFabric
From the Documentation page:
There are two tools (SQLAzureMW and SQLAzureFedMW) that can be used to migrate SQL Server / SQL Azure schema and data to a SQL Azure Federation.
The first thing that you need to do before running either of these tools is to create your ROOT database with your Federation. Note, that I would not SPLIT the Federation until after I had the objects (tables, views, stored procs …) migrated to the Federation. If the amount of data is small, you could also migrate your data at this time.Once you have your ROOT and Federation created, you can run SQLAzureMW. In the SQLAzureMW.exe.config file, you can preset source server and target server information. Here is an example of target server information:
<add key="TargetConnectNTAuth" value="false"/> <add key="TargetServerName" value="xxxxxx.database.windows.net"/> <add key="TargetUserName" value="user@xxxxxx"/> <add key="TargetPassword" value="TopSecret"/> <add key="TargetDatabase" value="AdventureWorksRoot" /> <add key="TargetServer" value="TargetServerSQLAzureFed"/>Valid values for TargetServer are:
- TargetServerSQLAzure = "SQL Azure"
- TargetServerSQLAzureFed = "SQL Azure Fed"
- TargetServerSQLServer = "SQL Server"
Here is another key that you will probably want to modify:
<add key="ScriptTableAndOrData" value="ScriptOptionsTableSchema"/>
Valid values for ScriptTableAndOrData are:
- ScriptOptionsTableSchema = "Table Schema Only"
- ScriptOptionsTableSchemaData = "Table Schema with Data"
- ScriptOptionsTableData = "Data Only"
This key tells SQLAzureMW if you want to migrate schema only, schema and data, or data only. You can also set this during runtime through the advanced options.
Note that the important thing is to set TargetServer to TargetServerSQLAzureFed. If you forget, you can always set this in the advanced options during runtime. Once you are ready, you can run SQLAzureMW and select the option to migrate a database. Select your source database and select the objects that you want to migrate. Once ready, hit Next a couple of times and have SQLAzureMW analyze and create TSQL scripts. Be sure to look for compatibility errors (like identity columns). You will have to fix before you proceed. The best way to do this is to copy the generated script to SSMS query windows. Edit the TSQL and remove any compatibility issues AND add “FEDERATED ON (cid=CustomerID” to the tables you want to federate on.You will have to modify cid to match your federation key and CustomerID to match your specific column to shard on. Once done, you can copy the TSQL from your query window and put it back into SQLAzureMW (if you have no data, you don’t need to go back to SQLAzureMW for second part, you can just change your connection in the query window and then execute your modified TSQL). If you choose to use SQLAzureMW for the second part (migration to SQL Azure Federation), replace the old TSQL with your modified TSQL and then click next. SQLAzureMW will prompt you for your SQL Azure Federation (it will be prefilled with data from the config file). Hit connect when you have verified information is correct. SQLAzureMW will then display Federations on the left side and Federation Members on the right side. If you have not done a split yet, then you should only see one Federation Member. Select that Federation member and then click next and SQLAzureMW will execute the TSQL against target server. With any kind of luck, you should be ready to SPLIT your Federation Members.
If you didn’t have SQLAzureMW copy your data during the initial phase, you can always use SQLAzureMW or SQLAzureFedMW to move your data at a later time. IF you SPLIT your Federated Members, then you will have to use SQLAzureFedMW for data movement.
…
George’s original SQL Azure Migration Wizard has been my favorite SQL Azure tool since v0.1 (see my Using the SQL Azure Migration Wizard v3.3.3 with the AdventureWorksLT2008R2 Sample Database post of 7/18/2010.) I’m sure the version for SQL Azure Federations will gain the same status. Stay tuned for examples when the SQL Azure Federations feature releases to the web later this year.
In the meantime, check out my detailed Build Big-Data Apps in SQL Azure with Federation cover article for Visual Studio Magazine’s March 2011 issue.
The Windows Azure Customer Advisory Team posted an SQL Azure Retry Logic library to CodePlex on 8/16/2011 (missed when published). From the Description:
Introduction
This sample shows how to handle transient connection failures in SQL Azure.
The Windows Azure AppFabric Customer Advisory Team (CAT) has developed a library that encapsulates retry logic for SQL Azure, Windows Azure storage, and the AppFabric Service Bus. Using this library, you decide which errors should trigger a retry attempt, and the library handles the retry logic.
This sample assumes you have a Windows Azure Platform subscription. The sample requires Visual Studio 2010 Professional or higher.
A tutorial describing this sample in more detail can be found at Retry Logic for Transient Failures in SQL Azure
TechNet Wiki.*SQL Azure queries can fail for various reasons – a malformed query, network issues, and so on. Some errors aretransient, meaning the problem often goes away by itself. For this subset of errors, it makes sense to retry the query after a short interval. If the query still fails after several retries, you would report an error. Of course, not all errors are transient. SQL Error 102, “Incorrect syntax,” won’t go away no matter how many times you submit the same query. In short, implementing robust retry logic requires some thought.
More Information
For more information, see [Windows Azure and SQL Azure Tutorials] http://social.technet.microsoft.com/wiki/contents/articles/windows-azure-and-sql-azure-tutorials.aspx [Last updated 7/8/2011]
* Thanks to Gregg Duncan for heads-up to this sample and the corrected tutorial location.
From the beginning of Windows Azure CAT Team’s the Retry Logic for Transient Failures in SQL Azure article:
SQL Azure queries can fail for various reasons – a malformed query, network issues, and so on. Some errors are transient, meaning the problem often goes away by itself. For this subset of errors, it makes sense to retry the query after a short interval. If the query still fails after several retries, you would report an error. Of course, not all errors are transient. SQL Error 102, “Incorrect syntax,” won’t go away no matter how many times you submit the same query. In short, implementing robust retry logic requires some thought.
The Windows Azure AppFabric Customer Advisory Team (CAT) has developed a library that encapsulates retry logic for SQL Azure, Windows Azure storage, and AppFabric Service Bus. Using this library, you decide which errors should trigger a retry attempt, and the library handles the retry logic. This tutorial will show how to use the library with a SQL Azure database.
Testing retry logic is an interesting problem, because you need to trigger transient errors in a repeatable way. Of course, you could just unplug your network cable, or block port 1433. (SQL Azure uses TCP over port 1433.) But for this tutorial, I’ve opted for something that’s easier to code: Before submitting a query, hold a table-wide lock, which causes a deadlock or a timeout. When the lock is released, the original query can be retried.
Prerequisites
This tutorial assumes you have a Windows Azure Platform subscription. For more information on creating a subscription, see Getting Started with SQL Azure using the Windows Azure Platform Management Portal.
This tutorial requires Visual Studio 2010 Professional or higher.
Download the project files.In this Article
Patrick O’Keefe described Tales from the Engine Room: Bulk Updates [to SQL Azure] in an 8/18/2011 post:
We have been tweaking the Project Lucy datastore again this week, and have found out a couple of things we think worth sharing. We found out the fastest way to update a lot of rows, was to do the complete opposite of what you’d think was the right thing to do.
Let me paint a scenario – we store wait stats data coming from Spotlight on SQL Server Enterprise customers in a fact table. One of the columns in the fact table is
wait_type, which contains the name of the wait. This was recognized as a bad idea some time ago, but we had other fish to fry.Recently our piscatorial problems have come back to bite us.
Since this column is an
nvarchar(100)it takes up beaucoup space, and since it is indexed multiple times, multiply that by the number of indexes. This results in a performance problem, since SQL Azure has to read lots more pages because there are less rows per page. Also we have an operational problem – not only are we are using more space that we want to (and space costs money), but any DDL operations on the table are expensive because of its size.
Hence, we find ourselves creating a
dim_wait_typesdimension table and instead of storing the wait type directly in the fact table, we store theidof the related row from the dimension table.
So far so good.
This database stuff is easy! We just create the dimension table and populate it, create a
wait_type_idcolumn in the fact table and write an update statement that updates thewait_type_idbased on the value of thewait_typecolumn. The update looks something like this:
update sqlserver_fact_waits_by_time set wait_type_id = 620, wait_category_id = 2 where wait_type = ''XE_DISPATCHER_JOIN''The table has close to 40 million rows in it, so a single update is out of the question. SQL Azure will throttle us based on log space long before the update completes. We have to break it down, so we decide to do it per wait type, since there are 635 SQL Server wait types (that we know about).
We start running 635 update statements….
- We find each statement does a full table scan, so we get hold of the plan and create the appropriate index
- We find we cannot create the index from on-prem – the CREATE INDEX statement takes too long and the socket between us the the datacenter keeps getting closed. The solution is to remote desktop to one of the roles in the running application, install SQL Server 2008R2 Express Tools and run the CREATE INDEX from there. Don”t forget to use the WITH (ONLINE=ON) option, otherwise SQL Azure will throttle you because you use too much log space
- Now that the index has been created, running the updates can commence. Chaos ensues. Because each update is doing a bunch of physical IO, all of our queued processing (end-users upload data around the clock) starts failing with timeouts.
- We turn queue processing off, and allow the updates to continue. Each one averages 3-5 minutes and there are 635 of them which is roughly 60hrs in total.
- Realization dawns that we cannot leave processing turned off for 3 days, so we stop the updates and turn queue processing back on. We need to find a way to trade time for concurrency. In other words, we don”t care if the updates take 3 days, but we need to be able to process incoming data whilst the updates are going on
A Solution
We went back to the original query plan for the update and found that 90% of the cost was in actually doing the update, very little cost was extant in actually finding the rows. Also and most significantly, we found that the amount of physical IO required scaled somewhat worse than linearly with the number of rows updated. The answer was to keep the actual number of rows updated small. So now our update looks something like this:
update
sqlserver_fact_waits_by_time
set
wait_type_id = 620, wait_category_id = 2
from
sqlserver_fact_waits_by_time wbt
inner join
(select top(2500) upload_id, id from sqlserver_fact_waits_by_time
where wait_type_id is null and wait_type = ''XE_DISPATCHER_JOIN'') wbt_top
on wbt.upload_id = wbt_top.upload_id and wbt.id = wbt_top.idThis new update statement takes under 20 seconds to run and we have a command line application run these for all wait types in a loop until there are no rows updated. Whilst these updates are running, user processing continues normally.
What we Learned
- Execute long running DDL operations from within the data centre
- Try to look for ways that allow you to trade time for maintaining concurrency
- On SQL Azure, doing a larger number of small updates (where each update touches less rows) took less elapsed time than doing a smaller number of large updates (where each update touches more rows)
Patrick O’Keefe shared his experiences in Tales from the Engine Room: Migrating to SQL Azure to the SQLServerPedia blog on 8/7/2011 (missed when published):
We’ve been very busy here at Project Lucy aver the last few weeks. We are “cup-runneth-over” with ideas on how to provide really cool and valuable stuff to our users – only problem was, we were getting bogged down in performance and data storage issues.
The way we were storing our data was not conducive to providing the sorts of features we wanted to provide and something had to change. After we re-architected our approach 4 times, with still no suitable solution, we decided our only two alternatives were between doing something stupid, or going broke.
How we used to do it
It is useful to look at how we did things up until a week ago in order to explain the changes so let’s look at an example – SQL Workload Analysis.
When we process a trace file (this applies to Spotlight on SQL Server uploads that include SQL Analysis data as well), we have a piece of code that detects a SQL event; we will use SQL:BatchCompleted as an example. Way back when we first went live we did some on the fly aggregation of these events but didn’t really store any event data because we had no way to display it. Fast forward to the end of Feb 2011 and we went live with the ability to explore the workload data within your upload using a grid and some ajax magic that allowed you to do multi-level grouping on the data in real-time.
In order to allow you to do grouping in real-time, we couldn’t store aggregates any more – we had to store every event. We could have stored each event as an entity in Table Services if we were prepared to live with the latency of fetching these entities. But we weren’t. You see, some of Project Lucy’s users upload trace files with hundreds of thousands of SQL events and pulling 200,000 entities out of table services is two orders of magnitude slower than a wet week. Another solution was required.
We found that when we stripped all of the text data (SQL, application name, login name, database name) out of the event data, it really wasn’t that big – certainly small enough to fit in memory. So that is what we did; as we were processing the data we put all of the text columns in look up tables and stored the even data in a big list. What we ended up with was a binary structure that had one big list for the event data, and numerous smaller hash tables for the category (SQL, application, database etc) data. This we then streamed out into a blob.
When a user came to the workload page and started interacting with it, we would check to see if we had the blob that contained the data we wanted to render for the user on the web server. If the file did not exist we would go get it from blob storage and cache it locally. Once the blob was local to the web server, we would load it into memory and perform the query user had requested using Linq. This worked surprisingly well and the response times even from blob files that contained many hundreds of thousands of events were acceptable.
The downside of this approach was that the data was stored one blob per upload. This meant that a few things were out of reach. Because a user can upload files from multiple servers in an upload, we had no easy way of getting data from just one server.
Also, we had no easy way of querying across time. We wanted to give users the ability to look at performance characteristics over time. In order to do this with our blob approach we would have had to load each blob in turn, get the rows we needed from it (after reading all the rows in it) and then merge those rows at the end. So we decided to invent our own database.
No seriously. We did. We spent some hours basking in warm delusions of our own grandeur, scribbling madly on a whiteboard.
We then came to our senses and decided that inventing our own database was somewhat foolish, and might distract us from our core objective of providing value to our users. So we decided to use SQL Azure to store our data. Now instead of a binary file, we store the event data in a big fact table and we store all the “category” (application, database etc) data in dimension tables. This now means that we can start to build features that allow querying across time – which is what we are working on right now.
What we learnt
As part of our migration to SQL Azure we learnt a few things:
- The database size limit is there to encourage you to think scale out, instead of scale up. This is a good thing – with scale out, you are not constrained by being on one machine. You need to design your scale out strategy from the start. We plan to use SQL Azure Federations when it ships, so our schema is optimised for this
- For tables that you are going to federate, you need to make sure there are no database scoped functions used. This means no identity columns. We used uniqueidentifier with a newid() default constraint
- Make sure you are religious about putting clustered indexes on tables. Operations against tables without clustered indexes will work in the development environment with a local database, but when you push into the cloud it will break
- Using table services for bulk inserts or updates involves high latency. You can only insert/modify 100 entities (rows) at a time and this takes 500ms – designs which involve big fetches of many rows from Table Services will be unsatisfactory
- If you are going to tempt fate and use Table Services, in order to fetch data from table services quickly enough (it takes 500ms per 1000 entities fetched) you have to be very careful about the partition and key values used when storing the data. (The partition+key must be unique for each entity; and you select multiple rows by partition.) The trick is to define your partition key so you can fetch multiple rows at a time, but not too many rows. Invariably this mean the same data will need to be stored multiple times, via different partition values. For example, for SQL Server wait statistics, you might need to store data partitioned by time (to service a trend over time chart) and store the same data partitioned by wait-name so you can compare different users with each other. As a consequence of storing the data multiple times, you end up writing a lot more (serialization-deserialization) code than you really feel you should need to
- Tables in Table Services store entities. An entity is an object with arbitrary properties. Just because you can store different entities in the same table doesn’t mean you should. We got clever with C# generics and got this to work and then decided it was just a rod for our own backs
- SQL Azure is expensive (we hope this changes) make sure you are indexing just enough. You can double your storage requirements by just creating an index – make sure that the index is useful
- High concurrency MERGE SQL statements that update multiple rows were problematic for us in SQL Azure. We had deadlocks, no matter what you try. The only way we were able to stop the deadlock was with a WITH TABLOCKX locking hint – not really a solution. Until Microsoft provide a way to get the deadlock graph from SQL Azure so you can really troubleshoot the problem rather than guess, we will be left with a cloying sense of dissatisfaction
- We’ve seen really impressive IO rates from our SQL Azure database (7,500 writes per second) – when you consider that this involves replication to two slaves, color us extra impressed
- Keep the persistence layer of your software simple. i.e. Where feasible, perform simple SQL select statements from multiple tables and then use something akin to LINQ in C# to merge and project that data. This will make life easier in future – when you decide to locate those tables within different Azure databases, or use federations. We encountered this issue. Our database grew to over 43Gb in size (50Gb is the limit, as at August 2011). We had to move one of our tables into a different database. Consequently, any SQL statements joining this table to other tables had to be achieved via LINQ instead. (There are no cross-database joins in SQL Azure)
- Long running SQL Statements like CREATE INDEX have to be run from within the data centre. This will drive you nuts. The internet will conspire against you to ensure that the socket will get closed while you wait for the query to finish
In general, our experience with SQL Azure has been a positive one, and now having taken the leap, we look forward to building out the new features we have planned in Project Lucy.
<Return to section navigation list>
MarketPlace DataMarket and OData
Brandon George posted AX 2012, PowerPivot and a Dash of OData Feeds on 8/17/2011:
Sometime back, I wrote about the importance of Microsoft PowerPivot as well as Open Data Protocol (OData) for Microsoft Dynamics AX 2012. All of those pointed to the fact of what AX 2012 would be able to do with such enabling technologies.
Well now finally we have the launch of Microsoft Dynamics AX 2012, and it's time to start cooking up some self-service BI with PowerPivot and OData feeds!
First it's very important to understand what it means for having data published as OData feeds, coming out of Microsoft Dynamics AX 2012. This starts with a query object within AX, as shown below.:
This is the basis for an OData feed being published, as it's similar the base for Document Services in AX 2012. This is were we stop though, and go a different path from Doucment services, for enabling these feeds. From here, we need to take and go into the Application Workspace, and navigate to: Organization Administration > Setup >Document Management > Document Data sources.
From here, we need to take and add a new entry. If we do not do this, then nothing will show as active in our list of possible OData feeds. In order to create go to File and then new to create a new document. Once you have the screen up, you should then select the data source type of Query, and select the query we created from the drop down. After that we click "Activate".
At this point, AX system service called ODataQueryService will be what publishes our new query document source we just setup. To test this out, you can open up a IE window, and type the following (Assumes your on the AOS): http://localhost:8101/DynamicsAx/Services/ODataQueryService/
When you do this, and Atom list of queries, that you have security rights too, will appear to work with. That's it for enabling a query to be published as an OData feed. We can start working with this right away yes?
Well, like all great baking stories there are little things that you need to understand so the final product comes out tasty and delicious!
So say, and in the real world this will be the case, that you have Excel and PowerPivot installed on a seperate machines from your AOS. Well if this is the case, and you wanted to work with your published OData feeds from the seperate machine, you might think, ok so instead of using localHost I would use the actual server name.Wrong! This will cause a 400 Bad Request to be generated, and the reason is the implementation of OData does not allow the use of named servers. You have to use the IP Address of the server in which the ODataQueryService is being published from. So the correct format, to access this from a remote server, or just really separate server than that which the AOS is running is as follows: http://xxx.xxx.xxx.xxx:8101/DynamicsAx/Services/ODataQueryService/
Now of course, any good cook knows that you replace the xxx with the actual values of the ip address itself, however for good measure I point that out to you here.Doing this, we are actually able to take and see the WSDL, after logging into the box when prompted for our AD creds. You do have to login the first time with your AD creds, however after that you should be fine.
Now that we have our WSDL showing up, we can launch into PowerPivot, and click to add Feed data sources.
Make sure to take and enter into the URI the same format as you did before, and click test connection. This should come back ready for your use, and hit next. In doing so we should see our new query document data source from within AX 2012, being published and therefore able to be consumed as an OData feed.
Now that we have this, we can work with our Query document data sources from within PowerPivot to create some wonderful self-service BI goods! It's pretty powerful stuff to be able to do this, and this adhere's to the security model within AX. Therefore if you have rights to a query then you will see it. Otherwise, you will not see it.
Alex James (@adjames) et al., posted Simple Web Discovery for OData Services to CodePlex on 8/12/2011 (missed when published). From the Project Description and Usage Overview:
This Simple Web Discovery (SWD) endpoint provides a uniform access point that tells you (1) where to authenticate and (2) where to get access tokens. It makes it easier to implement OAuth2.0 in a general way. It can also make other info available if need be. Written in C#.
More Details
A client may want to draw data from many different sources, but there is no standard way to find these two extra endpoints besides hard coding each of them for every single data source. We’d like to make this easier to enable richer uses of data.
- Consent Server: where the client actually authenticates and authorizes data access (Google, Facebook, etc.).
- Token Endpoint: where the client exchanges an Authorization code for an Access Token, or where they acquire a new Access Token with a Refresh Token.
The Simple Web Discovery protocol (hereafter SWD) outlines a mechanism for discovering the location(s) of services for a particular principal. In our case the principal will be the OData Service itself. Using a slightly modified version of this protocol, we can expose a standardized endpoint discovery method.
To make this as easy as possible, we are publishing code that will serve both of these necessary service URLs. We have created an Attribute that can just be added onto an existing OData service. It will add a Simple Web Discovery endpoint that provides all of the necessary endpoint information. The code is fairly general, so it can be useful in other contexts too. Let’s get started.
Note: The SWD protocol requires that the .wellknown endpoint extend directly from the root of the domain, but with an OData service this may not be possible without significant server configuration changes. This may not be possible for everyone, so in our case the .wellknow endpoint will extend from the OData service itself. Since we are extending from the service we already know our principal, it’s already in the URL. Therefore, it does not need to be specified as a query argument to the SWD endpoint. These changes make it extremely easy to add to an existing OData service.Simple Web Discovery Behavior
Adding a simple web discovery (SWD) endpoint is as easy as adding an attribute to your DataService. We will discuss the OAuthEndpoints class in the next section.[SimpleWebDiscoveryBehavior(typeof(OAuthEndpoints))] public class WcfDataService : DataService<WorldCupData> { // your code here }In this example, our service is located at …/WorldCupData.svc. By adding this attribute you will create a queryable endpoint at
…/WorldCupData.svc/$wellknown/simple-web-discovery
If we want to discover the usable consent endpoints for our service we just query the following URL:
…/WorldCupData.svc/$wellknown/simple-web-discovery?service=urn:odata.org/auth/oauth/2.0/consent
This returns a JSON string listing all of the allowed consent servers. It will look something like this:{ "locations" : [ "http://www.google.com", "http://www.facebook.com" ] }This gives the client application options to authenticate through many different supported services. For instance, if the client only supports Facebook authentication, they simply choose that particular endpoint and carry on.
Endpoints
We need some way to configure the locations returned by this new endpoint. This is done by extending the Endpoints class. The Endpoint class has a dictionary field mapping service names (keys) to endpoint lists (values). When you query the SWD endpoint for a particular service, it will look the service up in this dictionary and return the locations in the corresponding list. Let’s see it in action.Example 1
The best way to add your own endpoints is to extend the Endpoints class provided above. Once you have created your endpoint subclass, you can simply add the SimpleWebDiscoveryBehavior attribute to your service. The whole thing will look something like this:public class OAuthEndpoints : Endpoints { public OAuthEndpoints() : base() { this.AddEndpoint("consent", "http://www.google.com"); } } [SimpleWebDiscoveryBehavior(typeof(OAuthEndpoints))] public class WcfDataService : DataService<WorldCupData> { // your code here }Now when we can navigate to
…/WorldCupData.svc/$wellknown/simple-web-discovery?service=consent
This returns a JSON string listing all of the allowed consent endpoints. It will look like this:{ "locations" : [ "http://www.google.com" ] }Nice! We are pretty close to having a fully functional SWD endpoint.
Example 2
There is just one last thing. To make it more difficult to make a typo when entering keys into the EndpointDictionary, we have provided the static string values ConsentEndpoints and TokenEndpoints in the Endpoints class. This way you don’t need to type out those crazy strings for every endpoint you add.public class OAuthEndpoints : Endpoints { public OAuthEndpoints() : base() { this.AddEndpoint(Endpoints.ConsentEndpoints, "http://www.google.com"); this.AddEndpoint(Endpoints.ConsentEndpoints, "http://www.facebook.com"); this.AddEndpoint(Endpoints.TokenEndpoints, "http://www.bing.com"); } } [SimpleWebDiscoveryBehavior(typeof(OAuthEndpoints))] public class WcfDataService : DataService<WorldCupData> { // your code here }Now we will have precisely the behavior described at the beginning of this post:
…/WorldCupData.svc/$wellknown/simple-web-discovery?service=urn:odata.org/auth/oauth/2.0/consent
will return{ "locations" : [ "http://www.google.com", "http://www.facebook.com" ] }Furthermore, we’ve seen that you can add as many endpoints as you’d like. In this case we also included a token endpoints service, so
…/WorldCupData.svc/$wellknown/simple-web-discovery?service=urn:odata.org/auth/oauth/2.0/token-endpoint
will return{ "locations" : [ "http://www.bing.com" ] }Great, now you have listings of the consent endpoints and token endpoints! You can also add any other endpoint that you may need, so hopefully you can use this code again when OAuth3.0 or HotNewAuthProtocol becomes the cool thing to do.
<Return to section navigation list>
Windows Azure AppFabric: Apps, Access Control, WIF and Service Bus
<Return to section navigation list>
Windows Azure VM Role, Virtual Network, Connect, Traffic Manager, RDP and CDN
• Avkash Chauhan explained Windows Azure Traffic Manager (WATM) and App Fabric Access Control Service (ACS) in an 8/19/2011 post:
Recently I was asked if Windows Azure App Fabric Access Control Service (ACS) works with Windows Azure Traffic Manager or not.
The current CTP release of Windows Azure Traffic Manager is not supported with App Fabric Access Control Services.
As you may
haveknow,thatWindows Azure Traffic manager is in CTP now and offers many benefits to Windows Azure Hosted services.For more info on Windows Azure Traffic Manager visit:
Avkash Chauhan described Windows Azure Traffic Manager and using your domain name CNAME Entry in an 8/19/2011 post:
With the release of Windows Azure Traffic Manager, Windows Azure users, can point their domains such [as] www. companydomain.com to their Azure Traffic Manager policy domain name, such as companydomain.ctp.trafficmgr.com.
Above setting is equivalent to the following DNS record:
www.companydomain.com CNAME companydomain.ctp.trafficmgr.com
Once you have your application hosted in Windows Azure and registered domain name i.e. companydomain.com, you might have expectation with Windows Azure as:
- To respond to your domain name without the www in the name (http://companydomain.com only),
- And to have that site running through traffic manager.
Now the problem here is that above http://companydomain.com configuration won’t work currently because DNS does not allow for a CNAME record to have an empty name. Use of A records is not possible Windows Azure does not guarantee static IP address. An A record can be registered with an empty name, but the consequence of that is that you are then bound to a single IP address, and that then conflicts with how traffic manager works by using DNS redirection to an IP address in an appropriate data center. It is not the client that is wrong here – it is that you cannot combine a www-less domain name AND traffic manager.
<Return to section navigation list>
Live Windows Azure Apps, APIs, Tools and Test Harnesses
• Joel Foreman described Building and Deploying Windows Azure Projects using MSBuild and TFS 2010 in a 8/19/2011 post to his Slalom Consulting blog:
Windows Azure Tools for Visual Studio makes it very easy for a developer to build and deploy Windows Azure projects directly from within Visual Studio. This works fine for a single developer working independently. But for developers working on projects as part of a development team, there is the need to integrate the ability to build and deploy the project from source control via a central build process.
I thought this would be a good opportunity for me to dive into updating our build process for Windows Azure projects to be in line with these changes. The following post will cover configurations in Visual Studio, packaging from Visual Studio and the command line, packaging on a build server, and finally deploying from a build server.
Before I dug in, I came up with some goals for my Windows Azure build process. Here is what was at the top of my list:
- Being able to build, package, and deploy our projects to multiple Windows Azure environments from a build server using MSBuild
- Being able to retain copies of our packages and configurations for our builds in Windows Azure BLOB Storage for safe keeping, history, easy rollback scenarios, etc.
- Having a build number generated for a build and make sure that build number is present in the name of our packages, configs, and deployment labels in Azure for consistency
- Leveraging the Release Configurations in Visual Studio as much as possible as the designation of our different environments, since Web.Config transformations rely on this pattern already
- Being able to still package and deploy locally, in case there is an issue with the build server for some reason
Let’s start with a new Windows Azure Project in Visual Studio, with a MVC 3 web role. I quickly prepped the MVC project for deployment to Azure by installing the NuGet package for MVC 3 and turning off session state in my web.config. I will use this as my sample project.
Defining Target Configurations and Profiles
Release Configurations and Service Configurations
The first thing I want to examine is configuration files are now handled for deployments. I see that the Web.config has two release transformations that match up to the “Debug” and “Release” configurations that come with the project. I also see that there are two configs, a ServiceConfiguration.Local.cscfg file and a ServiceConfiguration.Cloud.cscfg file, that represent “Local” and “Cloud” target environments. I have found in my experience that each target environment for a web project needs its own transformation for a web.config and its own version of a ServiceConfiguration file. For example, the ServiceConfiguration might have settings such as the Windows Azure Storage Account to use for Diagnostics, but the web.config has the configuration section for your Windows Azure Cache Settings. I am sure there could be advantages to having the flexibility to have these differ, but for my project I want them to be consistent, in-line with our target environments, and to rely on the Release Configuration setting.
I added a Release Configuration for “Test” and “Prod” and removed the one for “Release” to mimic a couple of target environments. This was easily done using the Configuration Manager.
I used the “Update Config Transforms” option by right clicking on the web.config file to generate the new transforms.
Now I just needed to get the ServiceConfiguration files to fall in line. In Windows Explorer, I added ServiceConfiguration.Test.cscfg, and renamed the other two so that I had ServiceConfiguration.Debug.cscfg and ServiceConfiguration.Prod.cscfg. I then unloaded the Cloud Service project in Solution Explorer and editing the project file to include these elements.
After saving and reloading, you can see that all the configurations are showing up and named consistently.
Next, I needed to make sure these would package and publish correctly from Virtual Studio. I am happy to see that my changes are recognized and supported. There is a setting for which ServiceConfiguration to use when debugging.
For packaging/publishing my cloud project, I am prompted to select the configurations I want (and in this case I do not have variability in the two). The correct configuration file is staged alongside my package file.
The Service Definition File
In Hollander’s post, he mentioned some good reasons why you might want to have a different ServiceDefinition file for different environments, such as altering the size of the role. I also could envision other cases such as not having an SSL endpoint in a DEV environment, etc. I see this more as a nice to have than a must have. Feel free to skip this section if you are ok with a standard definition file for all of your environments.
For those interested, lets now tackle adding different ServiceDefinition files for our Release Configurations and making sure those are built as part of our package.
Hollander’s example shows how to use transformations on both the ServiceConfiguration file and ServiceDefinition file. In this example, I am simply going to replace the entire file, to fall in line with what is being done by Visual Studio for the ServiceConfiguration file.
In Windows Explorer, I added a ServiceDefinition.Test.csdef and ServiceDefinition.Prod.csdef, and renamed the existing one to ServiceDefinition.Debug.csdef for consistency. I updated the ccproj file for my cloud service to include those files like I did with the configuration files before, but added a separate section so that VS doesn’t complain about multiple definition files.
They now show up in Solution Explorer.
The part that is a little trickier is making sure the correct ServiceDefinition file for our Release Configuration is chosen when we are packaging/publishing our project. To understand this process, I started looking at the targets file that my project is using, Microsoft.WindowsAzure.targets located at C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\Windows Azure Tools\1.4, which is different from the targets file it used in Tom and Alex’s examples before the 1.4 tools update. I also starting running msbuild from the command line so that I could see the log file it was generating for the Publish process. Here is an example of the command I was using:
msbuild CloudService.ccproj /t:Publish /p:Configuration=Prod > %USERPROFILE%\Desktop\msbuild.log
On a side note, initially the command fails. In looking at the log file, it was because the variable $(TargetProfile) was not specified for the target “ResolveServiceConfiguration”, which looks for the correct ServiceConfiguration file to use.
I updated my command to the following and ran again:
msbuild CloudService.ccproj /t:Publish /p:Configuration=Prod;TargetProfile=Prod > %USERPROFILE%\Desktop\msbuild.log
This time my project build and packaged successfully and produced the correct configuration file. This property will come in handy later for our Build server configuration.
Now, back to the task of getting the appropriate definition file into our package. After more digging into the targets file and log file, its appears that a target service definition file, ServiceDefinition.build.csdef, gets generated from the original source definition file, and that one is used with cspack to create the package file. The target file is first created in the same directory as your project file, and then it is used with cspack and for reference can be seen in the .csx folder in your bin directory.
In looking back at the Microsoft.WindowsAzure.targets file, we can see that the ResolveServiceDefinition target sets these source and target variables. If I override this target to set the appropriate source definition file variable, then that one will be used. I added the following section to my CloudService.ccproj file to override this target and use the Configuration property as part of the source definition file name (could have also used TargetProfile).
I then ran some builds, packages, and publishes from the command line and from Visual Studio. Everything was ending up in the right place. To verify, you can dig into the CloudService.csx folder for our service located in bin\(Configuration) to find the definition file that was packed. For this “Prod” build we can verify that the correct definition file was used by looking at the contents of this folder:
Now I have my environment definitions and configurations in sync, and all relying on the same Release Configuration property. I am ready to starting building from our build machine.
Packaging and Deploying on a Build Machine
Build Machine Setup
I am using an existing instance of TFS 2010 for source control. I now need to configure a machine to be able to build my Azure project. I elected to set up a new virtual machine to serve as a new build server because of the Azure dependencies my build server will have. From some research, it appears that most people are installing Visual Studio, the Windows Azure SDK and the Windows Azure Tools for Visual Studio onto their build server in order to build Azure projects. I briefly looked at what it would take to not go down this path, but decided against it. If anyone has a more elegant solution (that doesn’t involve faking the Azure SDK installation), I am all ears. Due to these requirements, I felt it was best to use a separate VM as a build server.
After provisioning a new Windows Server 2008 R2 VM, I installed the Build Services only for TFS 2010. I did end up creating a new Project Collection on our TFS 2010 server and configured the build controller to be associated to that collection.
I was now ready to install the necessary dependencies to build Windows Azure projects. I installed Visual Studio 2010. I then used the Web Platform Installer to install the Windows Azure SDK and Windows Azure Tools for Visual Studio – August 2011. The Web Platform Installer did a good job of taking care of upstream dependencies for me.
After the installation completed, I double checked that the Microsoft.WindowsAzure.targets file was present on the VM.
Finally, I created a drop location on my build machine and updated the permissions to allow builds to be initially stored on this machine itself. My end goal is to have these builds being pushed to Windows Azure, but I figure it will be nice to verify locally on the machine first.
Packaging on the Build Machine
Back in Visual Studio on my laptop, I created a new Build Definition called “AzureSample-Prod”. I specified the Drop location to the share I created on the build VM. On the Process tab of the wizard I made sure to select my CloudService.ccproj file for my solution under “Items To Build” and also entered the “Prod” Configuration for which Configuration to build. I selected the cloud service project so I can package my solution for Windows Azure. The “Prod” Configuration represents my target environment I will be publishing to.
Next, remembering my experience running msbuild.exe from the command line before, I knew I had to set some MSBuild arguments. The field is located in the Advanced Properties window section on the Process tab. I added an MSBuild argument for $TargetProfile and set it to match my Release Configuration. I also specified a MSBuild argument for the Publish target to indicate that we want to create the package.
This Build Definition should successfully build and package my Azure project. I ran the build and it succeeded. I then went to the drop location that I specified to verify everything was packaged appropriately. To my surprise, the “app.publish” folder that should contain the .cspkg and .cscfg file that was packaged did not exist at the drop location. Strange. I looked over the MSBuild log file for any clues. I also found that the app.publish folder was created at the source build location on the build server. It just looked like it was getting left behind and not making it to the drop location. I am thinking it could be a timing issue for when contents are dropped at the drop location. This was not an issue prior to the 1.4 Tools for Visual Studio.
To get around this problem, I decided to add a simple custom target to my project file to copy the app.publish folder contents to the drop location after they are created. The condition checks the value of a new parameter, which I set as an MSBuild argument, so that this target doesn’t run when building/packaging locally in Virtual Studio.
My MSBuild Arguments property now looked like this:
A kicked off another build, and was happy to see the app.publish folder with package file and configuration file in my drop location. Note that my drop location happens to be on my build server, but it really could be another shared location.
Deploying from the Build Machine
The next task was to start deploying from my build machine directly to Windows Azure. Already having a Windows Azure Subscription, I created a new hosted service and storage account for my target environments. To be able to deploy, there are some extra configuration steps to complete on the build machine. In Hollander’s post, he outlines these steps:
- Create a Windows Azure Management Certificate and install it on the build machine and also upload it to the Windows Azure Subscription
- Download and install the Windows Azure Management Cmdlets on the build machine
- Create an AzureDeploy.ps1 powershell script and save it to the build machine
After completing these steps, I also made sure to update the execution policy for powershell on my build machine, knowing that the script would fail if the execution policy was restricted.
I continued following Tom’s instructions, and updated my project file to add a target that calls the AzureDeploy.ps1 script. I used the AzureDeploy.ps1 “as is” from his post. I did have to modify his “AzureDeploy” target. I had to update some of the parameter values to match my naming conventions and also to match the new app.publish folder where our package is created. I also reused the same “BuildServer” parameter that I used in my other target above. I eliminated the “AzureDeployEnviornment” parameter since we are leveraging the Release Configuration for this.
Finally, I came back to my Build Definition, and updated the MSBuild Arguments on the Process Tab to include the parameters for the Azure Subscription ID, the Hosted Service Name, the Storage Account Name, and the Certificate Thumbprint:
/t:Publish /p:TargetProfile=Test /p:BuildServer=True /p:AzureSubscriptionID="{subscriptionid}" /p:AzureCertificateThumbprint="{thumbprint}" /p:AzureHostedServiceName="{hostedservicename}" /p:AzureStorageAccountName="{storageaccountname}"I then kicked off a new build and logged onto the Windows Azure Management Portal to watch my service deploy!
Final Tweaks: Build Numbers, Deployment Labels, Deployment Slots, and Saving Builds in BLOB Storage
Now having my project deploying to Windows Azure, I noticed a few other tweaks I wanted to make to tighten this process up a bit. One of the things that I thought would be valuable was to have the Build Number generated by TFS during the build process be the Label on the deployment in Windows Azure. Then it would be very easy to identify what build is deployed. This turned out to be a little more complicated than I expected.
I knew that if I passed in the build number as a parameter to the AzureDeploy.ps1 script, then I could easily set that to the label for the deployment. But unfortunately, by default the build number value is not available to downstream targets. I found this post by Jason Prickett, which describes an easy update to make to the build template that exposes this value. I made a copy of the default build process template, edited the XML as indicated in that post, and saved the file as a new build process template. You then need to check this template into source control. I edited my Build Definition to use this new template rather than the default template.
Now that I had a variable containing the value of the build number, I updated the target to pass the $BuildNumber into the AzureDeploy.ps1 script. I then updated the script to accept this parameter and use it for the build label. The script already had a variable for “buildlabel” that was being used. I simply set its value to this incoming argument.
$buildLabel = $args[7]
I kicked off another build and went back to the Azure Portal. This time the deployment label matched the build number in TFS!
Next, I took a look at what was being stored in BLOB storage when deploying to Windows Azure with this process. My goal was to have both the package and service configuration file for a build be stored in BLOB Storage for historical purposes, and also to easily support rollback scenarios. I noticed that the Cmdlet being used was placing the package file in a private blob container called “mydeployments” in the storage account I specified. The name of the blob was a current date format plus the name of my package. Also, the configuration file was not present.
I wanted to see the package and configuration file here, and using a naming convention that included the build number so I could distinguish them appropriately. Let’s make this happen.
I opened the solution file for the Windows Azure Management Cmdlets. I looked at the source code for the cmdlet that was being called from the AzureDeploy.ps1 script. Inside NewDeployment.cs, I navigated to the NewDeploymentProcess method. It was easy to see where the package was being uploaded to blob storage. It was eventually calling to a static method called UploadFile in AzureBlob.cs. There it was applying a timestamp to all BLOBs as part of a naming convention, likely to ensure uniqueness. I decided to comment this out, because I plan to make my BLOB names unique myself by using the Build Number.
Next, back in the NewDeploymentProcess method, I added a snippet to also upload the configuration file to BLOB storage, right after the package file. Finally, I was prepared to move these changes onto the build server. I uninstalled the existing Cmdlets using an uninstall script that is provided with the cmdlets. I then removed them entirely from the build server. I zipped up and copied the updated version to the build server and then installed them once again.
The last remaining step was to have the package and configuration file names contain the build number from TFS. I once again modified the target that copied these files to the drop location. I simply included the $(BuildNumber) parameter that we exposed earlier as part of the destination file name.
After running a new build, I verified the contents of the “mydeployments” container. I was pleased again to see a package and configuration file labeled by build number.
This now matches the label on my hosted service that was just deployed, and also aligns with the build history in TFS and on the build server. All are now in sync, and centered around the Build Number generated from TFS. If I decided to change my naming convention for builds in my build definition, that change would be represented in these other areas as well.
For extra credit, I made one final improvement. Knowing that I may want to target either the Production or Staging deployment slot for different build definitions, I updated the AzureDeploy.ps1 script to accept deployment slot as an incoming parameter.
Here is a look at the final version of these targets in my CloudService.csproj file. You will be able to download the entire solution from a link at the end of this post.
Resources
I hope that this post is helpful in creating and customizing your build process for Windows Azure. If there are better ways to accomplish some of these things, please let me know. Credit to Tom Hollander and others for laying the groundwork for me to follow.
The solution, which includes the ccproj file with my custom targets, as well as the other files such as the AzureDeploy.ps1 script are available for download here.
• Petri I. Salonen warned Cloud ISV: Make sure you understand your ecosystem play – example of Intuit and Microsoft collaboration on software platforms to create a foundation for solution developers in an 8/19/2011 post:
I have written several times in my blogs about ecosystems and the role that ecosystems play. I recently run into an interesting article in the Redmond ChannelPartner with the header “Intuit Extends Cloud Pact with Microsoft”. As I am working with Microsoft ecosystem every single working day, I became interested what the article was all about. Intuit has been building a Partner Platform (IPP) that was reported by Mary-Jo Foley already back in January 2010. I am a longtime QuickBooks Online user so I have a pretty good picture of Intuit’s SaaS delivery model at least from 2003. I believe Intuit was one of the first software companies to introduce a full-blown accounting solution for the SMB market and my company still uses it every single day.
The idea behind this Intuit Partner Platform (IPP) is to help developers to build and deploy SaaS applications that are integrated with QuickBooks data and also to give huge exposure for the ISV on the marketplace that Intuit provides for its partners. This marketplace (Intuit App Center) has thousands of applications that can be used with QuickBooks and other QuickBooks third-party solutions.
Let’s look closer to why Intuit and Microsoft need each other. I read an interesting blog entry from Phil Wainewright that includes very interesting remarks about software platform that happens to be the topic of my Ph.D. dissertation (Evaluation of a Product Platform Strategy for Analytical Application Software). The blog entry from Wainewright includes following picture:
You can read more about this topic and download Wainewright’s report “Redefining Software Platforms – How PaaS changes the game for ISVs) for Intuit” and this can be found by following this link.
When you review the picture above in more detail, you will find interesting and relevant information how Windows Azure and Intuit IPP platform play together. According to Wainewright, the conventional software platform capabilities are all about functional scope of the development platform whereby cloud platforms add three additional distinct elements according to Wainewright: Multi-tenancy, Cloud Reach and Service delivery capabilities. The service delivery capabilities have to do with provisioning, pay-as-you-go pricing and billing, service monitoring etc. The multi-tenancy is typically not something that the PaaS platform provides automatically without the application developer building the multi-tenancy logic to the application. I still hear people saying that a legacy application that is migrated to the PaaS platform will automatically become multi-tenant. This is not true as each application has to be re-architected to take advantage of things such as scalability (application increases compute instances based on load).
The idea behind Intuit IPP platform according to Wainewrite is that Intuit has built service delivery capabilities that can be abstracted from the functional platform that is on the left hand side of the picture. The idea that Intuit had initially was to be able to provide support for any PaaS platform to be integrated to the IPP platform which I think is a good idea by not practical considering how fast the PaaS platforms are evolving and the amount of investments that are put into them.
One thing to remember is that all cloud platforms such as Windows Azure has already moved on the horizontal axis whereby the situation and clear cut separation between functional platform and service delivery capabilities is no longer that obvious. This also means that any Microsoft ISV that builds additional infrastructure elements to Windows Azure has to be carefully aligned with Microsoft product teams as there might be a danger to be irrelevant as some third-party functionality will be covered with the functional platform itself (PaaS platform) like Windows Azure. I have seen the same situation with some ISVs working with BizTalk extensions that suddenly have become part of BizTalk itself. Microsoft is very clear with its ISV partners that they should focus on vertical functionality or features that are unlikely to be part of the Microsoft platform in the short-term.
A new post from Jeffrey Schwartz on August 11th, 2011 explains how Intuit IPP and Microsoft Azure will be even more integrated as Intuit will drop its native development stack and instead “focus on the top of the stack to make data and services for developers a top priority” according to Schwartz. In reality this means that Intuit will invest heavily in Windows Azure SDK for IPP and make developing an app on Azure and integrating it to QuickBooks data and IPP’s go-to-market services easy and effective. Microsoft released some more information about this partnership in the Windows Azure blog. The two companies have launched a program for this called “Front Runner for Intuit Partner Program” that explains what the developers get by participating in the program. The site portrays three steps: Develop, Test and Market and there is a video that explains what it means.
So what should we learn from this blog entry? First of all, every development platform (PaaS etc.) will evolve and my recommendation for the ISV is to focus and invest on one that you think is here in the long run. I think this example from Intuit is a great example of a company that was initially in the race of competing in the PaaS space to some extent to conclude that the investments to keep the competition going is just too huge and this led to the conclusion to select Microsoft Azure as the foundation for IPP. Intuit will be much better off by focusing on building logic on-top of Windows Azure by participating in SDK development an ensuring that any solution specific development can be easily integrated into Windows Azure platform. Intuit will therefore focus on providing data and services for developers to use with Windows Azure PaaS platform.
Microsoft has been in the development tools and platform development since its foundation so they are much better off to do those kinds of massive investments. I think this is very smart from Intuit and this enables Intuit to have a scalable solution that developers can rely on even if the decision was not easy according to Liz Ngo from Microsoft. Alex Chriss (Director, Intuit Partner Platform) from Intuit explains this in his blog why Windows Azure is a good foundation for Intuit development. Also, Intuit provides a tremendous opportunity for ISVs like CoreConnext and Propelware report based on the blog from Liz Ngo.
Software ecosystem will continue to evolve and EVERY ISV has to figure out how its solutions will meld to be part of different sub-ecosystems. This will also require efficient and well-defined Application Programming Interfaces (API) from all parties to be able to create an integrated solution based on service oriented architecture (SOA).
Let me known if you know other good examples where software ecosystems mesh nicely with each other.
David Gristwood answered So, Who Else Is Using Windows Azure…? in the UK in an 8/19/2011 post:
If you are in the process of evaluating the Windows Azure platform, one of the questions you are may be asking yourself is, who else is using Windows Azure? Our Developer and Platform Group within Microsoft here in the UK has been working closely with a number of early adopters, many of whom are now live with applications and services running on Windows Azure:
CisionWire reported Adactus Ltd, Releases New Version of its Pulse Claims Management Software for Windows Azure on 8/19/2011:
London — 19th August 2011 — Adactus Ltd today announced it will launch a new version of its application, Pulse, which works with Microsoft Windows Azure to offer customers enhanced security, as well as innovative user interface features and reliability improvements.
“Our ISV community is alive with innovation, and we’re committed to helping our partners drive the next generation of software experiences,” said Ross Brown, Vice President of ISV and Solutions Partners for the Worldwide Partner Group at Microsoft. “Adding compatibility for the latest Microsoft technologies helps ISVs to stay ahead of the competition and give their customers access to cutting-edge technologies.”
“Adactus is excited to launch this new version of Pulse,” said James Smith, Head Of Software Development at Adactus Ltd. “Making our application compatible with Microsoft Windows Azure helps us offer our customers compelling benefits, including improved security and reliability features, tools to keep them connected to data stored on the Web, sophisticated management features and flexible access administration to improve mobile working.”
Pulse is a Microsoft approved cloud based application operating on the Windows Azure platform and offers unparalleled scalability, flexibility and security in claim management. Pulse is ideally suited to managing high claim volumes during claim surges experienced earlier this year. Commenting on this solution, James Smith, Head of Software Development at Adactus said “Our clients are demanding flexibility in the pricing and support models for claims technology solutions and Pulse has been built with that as its core objective. Volumes can vary dramatically based on a number of factors including the environmental conditions and the complexity of claim information required.
Yvonne Muench (@yvmuench) answered I’ve Built A Windows Azure Application… How Should I Market It? in an 8/19/2011 post to the Windows Azure blog:
We’ve heard from Independent Software Vendors (ISVs) that the technical aspect of building a Windows Azure application is sometimes the easier part. Coming up with the business strategy and transitioning a traditional company to Software-as-a-Service (SaaS) is often more challenging. My team, part of the Developer and Platform Evangelist (DPE) organization, works with ISVs who are building on Windows Azure. We’ve been talking to our early adopter ISVs to see what they're doing and how they’re mitigating the risks. We've found some interesting trends, which I highlighted in a session at the Worldwide Partner Conference (WPC) last month. Here’s a recap.
- UK: eTVMedia creates Enteraction, which enables businesses to “gamify” their applications
- Brazil: Connectt creates Zubit, which enables online brand managers to monitor social sites
- Mexico: Netmark creates WhooGoo, an online content and game platform for tweens.
The second trend focuses on packaging and pricing. Many ISVs evolve their businesses from within and restructure, retrain and hire new employees to succeed with SaaS. But because per-user, per-month subscription pricing can have a dramatic impact on traditional sales, billing, accounting and more, ISVs sometimes package and price their SaaS apps to mirror traditional on-premise pricing as closely as possible. For example, Xterprise built a new SaaS version of their RFID inventory solution. However, pricing for the SaaS app still uses an upfront license fee. There’s also an annual fee for support and maintenance – billed upfront. The main value for customer is not cost reduction, though there is some. Rather it’s a light footprint, ease of delivery and no need for IT on site. For Xterprise it’s essentially just a new SKU, with billing that is largely consistent with their on premise solution. Win, win.
My WPC panel session focused on three ISVs using the approaches above. David MacLaren, CEO of VRX Studios and MediaValet, had an existing business focused exclusively on the hospitality industry. When their new Windows Azure-based digital asset management solution began getting interest from diverse customers, his existing sales force didn’t have the expertise to effectively field requests that weren’t coming from hotels. He created a spinoff for MediaValet and hired new sales people. The spinoff now has the right sales and marketing personnel to serve a much more diverse customer base than the parent company, but it can still tap into VRX Studios for important resources. such as C-level executives, who split their time between the two companies. The spinoff also benefits by receiving free cash flow from the parent company, and sharing office space.
Markus Eilers, CEO of runtime and pulsd, is another example of creating a spinoff to market and sell a Windows Azure-based app. The parent company offers CRM and business processsolutions for medium sized businesses. pulsd was designed to reach the smallest SMBs. As a result, the direct sales approach they had been using was no longer tenable. They had to switch to marketing driven demand generation. By creating a spinoff, Markus was able to hire the right talent, including a new brand agency that helped create a good looking, easily navigable web site for effective digital marketing. Another important factor leading to the spinoff was the rapid update cycle of their SaaS apps. The requirements for rapid almost continuous updates had profound effects on developers, engineering and marketing. It required a very different approach from the parent company. Quite simply, it was easier to start anew than to change from within.
On the other hand, DMTI Spatial is changing their company from within. They provide digital mapping data and Geographic Information System (GIS) software to large enterprises in Telecommunications, Banking, Insurance and the Public Sector. They have already moved their flagship address recognition application, LocationHub, to Windows Azure, and they intend to move several more. Arthur Berrill, VP Technology explained that it hasn’t been easy. The sales force was very concerned with LocationHub moving to the cloud. Used to very large deals with very large commissions, they wondered how they could make money on the SaaS version. Additionally, accounting and billing asked how they could possibly bill for each transaction. So the company decided to price primarily through large blocks of transactions, billed up front. Think of it as a massive prepaid phone card. The result mirrors the billing size and frequency from the previous LocationHub and minimizes the needed internal change.
For those of you embarking on building SaaS apps on Windows Azure or advising ISVs that do so, here are some business strategy questions to consider:
- What customers will the SaaS applications serve? The same? New?
- What are the goals with the existing traditional on premise business? Keep it going? Retire it?
- How will the new applications be priced? Different or similar to the existing business?
DMTI Spatial serves the same customer base with the new version of LocationHub. They want to migrate each and every existing customer to this new version and retire the existing non-SaaS version within 2 years. And they're currently pricing in a very similar manner to traditional software licensing. As a result, they didn’t need to create a separate entity, but rather re-organized the company to align to the new goals and focused on internal change management to be successful.
With their new internal tool, MediaValet evolved to a product when they discovered external interest. It then logically evolved into a separate entity because it was focusing on a much broader customer base. Their existing team at VRX Studios, focused squarely on hospitality customers, couldn’t meet the needs of the new customers. With pulsd, they went from serving medium enterprises to very small businesses, which required marketing led demand generation instead of direct sales. Additionally, their pricing is dramatically different, using a freemium model – free entry-level apps, with fee-based versions after a certain threshold. As a result, the “learn, try, buy” experience is entirely self-serve. Lastly, their rapid product iterations had internal ripple effects but especially for engineering and marketing. Both VRX Studios and runtime created new entities in order to successfully pursue the opportunities created by their Windows Azure applications.
I think what our ISV’s are doing with Windows Azure is very interesting. and demonstrates that the cloud is definitely not just a technical decision. Stay tuned; I’ll continue to share things we learn and trends we’re seeing with ISVs using Windows Azure to grow their business. Also check out case studies on these panelists: VRX Studios, runtime, DMTI Spatial.
Yvonne is Group Manager, Azure ISV Market Insights, Azure ISV Incubation Team, Developer and Platform Evangelism Group:
Yvonne’s team provides business and technical readiness and co-marketing support to ISVs building on the Windows Azure platform; the team also provides research and insights about those ISVs.
The Microsoft Enterprise Library vNext team recently started a Windows Azure Integration Pack Forum:
<Return to section navigation list>
Visual Studio LightSwitch and Entity Framework 4.1+
• Paul Patterson described Microsoft LightSwitch – Creating a Custom Business Type in an 8/19/2011 post:
As a data centric application development tool, Microsoft Visual Studio LightSwitch 2011 provides a set of out of the box data types that satisfy most your application data requirement needs. Simple string, number, and date and time data types are probably the most typical data types that anyone would ever need to use. The following article demonstrates how to create and use a custom business type for your LightSwitch application.
Note that this post is based on what I learned from following along the MSDN walkthrough here: http://msdn.microsoft.com/en-us/library/hh290140.aspx
Data Types
In case you didn’t already know, LightSwitch natively offers .Net and SQL Server based data types. This is because the engine that builds data entities (the “tables”) based on the .Net Entity Framework and SQL Server technologies – which is why SQL Express is required and installed during the installation of the LightSwitch tool itself.
As an alternative to using the native data designers in LightSwitch, you can design your database entities outside of LightSwitch, and then add the new data source to your LightSwitch application. Doing this presents a few different challenges, but for scope for this article, I am going to assume you only want to learn about using business types using the native capabilities of LightSwitch.
So, as a refresher, here is a list of the data types that LightSwitch has available; when creating applications using LightSwitch to create entities (tables) via the built in designers.
- Binary – A variable length array of bytes (one “byte” is an 8-bit integer ranging in value from 0 through 255).
- Boolean – A value that is either True or False.
- Date – This is a value that is a date – hello!
- DateTime – A DateTime type is a value that contains a both the date and time.
- Decimal – A value with up to 28-29 digits after a decimal point.
- Double – A value that is 15-16 digits precision, with a floating decimal point.
- EmailAddress – an e-mail address (which is actually a string type).
- Image – The Image type is actually a Binary type that is treated as an image.
- Integer – A number between -2,147,483,648 and 2,147,483,648.
- Long Integer – A number between -9,223,372,036,854,775,808 and 9,223,372,036,854,775,807.
- Money – The Money type is actually a Decimal type, but is treated as monetary value in LightSwitch.
- Phone Number – Like the EmailAddress type, the phone number data type is actually a string, but is treated as a phone number in LightSwitch.
- Short Integer – A number between -32,768 and 32,767.
- String – any sequence of (Unicode) characters.
The list above shows what is natively available for data types from within LightSwitch. If you happen to look deeper into the SQL Server and .Net technologies used in LightSwitch, you’ll find that some of those data types don’t actually exist. The Email Address and Phone Number data types are examples of this. While the email address and phone number data types are available within the LightSwitch data designers, they are actually not native .Net data types.
Using LightSwitch extensibility features, Microsoft has provided custom data types that are based on native .Net data types. For example, the Phone Number data type is actually an extension of the String data type. The phone number is actually a value that is made up of string characters. The only difference is that LightSwitch will treat the phone number data type differently than a string.
By creating data types via the LightSwitch extensibility framework, custom data types can be configured with constraints and rules. In the example of the phone number data type, the phone number is constrained to contain a certain number of characters, such as only numbers and special characters that you would see in a phone number.
Business Types
A Business Type is essentially a data type that provides customized formatting, validation, and/or options to a data type. By extending an existing data type, a business type can be created to customize the way data is stored in the application.
Let’s use an example to better explain what a LightSwitch Business Type is, and how it is created and used in an application.
For the rest of this article, I am going to create a Business Type that I want to use for saving geographical information in my application. Specifically, I want to store location data in the form of latitude and longitude values. So I will create two Business Types named Longitude and Latitude.
Rules
First off, I need to understand the rules that apply to my new Business Types. Here are some descriptions, properties and rules for these new Business Types…
A Latitude is the angular distance of a location on Earth, South or North of the Equator. Usually written in degrees where the Equator is 0° (zero degrees), the North Pole is 90° north (90° N or +90°) and the South Pole is 90° south (90° S or -90°). 90° is the max on either side.
A Longitude is the angular distance of a location on Earth, East or West of the Prime Meridian . Similar to the Latitude, the Longitude is (typically) expressed in degrees where Longitudes east of the Prime Meridian are positive (+180° or 180° E) and those west of the Prime Meridian are negative (-180° or 180° W). 180° is the max on either side.
The measurement used for latitude and longitude locations can be measured entirely in degrees, or with minutes and seconds. For the sake of this article, I am only concerned with using a degrees only measurement. In other words – I am too lazy to write such a large Business Type and will take the easy way out on this one J
So, I begin…
Create the Extension Project
By the way, it’s assumed that you have the following prerequisites already installed…
- Visual Studio 2010 SP1 Professional or better.
- Visual Studio 2010 SP1 SDK
- Visual Studio LightSwitch 2011
- Visual Studio LightSwitch 2011 Extensibility Toolkit
First, I open Visual Studio and select to create a new LightSwitch Extensibility project (I am going to use the Visual Basic version).
After clicking the OK button, Visual Studio creates a solution based on templates available from the extensibility toolkit. You’ll notice in the Solution Explorer that there are a number of projects created.
Creating the Business Type
In the Solution Explorer of the new project, right-click the .LSpkg project, and select to add a New Item…
In the Add New Item dialog, select to add a Business Type, and give the new business type a name. In my example, I called my first business type Latitude.Isml…
Selected to add the new business type causes Visual Studio to do some magic based on the template selected. With the example I used above, here is what my Visual Studio looks like after I clicked the Add button…
Holy crap! That’s a lot of stuff. And I am not about to explain what each and everything in there does. How about we just move on and learn about each of the necessary elements as we work through the creation of the business type, okay?
With the shell of a business type created, the next thing to do is define what kind of business type it is going to be. A business type overrides an existing, or “base”, business type. For this Latitude business type I am going to use the Decimal base data type.
In the solution explorer, find the LocationCustomBusinessType.Common project. Location and open the Latitude.lsml. Edit the UndelyingType attribute of the SemanticType element to “:Decimal”..
Doing the above will tell LightSwitch that the data type used for the business type is actually a decimal, but will provide a user descriptive name of “Latitude” to the user when the user sees in the data type drop down list of the table designer.
Using a Default Control
Having created the new business type, I want to now define what should be used as a default control for viewing and editing the data for it.
In the LocationCustomBusinessType.Client project folder of the solution explorer, locate and open the LatitudeControl.xaml file within the Controls folder of the Presentation folder.
By default, the template created a simple textbox control for the new business type. This is fine for this particular business type. As an option, I could probably replace this textbox with something like a custom control specifically to deal with a decimal. Instead, I am going to continue to use the textbox control however I am going to add some logic directly here to so that I constrain the textbox to only allow for a decimal value.
In the XAML of the LatitudeControl.xaml file, update the XAML using the below information.
<UserControl x:Class="LocationCustomBusinessType.Presentation.Controls.LatitudeControl" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:slu="clr-namespace:Microsoft.LightSwitch.Utilities.SilverlightUtilities;assembly=Microsoft.LightSwitch.Client" ToolTipService.ToolTip="{Binding Description}"> <Grid> <Grid.ColumnDefinitions> <ColumnDefinition Width="*"/> </Grid.ColumnDefinitions> <TextBox Grid.Column="1" Text="{Binding StringValue, Mode=TwoWay}" Name="LatitudeTextBox" /> <slu:ValidatableContentControl ValidationDataSource="{Binding StringValue}" /> </Grid> </UserControl>Validating the Business Type
In LightSwitch, validating a value entered for a business type requires that the validation occur at both the presentation layer and the server. This is what the .Common project is for.
In the LocationCustomBusinessType.Common project right-click the project folder and select to add a new class to the project.
I named the class BusinessTypeExtensionModule.vb, and replaced the contents of the generated file with the following…
Here is the code that you can copy and paste from if you like…
Friend Module BusinessTypeExtensionModule Private Const Name As String = "LocationCustomBusinessType" Private Const ModulePrefix As String = Name + ":" Private Const AttributePrefix As String = ModulePrefix + "@" Friend Class Latitude Private Const Name As String = "Latitude" Public Const GlobalName As String = ModulePrefix + Name Friend Class ValidationAttribute Private Const Name As String = "LatitudeValidation" Public Const GlobalName As String = AttributePrefix + Name End Class End Class End ModuleMake sure to save everything.
Now right-click the same LocationCustomBusinessType.Common project and select to add a new class file named LatitudeValidation.vb. Here is what I added to this file…
Imports System Imports System.Collections.Generic Imports System.ComponentModel.Composition Imports System.Linq Imports Microsoft.LightSwitch Imports Microsoft.LightSwitch.Model Imports Microsoft.LightSwitch.Runtime.Rules Public Class LatitudeValidation Implements IAttachedPropertyValidation Public Sub New(attributes As IEnumerable(Of IAttribute)) Me.attributes = attributes End Sub Private attributes As IEnumerable(Of IAttribute) Public Sub Validate(value As Object, results As IPropertyValidationResultsBuilder) Implements Microsoft.LightSwitch.Runtime.Rules.IAttachedPropertyValidation.Validate If value IsNot Nothing Then ' Ensure the value type is a decimal. If GetType(Decimal) IsNot value.GetType() Then 'Throw New InvalidOperationException("Unsupported data type.") results.AddPropertyError("Value must be a decimal value between -90.0000000 and 90.000000") Else Dim decValue As Decimal = DirectCast(value, Decimal) ' First validation rule: value should be greater than 0. If decValue < -90 Then results.AddPropertyError("Value should be greater than or equal to -90") ElseIf decValue > 90 Then results.AddPropertyError("Value should be less than or equal to 90") End If End If End If End Sub End ClassAt this point you might be able to see some fundamental Model View ViewModel (MVVM) fundamentals being applied. The LatitudeValidation class implements the IAttachedPropertyValidation interface. The IAttachedPropertyValidation implements a Validate method which is called anytime the attached property (the business type value) is validated.
Now I have to add the attach the validation above to the business type itself.
Back in the LocationCustomBusinessType.Common project, I open the Latitude.Isml file. Update the file with the following new tags…
This new information is used to reference the validation code for the business type. It tells the Latitude “Model” that the value of the Latitude business type should be validated using the code added on the common project in the LatitudeValidation class.
Back in the LatitudeValidation.vb file of the LocationCustomBusinessType.Common project. Add the following class, named LatitudeValidationFactory class, after the existing LatitudeValidation class…
The above image is a bit difficult to read, so to make it easy for you, here is the entire file contents for you to copy and paste from…
Imports System Imports System.Collections.Generic Imports System.ComponentModel.Composition Imports System.Linq Imports Microsoft.LightSwitch Imports Microsoft.LightSwitch.Model Imports Microsoft.LightSwitch.Runtime.Rules Public Class LatitudeValidation Implements IAttachedPropertyValidation Public Sub New(attributes As IEnumerable(Of IAttribute)) Me.attributes = attributes End Sub Private attributes As IEnumerable(Of IAttribute) Public Sub Validate(value As Object, results As IPropertyValidationResultsBuilder) Implements Microsoft.LightSwitch.Runtime.Rules.IAttachedPropertyValidation.Validate If value IsNot Nothing Then ' Ensure the value type is a decimal. If GetType(Decimal) IsNot value.GetType() Then 'Throw New InvalidOperationException("Unsupported data type.") results.AddPropertyError("Value must be a decimal value between -90.0000000 and 90.000000") Else Dim decValue As Decimal = DirectCast(value, Decimal) ' First validation rule: value should be greater than 0. If decValue < -90 Then results.AddPropertyError("Value should be greater than or equal to -90") ElseIf decValue > 90 Then results.AddPropertyError("Value should be less than or equal to 90") End If End If End If End Sub End Class <Export(GetType(IValidationCodeFactory))> <ValidationCodeFactory(BusinessTypeExtensionModule.Latitude.ValidationAttribute.GlobalName)> Public Class LatitudeValidationFactory Implements IValidationCodeFactory Public Function Create(modelItem As Microsoft.LightSwitch.Model.IStructuralItem, attributes As System.Collections.Generic.IEnumerable(Of Microsoft.LightSwitch.Model.IAttribute)) As Microsoft.LightSwitch.Runtime.Rules.IAttachedValidation _ Implements Microsoft.LightSwitch.Runtime.Rules.IValidationCodeFactory.Create If Not IsValid(modelItem) Then Throw New InvalidOperationException("Unsupported data type.") End If Return New LatitudeValidation(attributes) End Function Public Function IsValid(modelItem As Microsoft.LightSwitch.Model.IStructuralItem) As Boolean Implements Microsoft.LightSwitch.Runtime.Rules.IValidationCodeFactory.IsValid Dim nullableType As INullableType = TryCast(modelItem, INullableType) ' Get underlying type if it is a INullableType. modelItem = If(nullableType IsNot Nothing, nullableType.UnderlyingType, modelItem) ' Ensure that type is a decimal semantic type, or that a type inherits from it. While TypeOf modelItem Is ISemanticType If String.Equals(DirectCast(modelItem, ISemanticType).Id, BusinessTypeExtensionModule.Latitude.GlobalName, StringComparison.Ordinal) Then Return True End If modelItem = DirectCast(modelItem, ISemanticType).UnderlyingType End While ' If the conditions aren't met, LightSwitch will not display the validation rule for ' this model item. Return False End Function End ClassAnd that is pretty much all I needed to do.
Testing
Testing the custom business type is easy. Right-click the My Project item in the LocationCustomBusinessType.Vsix project, and select Open.
In the LocationCustomBusinessType.Vsix window;
- Update the Start external Program value to be “C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\devenv.exe” (no quotes). This should be the location of the Visual Studio IDE executable, And
- Add “/rootsuffix Exp” (no quotes) to the Command line arguments text box.
Also, make sure that the LocationCustomBusinessType.Vsix project is set as the startup project in the solution.
Configuring the above will cause a new instance of Visual Studio to launch, and install the Vsix (the Visual Studio Extension) for the purpose of testing within a new LightSwitch application. Pretty sweet huh!?!
So, having said that, let’s fire this up! Hit F5 to start debugging.
With the new instance of Visual Studio open, select to create a new LightSwitch application. In my example, I created a new LightSwitch application named LSTestNewBusinessType.
Once the new project is created, open the project properties window and select to view the Extensions tab. In the Extensions tab the LocationCustomBusinessType should show as an available extension. Check the extension too enable the extension for the project (no need to check the Use in new projects checkbox, we are only testing right now).
With doing the above, I should now be able to use my fancy new custom business type. So, I create a new table and presto, the new business type is available as a selection from the list of types available…
Saweet! Now to prove that the validation is working, I created a simple ListDetails screen and started playing…
Okay, that was just way to[o] easy.
Conclusion
So this was an extremely simple example of a creating a custom business type. If I had a few days of uninterrupted time, I could have (and kinda want to) build a very intuitive geographical location business type – complete with options for view different formats of locations and even some mapping…. but I don’t have that kind of time, so go hard!
None-the-less, this was a fun exercise and definitely worth doing. I learned a lot, I am sure you can, to[o].
Beth Massi (@bethmassi) announced Metro Theme Released! Plus More Themes from the Community in an 8/19/2011 post to the Visual Studio LightSwitch Team (@VSLightSwitch) blog:
Last night the LightSwitch Team released the Metro Theme for Visual Studio LightSwitch on the Visual Studio Gallery. You can also download it directly from the Extension Manager inside Visual Studio LightSwitch.
Download the Metro Theme for Visual Studio LightSwitch
More Themes from the Community
There are more and more LightSwitch extensions being built and released on Visual Studio Gallery! Check out these latest themes from the community and our partners.
- Luminous Themes by Yann Duran (Luminous Software)
- In4MA Theme by Jamil MilheM
- NetAdvantage for Visual Studio LightSwitch 2011 Vol. 1 Light (Free)
This free pack of controls from Infragistics contains two free themes and handy controls. Check out more extensions from Infragistics here.- Want to build your own themes and extensions? Visit the Extensibility tab on the LightSwitch Developer Center to learn how.
Paul Patterson explained Microsoft LightSwitch – Installing Themes in an 8/19/2011 post:
The extensibility of Microsoft Visual Studio LightSwitch will certainly be the driver to a great many third party features to be offered in the near future. As such, it is worth a post to help understand how to install one of these extensibility features – themes. Here is how…
Themes are made available to LightSwitch through the extensibility framework that is part of LightSwitch. More specifically, theme extensions are created and then packaged as installable extensions for the Visual Studio Integrated Development Environment (IDE), which is used in developing LightSwitch applications.
In a nutshell, using the Visual LightSwitch Extensibility Toolkit, a theme is created and then packaged up as .VSIX file. When downloaded and executed, the VSIX file is registered as an extension for Visual Studio. When Visual Studio is launched for LightSwitch, the extension is then discovered and loaded as something to use for a LightSwitch application. Once loaded, the developer can select to use the extension.
Here is how an example theme extension is loaded and used…
Method 1 – Manual Download and Installation
Prerequisite: If Visual Studio is open, close it.
Locate the downloadable file, a .vsix file, and save it to your computer.
Double click the file, or right-click the file and select Open. The following example shows a Microsoft.LightSwitch.Metro.Theme.vsix file that I downloaded onto my hard drive…
In the Visual Studio Extension Installer that opens, make sure the Microsoft LightSwitch is checked, and then click the Install button.
The installer will notify you that the installation was complete…
Method 2 – Using the Extension Manager
Microsoft has a nifty little tool that developers can use directly from Visual Studio. It is called the Extension Manager. The Extension Manager is a tool that provides for some searching and installing of extensions, such as LightSwitch Themes. Themes that are published to the online Visual Studio Gallery are accessible via the Extension Manager tool.
To launch the tool, start up Visual Studio and select Tools, and then Extension Manager from the file menu…
In the Extension Manager window, select the Online Gallery item on the left. Then enter “LightSwitch Theme” in the search box in the upper right. While typing, the Extension Manager will dynamically search for matches. In the Search Results panel in the middle, locate the theme and click the Download button…
A download status dialog will appear, indicating the status of the download.
Once downloaded, the installer will automatically launch. Click the install button to install the extension.
Once the installation is completed, your web browser may open to the site containing information about the extension, for your review.
Back in Visual Studio, the Extension Manager will be refreshed to show the newly installed extension. The Extension Manager will also notify you that Visual Studio must be restarted. Click the Restart Now button.
Using the Theme
With the extension installed, the theme can easily be implemented. Open up LightSwitch and select to either create a new project, or open an existing one. In the solution explorer of the project, select to view the project properties. For example,
In the properties window, select the Extensions tab. You will see that the new theme extension is installed. To enable the theme for use by the project, select the theme by turning on the checkbox beside it. To ensure the theme is also automatically available for other projects, check the Use in new projects checkbox also.
Finally, to apply the theme to the project, select the General Properties tab from the same properties window. Update the Theme property by selecting the new them from the theme dropdown field. In the example below, I selected the LightSwitch Metro Theme that was just installed and enabled.
Finally, run the project. If all went as planned, the new theme should be enabled and used in the application…
That’s a great lookin’ app!!
That was pretty easy eh? Installing most extensions is just as easy. When I start messing around with other extensions I’ll be sure to also post how to install and use them too.
Nicole Haugen reminded LightSwitch developers on 8/19/2011 that How To: Handle Database Concurrency Issues (Nicole Haugen) is available on her blog:
Concurrency issues are common in multi-user applications that have a database on the backend.
When a conflict is detected, a “Data Conflict” screen is automatically displayed to the user allowing them to choose how to resolve the conflict
Or, for a more customized solution, code can be written so that the application resolves the conflict without requiring interaction from the user
To find out more information on how to handle concurrency issues using LightSwitch, refer to my recent blog post called "How To: Handle Database Concurrency Issues".
Also, check out other "How To" posts on my blog.
Return to section navigation list>
Windows Azure Infrastructure and DevOps
Bruce Kyle reminded developers about Deals for Getting Started on Windows Azure Extra Small Compute in an 8/19/2011 post to the US ISV Evangelism blog:
The price of a Windows Azure extra small compute instance will be reduced by 20 percent beginning October 1. You can also use extra small compute hours to power small instances at a ratio of three extra small compute hours per one hour of small compute.
For details, see the Windows Azure team blog post Announcing Simplified Data Transfer Billing Meters and Swappable Compute Instances.
To determine the resources you receive with the various instances, see Windows Azure Compute.
Cory Fowler (@SyntaxC4) explained How to Speed up Windows Azure Deployments in an 8/18/2011 post:
The most common question I get asked about Windows Azure [aside from cost] is “How do I speed up my Deployment”. In this post I’d like to address this question and hit on a few things that most people don’t really think about in terms of what it means to deploy to Windows Azure.
Upgrade Hosted Service instead of New Deployment
Gotchas
Cannot:
- Change the Type of Role that is Deployed: Web, Worker or VM Role
- Increase the Size or Number of Instances of your Service
- Change Configuration Settings or Values
- Add/Change the number of Input or Internal Endpoints or their Values
- Add/Change Certificates deployed to your instance
- Add/Change Local Resources for your instance
Limit Your Startup Tasks or Move to a VM Role
Startup tasks are one of my favorite additions to the Windows Azure Deployment Model. They introduced the ability to add required functionality to the Guest OS which would have been rather tedious or almost impossible to set up using C# during the Role Startup [Read more about the Windows Azure Role Startup Life Cycle].
Even though I love startup tasks so much you need to learn when enough is enough. Even though there is a timeout for startup tasks, any lengthy execution of a startup task does effect the startup time of your role.
Introducing the VM Role
With the Role Startup Cycle being a continuous process [occurs every time the role is recycled or scaled out]. The VM Role takes the repetition out of the process as additional functionality or components can be pre-loaded/configured then the entire VM can be uploaded in order to replace the typical Windows Azure Guest OS.
Learn how to Create a VM Role and get rid of all of the overhead of countless installations of the same software in your critical business application.
VIP Swap to Avoid Downtime or Update Deployment
The benefit of the VIP [Virtual IP] SWAP is it allows a brand new service to be deployed [in the staging slot of your Windows Azure Environment] then Upgraded to the Production Slot to be accessed by the World. This provides the benefit of minimal downtime as the VIP Swap operation does an internal DNS Change within the Windows Azure Platform. If the deployment requires a to change the physical deployment of your application [which cannot be performed during an in-place upgrade] the VIP swap is a perfect scenario to deploy an enhancement to your application.
To do a VIP Swap you must have at lease an application deployed to the Staging Slot. [Note: A deployment cannot be VIP swapped from Production to an empty Staging Slot.]
Windows Azure Accelerator for Web Roles
Although this functionality is not officially supported by Microsoft it provides the ability to quickly deploy or update multiple websites deployed to Windows Azure. If you overload your Web Roles by adding multiple Websites using the full IIS features in Windows Azure, this is definitely a no brainer.
The Windows Azure Accelerator for Web Roles deploys websites to a Windows Azure Storage account which is frequently polled by a Worker Process [overridden Run method in the RoleEntryPoint] in your Web Role. When changes are found in the Storage Account, the Worker Process pulls the new bits into the Directory in which the website is set up in IIS within your Windows Azure Web Role.
Keep in mind that the Windows Azure Accelerator for Web Roles is a solution that is deployed to the Web Role Instance, which means regardless of if you have an application deployed through it you will be playing for your instance to be deployed. The full time deployment of this Accelerator does give the advantage of a deployment time of mere seconds.
Conclusion
Hopefully this post has helped you understand how to potentially increase deployment times in Windows Azure.
Adam Hall described Working with the Orchestrator Web Service–evidence and guidance from the CEP! in an 8/18/2011 post to the System Center Team blog:
One of the outcomes we look for in the running of the Evaluation Programs is the emergence of leaders in the community. We encourage and reward the CEP participants for their efforts in helping us build the communities around the System Center solutions, and each time we run a program we discover new rockstars who are deserving of some recognition.
Today I want to showcase the work that Damian Flynn has been doing with Orchestrator in the Orchestrator Evaluation Program.
There are 2 significant contributions that Damian has made in recent weeks. Firstly, he has created and published an Integration Pack for F5. You can read about the IP and I hope that if you use F5 solutions that you investigate his work and find it useful.
Secondly, Damian has engaged with the Orchestrator engineering team and learnt how to work with the new web service that is part of Orchestrator. The new web service is a critical new component and it’s great to see the interest that people have shown in wanting to know how to work with it. We are still building the official documentation and guidance, so Damian took the initiative and has written a series of articles about it!
You can read his articles on Orchestrator here. He starts with an article outlining the web service itself and how to work with it in Visual Studio, follows this up with guidance on how to start a Runbook via the web service and then closes out with the injection of Runbook parameters when doing the initiation.
Thank you Damian for your efforts and initiatives, and I hope this serves as encouragement for anyone else out there who wants to get involved!
<Return to section navigation list>
Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds
Young Chow posted Announcing Upcoming TechNet Events in US East Region on 8/19/2011:
- Get Ready for the Cloud with Hyper-V (pre-event intro session) 60 mins
- The IT Pro's Heaven: The Private Cloud 75 mins
- SCVMM 2012: From Virtualization Management to Private Cloud 75 mins
- Lunch Break
- Getting Started Building Your Private Cloud 75 mins
- You've Got a Cloud: Familiar Tools to Manage It! 60 mins
- Bonus Session and Wrap-Up 30 mins
Notice the Hyper-V session is intended for those who are not familiar with or needs a refresh course of Hyper-V. We will go through the essential steps and operations to set up a Hyper-V environment for running a virtualization solution. You will need to understand the fundamentals of Hyper-V to fully benefit from most of the sessions. We are going to make it as easy and convenient as possible for you to master Hyper-V and subsequently private cloud, at the same time you will need to do your part. So, please come prepared.
What you will get from this full day event is sufficient technical depth for you to start building up your skill portfolio of cloud computing, particularly on private cloud. You will have all the information needed to learn, build, and evaluate a private cloud yourselves. As needed, you should consider acquiring hardware with hardware virtualization and with 8 or more GB RAM to build a test lab that is affordable and with enough resources to do some adequate testing.
If not already, do access and download the following free trainings and trials and start getting yourselves better with the following training and software which will facilitate your learning on private cloud:
- Microsoft Virtual Academy – highly recommended, a lot of great contents here
- Windows Azure Applications
- Windows Server 2008 R2 w/ SP1
- Hyper-V Server 2008 R2 SP1
- System Center Virtual Machine Manager 2008 R2 SP1
- System Center Virtual Machine Manager 2012 Beta
- System Center Operations Manager 2007 R2
- System Center Operations Manager 2007 R3
- System Center Configuration Manager 2012 Beta 2
Bookmark and watch closely on the announcements made in http://technetevents.com on the event cities and dates. And follow your local IT Pro Evangelist on twitter and keep yourself informed of the latest information.
- Bob Hunt (bobh@microsoft.com, @bobhms) covering NY, NJ, PA, Southeastern CT, DE
- John Baker (john.baker@microsoft.com, @johnbake) covering GA, SC, AL
- Blain Barton (blainbar@microsoft.com, @blainbar) covering AL, MS, and FL
- Yung Chou (yung.chou@microsoft.com, @yc8050) covering DC, VA, MD, WV, NC
- Dan Stolts (dstolts@microsoft.com, @itproguru) covering ME, VT, NH, MA, RI, CT
And we look forward to seeing you all at the events.
Notice that the date is after the BUILD Conference (9/13 - 9/16/2011), so the sessions can take advantage of content whose nature is being withheld so far from attendees. Hopefully the events will cover hybrid cloud infrastructure with more details on System Center App Controller (formerly Project “Concero.”)
<Return to section navigation list>
Cloud Security and Governance
No significant articles today.
<Return to section navigation list>
Cloud Computing Events
See Young Chow posted Announcing Upcoming TechNet Events in US East Region on 8/19/2011 in the Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds section above.
<Return to section navigation list>
Other Cloud Computing Platforms and Services
• Scott M. Fulton, III reported Macheen's Mobile Cloud-in-a-Box Goes Live with Sprint in an 8/19/2011 post to the ReadWriteCloud blog:
Here's a take on a cloud-based service you might not have considered (something new to attach an "-aaS" to): Suppose you're the producer or manufacturer of a class of device that should have Internet connectivity. One hypothetical possibility: an interactive touch panel for the back seats of taxicabs. Perhaps you've acquired some ARM licenses and you're ready to stamp out the device and put a case on it. But you're not in a position to house the data center or make a deal with broadband carriers to supply the connectivity, or maybe even to write all the back-end software that connectivity would require.
Premiering last February in Europe, a new company called Macheen is assembling a global cloud service that caters to an entirely new market: outsourced back-end support with connectivity built-in. Yesterday, the company went live with broadband service to the U.S. via Sprint, with the goal of endowing classes of portable devices of all sizes with built-in, cloud-based online support and services channels.
"We help with Internet-included devices - things like tablets, laptops, portable gaming devices, e-book readers, helping the device makers and their retailers include Internet capability hot-out-of-the-box, under their own brand and their own go-to-market channels," Macheen CEO Richard Schwartz told reporters at the CTIA 2011 conference last April. Dell has already signed on in Europe, Schwartz added, endowing certain of its laptops with instant connectivity to Dell services through Macheen's support system.
As a Macheen white paper tells prospective customers, "You partner with Macheen for the global rollout of connected devices you manufacture or retail. Devices access service via built-in or external modems, with optional customized client interfaces. Working with us, you define the offering to fit your business model. You market through your existing brand sales channels; we take the service risk and share royalties with you."
Last June in Los Angeles, Schwartz participated in a roundtable discussion hosted by White House Chief Technology Officer Aneesh Chopra, the subject of which could give you a big idea of where Macheen is going with this. The subject was putting broadband networks to use for public safety officials and first responders. As Chopra wrote last June, "The discussion provided helpful input for the proposed R&D portion of the public safety investment, as called for in the President's proposal, with emphasis on opportunities for pre-competitive R&D collaboration to accelerate the pace of innovation in delivering a cost-effective, secure, reliable network that meets public safety-specific requirements."
The U.S. Government has had difficulty of late in working out a communications scheme for first responders to be able to reuse public frequencies recently reclaimed from the UHF television spectrum. A cloud-based, broadband connectivity alternative based on "hot-out-of-the-box" components where the problems of managing communication are already solved, could revolutionize government's entire way of thinking about solving this problem.
This appears to me to be an interesting service, but existing cloud providers are likely to adopt such connectivity features, if Macheen’s service gains significant market share.
Carl Brooks (@eekygeeky) reported Amazon cloud measures up to enterprise at last in an 8/19/2011 post to SearchCloudComputing.com:
Almost since its inception, Amazon Web Services has been after the enterprise IT crowd, mainly because that's where the money is in IT.
Amazon is a long way from where it started, with a user base of open source nerds, technologists and the odd, off-the-wall startup.
Just ask Dell, Microsoft, Oracle or HP whether they prefer to sell to Joe Consumer, who's going to buy one computer or printer on margin prices, or to Enterprise Eddie, who's going to buy 100 computers and a $75,000 printer with a service contract to boot.
But it's been a tough sell; despite the powerful lure of easy-on, hands-off, pay-by-the-penny computing, members of the genus grumpius adminius are typically not interested in having less control of their IT resources.
Instead IT admins are interested in compliance with regulations, controlling the chaos factor and being able to get management out of their hair. Putting the company servers "in the clouds" does not appeal to enterprises; in fact, it generally sounds like a poor idea right out of the gate. But Amazon's cloud has come a long way, and signs indicate that it may have arrived.
Of course, it's always going to be a mixed bag. Cloud computing infrastructure is simply off the table for many enterprises, no matter how enterprise-friendly it's become. Mike Rigsby, network administrator at Beaverton, Ore.-based Vanguard EMS, says that owning their infrastructure is predicated on a pretty cut and dried business need. "We are an electronics manufacturing services (EMS) provider -- a contract circuit board manufacturer, in less confusing terms -- with military, aerospace, and medical customers," he said via email.
As such, data security and control are the top concerns for Rigsby, and there's so much gray area when it comes to what's regulated and what's not, it's easier on several fronts to do everything themselves. Rigsby said his infrastructure needs are not extensive and best practices in storage, backup and availability are easy to maintain. He's not a head-in-the-sander, though. Rigsby's shop has been using virtualization and management systems for 10 years and stays abreast of technology.
"We have server-backed applications, network-hosted drives for data storage, an email server, fault tolerant clusters, a backup solution," he said, all things touted as cost-effective to run in the cloud these days. Rigsby doesn't fault the AWS model, he just doesn't need it. However, that's a massive advance from the early days of AWS, which couldn't come close to that kind of basic, well-understood sophistication.
Discrete cloud meets unique enterprise needs
All of this has changed permanently. For instance, AWS just launched a discrete cloud environment specifically aimed at users like Rigsby, with complex regulatory needs. The GovCloud Region(U.S. only) is "designed to meet the unique regulatory requirements of the United States Government," and supports the International Traffic in Arms (ITAR) data standards. That's one example of how AWS has grown, to the point it can now segment its own service line to meet special needs.
But it has also changed access in fundamental ways that have led to novel uses for enterprise. Andrew Bartels, CTO of insurance brokerage PSA Insurance and Financial Services, said that AWS is a sterling, enterprise-grade infrastructure service. But it's not for running his line of business applications, it's more for disaster recovery (DR) planning.
"What I use AWS for is actually DR. The amount of money and time that's been put into creating multiple copies of our infrastructure just wasn't viable anymore," he said. Traditional DR planning and provisioning was expensive and broken, Bartels added; even after dropping many tens of thousands of dollars down a hole for DR hosting, he had close to no visibility into that infrastructure and no easy way to test it out or keep tabs on it.
Two factors went into Bartels' adoption of a cloud computingservice. First was bandwidth. His Maryland offices upgraded from a T-1 connection to a 100 MB fiber link to his provider a while back. That gave him the opportunity to look realistically at availability and accessibility for cloud computing services. Bartels said that opened the door to cloud services. "It's just not viable to deploy production systems when those systems are pushed through a [3 MB] pipe," he said.
The second ice breaker came only this year, with the general availability of Virtual Private Cloud (VPC) on AWS and the ability to functionally recreate his network topology. Bartels said that allowed him to look at DR in a whole new way.
Bartels said he can recreate his entire network, server images, connections, IP addresses and all, in minutes, update as needed, and then spin them all down and store the images and configuration for pennies a month. He also uses the Amazon Simple Storage Service (S3) for archive data and backups.
"When you talk about a disruptor, I can't think of anything more disruptive than S3. We were literally spending tens of thousands of dollars on proprietary backup," he said.
Bartels said he doesn't run day-to-day infrastructure at AWS because he's already got more than adequate server resources on-premises and the cost and benefit for a migration isn't there. However, he doesn't see any functional barrier to doing so in the future, especially since VPC supports more configurations after an August update. "I would love to move my entire infrastructure to Amazon cloud but it doesn't make financial sense given that I don't have a scalability need," he said.
The advent of the AWS Virtual Private Cloud in 2009 weakened any resistance to using Amazon Elastic Compute Cloud (EC2) many that IT professionals held; they went from categorically opposed to mildly interested in experimenting. Earlier this year, Amazon announced the availability of standard virtual private networking (VPN) features on the VPC, like being able to assign subnets, IP addresses and control routing behavior -- long-standing practice for any hosting provider or colocation facility, but not trivial for AWS' cloud environment. Amazon's ability to support common networking gear still limits this, according to Bartels.
"Amazon until recently required you to do you VPN with BGP [border gateway protocol] and they didn't allow you to run multiple tunnels. That was very scary to people," Bartels said. "I think what they did was secretly pick two vendors, Cisco or Juniper, and said we'll support these two."
However, in early August, AWS announced availability for new VPC features and identity and access management tools that did away with a lot of those objections. Now, with the exception of dedicated, exclusive hardware for individual users, AWS has caught up to traditional hosting providers and secure VPN connections. …
Carl continues with “AWS secures its footing with enterprise IT” and “Fueling the cloud computing spark” topics.
More from SearchCloudComputing on Amazon and the cloud:
Full disclosure: I’m a paid contributor to SearchCloudComputing.com.
Werner Vogels (@Werner) explained Driving down the cost of Big-Data analytics with Spot Instances in an 8/18/2011 post to his All Things Distributed blog:
The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud. Many of our Big-Data customers already saw a big drop in their AWS bill last month when the cost of incoming bandwidth was dropped to $0.00. Now, given that historically customers using Spot Instances have seen cost saving up to 66% over On-Demand Instance prices, Amazon EMR customers are poised to achieve even greater cost savings.
Analyzing vast amounts of data is critical for companies looking to incorporate customer insights into their business, including building recommendation engines or optimizing customer targeting. Hadoop is quickly becoming the preferred tool for this type of large scale data analytics. However, Hadoop users often waste significant intellectual bandwidth on managing clusters and running Hadoop jobs rather than focusing on creating value through analytics. Amazon Elastic MapReduce takes away much of this muck by providing a hosted Hadoop framework that enables businesses, researchers, data analysts, and developers to easily and efficiently spin up resizable clusters for distributed processing of large data sets.
An interesting observation is that data analytics is no longer the purview of large enterprises. Every young business launching today knows they must integrate data collection and analytics from the start. In order to compete in today’s market, these companies must have a deep understanding of their customers’ behavior, allowing them to continuously improve how they serve them. Launching a business with a minimally viable product and then rapidly iterating in the direction that customers lead them is becoming a standard approach to success. However, this cannot be done without efficient, scalable data analytics. Many of these startups are using Hadoop for data processing and Amazon Elastic MapReduce is the ideal environment for them: it provides instant scalability and lets them focus on analytics while EMR handles the hassle of running the various Hadoop components. Given the initial shoestring budget of many of these new companies, driving down the overall cost of analytics using Spot Instances is a huge benefit.
There are three categories of instances in an Amazon EMR cluster: 1) the Master Instance Group which contains the Hadoop Master Node that schedules the various tasks, 2) the Core Instance Group which contains instances that both store the data to be analyzed and run map and reduce tasks, and 3) the Task Instance Group which only runs map and the reduce tasks. For each instance group, you can decide to use On-Demand Instances (possibly from your Reserved Instances pool) or Spot Instances. If you choose to use Spot Instances you provide the bid price you are willing to pay for each instance in that group. If the current Spot Price is below the bid price, the Instance Group will launch. The instance groups in which Spot Instances are appropriate depends on the use case. For example, for data-critical workloads you might decide to run only the Task Group on Spot Instances, with the Core Group on On-Demand, while if you are performing application testing you may decide to run all Instance Groups using Spot Instances.
If you want a quick introduction on how to get started with mixing Spot Instances with On-Demand Instances in an Amazon EMR cluster, watch this Getting Started Video. More details can be found in the Spot Instances Section of the Amazon Elastic MapReduce Developer Guide. The posting on the AWS developer blog also has some more background.
Jeff Barr (@jeffbarr) described Four New Amazon EC2 Spot Instance Videos in an 8/18/2011 post:
If you have read about the Amazon EC2 Spot Instances but are not quite sure how to get started, how to make cost-effective bids, or how to build applications that can manage interruptions, I've got a treat for you. We've put together a trio of short videos that should give you all of the information you'll need to get started. Here is what we have:
Getting Started with EC2 Spot Instances - This video shows you how to launch, view, and cancel spot instance requests using the AWS Management Console. You will see how to log in to the console, select an AMI, and create a bid.
- Spot Instance Bidding Strategies - This video shows how customers choose bid strategies that have resulted in savings of 50-66% from the On-Demand pricing. Four strategies are
covered: bidding near the hourly rate for Reserved Instances, bidding based on previous trends, bidding at the On-Demand price, and bidding higher than the On-Demand price. Watch this video to learn when it is appropriate to use each of these strategies. The video also references a particle physics project run by the University of Melbourne and the University of Barcelona on Spot Instances. More information about this project is available in a recent case study.
Spot Instance Interruption Management - This video outlines multiple ways to create applications that can run in an environment where they may be interrupted and subsequently restarted. Strategies include using Elastic MapReduce or Hadoop, checkpointing, using a grid architecture, or driving all processing from a message queue.
- Using EC2 Spot Instances With EMR - This video shows how to run Elastic
MapReduce jobs on a combination of On-Demand and Spot Instances to get work done more quickly and at a lower cost. The includes guidelines for choosing the right type of instance for the Master, Core, and Task instance groups.
I hope that you find these videos to be interesting and informative, and that you are now ready to make use of EC2 Spot Instances in your own application.
<Return to section navigation list>
Atom 1.0 site feed
0 comments:
Post a Comment