Thursday, May 31, 2007

Matt Warren's "The Origin of LINQ to SQL"

Read the inside story of how LINQ to SQL and LINQ itself came to be, learn how ObjectSpaces was drawn into the black hole of WinFS, and discover that Microsoft's second attempt to productize an object/relational mapping (O/RM) tool started "the political nightmare that became [Matt's] life for the next three years."

The first major event after the Microsoft Business Division (MBD) absorbed the Server and Tools Business (STB) unit was to deep-six this year's Professional Developer's Conference. What's next on MBD's chopping block?

  • LINQ to SQL?
  • LINQ to Entities, Entity Framework, and Entity Data Model?
  • LINQ itself?

Update: 6/2/2007: ADO.NET Architect Mike Pizzo states in a comment to my May 29, 2007 Defining the Direction of LINQ to Entities/EDM post's items #1 and #4:

The bottom line on Entity Framework and LINQ to SQL was that they are each designed around a different set of priorities; LINQ to SQL is designed around rapid development, with direct (1:1) mapping, while the Entity Framework is designed around exposing a rich conceptual model with flexible mappings to extended ADO.NET Data Providers. We considered not shipping one or the other of these to reduce confusion, but ultimately decided that neither stack fully addressed the customer commitments and expectations set by the other. So we will ship both; LINQ to SQL in Orcas and the Entity Framework in an update to .NET Framework 3.5 in first half 2008. Over time we will look to customers to help us decide whether it makes sense to continue to invest in enhancing both stacks or merging into a single stack over time.

It’s an undeniable fact that Microsoft has fumbled the ball repeatedly when it comes to delivering an O/R solution. The biggest reason we didn’t ship ObjectSpaces was not technical, but rather a lack of recognition at a management level of the importance of providing such a solution. The only way for ObjectSpaces to gain traction was by aligning with something bigger (i.e., MBF or WinFS). The biggest thing that’s changed (thanks largely to the success of Hibernate) is that management now gets the importance of a good O/R solution. The support for LINQ, LINQ to SQL, and the Entity Framework at the executive level is what ObjectSpaces never enjoyed, and what gives me confidence that we will finally shed the albatross that’s been hanging around our neck since .NET Framework 1.0.

I'd venture a guess from the tenor of his post that Matt's "political nightmare" of the last three years will be sweet dreams compared to the future under MBD's aegis.

Don't miss it.

Note: Dare Obasanjo ends his Surviving WinFS: How LINQ to SQL Came to Be post on this positive slant:

This takes me back. I had friends who worked on ObjectSpaces and it was quite heart breaking to see what internal politics can do to passionate folks who once believed that technologies stand based on merit and not who you know at Microsoft. At least this story had a happy ending. Passionate people figured out how to navigate the internal waters at Microsoft and are on track to shipping a really compelling addition to the developer landscape.

I'm a strong proponent of LINQ to SQL as an alternative to the delayed and overweight Entity Framework, so I hope Dare's right. I'm concerned, however, that LINQ to SQL, whose origins are in the C# team, might be sacrificed on the alter of the ADO.NET team's Entity Framework.

Backstories from the OakLeaf Blog

A Ribbon UI for Visual Studio Code Name Orcas?

SQL Server "Katmai" to Deliver Entity Data Platform and Support LINQ

Déjà Vu All Over Again: Entity Framework Cut from Orcas

Orcas December 2006 CTP Coming with New LINQ/EDM Bits

Wednesday, May 30, 2007

SqlMethods Stealth Class Enables LIKE and DATEDIFF for LINQ to SQL

Anders Hejlsberg took the wraps off LINQ to SQL's SqlMethods class hidden in the System.Linq.Data.SqlClient namespace with this ADO.NET Orcas forum post. Google and Windows Live searches for linq sqlmethods return only one unique hit. Yahoo! Search returns 3 hits, one of which is the result of a typo and the other two duplicate the primary Google hit with omitted results included.

Orcas Beta 1's online help documents the SqlMethods class and its Like and DateDiff methods but doesn't provide sample use cases.

The Like Method and T-SQL LIKE Operator

The SqlMethods class includes two overloads of the Like method that the Canonical Query Tree (CQT) sends as T-SQL directly as the LIKE operator:

Dim matchExpression As String
Dim pattern As String
Dim returnValue As Boolean

returnValue = SqlMethods.Like(matchExpression, pattern)

And

Dim matchExpression As String
Dim pattern As String
Dim escapeCharacter As Char
Dim returnValue As Boolean

returnValue = SqlMethods.Like(matchExpression, pattern, escapeCharacter)

The .NET String methods provide Contains(chars), StartsWith(chars), and EndsWith(chars), which the CQT translates to LIKE '%chars%', LIKE 'chars%', and LIKE '%chars' operators, respectively. Note that Contains(chars)isn't the same as the Contains(Of T) standard query operator that determines whether an ICollection(Of T) contains a specified value.

However Contains(chars) doesn't support multiple character sets separated by wildcards, such as LIKE '%abcd%efgh%', which are valid in T-SQL. Here's a LINQ to SQL VB query that uses the SqlMethods.Like method:

Dim Query = From c In dcNwind.Customers, o In c.Orders, _ 
d In o.Order_Details _
Where SqlMethods.Like(c.CompanyName, "%fred%kist%") _
Order By o.OrderID Descending _
Select c.CustomerID, c.CompanyName, o.OrderID, _
o.OrderDate, d.Product.ProductName, d.Quantity 

The preceding query sends the following parameterized T-SQL query to an updated version of the the Northwind sample database:

SELECT [t0].[CompanyName], [t0].[CustomerID], [t1].[OrderDate], [t1].[OrderID], [t3].[ProductName], [t2].[Quantity]
FROM [dbo].[Customers] AS [t0], [dbo].[Orders] AS [t1], [dbo].[Order Details] AS [t2], [dbo].[Products] AS [t3]
WHERE ([t3].[ProductID] = [t2].[ProductID]) AND ([t0].[CompanyName] LIKE @p0) AND ([t1].[CustomerID] = [t0].[CustomerID]) AND ([t2].[OrderID] = [t1].[OrderID])
ORDER BY [t1].[OrderID] DESC
-- @p0: Input NVarChar (Size = 11; Prec = 0; Scale = 0) NOT NULL [%fred%kist%]

 which returns the following rows (formatting added):

CustID	OrderID	OrderDate	Quan	ProductName
ALFKI 11011 04/09/2007 40 Escargots de Bourgogne
ALFKI 11011 04/09/2007 20 Flotemysost
ALFKI 10952 03/16/2007 16 Grandma's Boysenberry Spread
ALFKI 10952 03/16/2007 2 Rössle Sauerkraut
ALFKI 10835 01/15/2007 15 Raclette Courdavault
ALFKI 10835 01/15/2007 2 Original Frankfurter grüne Soße
ALFKI 10702 10/13/2006 6 Aniseed Syrup
ALFKI 10702 10/13/2006 15 Lakkalikööri
ALFKI 10692 10/03/2006 20 Vegie-spread
ALFKI 10643 08/25/2006 15 Rössle Sauerkraut
ALFKI 10643 08/25/2006 21 Chartreuse verte
ALFKI 10643 08/25/2006 2 Spegesild

The DateDiff Methods and T-SQL DATEDIFF Function

The SqlMethods class includes seven DateDiffInterval methods, where Interval is Year, Month, Day, Hour, Minute, Second, and Millisecond. (Surprisingly, Quarter and Week intervals are missing.) Each interval method has two overloads, one that accepts a pair of DateTime and one that accepts a pair of Nullable(Of DateTime) datatypes, as shown here:

Dim startDate As DateTime
Dim endDate As DateTime
Dim returnValue As Integer

returnValue = SqlMethods.DateDiffYear(startDate, endDate)

And

Dim startDate As Nullable(Of DateTime)
Dim endDate As Nullable(Of DateTime)
Dim returnValue As Nullable(Of Integer)

returnValue = SqlMethods.DateDiffYear(startDate, endDate)

The query to return rows for all products shipped in the past month to Brazil is:

Dim Query = From c In dcNwind.Customers, o In c.Orders, _ 
d In o.Order_Details _
Where SqlMethods.DateDiffMonth(o.OrderDate.Value, Today) = 1 And _
c.Country = "Brazil" _
Order By o.OrderID Descending _
Select c.CustomerID, c.CompanyName, o.OrderID, _
o.OrderDate, d.Product.ProductName, d.Quantity

which sends the following T-SQL statement:

SELECT [t0].[CompanyName], [t0].[CustomerID], [t1].[OrderDate], [t1].[OrderID], [t3].[ProductName], [t2].[Quantity]
FROM [dbo].[Customers] AS [t0], [dbo].[Orders] AS [t1], [dbo].[Order Details] AS [t2], [dbo].[Products] AS [t3]
WHERE ([t3].[ProductID] = [t2].[ProductID]) AND (DATEDIFF(Month, [t1].[OrderDate], @p0) = @p1) AND ([t0].[Country] = @p2) AND ([t1].[CustomerID] = [t0].[CustomerID]) AND ([t2].[OrderID] = [t1].[OrderID])
ORDER BY [t1].[OrderID] DESC
-- @p0: Input DateTime (Size = 0; Prec = 0; Scale = 0) NOT NULL [5/30/2007 12:00:00 AM]
-- @p1: Input Int (Size = 0; Prec = 0; Scale = 0) NOT NULL [1]
-- @p2: Input NVarChar (Size = 6; Prec = 0; Scale = 0) NOT NULL [Brazil]

and returns the following rows:

CustID	OrderID	OrderDate	Quan	SkuName
RICAR 11059 04/29/2007 30 Konbu
RICAR 11059 04/29/2007 12 Alice Mutton
RICAR 11059 04/29/2007 35 Camembert Pierrot
HANAR 11052 04/27/2007 30 Ipoh Coffee
HANAR 11052 04/27/2007 10 Sirop d'érable
GOURL 11049 04/24/2007 10 Chang
GOURL 11049 04/24/2007 4 Queso Manchego La Pastora
COMMI 11042 04/22/2007 15 Gula Malacca
COMMI 11042 04/22/2007 4 Sirop d'érable
HANAR 11022 04/14/2007 35 Teatime Chocolate Biscuits
HANAR 11022 04/14/2007 30 Gudbrandsdalsost

There undoubtedly are additional T-SQL-specific operators and functions that would be useful if the ADO.NET team decides to expand the methods exposed by the SqlMethods class. Any suggestions?

Technorati tags: , , ,

Tuesday, May 29, 2007

Defining the Direction of LINQ to Entities/EDM

The ADO.NET team today posted an invitation targeted at "experienced IT developers" among Tech*Ed 2007 attendees to "attend a focus group on LINQ/EDM ... to help define the direction of LINQ to Entities." I'm not attending Tech*Ed because Microsoft moved it from New Orleans, a city I love almost as much as San Francisco, to Orlando, which I detest. (I'd rather travel to Flint, Michigan, than Orlando, Florida.)

Update 6/7/2007: LLBLGen Pro's Frans Bouma posted SqlServer 2008: Does it or doesn't it have the Entity Framework?, which supports my request for read-only foreign key values (#9) and a comparison of the Entity Framework's current and short-term future feature set with other popular object/relational mapping (O/RM) tools (#2). Ayende (Oren Eini) replied with Phsycic Debugging from Blogs, which contains an analysis of why it might be difficult for the ADO.NET team to deliver foreign key values without retrieving the corresponding object.

Update 6/2/2007: ADO.NET architect, Mike Pizzo responded in detail to my feedback in this comment.

I won't be able to join the focus group, so I'll add here the two cents I would have provided there. Following are what I consider the more important actions required to make the Entity Framework, Entity Design Model, Entity Client, and LINQ to Entities a competitive O/RM tool:

  1. Come clean with positioning of EF/EDM versus LINQ to SQL by releasing the retracted white paper that compared features of these technologies. (Julie Lerman mentioned an ADO.NET Team post that "addresses one of the biggest questions about querying against the Entity Framework - what are the pros & cons of Entity SQL vs. LINQ" that's disappeared. She says, "In discussing this, the post also lays out a lot of great background info.")
  2. Provide a similar white paper comparing the projected feature set of EF v1.0 and, perhaps, EF vNext with that of NHibernate, NPersist, and commercial OR/M tools such as LLBLGen Pro, Wilson ORMapper for .NET, et al.
  3. Recommit to Mike Pizzo's fervent promise of RTM with a graphical EDM Designer in the first half of 2008, not "in the Katmai timeframe."
  4. Deal openly with Mats Helander's contention that Microsoft is crying "O/R wolf" and won't release the EF/EDM because of the "bad press" ensuing from tools that "don't work or underperform."
  5. Respond formally to the concerns about lack of persistence ignorance and other issues voiced by the "O/RM Mafia" at the last MVP Summit in Redmond.
  6. Provide a detailed development path to and timeline for the version that enables generating the database schema from a domain model, rather than developing the domain model from the database schema.
  7. Eliminate the inability to navigate associations having composite primary keys. (LINQ to SQL doesn't have a problem with composite primary keys).
  8. Implement new object addition and all other advertised features that throw "Not yet implemented" exceptions.
  9. Provide read-only access to foreign key values.
  10. Define the support for n-tier architectures and provide non-trivial, loosely-coupled sample projects with WCF.
  11. Deliver a log feature similar to LINQ to SQL's that captures the [not just T-]SQL sent to the server.
  12. Provide a more friendly method of responding to concurrency exceptions, such as that provided by LINQ to SQL.
  13. Commit to writing an Oracle managed provider if Oracle is reluctant to do so. Developers need proof that EF/EDM/LINQ to Entities is database-agnostic. [Oracle is now on board and participating in a Tech*Ed 2007 chalktalk about EntityClient providers.]
  14. Let developers know when they can expect the first CTP and the approximate schedule of CTPs/betas to follow.*
  15. Provide detailed change lists for each CTP.
  16. Accelerate work on adding DML constructs (INSERT, UPDATE, DELETE) to Entity SQL/Entity Client.
  17. Provide an analysis of EF/EDM in terms of Martin Fowler's Patterns of Enterprise Application Development and compliance with domain-driven design (DDD) and test-driven development practices as Ian Cooper did for LINQ to SQL in his Being Ignorant with LINQ to SQL essay.

Update 6/14/2007: Added item 17 as promised in Ian Cooper Takes on DDD, TDD and PI with LINQ to SQL.

I'll update this list as I think of additional issues or missing features. If you have an EF bug you'd like squashed or a feature you'd like to see in EF/EDM/LINQ to Entities, please leave a comment.

* Note: ADO.NET's Zlatko Michailov said on May 21, 2007, "The CTP is scheduled for June" in this ADO.NET Orcas forum thread. However, future CTP drop schedules should be posted in the ADO.NET Team blog for better visibility to interested parties via syndication.

P.S. The sign-up site includes this sentence about the focus group session: "This session will describe our current thinking about the LINQ/EDM APIs we will offer with the Orcas release of Visual Studio." [Emphasis added.]

Last I heard, EF/EDM/LINQ to Entities had been removed from the Orcas release.

Update 5/30/2007: Added five topics.

Sunday, May 20, 2007

LINQ to DataSets Rehabilitates VB's Bang Operator

Visual Basic's bang (!) operator (a.k.a. pling and officially the dictionary lookup operator) seldom appears in VB.NET code. The purpose of the bang operator (alias the bang separator) is to specify members of string-keyed collections, such as fields of an ADO.COM Recordset, as in:

rsCusts!CustomerID = "BOGUS"
rsCusts!CompanyName = "Bogus Software, Inc."
rsCusts!ContactName = "Joe Bogus"

The preceding shaves a few keystrokes from "parens-and-quote" syntax, which relies on Fields being the default property of a Recordset:

rsCusts("CustomerID") = "BOGUS"
rsCusts("CompanyName") = "Bogus Software, Inc."
rsCusts("ContactName") = "Joe Bogus"

which, in turn, is shorthand for the fully-qualified syntax:

rsCusts.Fields("CustomerID") = "BOGUS"
rsCusts.Fields("CompanyName") = "Bogus Software, Inc."
rsCusts.Fields("ContactName") = "Joe Bogus"

One reason for bang's disappearance is that ! got a bum rap from VB.COM and VBA developers because amateur coders (especially Access rookies) used bang in place of dot (.) to specify properties of other objects, such as forms and controls. Doing this kills IntelliSense and compile-time checking. In fact, ! fell into such disrepute that many developers, such as this "VB Expert," believed that VB.NET didn't support the bang operator. Some folks claimed that performance suffered when ! was tokenized to the paren-quote syntax, although I never found anyone who reported testing the performance hit.

Paul Vick's July 15, 2003 Dictionary Lookup Operator post explains how the bang operator works and, in a comment, it's equivalent in C#—x!y : x["y"].

C# Joins and Associations in Typed and Untyped DataSets

LINQ to DataSets is most likely to be used in place of DataViews where joins between related tables are required. (DataViews don't support joins or projections (SELECT lists), so you need LINQ to DataSets or a custom DataView class, such as the JoinView from MSDN, to accommodate them.) I needed to verify the differences in the syntax for LINQ to DataSet queries over typed or untyped multi-table DataSets that use joins or navigate associations and include value types (specifically DateTime) with nulls. Erick Thompson wrote a Nulls - LINQ to DataSets Part 3 treatise in his LINQ to DataSet Documentation Series but didn't mention nullable datatypes in conjunction with the added DataRow.Field<T> collection.

Here's the C# test query for the typed DataSet with joins, which returns nine rows with values from each of four related Northwind tables, including one null ShippedDate value to test null handling:

var query1 = from c in dsNwindT.Customers
             join o in dsNwindT.Orders on c.CustomerID equals o.CustomerID
             join d in dsNwindT.Order_Details on o.OrderID equals d.OrderID
             join p in dsNwindT.Products on d.ProductID equals p.ProductID
             where p.ProductID == 2 && c.Country == "USA"
             orderby o.OrderID descending
             select new { c.CustomerID, c.CompanyName, o.OrderID,
                          ShippedDate = o.Field<DateTime?>("ShippedDate"),
                          p.ProductName, d.Quantity };

With the exception of the code to supply the ShippedDate value (emphasized), the preceding query statement is identical to the corresponding LINQ to SQL query (just replace the dsNwindT DataSet with a DataContext.

You lose strong typing of related DataRows when you navigate associations (ChildRows or ParentRows) between related DataTables instead of joining the DataTables. This requires FieldName = DataTable.Field<DataType>("FieldName") expressions for non-nullable and FieldName = DataTable.Field<NullableType>("FieldName") expressions for nullable fields for all but the topmost table of the hierarchy:

var query2 = from c in dsNwindT.Customers
             from o in c.GetChildRows("FK_Orders_Customers")
             from d in o.GetChildRows("FK_Order_Details_Orders")
             from p in d.GetParentRows("FK_Order_Details_Products")
             where p.Field<int>("ProductID") == 2 && c.Country == "USA"
             orderby o.Field<int>("OrderID") descending
             select new {
                 c.CustomerID, c.CompanyName,
                 OrderID = o.Field<int>("OrderID"),
                 ShippedDate = o.Field<DateTime?>("ShippedDate"),
                 ProductName = p.Field<string>("ProductName"),
                 Quantity = d.Field<short>("Quantity") };

Join syntax for untyped DataSets becomes quite cumbersome because of the need to apply the AsEnumerable() method to each DataTable and DataTable.Field<DataType>("FieldName") to the joined fields:

var query3 = from c in dsNwindU.Tables["Customers"].AsEnumerable()
             join o in dsNwindU.Tables["Orders"].AsEnumerable() on
                 c.Field<string>("CustomerID") equals
                 o.Field<string>("CustomerID")
             join d in dsNwindU.Tables["Order_Details"].AsEnumerable() on
                 o.Field<int>("OrderID") equals d.Field<int>("OrderID")
             join p in dsNwindU.Tables["Products"].AsEnumerable() on
                 d.Field<int>("ProductID") equals p.Field<int>("ProductID")
             where p.Field<int>("ProductID") == 2 &&
                 c.Field<string>("Country") == "USA"
             orderby o.Field<int>("OrderID") descending
             select new {
                 OrderID = o.Field<int>("OrderID"),
                 ShippedDate = o.Field<DateTime?>("ShippedDate"),
                 ProductName = p.Field<string>("ProductName"),
                 Quantity = d.Field<short>("Quantity") };

The added typing for untyped DataSets with associations instead of joins (emphasized) isn't that great:

var query4 = from c in dsNwindU.Tables["Customers"].AsEnumerable()
             from o in c.GetChildRows("FK_Orders_Customers")
             from d in o.GetChildRows("FK_Order_Details_Orders")
             from p in d.GetParentRows("FK_Order_Details_Products")
             where p.Field<int>("ProductID") == 2 &&
                c.Field<string>("Country") == "USA"
             orderby o.Field<int>("OrderID") descending
             select new {
                 CustomerID = c.Field<string>("CustomerID"),
                 CompanyName = c.Field<string>("CompanyName"),
                 OrderID = o.Field<int>("OrderID"),
                 ShippedDate = o.Field<DateTime?>("ShippedDate"),
                 ProductName = p.Field<string>("ProductName"),
                 Quantity = d.Field<short>("Quantity") };

But the added typing probably would trouble C# purists who substitute lambda functions for enhanced query syntax because doing so saves a few characters.

VB Associations in Typed and Untyped DataSets

When skimming the sample queries in the "LINQ over DataSet for VB Developers" whitepaper for the May 2006 LINQ preview, I noticed extensive use of the bang operator in DataTable!FieldName expressions taking the place of FieldName = DataTable.Field<DataType>("FieldName") and FieldName = DataTable.Field<NullableType>("FieldName") expressions. Strangely, there's no mention of "bang" or "dictionary" in the nine-page document.

Unfortunately, Orcas Beta 1's VB compiler doesn't implement the Join keyword and joins using SQL-89 equi-join syntax are limited to two tables. Thus, my tests with VB are limited to navigating associations.

Here's the VB version of C# query2, which returns objects rather than designated data types for fields other than CustomerID and CompanyName:

Dim Query2 = From c In dsNwindT.Customers, _
             o In c.GetChildRows("FK_Orders_Customers"), _
             d In o.GetChildRows("FK_Order_Details_Orders"), _
             p In d.GetParentRows("FK_Order_Details_Products") _
             Where CInt(p!ProductID) = 2 AndAlso c!Country.ToString = "USA" _
             Order By o!OrderID Descending _
             Select c.CustomerID, c.CompanyName, o!OrderID, _
                 o!ShippedDate, p!ProductName, d!Quantity

Notice that no special treatment of the ShippedDate field is required to handle null values. However, if you attempt to strongly type ShippedDate with a CType(o!ShippedDate, Nullable(Of DateTime)) expression, a runtime error occurs when iterating the row with the null value.

And here's the VB version of C# query4, which returns objects for all fields:

Dim Query4 = From c In dsNwindU.Tables("Customers"), _
             o In c.GetChildRows("FK_Orders_Customers"), _
             d In o.GetChildRows("FK_Order_Details_Orders"), _
             p In d.GetParentRows("FK_Order_Details_Products") _
             Where CInt(p!ProductID) = 2 AndAlso c!Country.ToString = "USA" _
             Order By o!OrderID Descending _
             Select c!CustomerID, c!CompanyName, o!OrderID, _
                 o!ShippedDate, p!ProductName, d!Quantity

Notice that the AsEnumerable() method is missing from the DataTable identifier in the first line. The C# query won't compile without it but the VB compiler doesn't care if it's present or not.

I believe it's a good bet that VB's Join syntax for untyped DataSets in Beta 2 will look like this, which is considerably more concise than that of the C# version:

Dim Query4 = From c In dsNwindT.Customers
             Join o In dsNwindT.Orders On c!CustomerID Equals o!CustomerID
             Join d In dsNwindT.Order_Details On o!OrderID Equals d!OrderID
             Join p in dsNwindT.Products On d!ProductID Equals p!ProductID
             Where CInt(p!ProductID) = 2 AndAlso c!Country.ToString = "USA"
             Order By o!OrderID descending
             Select c!CustomerID, c!CompanyName, o!OrderID, o!ShippedDate,
                 p!ProductName, d!Quantity

Update 5/18/2007: According to VB Team member Amanda Silver, VB's Join syntax uses the Equals operator (equals in C#) instead of the = operator to make explicit that the expression uses Object.Equals as the comparison operator.

Conclusion: C# purists castigate VB for its verbosity and some condemn VB code as prolix. VB's syntax for LINQ queries against joined or associated tables is equally or more concise than that of C#.

Update 5/24/2007: Paul Vick points out in his Javascript in VB ... post that "the Silverlight-based Javascript compiler that we released in Silverlight 1.1" as well as the VBx compiler for VB 10.0 is written in VB. Miguel de Caza mentions in his Compiler Lab: Second Day post:

Their new DLR-based Javascript compiler (that implements the current ECMA spec) was written by two developers in four months. Nandan Prabhu was at the conference and he explained that if they were to rewrite it today it would probably take three months for a programmer to implement as such a person would not have to keep up with the daily changes in the DLR [Dynamic Language Runtime].

It will be interesting to see how the final implementation of VB's Join keyword handles null value types in strongly typed projections.

Update 5/22/2007: You'll need to use the FieldName = DataTable.Field<DataType>("FieldName", DataRowVersion.Version) overload of the Field method in both VB and C# if you want to filter by a member of the DataRowVersion enumeration: Current, Default, Original or Proposed. (The "LINQ to DataSet for C# Developers" whitepaper describes the DataRowVersion overloads, but doesn't include a sample of ther use. "LINQ to DataSet for VB Developers" doesn't even mention these overloads.)

Friday, May 18, 2007

Erik Meijer and LINQ 2.0 - Round 2

Microsoft's peripatetic LINQ theorist/evangelist Erik Meijer delivered the second edition of his "LINQ 2.0: Democratizing the Cloud" presentation to the XTech 2007 Conference in Paris on May 16, 2007. The session's overview demonstrates that LINQ 2.0 is more than Erik's vision for "simplifying the development of data-intensive distributed applications."

Erik observes:

One painful problem that is yet unsolved is the “last mile” of data programming namely the mapping between data models at the edges, in particular the mapping between relational data and objects.

My response? "Ain't that the truth!"

Jeni Tennison wrote a brief review of Erik's session to which Mike Champion replied.

Update 5/22/2007: Erik also presented XML and LINQ: What's new in Orcas and beyond at XTech 2007.

What's new in round 2 is that LINQ 2.0 will address an issue that plagues the current (delayed) version of the Entity Framework (EF) and Entity SQL (eSQL)—lack of update capability.

Object/Relational Mapping and Updatable Views

ADO.NET program manager Zlatko Michailov wrote in his Entity Client post of February 14, 2007:

Entity SQL will not support any DML constructs – INSERT, UPDATE, or DELETE in the Orcas release. Data modification could still be done through ObjectContext, or through the native SQL of the target store. It is our goal to implement DML support but for now it’s more important to verify we’ve built the right query language. [Emphasis added.]

You can read more about this issue in Bob Beauchemin's Re: UPDATE, INSERT, DELETE in eSQL? thread of September 14, 2006 in the ADO.NET Orcas forum. Pablo Castro replied on September 15, 2006:

We want to do DML for Entity SQL over the mapping layer, it's just that we'll not get to it in the upcoming release.

Just three weeks before Zlatko's belated admission, ADO.NET data architect Mike Pizzo had written in his January 23, 2007 Data Access API of the Day - Part IV (Programming to the Conceptual Model…) post:

These rich conceptual schemas are exposed and queried through an "Entity Client".  The Entity Client is an ADO.NET Data Provider that builds queries against storage-specific providers using client-side read/write "views". Queries and updates written against these conceptual views are expanded by the Entity Client and executed as queries against underlying storage-specific providers.  All the actual query execution is done in the store (not on the client), and the results are assembled into possibly hierarchical, polymorphic, results with nesting and composite members. [Emphasis added.]

Updating views isn't a walk in the park, as Microsoft Research's Sergei Melnik writes in his August 23, 2006 "Mapping-Driven Data Access" position paper:

[T]o date there is no elegant and widely accepted mechanism
for specifying view update behavior and processing updates
in mapping-driven data access scenarios. As a consequence, commercial
database systems offer poor support for updatable views.

Sergei goes on to say, about three weeks before Pablo's announcement that updates wouldn't make it into the first EF release:

The database architects and researchers at Microsoft have been working on an innovative mapping architecture which aims to address the challenges identified in the previous sections. It exploits the following ideas:

• Mappings are specified using a declarative language that has well-defined semantics and puts a wide range of mapping scenarios within reach of non-expert users.
• Mappings are compiled into bidirectional views, called query and update views, that drive query and update processing in the runtime engine.
• Update translation is done using a general mechanism that leverages view maintenance, a mature database technology [5, 13].

The new mapping architecture enables building a powerful stack of mapping-driven technologies in a principled, future-proof way. Moreover, it opens up interesting research directions of immediate practical relevance.

It has been implemented in the upcoming release of the ADO.NET Entity Framework, which provides a mapping-driven data access layer for .NET applications. [Emphasis added.]

It appears that view maintenance turned out to be a bit harder to implement than Sergei, Mike Pizzo and the ADO.NET team estimated. LINQ to SQL now supports updates with dynamic SQL and stored procedures, but it offers only one-to-one mapping of tables to entities and has limited inheritance support. (The Beta 1 version also has a few bugs in its stored procedure support).

Abandon eSQL and Move to LINQ 2.0 for Updates and Queries?

Here's Erik's grand plan for LINQ 2.0, O/R mapping, and updatable views:

Just as we provided deep support for XML in Visual Basic, in LINQ 2.0 we hope to directly support relationships and updatable views at the programming-language level. In that case, we only need a very thin layer of non-programmable default mapping at the edge between the relation and object world and allow programmers to express everything else in their own favourite LINQ-enabled programming language. The result is that just as LINQ 1.0 side-stepped the impedance mismatch “problem” with something better (monads and monad comprehensions), LINQ 2.0 will sidestep the mapping “problem” with something better (composable and programmable mapping).

If LINQ 2.0 can deliver "composable and programmable mapping," why add DML commands to a non-standard, proprietary SQL dialect—eSQL? If LINQ 2.0 is two years away, as Erik estimates, and EF's RTM has been delayed until 2007H1 (or later), why not just let EF [N]hibernate for another year, then release with LINQ 2.0 for Entities?

However, if "we only need a very thin layer of non-programmable default mapping", perhaps Erik doesn't intend LINQ 2.0 to supplement EF and LINQ to Entities, but instead to replace them. I don't consider three layers of complex XML mapping documents to be "a very thin layer."

The First Round

In his original "Democratizing the Cloud" session at QCon 2007 Conference (London) on March 15, 2007, Erik introduced the term LINQ 2.0 but concentrated on automatically refactoring data intensive apps to n-tier projects for the Web that support all client devices. None of the bloggers who reported on Erik's session covered it thoroughly, but my "LINQ 2.0" in Early Returns from QCon 2007, London post has links to as many as I could find at the time. Tim Anderson's Getting to grips with LINQ 2.0 review is the most comprehensive.

In round 2, Erik reiterated his dynamic refactoring proposal, and added data concurrency and synchronization to the mix. His overview doesn't mention how or whether Plinq and Synchronization Services for ADO.NET fit into his concurrency and synchronization vision. However, my take is that "composable and programmable mapping" has the potential to offer more immediate benefit to data-oriented .NET developers than tool-driven n-tier/device refactoring or solving concurrency/sync problems.

Round 3

The next round of "LINQ 2.0: Democratizing the Cloud" is scheduled for June 14, 2007 at DevDays Europe 2007 in Amsterdam's RAI convention center. In addition to Erik, DevDays features a panoply of .NET heavy hitters, such as Scott Guthrie, Francisco Ballena, Dino Esposito, Ingo Rammer, Gert Drapers, Jan Thielens, Daniel Moth, Tim Sneath, and Alex Thissen. Molly Holzschlag will be "Gazing into the Future of Web Development."

Here are the LINQ and Entity Framework-related sessions at DevDays Europe 2007:

Round 4

On June 15, 2007, Erik's giving the same presentation as an invited talk (keynote) at XMIME-P 2007, the 4th International Workshop on XQuery Implementation Experience and Perspectives, in Beijing. Erik's Amsterdam presentation is scheduled for June 14, 2007. (Hope he makes it to Beijing in time. Erik says his 9:20 PM flight from AMS to PEK will get him to the the airport at 8:55 AM. Hope he gets to the venue on time. (PEKing airport is about 10-12 miles from Beijing's center.) Update 6/16/2007: Erik says he "made it to his hotel before 9:00 AM, checked in, took a shower, and spent a nice day at the workshops. Both talks went very well." Apparently, his route provided a substantial tailwind.)

Here's the blurb:

The web is rocking the world of developers. Our customers love consistency. They want to have the same rich experience, anywhere, any time, on any device. Our sales people love market share.They want no platform that cannot leverage their web services. We as developers have embraced agile methods. We want to keep our options open as long as possible and create software incrementally by successive refactorings. This surely sounds like a contradiction, another impossible triangle just like infamous Relations-Objects-XML trio we tackled with LINQ 1.0. As the Dutch artist MC Escher once said "Only those who attempt the absurd will achieve the impossible". With LINQ 2.0, we are trying to stretch the .NET framework to cover the Cloud such that it will become possible to incrementally and seamlessly design, develop, and debug complex distributed applications using your favorite existing and unmodified .NET compiler and deploy these applications anywhere.

Thanks to Mike Champion (still writing for the the XML Team) for bringing Erik's round 2 presentation to my attention. I believe this is the first of his recent LINQ presentation I didn't catch before Erik gave it.

A Ribbon UI for Visual Studio Code Name Orcas?

Mary Jo Foley reports today:

On May 18, Microsoft announced that it is moving its Server and Tools unit out from under Platforms and Services to the Microsoft Business Division (MBD). That means Senior Vice President Bob Muglia’s group now falls under MBD President Jeff Raikes’ organization. Muglia will report directly to Raikes as part of the shuffle.

Microsoft also is moving its Developer and Platform Evangelism team under Corporate Vice President Sanjay Parthasarathy to the Server and Tools Business unit under Muglia.

As "The Life of Riley" protagonist William Bendix would say, "What a revolting development this is!"

Update 5/21/2007: Gavin Clarke of Info2, a computer-industry analysis firm, added an insightful comment that links to his 5/21/2007 Solving the Microsoft re-org puzzle post. Gavin is more optimistic than I about the future for Servers and Tools as part of the Office (MBD)group.

Update 5/18/2007, 5:00 PM PDT: Mary Jo's source for her report apparently is an e-mail to Microsoft employees first reported by Seattle Post-Intelligencer reporter Todd Bishop and later confirmed by Waggener*Edstrom. Joe Wilcox just posted an analysis of the reorg.

Tuesday, May 15, 2007

Changes Coming to LINQ for SQL

Beta 2 will bring a substantial number of changes to Orcas's LINQ-related features. Following are a few of the forthcoming changes mentioned in the LINQ Project General and ADO.NET vNext forums.

Simplifying Data Tracking in n-Tier Projects

Matt Warren replies to questions in the very long LINQ Project General forum's Update Existing Disconnected Objects thread (4/12/2007):

There is more work coming that makes 3-tier scenarios easier. The [current] re-attach and playback API's are included because they are the bare minimum required to make any reconnecting work at all, however cumbersome it may seem at this time.

  • There is an upcoming feature that automatically databinds LINQ to SQL objects directly to webforms that use these API's to communicate changes back into the DataContext.
  • There is also a technology in the works that automatically serializes LINQ to SQL objects to other tiers, change tracks and data binds the objects for you on that tier and serializes the objects back, solving the cyclic reference serialization problem and the indescribable schema problem for objects with change sets; which seems to match what many of you are describing.

The mechanism that does the change tracking on the client is similar to a mini-connectionless DataContext. It is a type that packages up all the objects, lists of objects and graphs that you want to send to the client. It serializes itself and everything you've given it automatically. (It implements IXmlSerializable.)

  1. On the client, the same class also manages change-tracking for all the objects that were serialized with it.
  2. When serialized again (on the trip back) it serializes both the objects and their change information that it logged.
  3. Back on the middle tier, you just make one call to re-attach the whole package to a new DataContext instance and then call SubmitChanges.

[Formatting added.]

I'm looking forward to test-driving the mini-connectionless DataContext when Orcas Beta 2 arrives (sometime between July and September, according to Microsoft sources).

Update 6/28/2007: The "mini-connectionless DataContext" that Matt Warren described in his April 12, 2007 post to the Update existing disconnected objects (dlinq) thread in the LINQ Project General forum won't be in VS 2008's LINQ to SQL implementation. See Matt's June 18 and 27, 2007 posts to the LINQ: Next CTP thread and my Rico Mariani's DLinq Performance Tips (Part 2) and Compiled Queries post of 6/28/2007.

The O/RM Designer's O/R Provider Property Will be Gone in Beta 2

According to Microsoft's Jay Hickerson in an answer to questions in the forum's LINQ to SQL designer "O/R Provider" property thread:

The O/R Provider was originally placed there so that we could generate code that would work correctly with databases other than MS SQL Server at runtime. However, the runtime support for this did not materialize in time to get it into the designer for Orcas. The property has been removed for Beta 2 and there will not be support for different providers from the designer. We are considering adding some level of support back in a future release, possibly a service pack.

Lack of support for databases other than SQL Server is the subject of my recent Future LINQ to SQL Support for Multiple Databases? post.

LINQ to SQL Integration with Windows Communication Foundation

Matt Warren says the following about WCF DataContract and DataMember attributes in answer to a question about DLINQ as WCF service in the LINQ Project General forum:

A future version of SqlMetal.exe will generate DataContract and DataMember attributes on your entities for you. Putting them on the DataContext won't work because the DataContext is not serializable.

[Thanks to Julie Lerman for the preceding link.]

Jim Wooley provides detailed instructions for adding WCF attributes as Custom Attributes properties with the LINQ to SQL O/RM tool's Attribute Editor dialog. Adding attributes in the designer prevents them from disappearing when you refresh the partial class code.

Microsoft should provide complete, non-trivial sample LINQ to SQL and Entity Framework applications that take full advantage of WCF in the Beta 2 timeframe.

Repairing the Union Operator

The following T-SQL code demonstrates with the Northwind Customersa and Suppliers tables how to write UNION queries with literals in SELECT lists based on SQL89-style equi-joins:

SELECT c.City AS City, c.CompanyName, 'Customer' AS [Type]
FROM Customers AS c, Suppliers AS s
WHERE c.City = s.City
UNION SELECT s.City AS City, s.CompanyName, 'Supplier'
FROM Customers AS c, Suppliers AS s
WHERE c.City = s.City
ORDER BY City

Executing the preceding statement generates the following output:

City      CompanyName                                    Type
--------- ---------------------------------------------- --------
Berlin    Alfreds Futterkiste                            Customer
Berlin    Heli Süßwaren GmbH & Co. KG                    Supplier
London    Around the Horn                                Customer
London    B's Beverages                                  Customer
London    Consolidated Holdings                          Customer
London    Eastern Connection                             Customer
London    Exotic Liquids                                 Supplier
London    North/South                                    Customer
London    Seven Seas Imports                             Customer
Montréal  Ma Maison                                      Supplier
Montréal  Mère Paillarde                                 Customer
Paris     Aux joyeux ecclésiastiques                     Supplier
Paris     Paris spécialités                              Customer
Paris     Spécialités du monde                           Customer
Sao Paulo Comércio Mineiro                               Customer
Sao Paulo Familia Arquibaldo                             Customer
Sao Paulo Queen Cozinha                                  Customer
Sao Paulo Refrescos Americanas LTDA                      Supplier
Sao Paulo Tradição Hipermercados                         Customer
(19 row(s) affected)

LINQ uses similar syntax and the Union/union operator to generate the preceding output. Here's the C# code:

StringBuilder sbQuery = new StringBuilder();

var query = (
from c in dcNwind.Customers
join s in dcNwind.Suppliers on c.City equals s.City
select new { c.City, c.CompanyName, Type = "Customer" })
.Union(
from c in dcNwind.Customers
join s in dcNwind.Suppliers on c.City equals s.City
select new { s.City, s.CompanyName, Type = "Supplier" });

foreach (var u in query)
{
    sbQuery.Append(u.City + "\t" + u.CompanyName + "\t" +
        u.Type + System.Environment.NewLine);
}
// Write to the text box
txtResult.Text = sbQuery.ToString();

However, the preceding code returns Customer to all rows, instead of the five Supplier instances in the T-SQL query resultset.

According to Matt Warren in a response to my forum post this bug will be fixed in Beta 2.

I would have posted a VB version of the preceding code, but VB won't support the Join keyword until Beta 2.

Missing VB LINQ Keywords

Amanda says in this Where is gone the Group By operator in VB? and and Let Keyword for Visual Basic posts that the following LINQ keywords will be added to VB 9.0 in Beta 2:

  • Join
  • Group By
  • Group Join
  • Skip [While]
  • Take [While]
  • Aggregates (Sum, Min, Max, etc.)
  • Let

as well as these language features:

  • Nullable shortcut syntax (?)
  • Lambda Expressions
  • Partial Methods
  • Ternary If operator*

* Update 6/15/2007: Jim Wooley reported in a comment that a ternary If operator (similar in function to IIf or inline If) will be added per Paul Vick's IIF becomes If, and a true ternary operator post of 5/8/2007.

LinqDataSource Control for ASP.NET

Beta 2 should deliver ASP.NET's LinqDataSource control that Anders Hejlsberg demonstrated at MIX07.

Problems with Stored Procedure Updates in the O/R Designer

This long LINQ to SQL: Problem with Use of Stored Procedures for Insert, Update, and Delete thread in the LINQ Project General forum lists four issues that affect replacing dynamic SQL with stored procedures for table updates in the O/R Designer:

  1. Code generation adds an underscore to reserved words, such as Order in VB which is missing in method calls.
  2. Delete funcitons have an extraneous current parameter.
  3. Update functions have current and original parameter values reversed (discovered by Jim Wooley and Mike Taulty).
  4. To date, I've been unable to match INSERT sprocs with the required signature for tables with identity PKs.

Presumably, these problems will be fixed in Beta 2. Matt Warren says early in the thread:

The problems with the designer code-gen have been resolved and should appear in the beta2 CTP or possibly sooner.

However, I haven't received any direct assurance that the specific problems in items 2 through 4 of the preceding list will be addressed. The party line is: "Databinding will be greatly improved in Beta 2."

I'll add more new features and improvements coming in Orcas Beta 2 to this list as I learn about and/or confirm them.

Katmai CTP Scheduled for June

Francois Ajenstat, Director of Product Management for SQL Server said at about 08:45 into a Podcast interview at the first Microsoft Business Intelligence Conference in Seattle:

There will be a first CTP [of Katmai] coming out shortly, probably within the next month. And so customers will be able to start playing with Katmai, learning it, and providing feedback. [Emphasis added.]

The Business Intelligence Conference's resources page has links to Webcasts of the Jeff Raikes, Ted Kummert, and Steve Ballmer keynotes.

Following are cues for LINQ, Entity Framework, and OCS topics in Ted Kummert's keynote:

  • 46:00 LINQ in Katmai
  • 46:25 Entity Data Platform
  • 47:25 Occasionally Connected Systems
  • 57:18 First public Katmai demonstration
  • 57:35 Spatial data types and indexing (planar and geodetic)
  • 59:00 Virtual Earth mashup with the spatial data type

For more details on the relationship between Katmai's Business Data Platform and the former Orcas Entity Framework, Entity Data Model, and related components, see SQL Server "Katmai" to Deliver Entity Data Platform and Support LINQ.

Update 5/23/2007: Francois Ajenstat's last name is misspelled as "Ajenstad" on the Resources page and corrected here.

Saturday, May 12, 2007

Synchronization Services Runtime Beta 1 for ADO.NET 2.0 Is Available

Rafik Robeal announced in his Just Published! Microsoft Synchronization Services for ADO.NET Beta 1.0 post of May 12, 2007 that the Beta 1 version of the Sync Services runtime for .NET 2.0+ and Visual Studio 2005 is available here. These bits are the same as those for the Sync Services runtime in Visual Studio "Orcas" Beta 1.

Updated documentation for runtime Sync Services, which also applies to the Orcas Beta 1 implementation, is available here. (It must be a cold day in hell, because the documentation preceded the binaries.)

You'll need to make a few changes to using/Imports directives and altered object and property names than have changed from the earlier versions used in Rafik's samples.

Update 5/17/2007: Rafik has updated his six sample applications for the Beta 1 runtime release.

The runtime doesn't include the graphic Sync Designer that opens when you add Orcas Beta 1's Local Database Cache template to a VB or C# project. My A Sync Services Bidirectional Test Harness blog of March 23, 2007 describes a Windows form app that synchronizes the Northwind Orders and Order_Details tables between an SQL Server 2005 Express (SSX) instance and an SQL Server 2005 Compact Edition v3.5 (SSCE) database.

Note: Converting the Test Harness from the Orcas March 2007 CTP to Orcas Beta 1 failed due to issues with the Sync Designer's generation of the Northwind.sdf database. Specifically, the Sync Designer didn't add the required LastEditDate and CreationDate fields to the SSCE Orders and Order_Details tables, which causes replication to fail with error messages about the missing fields. For more information see my Sync Designer/SSCE Version Problems with Orcas Beta 1 post in the Microsoft Synchronization Services for ADO.NET forum and Bad Error Message Connecting to SSCE v3.5 from Orcas Beta 1 in SSMSX SP2 under Vista post in the SQL Server Compact Edition forum.

Friday, May 11, 2007

Using the Entity Framework with IronPython 1.1 in Project Jasper

Andrew Conrad's A walk through the Jasper API with IronPython - Part 1 post shows you how to get Project Jasper running with IronPython 1.1. I'm far from a Python expert, so I thought I'd give Andrew's step-by-step instructions a test drive. Here's what I did:

Setup

  1. Downloaded Jasper: MIX 2007 CTP release (MicrosoftJasperCTP.msi, 14.0 MB) and installed it on my test machine that runs the prerequisite Orcas Beta 1 and an SQL Express instance with the Northwind sample database installed.
  2. Downloaded the IronPython 1.1 runtime binary from CodePlex (IronPython-1.1-Bin.zip, 973 KB) and expanded it to my \Program Files folder with the Use Folder Names checkbox marked. Installing IronPython 1.1 adds an ...\IronPython-1.1 folder with \Doc, \Lib, and \Tutorial folders.
  3. Ran ipy.exe -X:TabCompletion from a command prompt to open the interactive console. (There's also an ipyw.exe that runs IronPython without a console.)
  4. Changed ipy.exe's default background color to R=222, G=112 and B=8 and text color to R=245, G=222, and B=179 to match my blog colors.

Obtaining a DynamicContext

  1. Typed clr.AddReferenceToFileAndPath("C:\\Program Files\\Microsoft Codename Jasper CTP\\Binaries\\Microsoft.Jasper.CTP.dll") to add the Jasper reference.
  2. Typed the Microsoft.Jasper import * directive.
  3. Typed connString = "Provider='System.Data.SqlClient';Provider Connection String='Initial Catalog=Northwind;Data Source=.\SQLEXPRESS;Integrated Security=SSPI';Generate Default EDM=True" to create the connectionString variable.
  4. Typed nw = DynamicContext.CreateDynamicContext(connString) to create a DynamicContext object for the Northwind sample database. It took about 13 seconds to generate the default Entity Data Model (EDM) for the Northwind tables and associations. This is about half the approximately 30 seconds it took to in Sam Drucker and Shyam Pather's DEV18 - Rapidly Building Data Driven Web Pages with Dynamic ADO.NET MIX07 video (19:17 to 19:47).

Note: All instructions must be entered at the console's >>> prompt as a single line of text. The complete connection string doesn't appear in Andy's post (even in 1280 x 960 resolution) and needs to be changed if you're running SQL Server Express.

At this point, I was ready to exercise the DynamicContext object:

Spelunking the DynamicContext Object

To prove that the ObjectContext's entity sets were collections of the Query type, I typed print type[nw.Products], which returned Query. I then went on to test a few more of Andy's examples. I noticed a brief delay when assigning a new Query collection member to a variable with statements such as product = nw.Products[0] or category = product.Category. Associations are navigable as shown in the last two queries.

Querying with Entity SQL

Entity SQL (eSQL) is the second approach to querying objects exposed by DynamicContext:

If IntelliSense was enabled in the console, as implied in Andy's post, it didn't work for me.

Supporting LINQ in Jasper

LINQ is Jasper's third query technology. Unfortunately, Jasper doesn't support LINQ to Entities currently. In response to a question in the Project Codename: Jasper forum, Carl Perry said the following:

We do support LINQ in this CTP release of Jasper, however only LINQ to Objects is supported.  While we were working on this CTP we did work to have our dynamically generated classes and collections support LINQ;  essentially providing the ability to translate LINQ queries to the backend query language.  This support is available to in the Entity Framework in the form of LINQ to Entities.  Unfortunately, due to limitations in VB.NET 9.0 and Iron Python 1.1, we were unable to support translation of these LINQ queries to appropriate backend query language.  Supporting LINQ in our dynamic languages (VB.NET, Iron Python, Iron Ruby, etc) is extremely important and we are currently working with teams across the company to get this working. [Minor corrections made.]

Andy Conrad further clarifies the LINQ support situation in response to a Jasper forum rant against VB:

I would suggest that folks give VB a chance occasionally. IMHO, particularly with the new features coming on line in VB9 (lambda expressions, LINQ) and beyond (true dynamism) I think VB is becoming a very compelling dynamic language. And based on constantly switching between languages while working on Jasper, between Python, Ruby, and VB – I can’t say I can pick one over the others.

That said, the DLR will initially support four different .NET languages (IronRuby, IronPython, VBX, and managed Javascript) and we are designing the Jasper framework to work in all, so the developer will be free to pick their dynamic language of choice.

A couple of clarifications – Linq is supported by IronPython 2.0 but not IronPython 1.1. Since the initial CTP of Project Jasper is not integrated with the DLR, we only support VB 9 and Iron Python 1.1 – and only the former supports Linq.

Also, Project Jasper CTP is built on top of .NET framework 3.5 Beta 1 (Orcas release). We could have built it over .NET framework 2.0, but we wanted Linq support. As I said before, as on the next CTP release Jasper will be built over the DLR which is targeted as a post Orcas release.

The Project Jasper CTP download includes VB and IronPython 1.1 samples in the \Program Files\Microsoft Code Name Jasper\Samples folder. If you're running Vista, you must move the samples outside the \Program Files folder.

Update 5/12/2007: Andy Conrad's Some comments about comments and other ramblings about Project Jasper post of May 11, 2007 explores developers' unease with frameworks and scaffolding that substitute built-in processes for programmer-authored code. (I find myself a bit uncomfortable not having direct control of the database connection when using DataContext or DynamicContext objects.) However, this perceived lack of developer control certainly hasn't impeded widespread adoption of Ruby on Rails and ActiveRecord for projects similar to those that Jasper appears to target.

Andy links to a secretGeek post that takes Microsoft to task for 1) not recognizing the contributions of others to dynamic languages and the convention over configuration concept (specifically David Heinemeier Hansson, the original developer of Ruby on Rails, and 2) limiting Jasper to SQL Server 2005 databases only.

Credit Where Credit is Due

I don't fault Microsoft for not thanking Hansson by name any more than I'd complain that download pages for SQL Server 2005 don't include "thanks to Dr. E. F. Codd for his contributions to relational database theory and design." (It's clear to me that Blinq was inspired by Ruby on Rails scaffolding, and I inferred this in my June 2006 Generate Data-Based Web Sites With Blinq post and Visual Studio Magazine article about Polita Paulus's original Bling implementation. Blinq was based on LINQ to SQL, not the Entity Data Model.)

RDBMS Xenophobia in Project Jasper and Related Technologies

I do agree with secretGeek and the post's commenters that restricting Jasper to SQL Server is a bad idea, just as bad as—or worse than—doing the same to LINQ to SQL (see my Future LINQ to SQL Support for Multiple Databases? post of April 19, 2007). Microsoft touts the Entity Framework as accommodating databases other than SQL Server 200x if the RDBMS vendor is willing to invest in writing an EntityClient data provider. In a 4/21/2007 update to that post I noted that Microsoft held an on-campus seminar to aid other RDBMS vendors in extending their .NET data providers with EntityClient features. IBM, for one, says it intends to support LINQ, presumably via the Entity Framework.

If Microsoft intends to live up to its commitment that a primary feature of the Entity Framework is "The ability to query relational stores other than the Microsoft SQL Server family of products" it behooves the ADO.NET team to extend this capability to Entity Framework derivatives, such as Jasper and Astoria (which also has an SQL Server 2005 requirement for the toolkit). Microsoft must clarify that the SQL Server 2005-only limitation applies only to the early "incubator project" releases and not to the final release, if there is one. Failing that is "crying wolf"—no developer or RDBMS vendor will believe that the Entity Framework will be RDBMS-agnostic when Microsoft says Katmai will deliver the Entity Data Platform

Update 5/22/2007: Andy's response to my comment in his Some comments about comments and other ramblings about Project Jasper post clarifies that Astoria's and Jasper's SQL Server-only restriction is a temporary expedient to get the samples "out the door." Here's the part of the response relating to database agnosticism:

Jasper being a Sql Server only framework is just a limitation of the first CTP.  If you look at the Jasper download package, we install private copies of System.Data.dll and System.Data.Entity.dll (renamed for the CTP) because we required some Entity Framework functionality not available in the Beta 1 Orcas down load. As a result, because of the ADO.net provider factory design and the fact we didn’t want to mess with config files – we decided to hard code Sql Client as the only provider for the initial CTP. This was a hard decision, but we wanted to get the preview out the door.  And since Sql Express is free – we could at least give folks a chance to play around with Jasper.  In house we actually have had Jasper working over some of the non-Sql Server Entity Framework providers and even over Access and DataSets. Non-Sql Server provider support is very high on our list for CTP2.

Also of note, Katmai (next version of Sql Server) for the Entity Framework is a ship vehicle not a restriction on what providers are supported. Astoria and Jasper are currently very early in the product cycle (still in incubation) but I don’t see any reason why these technologies would only support Sql Server. [Emphasis added.]

Hopefully, Katmai won't be the only ship vehicle for the Entity Framework.

Mike Taulty Posts Six New LINQ to XML Video Segments

As promised in his original post for the LINQ to Objects and LINQ to SQL video segments, Mike Taulty has posted a trio of clips in both C# and VB. Mike is a member of Microsoft UK's Developer & Platform Group who frequently blogs on LINQ-topics.

C# Segments:

  1. Querying XML (17:24)
  2. Namespaces (11:26)
  3. Creating XML (13:22)

VB Segments:

  1. Querying XML (12:06)
  2. Namespaces (5:37)
  3. Creating XML (15:03)

Notice that the length of the Quering XML and Namespaces segments are five minutes shorter for VB than C#. Does this mean that querying XML and dealing with XML namespaces is that much simpler in VB than C#?

Note: Mike completed his 18-segment LINQ to SQL series on May 10, 2007.

Technorati tags: ,

Wednesday, May 09, 2007

SQL Server "Katmai" to Deliver Entity Data Platform and Support LINQ

Microsoft's announcement this morning that 2008 is the target release date for SQL Server "Katmai" (presumably SQL Server 2008) might explain why the ADO.NET team dropped the Entity Framework (EF) from Visual Studio "Orcas" and postponed EF's target release date to the "first half of 2008."

Mike Pizzo, an Architect on the Data Programmability team, announced that EF was cut from Orcas in an April 28, 2007 (Saturday) post, ADO.NET Entity Framework Update, which contained the following blurb about the Entity Data Model/Platform and a teaser for the "Astoria" and "Jasper" incubator projects:

Microsoft will be leveraging the Entity Data Model in future versions of Microsoft products such as SQL Server. This Data Platform vision enables customers to leverage their investment in a common conceptual entity model across product lines. Stay tuned for announcements starting next week at Mix around new products building on the ADO.NET Entity Framework. [Emphasis added]

In his  Microsoft's Data Access Strategy post that followed the announcement by three minutes, Mike asks "Does Microsoft have a Data Access Strategy?" and then gives the following answer:

Yes, it turns out we do. Microsoft envisions an Entity Data Platform that enables customers to define a common Entity Data Model across data services and applications. The Entity Data Platform is a multi-release vision, with future versions of reporting tools, replication, data definition, security, etc. all being built around a common Entity Data Model. [Emphasis added.]

Update: 5/25/2007: As an IBM senior executive used to say, "Vision without execution is hallucination." (From the Microsoft Architectural Journal's "Architecture Journal Profile: Don Ferguson" interview. Don Ferguson is a former IBM Fellow who moved to Microsoft as a Technical Fellow in early 2007.)

The newly released SQL Server Code Named "Katmai" data sheet's "Develop with Entities" section says:

As part of the next generation of the ADO.NET data access framework, developers will be able to access their data by defining business entities like Customers, Orders and Products instead of tables and columns using the Entity Data Model. Query and retrieve these entities natively within any .NET language with the introduction of LINQ. These services enable developers to work within the logical entity model while administrators can define the physical implementation of the model as tables and columns. [Emphasis added.]

According to InformationWeek's May 9, 2007 Microsoft Gives Peek At Next Version Of SQL Server article by J. Nicholas Hoover from the first Microsoft Business Intelligence Conference 2007:

The next SQL Server also will include the "entity data platform," ostensibly the ADO.Net entity framework,which makes it so. [I]nstead of having to deal with raw data in a relational table, developers can easily deal with data on an object level—for example, as a customer or product object—regardless of whether the information and data that defines the object is in multiple tables.

There also will be support for Language Integrated Query, or LINQ, which simplifies the process required for developers to query databases.

Katmai additionally will introduce a technology called "occasionally connected systems" to provide core services that allow developers to synchronize devices with data whenever they're connected to a network. [Emphasis added.]

Mary Jo Foley's May 10, 2007 Making sense of Microsoft’s myriad database projects post to her All About Microsoft blog quotes Francois Ajenstat, director of SQL Server:

At the core of these different data-programmability projects is the concept of a common entity data model. EDM — a way of describing entities and their relationships — isn't new; it has been used by database programmers for more than 30 years. Ajenstat described EDM as a way of allowing programmers to program at a logical level, instead of a physical level.

Microsoft is building an entity data model framework around its ADO.Net technology that is designed to raise the level of abstraction for database programmers. Up until recently, the ADO.Net Entity Framework was slated to be part of the first release of Visual Studio Orcas; now the entity framework won't be released until some time in 2008, as a post-Orcas update to the .Net Framework, Ajenstat confirmed.

Microsoft's goal is to enable LINQ, its Language Integrated Query extensions to the .Net Framework, to query the new ADO.Net Entity Model. Microsoft wants to simplify SQL Server developers' lives by freeing them up from having to understand SQL the language, said Ajenstat.

Paul Flessner, then Senior Vice President of the Server Applications, alluded to support for LINQ in his SQL Server 2005 Update of April 6, 2006 with the following sentence under his "Beyond Relational" topic:

We will strive to deliver the best platform for integrated storage, and advanced applications such as spatial data, while also making it dramatically easier for you to build data-driven applications, without needing to invest significant resources in bridging the gap between data and programming language data structures. [Emphasis added.]

Katmai and Entity Framework Dependencies?

CRN's Barbara Darrow wrote the following two days before EF's Orcas swansong:

Microsoft may synchronize its "Longhorn" Windows Server release with its promised SQL Server and Visual Studio updates, even if that means delaying Longhorn's availability slightly, CRN has learned.

I doubt very much that Longhorn will have dependencies on SQL Server and Orcas, or vice versa. However, EF might end up with a dependency on Katmai, despite Mike Pizzo's insistence that EF will have:

The ability to query relational stores other than the Microsoft SQL Server family of products.

Without Orcas as a release vehicle, EF is an orphan. Will Katmai adopt EF as an SQL Server toolset and delay EF's debut to the Katmai RTM date?

Update 5/23/2007: According to a comment to a blog post by ADO.NET team member Andrew Conrad, "Katmai (next version of Sql Server) for the Entity Framework is a ship vehicle." [Emphasis added.]

Mike Pizzo was adamant about the delayed EF shipping in the 2008H1. Here's his committment in ADO.NET Entity Framework Update:

Based on the need to align with requirements from key internal partners that are building on the Entity Framework, along with the need for a better tool experience, we have decided to ship the ADO.NET Entity Framework and Tools during the first half of 2008 as an update to the Orcas release of the .NET Framework and Visual Studio.

and in Microsoft's Data Access Strategy:

A few months after the shipment of Orcas, and within the first half of 2008, Microsoft will release the ADO.NET Entity Framework as an extension to the Orcas version of the .NET Framework.

Stay Tuned

There's more coming from Christian Kleinerman's A First Look at SQL Server codename ‘Katmai’ Database Engine presentation on Monday, May 8, 2007 at SQLLive! Orlando that promises:

[A] high level overview of some of the enhancements to the relational database engine in SQL Server code name "Katmai".

and Ted Kummert's keynote on Thursday, May 10, 2007, which should shed more light on Katmai's Enterprise Data Platform, LINQ, and OCS (synchronization) features. Kummert took over Paul Flessner's duties in October 2006. According to Mary Jo's post:

All of what Microsoft lumps under "data programmability" falls under the domain of Ted Kummert, the Microsoft Corporate Vice President in charge of the data and storage platform division, Ajenstat said.

Update 5/15/2007: Microsoft posted a Webcast of Ted Kummert's keynote, and Podcasts of interviews with Francois Ajenstad, Director of Product Management for Microsoft SQL Server and Ram Ramanathan, Senior Product Manager for Microsoft SQL Server Data Warehousing and Data Integration. Arjenstad says there will be a "CTP of Katmai coming out shortly, probably within the next month" (see Katmai CTP Scheduled for June.)

Here's an excerpt from Mark Whitehorn's The four pillars of Katmai report of May 11, 2007 for the Reg[ister] Devloper:

Dynamic development. Ted talked about a fascinating development in Katmai, the introduction of an ER modelling layer. This provides a logical layer which sits between application developers and the relational tables – allowing them to interact with the data more easily.

Fans of ER modelling will want to know that this is not (at least in Katmai) a design tool – you cannot develop an ER model and then generate the schema. However, Microsoft sees this as a first step and there is a strong possibility that design capabilities will appear in later versions. Database engines in general have been crying out for this kind of functionality for years so this first tentative step is very welcome.

Mark and Seattle Post-Intelligencer reporter Todd Bishop, along with other members of the trade press, were excluded from Steve Ballmer's Friday keynote. Here's a long story from Mark and a briefer report from Todd. I'd characterize the exclusion as a very unwise attempt at a commercial version of prior restraint.

Update 5/15/2007: Microsoft posted a Webcast of Steve Ballmer's keynote, so the preceding tempest in a teacup is moot.

Update 5/14/2007: Added excerpt from SQL Server 2005 Update from Paul Flessner of April 6, 2006, The four pillars of Katmai report of May 11, 2007, and stories about the Steve Ballmer keynote fiasco.

Update 5/11/2007: Patrice Luong's detailed post from the Microsoft BI Conference didn't mention the Entity .

Update 5/10/2007: Added links to Mary Jo Foley's post.

Related OakLeaf Posts

Entity Framework

Occasionally Connected Systems, Sync Framework/Systems, and Sync Designer