LINQ Changes from Orcas Beta 1 to VS 2008 Beta 2

There's a substantial amount of blogging going on about new features and changes to LINQ to Objects and its domain-specific versions in Visual Studio 2008 Beta 2. This post is intended as a central anthology of changes between the two betas with links to the original sources.

LINQ to Objects (Generic LINQ)

NEW VB LINQ-Related Keywords and Features

VB 9.0 gets the missing LINQ keywords promised by Amanda Silver and noted (except Into) in my May 15, 2007 Changes Coming to LINQ for SQL post:

• Join
• Group By
• Group Join
• Into
• Skip [While]
• Take [While]
• Aggregates (Sum, Min, Max, etc.)
• Let

VB 9.0 now supports the following LINQ-related features mentioned (except closures and mutable anonymous types) in the preceding post:

• Nullable shortcut syntax (int?, Double?)
• Lambda functions (the VB team calls them "inline functions")
• Partial methods
• Closures
• Ternary If operator
• Mutable anonymous types when using New With { ... } syntax *

* C# anonymous types are immutable, as are conventional VB anonymous types created by the LINQ Select clause and and implied Select clauses. You can make members of mutable VB anonymous types immutable by specifying their Key value, as in New With { a.Key, b.Key }. For more information on VB's new mutable anonymous types, see Paul Vick's Mutable and immutable anonymous types, and keys post of May 11, 2007.

VB's Group Join and Group By statements require Into Group rather than C#'s Into identifier expression, as reported my 7/28/2007 Strange Change to VB Join and Group By Expressions in VS2008 Beta 2 post.

VB MVP Bill McCarthy reports in a comment to Strange Change to VB Join and Group By Expressions in VS2008 Beta 2 that VB also supports Group Join identifier1 ... Into identifier2 = Group and Group By's Group prefix1 By prefix1.identifier1 Into identifier2 = Group expressions.

There's a bug in VB's implementation of the IEnumerable.Count() method. The <Extension> attribute doesn't override the default Count() operator for a List(Of T) in Beta 2. The following C# code works fine (OrderList is 100 Northwind order elements in an List<Order>):

int countUsOrders = OrderList.Count(c => c.ShipCountry == "USA");

The following VB statement won't compile:

Dim CountUsOrders2 As Integer = OrderList.Count(Function(c) c.ShipCountry = "USA")

It throws a "'Public ReadOnly Property Count() As Integer' has no parameters and its return type cannot be indexed" error. These two statements compile and run as expected:

Dim CountUsOrders1 As Integer = OrderList.Where(Function(c) c.ShipCountry = "USA").Count()

Dim CountUsOrders As Long = OrderList.LongCount(Function(c) c.ShipCountry = "USA")

See my Strange Behavior of VB 9.0 Count() Operator with Predicate LINQ Project General forum post and bug report 291116.

Update 8/9/2007: Jonathan Aneja, a program manager on the VB Team says in response to the bug report:

In VB Properties shadow-by-name, so this is actually by design. While it makes this case a bit inconvenient, you can work around it by calling LongCount, or by using non-Extension method syntax and calling Enumerable.Count(OrderList, ...).

It's unfortunate that VB programmers must special-case the most popular of built-in aggregate operators. (Timothy Ng provides a more detailed response to my forum message.) 

Mike Taulty notes in MMmmmmm. VB 9.0 in Beta 2 the new VB-specific query expression syntax for Skip and Take. He says:

There's an awful lot going on with VB in 9.0 - truly great stuff. Makes me feel like converting! [Emphasis added.]

Non-VB LINQ Issues

IQueryable has been refactored into IQueryable and IQueryProvider (see below.)

No Change: Joe Albahari reports that the NotSupportedException: Aggregate operators are not supported in projections bug he found in Beta 1 is still present in Beta 2, according to the Bug - LINQ to SQL aggregations in projections thread in the LINQ Project General forum.

The ToLookup() standard query operator returns the ILookup<TKey, TSource> or ILookup<TKey, TElement> interface, not the expected Lookup<TKey, TSource> or Lookup<TKey, TElement> type. C# requires an explicit cast to Lookup<,>; VB performs an implicit cast. See Jomo Fisher's response to my ToLookup() Operator Doesn't Behave As Expected in C# Query Expression message in the LINQ Project General forum. (This behavior might not have changed in Beta 2.)

Matthieu Mezil reports in a comment to this post:

System.Data.Entity.dll becomes System.Data.DataSetExtensions.dll with Beta 2. So if you used DataSet with Orcas Beta 1, you must change [the] reference in your project.

Off topic, Bill has a fix for the VB and C# code samples that won't open by double-clicking the .sln file to change "Visual Studio Codename Orcas" to "Visual Studio 2008" in the subdirectories of the folder you run the code from.

Update 7/31/2007: Added VB's new mutable anonymous types. Update 8/2/2007: Updated and moved Matt Warren's IQueryProvider series, added Mike Taulty's salute to VB 9.0, and my question about the ToLookup() standard query operator. Update 8/6/2007: Added VB 9.0 IEnumerable.Count() bug and Matt Warren's part 5. Update 8/8/2007: Added System.Data.Entity.dll change to DataSetExtensions.dll from Matthieu Mezil. Update 8/9/2007: Categorized LINQ to Objects issues and updated Microsoft response to VB Count() issue.

IQueryable, IQueryProvider, and Expression Trees

Matt Warren, principal architect in the C# group and LINQ to SQL protagonist, advises in his LINQ: Building an IQueryable Provider - Part I post of July 30, 2007 that IQueryable has been refactored into IQueryable and IQueryProvider in Beta 2. This is a breaking change for all third-party LINQ query providers.

On the other hand, current and erstwhile third-party LINQ query provider writers will finally get some help from Matt's informal documentation series. (Near the end of the comments, Frans Bouma says he's starting work on a query provider for his LLBLGenPro object/relational mapping (O/RM) tool.)

Matt's LINQ: Building an IQueryable Provider - Part II shows you how to create a very simplified version of the LINQ to SQL provider that translates a LINQ query into SQL command text and and translates the result of executing the command into objects. Matt also provides (by popular demand) a public implementation of the internal ExpressionVisitor class from the System.Linq.Expressions namespace. (Jomo Fisher says in an update to his May 23, 2007 Dealing with Linq’s Immutable Expression Trees post that Matt's implementation "is pretty close to the one we use internally.") Frans Bouma comments, "[The] design is flawed, and the root cause is the fact that it's a combination of concerns: a query and an enumerable resultset" that include a dependency on the database engine. (Updated 7/31/2007)

Matt continues by adding support (or a fix) in LINQ: Building an IQueryable Provider - Part III for translating local variables, such as:

string city = "London";
var query = db.Customers.Where(c => c.City == city);

This turns out to be more of an exercise than expected because local variables in expression trees turn into a compiler-generated class to hold the local variable. (You've seen these compiler generated classes if you've examined query expressions with Reflector.)

So Matt provides the code for an Evaluator class, which he says "exposes a static method ‘PartialEval’ that you can call to evaluate these sub-trees in your expression, leaving only constant nodes with actual values in their place." Nice work!

Not willing to leave well enough alone, Matt muddies the waters today with LINQ: Building an IQueryable Provider - Part IV, in which he implements an improved Select translator with a new ColumnProjector class and a visitor that builds an SQL SELECT clause. What could be next? And where's Frans Bouma? (Update 8/3/2007: See Frans Bouma's new comment to this post.)

Matt comes through with yet another refactoring exercise to solve member access (projection) problems with method calls that reverse the order of Where and Select and add a member that doesn't correspond to a column. This requires a new set of expression node types to abstract SQL (TableExpression, ColumnExpression, and SelectExpression) and LINQ (ProjectionExpression.) There's much more rework in LINQ: Building an IQueryable Provider - Part V, but fortunately Matt includes a zip file of the source code to date (Query5.zip), which compiles under Beta 2 and runs a simple demo query in the console. Matt says, "[I]t works no matter how many Select’s or Where’s I add, no matter how complex I make each projection." At this point in the game a flow diagram would be a big help. (Update 8/6/2007)

The djini must be out of the bottle because Matt, Luca Bolognese, and Charlie Calvert produced a 40-minute "LINQ [to SQL] Pipeline" video and released it to SoapBox yesterday afternoon. The first 15+ minutes is devoted to diagramming the translation of LINQ queries to T-SQL text strings. Other topics include diagramming the SubmitChanges operation, and reminiscing about the history of LINQ to SQL and its ObjectSpaces roots. Matt and Luca have interesting posts about their relationship, LINQ, LINQ to SQL and ObjectSpaces. (P.S. The ObjectSpaces PDC alpha preview was code-named "Orca." Object Spaces was in Whidbey Beta 1 and removed from Whidbey Beta 2. For the tragic tale of ObjectSpaces demise at the hands of WinFS, see my Microsoft Bites the Bullet: WinFS Bites the Dust post of June 26, 2007.) (Update 8/7/2007)

Part 6 uncovers another defect in the now-adolescent application: Subqueries in projections throw exceptions. In LINQ: Building an IQueryable Provider - Part VI, Matt takes on the case where projection nodes appear in selector expressions with nested queries. He solves the problem with "lots of extra little queries," which require MARS (MultipleActiveResultSets=true in the connect string) so the Query6 implementation won't work with SQL Server 2000. Version 6's source code gets the DataContext.Log property so you don't need Profiler to check the T-SQL emitted. (Update 8/10/2007)

LINQ to SQL

Joe Albahari reports that EntitySets still don't support IQueryable and Jomo Fisher says this will continue to be the case in RTM, according to the August 6, 2007 Why does EntitySet no longer implement IQueryable? and the August 15 What do others think of the crippled new EntitySet?post in the LINQ Project General forum. (EntitySets supported IQueryable in the LINQ May 2006 CTP.) Lack of support for IQueryable reduces composability of queries over EntitySets. Update 8/15/2007.

Creating an SQL Server database with SQLMetal.exe now generates a user instance (local .mdf file) instead of attaching the database to a SQL Server instance. (See Jay Hickerson's answer to Joe Rattz in the In Beta2, SQLMetal no longer attaches the database thread of the LINQ Project General forum. Update 8/7/2007.

Beta 2 requires attaching every object individually to the DataContext. Beta 1 and earlier attached the specified object and all reachable associated objects. (See Matt Warren's answer in the Attach, SubmitChanges doesn't save child objects thread of the LINQ Project General forum. Update 8/10/2007.)

VB MVP Jim Wooley contributes his July 30, 2007 list in LINQ to Sql changes from Orcas Beta 1 to Visual Studio 2008 Beta 2.

Jim Wooley also observed in the LINQ to SQL Entities as WCF DataContracts post in the LINQ Project General forum that Beta 2 removes the capability to set custom attributes for WCF in the LINQ to SQL Designer, but Dinesh Kulkarni says in an answer:

You can specify that you want [WCF] DataContract/DataMember attributes in designer or SqlMetal.

In designer, just right click on the design surface and specify "Unidirectional" in property grid for "Serialization Mode".

For SqlMetal, the /serialization:Unidirectional does the same trick.

Jim reports that the expected name change of the DataShape object to DataLoadOptions mentioned in messages in the LINQ Project General forum has occurred. Note that DataContext.Shape property names changes to DataContext.LoadOptionsShape also. (Updated 8/1/2007)

Mike Taulty adds more detail about the changes in his VS 2008 Beta 2 - LINQ to SQL post of July 30, 2007. 

Dinesh Kulkarni's July 26, 2007 LINQ to SQL (fka DLinq): What's new in Orcas beta2 post (unearthed by Julie Lerman) describes 15 LINQ to SQL improvements and new features. Dinesh would have better said "SQLMetal.exe supports SQL Server CE" in item 7 and I the same in the post's title, because the LINQ to SQL Designer doesn't work with SSCE in Beta 2 and won't in RTM. (See LINQ to SQL to Support SQL Server Compact Edition.)

According to a post by Dinesh in the LINQ Project General Forum:

1. GetQueryText() is superseded by GetCommand() which gives you the SqlCommand(). This allows programmatic access in addition to getting the text.
2. GetChangeText() is superseded by GetChangeSet() which gives you access to the the set of changed objects. Again this allows programmatic manipulation including complete takeover of SQL command generation or additional/alternate actions.

Dinesh's July 29, 2007 Orcas beta2: what's where (directories I mean)? post clarifies the location and provision of LINQ to SQL-related files and features:

So here is where you can find LINQ to SQL related files.

• New .NET Framework 3.5 DLLs (green bits) are in: Program Files\Reference Assemblies\Microsoft\Framework\v3.5\
• SqlMetal.exe is in Program Files\Microsoft SDKs\Windows\V6.0A\Bin\

Another source of confusion - designer vs. SqlMetal

• The visual designer for generating mapped LINQ to SQL classes is a part of Visual Studio 2008. You need to install some SKU of Visual Studio to use it.
• SqlMetal.exe is a command line tool that is a part of the Windows SDK and lives in the same directory as similar command-line tools like Xsd.exe

Scott Guthrie's LINQ to SQL (Part 5 - Binding UI using the ASP:LinqDataSource Control) post of July 16, 2007 is a tutorial for connecting the new LINQDataSource control for ASP.NET to LINQ to SQL's graphical object/relation mapping (O/RM) tool and taking advantage of LINQ to SQL as a data access layer (DAL) with paging, validation, and other business-related features. Andres Aguiar has an interesting commentary on the LinqDataSource control and its incestuous relationship with LINQ to SQL in his July 30, 2007 Digging into LinQDataSource post.

Scott also posted a tutorial for using the LINQ to SQL Debug Visualizer. He says:

The LINQ to SQL Debug Visualizer isn't built-in to VS 2008 - instead it is an add-in that you need to download to use.  You can download a copy of it here.

The SQL Debug Visualizer is included in the CSharpSamples.zip file located in the \Program Files\Microsoft Visual Studio 9.0\Samples\1033 folder. However, the ReadMe.html page's recommended deployment folder for the DLL file is incorrect. Scott's recommended location is \Program Files\Microsoft Visual Studio 9.0\Common7\Packages\Debugger\Visualizers\, which is correct. That folder contains the DataSetVisualizer.dll file, but requires Vista users to have Administrator privileges. Here's a link to my bug report on the issue. Microsoft says either location is OK, but Documents ... didn't work for me. (Update 8/4/2007)

Dan Wahlin's Video: First Look at Visual Studio .NET 2008 and the LinqDataSource shows you how to use the LINQ to SQL Classes template with the LINQDataSource and common databound controls.

Jim Wooley reports in his August 6, 2007 Use the new LINQ "Contains" extension method for the SQL "IN" clause post that LINQ to SQL Beta 2 translates LINQ queries that use the "reverse mode" of IEnumerable<T>'s Contains() extension method into an T-SQL IN() clause. For example, the following LINQ to SQL query: 

Dim strStates As String() = New String() {"CA", "OR", "WA"}
Dim States = From St In strStates Select St
Dim WestCoastCustomers = dc.Customers.Where(Function(c) States.Contains(c.Region))

Generates a T-SQL string for a single query with the equivalent of an IN('CA', 'OR', 'WA') clause. (Update 8/7/2007)

The Union/union operator bug is fixed (see Changes Coming to LINQ for SQL.) This bug was in the way the LINQ to SQL expression tree handled translation of literal column names, not in LINQ's standard query operator.

LINQ to XML

I haven't seen any news on this topic so far from the Microsoft XML Team's WebLog or Eric White's Blog.

Hopefully, Ralf Lämmel will update his LINQ to XSD preview for Beta 2 shortly.

LINQ to DataSet

Daniel Moth, a Microsoft UK developer evangelist and .NET CF expert, offers a July 27, 2007 LINQ to DataSet post that describes the change in the location of the LINQ to DataSet extensions from System.Data.Entity.dll to System.Data.DataSetExtensions.dll. Daniel also provides class diagrams which illustrate the six infrastructure-supporting types, as well as the three extension methods to aid working with typed and untyped DataSets: DataTableExtensions, TypedTableBaseExtensions and DataRowExtensions.

Update 8/9/2007: Added link to Daniel Moth's page and changed DataSets to DataSet.

I'll update this post as I uncover more changes myself or from others. Please add a comment if you have and changes to contribute.

Technorati tags: LINQ, LINQ to SQL, LINQ to XML, LINQ to DataSets, VS 2008 Beta 2, VB 9.0, C# 3.0, LinqDataSource

LINQ to SQL to Support SQL Server Compact Edition

Despite previous statements to the contrary, LINQ to SQL will support SQL Server Compact Edition (SSCE), according to a blog post by Dinesh Kulkarni, program manager for LINQ to SQL (formerly DLinq). Microsoft's earlier plan was to limit LINQ to SQL to to use with SQL Server 200x [Express] and deliver only LINQ to Entities support for SSCE within the Entity Framework (EF). Dinesh's July 26, 2007 LINQ to SQL (fka DLinq): What's new in Orcas beta2 post included "SQL Server Compact (fka SQL CE) support" as number 7 in his 15-item list of new and improved LINQ to SQL features in VS 2008 Beta 2.

Important Update 7/30/31: Dinesh appears to have an overly generous definition of "support" and I extended this to my post's title. What he and I should have said is that SQLMetal.exe supports SSCE. Here's a quote from Dinesh's next-day post to the LINQ Project General forum's Sql Server CE and linq (Beta 2) thread:

Unfortunately, LINQ to SQL designer will not be able to support SSC in beta2 or even RTM of VS 2008. Let's see what can be done after that! [Emphasis added.]

BTW, before you run into this - there is an unfortunate known bug if you use SqlMetal against the Northwind.sdf shipped with beta2 SSC. Due to a space in one of the foreign key names, you will get an error. We are working on posting a Northwind.sdf that is closer the the "classic" Northwind database so you can try out SqlMetal and LINQ to SQL run-time on that.

Needless to say we have fixed the bug but unfortunately it didn't happen in time for beta2.

SSCE v3.5 enables developers to use SSCE's lightweight, high-performance embedded database for object persistence in specialized applications. SSCE is a single-user relational database that runs in the client application's process and supports multiple replication and synchronization protocols with SQL Server and, potentially, other databases. The production release of SSCE v3.5 will ship with VS 2008.

I had been agitating for SSCE support by LINQ to SQL since I began testing and writing about SSCE 3.5 and the new Synchronization Services runtime earlier this year. LINQ to SQL is a much lighter weight data access layer than LINQ to Entities plus the EF; thus LINQ to SQL is the preferred partner for SSCE.

However, the word from Microsoft in March and April 2007 was that LINQ to SQL was SQL Server 200x only. Here are a couple of related statements extracted from my April 19, 2007 Future LINQ to SQL Support for Multiple Databases? post.

A Microsoft program manager (4/2/2007, private correspondence):

"Our Orcas plans do not include 'LINQ to SQL' Server Compact Edition in Orcas. LINQ to SQL is only SQL Server."

Steve Lasker, SQL Server Compact Edition program manager (3/1/2007, reply to blog comment):

"We don't yet have a LINQ to SQL (DLINQ), but we do have ADO.NET V3 Entities working over SQLce. If you install the Sync Services CTP, you'll see we install a Entity dll for SQLce as well. This CTP isn't directly compatible with the Orcas Feb CTP, so don't expect to do anything with it just yet. It's possible the Feb CTP that was just released would enable Entities over SQLce, but I have to check to see what made it into that build."

Update 7/31/2007: New SSCE 3.5 Beta 2 documentation is available that VS 2008 Beta 2 doesn't install. Here are the links from Steve Lasker's SQL Server Compact 3.5 B2 & Sync Services for ADO.NET Documentation released post:

• SQL Server Compact 3.5 Documentation
• Sync Services for ADO.NET Documentation and Samples

The Synchronizer blogger Rafik Robeal says in today's Sync Services Hit Beta 2.0 Milestone post that he'll update his Sync Services samples this week. Rafik also points to sync batching capabilities as a new Sync Services Beta 2 feature.

Update 8/10/2007: Version 1.0 of Sync Services doesn't have server-side push capabilities because the sync server is unaware of the location of sync clients. As an alternative to clients repeatedly pinging the server, Steve Lasker proposes an independent Notification to Pull system to send messages to sync services clients that the server has new data to deliver. In one implementation clients use a "Here I Am" Web service, similar to that used for roaming cellular phones, to track client addresses.

Here are links to my more recent articles and blog posts about SSCE and SSCE Sync Services, which also is part of VS 2008:

• Exploit SQL Server Compact Edition and Occasionally Connected Systems, June 4, 2007
• Update Local Data Caches with Sync Services, May 25, 2007
• SQL Server Compact Edition and Synchronization Updates, April 2, 2007
• A Sync Services Bidirectional Test Harness, March 23, 2007
• SSCE Sync Designer Q&A and Screencast, March 22, 2007
• New SQL Server Compact Edition Article, March 20, 2007
• Sync Services for ADO.NET Overview, March 19, 2007
• Guided Tour of Orcas's Sync Services Designer for SSCE, March 17, 2007

LINQ to SQL support for SSCE, such as it is, unquestionably was the right decision. However, designer support for SSCE should receive a high post-RTM priority and an out-of-band Web release.

Technorati tags: LINQ to SQL, LINQ, SSCE, SQL Server Compact Edition, Sync Services, SSCE Sync Services, VS 2008, VS 2008 Beta 2

P.S. Thanks to Julie Lerman for alerting me to Dinesh's post. Dinesh hadn't posted any new technical data about LINQ to SQL for more than a year so his blog wasn't on my reader's list.

Strange Change to VB Join and Group By Expressions in VS2008 Beta 2

Last week, I spent some time testing an assortment of C# and Visual Basic versions of LINQ query expressions having Join ... Into and Group By ... Into clauses. Until yesterday, I was using the Orcas June 2007 CTP, which finally re-enabled the GroupJoin and GroupBy standard query operators in VB projects.

Yesterday, I installed an instance of VS 2008 Beta 2 on a Virtual Server 2005 R2 SP2 VM running Windows Vista Premium as the host OS and moved my test code to the new VM. My collected C# queries compiled and ran as expected, but the VB compilation wouldn't compile.

Change to Group Join Syntax

The problem for a Group Join query expression turned out to be the expressionList identifier co following the Into keyword, as shown here:

The error message was "Definition of method 'co' is not accessible in this context" and IntelliSense added method-style parenthesis to the identifier. There is no indication in the February 2007 Overview of Visual Basic 9.0 white paper that the VB and C# [Group ]Join/join and Group By/groupby syntax differed by more than keyword capitalization and and space stripping.

Note: The current (and almost final, 500-page) C# Language Specification Version 3.0 covers translation of C# query expression syntax to anonymous (lambda) method calls in section 7.15, starting on page 209.

Based on my previous experience with VB GroupJoin...Into and C# Join...Into expressions, I had added an identifier for an expected range variable, as in this C# equivalent that compiles and produces the expected result in both Orcas June 2007 CTP and VS 2008 Beta 2:

I went back to the VB version, put the caret on Group Join and pressed F1. Thankfully, context-sensitive help was working and I uncovered this interesting paragraph:

expressionList
Required. One or more expressions that identify how the groups of elements from the collection are aggregated. To identify a member name for the grouped results, use the Group keyword (<alias> = Group). You can also include aggregate functions to apply to the group.

I also took a look at the newly added Join examples in the VB Project LINQ Sample Query Explorer. The Group Join example used the Group keyword as the Into identifier. Here's the VS 2008 VB editor's enhanced IntelliSense list and tooltip for the Into keyword's <group alias> = assignment:

Note: Scott Guthrie's Nice VS 2008 Code Editing Improvements post covers most new VB and C# editor features.

Changes from co to Group in two places solved the problem, as shown here in the final VB query expression:

Note that Group is a keyword in this context.

Here's the output in a simple test harness:

Click for Larger Image

Update 7/30/2007: VB MVP Bill McCarthy reports in a comment to to this post that VB also supports Group Join identifier1 ... Into identifier2 = Group and Group By's Group prefix1 By prefix1.identifier1 Into identifier2 = Group expressions.

Online help doesn't include this extended syntax in its Group Join or  Group By topics.

Update 7/31/2007:  The runtime exception reported with the Group By implementation was my error. I successfully reused the Query11 variable for two forms of the Group Join query but using the Query16 variable for two forms of the Group By query caused a type conflict (see below).

Change to Group By Syntax

A similar problem occurred with a query expression containing Group By ... Into clause, in this case with the c identifier. Here's the original VB version with the iterator code, because the iterator gets a significant change:

And here's the C# version which compiles and runs as expected in both VS versions:

The Group By select new {...} clause isn't optional for C# query expressions.

Note that the final version uses a StringBuilder object to populate the text box shown here:

Click for Larger Image

Finally, here's the modified VB version that compiles and runs in VS 2008 Beta 2:

Update 7/30/2007: Here are the modifications that compiled but wouldn't run for me:

Query16 = From c In CustomerList _
          Order By c.Country, c.CustomerID _
          Group c By c.Country Into g = Group _
          Select New With {Country, g} 'Not optional

The runtime error message I received was:

System.InvalidCastException was unhandled
  Message="Unable to cast object of type '<SelectIterator>d__d`2[VB$AnonymousType_12`2[System.String,System.Collections.Generic.IEnumerable`1[QueryOperatorsVB.MainForm+Customer]],VB$AnonymousType_13`2[System.String,System.Collections.Generic.IEnumerable`1[QueryOperatorsVB.MainForm+Customer]]]' to type 'System.Collections.Generic.IEnumerable`1[VB$AnonymousType_11`2[System.String,System.Collections.Generic.IEnumerable`1

Update 7/31/2007: As noted in the preceding section, the cause of the problem was using the Query16 variable for two forms of the Group By query, which worked for Group Join but not Group By. Notice that Dim is missing in the preceding code. Adding Dim and giving the query a new name solved the problem, and made the Query16 statement optional.

The type conflict was between the immutable anonymous type returned by the original version's Select Country, Group Join clause or the implied Select clause and the mutable anonymous type returned by the Select New With {Country, g} clause. For more information on VB's new mutable anonymous types in VS 2008 Beta 2 see Paul Vick's Mutable and immutable anonymous types, and keys post of May 11, 2007.

Bill gave another sample of an even more flexible Group By syntax:

Query16 = From c In CustomerList _
          Order By c.Country, c.CustomerID _
          Group c By c = c.Country Into g = Group _
          Select c, g} 'Optional

which resembles a C# lambda function (but isn't; the > is missing). In this case, you can assign an identifier to both the key and the group items.

Note: As you'd expect, LINQ queries over small collections of in-memory objects are fast. Seventeen sample queries execute against objects created from the Northwind Customers table (91 rows) in 124 milliseconds (when you append all text to a StringBuilder object and use it to populate the text box after executing the last query.

Conclusion

LINQ's objective is to provide a common set of Standard Query Operators for multiple data domains. That said, however, doesn't mean that developers can expect a common query expression syntax for multiple .NET languages.

The first major departure between C# and VB LINQ features was VB's added explicit XML authoring capability. VB supports Group as a keyword but C# doesn't. It will be interesting to see how far the new dynamic languages—IronPython and IronRuby—will depart from the C# query expression translation specification.

Note: Iron Python 2.0 will support LINQ to Objects (based on its presence in the System.Core namespace) and LINQ to XML is promised later for the DLR.

Important other VB discovery: ByVal is no longer required as a prefix to parameter names of VB lambda (inline) functions, which first arrived in the Orcas June 2007 CTP.  ByVal's presence won't throw an exception but if you edit an expression containing it, ByVal disappears and the editor won't let you add it back.

Technorati tags: LINQ, LINQ to Objects, LINQ to SQL, LINQ to XML, LINQ to Entities, LINQ to DataSets, VB 9.0, C# 3.0, Query Expressions, Query Comprehensions, VS 2008 Beta 2, Visual Studio 2008

Layer Data Access with LINQ to SQL

1105 Media's July 24, 2007 .NETInsight newsletter features my "Layer Data Access with LINQ to SQL" article from the July 2007 issue of Visual Studio Magazine. The feature-length article digs into LINQ to SQL's origins in Microsoft's ill-fated ObjectSpaces project, covers its object/relational mapping (O/RM) features, demonstrates the graphical O/RM designer, and describes the classes that LINQ to SQL autogenerates from SQL Server 200x databases.

The project's sample code is based on Orcas Beta 1, but should be operable with the shortly forthcoming Visual Studio 2008 Beta 2. (Click the "get the code" link under the "Article Tools" heading to download VS0707rj.zip. Requires free registration.)

The newsletter also has links to my brief Introducing the Entity Framework and LINQ and Hot at Tech●Ed 2007: SQL Server 2008 articles from the newsletter's Tech●Ed 2007 issues.

Technorati tags: LINQ to SQL, LINQ, Entity Framework, SQL Server 2008, 1105 Media, Visual Studio Magazine

LINQ Update for the Week of July 16, 2007

It's been uncharacteristically quiet on the LINQ front since Scott Guthrie announced that VS 2008 Beta 2 was due by the end of the month. Most LINQ aficianados have been waiting for the Beta 2 bits to drop. But there were a few items worth covering this week, to wit:

Scott Guthrie Demos the ASP.NET LINQDataSource Control

Scott's lengthy LINQ to SQL (Part 5 - Binding UI using the ASP:LinqDataSource Control) is the latest in his detailed demonstrations of LINQ to SQL in WebForms. Like almost all other code examples provided by the VS teams in the past two months, this episode requires VS 2008 Beta 2. (The Orcas June 2007 CTP doesn't include the LINQDataSource control, formerly named the LinqDataSource control.)

Frans Bouma, whose LLBLGen Pro object/relational mapping (O/RM) tool competes with LINQ to SQL, complains in a July 17, 2007 comment that:

1. Displaying human-readable names for numeric foreign key values requires two SQL queries per row if lazy loading is used.
2. The LINQDataSource control is tied to LINQ to SQL and won't interoperate with third-party O/RM tools that implement "generic" LINQ features.
3. Grouping entities makes them read-only (Comment: Grouping data rows makes them read-only, too).
4. Strings "inside the eval calls" are brittle and will make the page "fall apart at runtime."

Scott sets the record straight in this reply comment of the same date:

1. Eager loading (pre-fetching) is a LINQ to SQL option that you implement with the Span property. Scott will cover this in a future post.
2. The LINQ DataSource will support for additional O/RM tools "in the future." First on tap will be LINQ to Entities (which is running now as a prototype) and the ASP.NET team will release the source code. Scott says (with tongue in cheek?), "It would obviously be great to get support for it with additional OR/M's like LLBLGen."
3. If the data has been made read-only by a grouping operation, the UI provides visual clues.
4. An alternative to Eval is to use code-behind to set values.

Scott confirmed in this reply comment that "VS 2008 and .NET 3.5 Beta2 should be available for download in a little over a week."

Here are links to the preceding four episodes:

• Part 1: Introduction to LINQ to SQL
• Part 2: Defining our Data Model Classes
• Part 3: Querying our Database
• Part 4: Updating our Database

Bart DeSmet Frees SharePoint Developers from CAML Coding

Bart's worked hard for the past three weeks to get the latest update to LINQ to SharePoint ready to test before his move from Belgium to Redmond this fall. He's just posted the Alpha 0.2.2.0 version to his CodePlex site.

His new installer for the LINQ to SharePoint binaries is set up for Orcas Beta 1 and might not work with Orcas June 2007 CTP or VS 2008 Beta 1. If I have the time, I'll test it with the CTP. The 0.2.2 Alpha Interim Overview post (above) includes fully illustrated installation instructions.

Bart says:

Another interim release, likely 0.2.3 and planned for mid August, will cover:

• List attachments
• Additions to SPML and SpMetal improvements (generate SPML, consume SPML) + improvements for the entity wizard
• Grouping support (under investigation)
• Support for User fields (under investigation)

Erik Meijer Gives Yet Another LINQ Lecture on Channel9

VB evangelist Beth Massi tapped Erik Meijer for a 37-minute Erik Meijer Shows Us His Favorite Visual Basic 9.0 Feature - XML Literals interview. Here's Beth's description of the coverage:

In this interview Erik Meijer, an Architect on the Data Programmability team, shows us how LINQ to XML works in the next version of Visual Basic, 9.0. He talks about some of the history behind the features as well as how XML literals, unique to Visual Basic, are implemented. He takes us on an adventure on how to efficiently work with XML using XML IntelliSense and an Office Open XML Excel document, pointing out a lot of tidbits of information along the way that only Erik could show us.

Click here to display recent OakLeaf posts on the peripatetic LINQ promoter and LINQ 2.0. Click here for all Erik Meijer posts on the OakLeaf site (from September 2005).

Implementing a Many-to-Many Relationship in LINQ to SQL

Mitsuru ("Mitsu") Furuta, a developer evangelist for Microsoft France, has written sample code for creating many-to-many associations between LINQ to SQL EntitySet objects. The lack of support for direct one-to-many associations (without relation tables, such as Northwind's Order_Details table) has disappointed many potential users of LINQ to SQL, the Entity Framework and LINQ to Entities.

Using Mitsu's proxy class approach that's described in his How to implement a many-to-many relationship using Linq to Sql? post, you can write:

foreach (Product p in order.Products) ... ; 

Instead of

foreach (Order_Detail od in order.Order_Details)
Product p = od.Product ... ;

Mitsu's C# sample code runs under Orcas Beta 1 and includes IList and IBindingList implementations for data binding.

Slightly Off-Topic: A Strange Proposal to Ease VB Developers' C# Envy

Paul Vick's 7/20/2007 lowercase keywords? post wonders "how much the fact that we uppercase [VB] keywords affects our perception?" Of course, the question is "Perception of what?" His post starts with:

One of the raps that VB sometimes gets is that we're too "verbose."

In general, I prefer VB's verbosity to C# tersity by symbology. However, this preference declined dramatically when having to write .Where(Function(ByVal c) c.Country = "USA") instead of .Where(c => c.Country == "USA") or the like a few hundred times. On the whole, I prefer C# syntax for lambda expressions.

I believe what Paul meant by "our perception" is "our perception [of VB as a second-class language]" because his post implies that changing from VB's Pascal-case (proper-case) to lower case keywords will deliver a dose of (imagined) C# cachet to the Visual Basic language. Implicit in this misunderstanding is that typing lower-case keywords will infuse VB developers with the (imagined) élan of C# programmers.

I believe that VB has acquired the second-class language stigma as the result of Microsoft's developers writing 100% C# unless instructed or paid to do otherwise. (A possible exception is Erik Meijer, who's an architect not a developer.)

Another Microsoft contribution to the perception of VB's inferiority to C# is late adoption or omission of new C# language features. For example, VB's support for lambda expressions and the LINQ aggregate (Sum, Min, Max, etc.), Group By, Group Join, Join, Let, Skip [While] and Take [While] keywords didn't arrive until the Orcas June 2007 CTP but they were present in C# during 2006's LINQ CTPs.

Another factor is lack of sufficient resources devoted to VB in Visual Studio 2008. According to Paul's June 27, 2007 What's in VB 2008? What's out? post:

For VB 2008, we will only support initializing read-write fields of non-collection objects [... and] only support expressions in lambdas, not statements.

Our original plans, going back to PDC05, included several more features for object initializers, such as being able to write to read-only properties, as well as collection and array initializers. In the end, the schedule for VS 2008 was not sufficient to implement these features with a high degree of confidence. Which unfortunately means that they will have to wait to a release beyond VS 2008.

Finally, C# bigotry by some .NET developers and writers contributes to VB programmers' perception of language inferiority. For example, here's Kevin Hoffman's [The .NET Addict] lead-off to his Jasper and IronPython in an ASP.NET Application post in the Project Codename: Jasper forum:

Every single demo that I've seen of Jasper has involved people using ipy [IronPython] to dynamically interrogate the database, which is fine. Also, the ASP.NET demos of Jasper have used VB as the code-behind language.

Here's the problem: both of those scenarios that I will never use in production. I will not be using VB in production, nor will I be using ipy in production.

What I'd like to see is:

- an ASP.NET application with IronPython as the code-behind
- an ASP.NET application with Ruby as the code-behind

Microsoft's Andrew Conrad replied:

Just curious, but are you saying you would be willing to put an IronPython or Ruby app into production, but not the same app written in VB?  I would be curious to hear the reasons why?

To which Kevin responded:

I'm not saying that VB isn't a production-quality language. I'm saying that in the circles of people I associate with, no one will use VB. I don't know a single programmer who would willingly choose to write their code in VB. I make no claims as to the prejudicial attitudes of said programmers - only that I no [sic] of no one personally who prefers VB as a development language.

That said, in that same group of people, many of them find Ruby to be hip and cool, and Python to be equally cool and powerful.

But Kevin Hoffman's bio reads, in part:

Kevin Hoffman has been programming since he was 10 and has written everything from DOS shareware to n-tier, enterprise Web applications in VB, C++, Delphi, and C. [Emphasis added.]

and he's reported by Amazon.com to be the co-author of:

• ASP.NET Website Programming: Problem - Design - Solution, Visual Basic .NET Edition
• Professional ADO.NET with VB.NET
• Pro ADO.NET with VB .NET 1.1
• Professional ASP.NET Web Services with VB.NET
• Visual Basic .NET Deployment Handbook

Apparently, Kevin once thought VB was "cool" even if not "hip."

To its credit, Microsoft hired Beth Massi to evangelize VB, and Beth has done an amazing job of increasing the exposure of VB with Live from Redmond Webcasts and other video presentations, blog posts, and articles. Her July 17, 2007 LINQ Articles and More from the VB Team post summarizes recent works. The Visual Basic Team blog now appears to be much more active than Charlie Calvert's (C#) Community Blog. The technical level of the VB Team's articles has increased recently with single and serial posts by Kevin Halvorson, Matt Gertz, Jared Parsons, and Scott Wisniewski, although the LINQ Cookbook series would benefit from a dose of VB best practices.

Update 7/22/2007: Added Mitsu's many:many associations post and Paul Vick's lower-casing VB post.

Technorati tags: LINQ to SQL, LINQDataSource, LINQ, LINQ to SharePoint, LINQ to XML, Erik Meijer, VB 9.0, O/RM, Object/Relational Mapping.

New LINQ with Visual Basic Articles

The VB Team has lagged the C# group in implementing LINQ-related keywords and language features throughout the VS 2008 beta process. This situation is understandable because the C# group is in charge of the core LINQ specification, so the VB compiler group can't complete their work until the C# implementation is fully baked.

The same has been true for LINQ documentation, but it's less understandable. For example, the May 2006 .NET Standard Query Operators white paper has only C# 3.0 sample code. Until Wednesday, I hadn't seen a single working example of an expression tree or custom LINQ implementation written in VB; they were all C#-based.

LINQ to WDS

The tide may be turning with Kevin Halverson's LINQ to WDS project. (WDS = Windows Desktop Search.) In How to implement IQueryable (Part 1) and How to implement IQueryable (Part 2), Kevin delves into a detailed explanation of how to create a custom LINQ implementation that lets you execute queries against the file system with WDS's OLE DB driver. WDS is part of Vista, but you can download versions for Windows XP and Windows Server 2003 here. The project translates simple LINQ Select and Where clauses into SQL statements like this:

SELECT System.ItemPathDisplay FROM SystemIndex WHERE NOT CONTAINS(System.ItemType, 'folder') AND (System.DateCreated >= 'EarliestDate' AND System.FileName LIKE 'Expression')

Kevin's Part 1 post provides a link to download the complete source code, which totals about 500 lines. The UI is a bit Spartan: A message box wrapped by a For Each loop substitutes for the usual console output.

Note: Kevin says the project requires Beta 2 because IQueryable was been refactored after Beta 1 released. However, it works for me on the Orcas June 2007 CTP.

VB Query [In]Comprehensions

Scott Wisniewski's "Basic Instincts: Query Comprehensions" column in the August 2007 issue of MSDN Magazine covers VB's query translation, lazy evaluation, the Select, SelectMany, Where and Join query operators, and expression trees in much greater depth than I've seen before (other than in Kevin's two posts).

Scott includes a sample project that instruments most of the 48 (or 50 if you count TakeWhile and SkipWhile separately) Standard Query Operators with PerfMon counters. The current version throws an error when you open the solution because of a missing unit Test.vbproj project file. You can fix that error by removing its entry from Microsoft.Linq.Instrumentation.sln.

There's a 45-KB EULA but not even a Readme file to explain how to use the WrappedEnumerable, WrappedEnumerator, and WrappedOrderedCollection classes. I couldn't find a single comment in the project. This makes the sample project incomprehensible to most LINQ neophytes.

The VB Team must have been confident that Beta 2 would be available by the magazine's posting date because Scott's project does require Beta 2. (It needs Beta 2's IOrderedSequence interface and CreateOrderedSequence method to compile; they aren't in the Orcas June 2007 CTP.)

Maybe Scott will add some documetation to his Shameless Plug (Scott Wisniewski) post.

Technorati tags: LINQ to WDS, LINQ, VB 9.0, Expression Trees

VS 2008 Beta 2 Due by End of July, RTM by End of Year

Hard on the heels of the Orcas June 2007 CTP, you can expect the Visual Studio 2008 Beta 2 bits to be released by July 30, 2007. According to Scott Guthrie's reply to a comment in his July 9, 2007 VS 2008 Nested Master Page Support post:

VS 2008 Beta2 should be available for free download in the next three weeks. It will support a go-live license. [Emphasis added.]

Doug Seven, project manager for Visual Studio Team System, says in his July 10, 2007 Windows Server 2008, Visual Studio 2008 and Microsoft SQL Server 2008 Joint Launch Announced post:

While the launch events are scheduled to kick off on February 27, 2008, Visual Studio 2008 will be released before the end of the year.

Update 7/12/2007 11:30 p.m. PDT: Bill McCarthy set me straight (again) about the "bug" below in a comment. I had forgotten to add the Of preposition for an (Of DateTime?) type assignment in the translation from C# <DateTime?> to VB. (I canceled the bug report.) Guess I've been writing too much C# code lately (or I need more sleep.)

Hopefully, "fixed in Beta 2" promises for by the LINQ to SQL team will be kept and the bug I reported with nullable field values in LINQ to DataSet Select projections for VB projects will be corrected in Beta 2.

As Bill noted in his Connect bugs scare me post today, the VB Team member handling bug reports didn't catch the missing Of preposition either. Here's the "resolution" proposed:

Thanks for your feedback. We have reproduced this bug on Visual Studio Codename Orcas June CTP, and we are sending this bug to the appropriate group within the VisualStudio Product Team for triage and resolution.

I agree with Bill that the compiler error messages should suggest adding Of.

SQL Server 2008 (formerly codenamed "Katmai") will "launch" in February 2008 but won't release to manufacturing until the second quarter of 2008, according to today's ARE You Ready to Rock? Launch 2008--it's official! post by Michael J. Murphy.

Technorati tags: VS 2008 Beta 2, LINQ, SQL Server 2008, Katmai

Orcas and Entity Framework June 2007 CTPs - New Features

Despite Microsoft's deafening silence (so far) about the availability of the Orcas June 2007 CTP, a few pilgrims are spelunking the VM for new features. Folks are using Danny Simmons' new features list to test-drive the Entity Framework June 2007 CTP under the original Visual Web Developer Express (VWDX) June 2007 CTP as well as the full Orcas June 2007 CTP.

Update 7/10/2007: Here's the apparent explanation for Microsoft's deafening silence. The original Microsoft UK Visual Studio "Orcas" June 2007 CTP post reappeared this morning with the following caveat:

The Product Group have released an newer version of Visual Studio, but they aren't releasing it as a CTP. It is aimed at fixing a couple of problems for some targeted customers. The reality of this isn't tested or up to the quality of a proper CTP. You can still use this download, but it has some problems and it isn't supported for TFS. I would recommend waiting until Beta 2. Sorry for any confusion I have caused. [Spelling corrected.]

If the download page says it's a CTP and the four Readmes say it's a CTP, they released it as a CTP. The post's headline now calls it a "pre-CTP."

Tamir Khason of Microsoft Israel posted Orcas June CTP for limited beta is available on July 4, 2007 with this announcement:

After releasing June CTP of .NET framework 3.5, June CTP bits of Visual Studio 2008 (Team Suite and Visual Web Developer) are also ready for limited beta as Layout or VPC image.

and invited readers to contact him for downloading details.

Visual Basic 9.0 Language Features

VB MVP Bill McCarthy's What's new in June Orcas CTP for VB post lists the following new language features:

• Lambda functions
• Nullable types get ? shorthand for Nullable(Of ...) and operator elevation
• Ternary If() function
• Join, Group By and other missing LINQ Standard Query Operators

Bill also mentions differences in C# 3.0 and VB 9.0 behavior in his Having fun with "Group-Ease", Change notification with Orcas data model and Keyed Anonymous types in VB9 posts.

VB 9.0 Lambda Functions and LINQ Method Call Syntax Issues

Online help's "Lambda Expressions [VB]" topic provides the following as a code example in the "Lambda Expressions in Queries" subtopic for method call query syntax:

Dim londonCusts As IQueryable(Of Customer) = _
    db.Customers.Where(Function(c) c.City = "London").Select(c)

If you create a new NorthwindDataContext that contains a Customer object and assign it to db, then add the above code to a Sub or Function, it won't compile. The c in the Select(c) selector isn't visible, which results in a "Name 'c' is not declared" error.

The following version with type inference compiles and returns the expected result to an iterator:

Dim londonCusts = _
    db.Customers.Where(Function(ByVal c) c.City = "London") _
                .Select(Function(ByVal c) c)

Notice that the ByVal directive is missing from the help topic's sample; if you don't type it, the compiler adds ByVal for you.

The preceding code seems absurdly verbose compared to the corresponding C# method call syntax:

var londonCusts = db.Customers.Where(c => c.City == "London")
                              .Select(c => c);

The following C# query chains an OrderBy lambda:

var londonCusts = db.Customers.Where(c => c.City == "London")
                              .OrderBy(c => c.ContactName)
                              .Select(c => c);

Here's the corresponding VB version:

Dim londonCusts = _
    db.Customers.Where(Function(ByVal c) c.City = "London") _
                .OrderBy(Function(ByVal c) c.ContactName)
                .Select(Function(ByVal c) c)

The problem with chaining the additional VB lambda expression is that you get the wrong IntelliSense's List Members collection (Equals, GetHashCode, GetType, ReferenceEquals and ToString instead of the Customer object's properties) for the intermediate OrderBy and other chained functions between Where and Select. C# IntelliSense has no such defect.

Join Operator Works But Union Method Is Still Broken

VB 9.0 finally acquired the Join operator, but the following code still returns the literal 'Customer' for every row, not a mix of 'Customer' and 'Supplier' values. I described this problem in my May 15, 2007 Changes Coming to LINQ for SQL post. Here's the VB code with Joins:

Dim sbQuery As New StringBuilder

Dim Query = ( _
From c In dcNwind.Customers _
Join s In dcNwind.Suppliers On c.City Equals s.City _
Select New With {c.City, c.CompanyName, .Type = "Customer"}) _
.Union( _
From c In dcNwind.Customers _
Join s In dcNwind.Suppliers On c.City Equals s.City _
Select New With {s.City, s.CompanyName, .Type = "Supplier"})

' Append list header
sbQuery.Append("City" + vbTab + "Company Name" + vbTab + "Type" + vbCrLf)
' Append query results
For Each u In Query
    sbQuery.Append(u.City + vbTab + u.CompanyName + vbTab + u.Type + vbCrLf)
Next

Matt Warren said on May 15, 2007 in response to my Strange Behavior of LINQ Union Operator message in the LINQ Project General forum, "[T]here is a bug on how we treat literals in a union.  We are fixing it."

I plan to do more testing of Join, Group By and other previously missing LINQ Standard Query Operators and will update this post accordingly.

Update 7/11/2007: Extensive tests of multiple Joins with typed and untyped DataSets indicate that it behaves as expected.

Julie Lerman Explores the Entity Framework's New Feature Set

MVP Julie Lerman provides in her illustrated Entity Framework, Some Before & After tidbits post:

• Illustrated examples of fixes for the missing foreign key values,
• Views and stored procedures now appear in the Entity Data Model Wizard
• New support for stored procedures
• Replacement of default constructors with Public Shared Function CreateObjectName functions.

Breaking Changes in .NET Fx Libraries (Update: 7/11/2007)

LINQ to SQL: INotifyPropertyChanged and Column properties have changed, which generates 100+ errors in small projects. You'll need to recreate your LINQ to SQL classes with the renamed LINQ to SQL Classes template.

LINQ to SQL: Jim Wooley reports in a comment that the System.Data.Linq.AttributeMappingSource and System.Data.Linq.XmlMappingSource classes have moved to System.Data.Linq.Mapping.AttributeMappingSource and System.Data.Linq.Mapping.XmlMappingSource respectively. [Note: MappingSource changed to AttributeMappingSource and XmlMapping changed to XmlMappingSource, and namespace prefixes added.]

LINQ to DataSet: The System.Data.DataTableExtensions namespace has moved from System.Data.Entities.dll to System.Data.DataSetExtensions.dll so you must change references.

LINQ to XML: Fixing "Could not load type 'System.Linq.Func`2' from assembly 'System.Core, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'." exceptions requires adding a reference to System.Linq.DataSetExtensions.dll. This is probably a temporary expedient for the June 2007 CTP.

Problems with the Orcas June 2007 CTP

Julie reported problems running the Entity Framework June 2007 CTP and I encountered an issue running the extended Northwind database for the C# LINQ to Entity samples under the Orcas June 2007 CTP  in this thread in the ADO.NET Orcas forum.

If you've found and tested new language features (especially that support LINQ), please add a comment with a link to your post.

Technorati tags: Orcas June 2007 CTP, Orcas, Entity Framework June 2007 CTP, Entity Framework, Entity Data Model, Lambda Expressions, VB 9.0, C# 3.0, LINQ to Entities, LINQ, IntelliSense

Michael Stonebraker on Accessing Data as Entities and Low-Latency Streaming

The May/June 2007 issue of the Association for Computing Machinery's ACM QUEUE magazine includes a wide ranging interview: "A Conversation with Michael Stonebraker and Margo Seltzer: Relating to Databases." Stonebraker was the primary force behind UC Berkeley's development of the Ingres relational database, which provided the foundation for the PostgreSQL open-source database, and is an adjunct professor of computer science at MIT. Margo Seltzer is professor of computer science at Harvard University and one of the founders of Sleepycat Software, publishers of the Berkeley DB embedded database engine. Ingres Corporation is owned mostly by Computer Associates and Oracle acquired Sleepycat Software in 2006.

Stonebraker's major premise is that today's one-size-fits-all relational OLTP and OLAP databases/applications, which he calls legacy code sold by elephants, will be displaced by in important vertical markets by new technology sold by start-ups, which Seltzer calls mice. Along the way, he attributes the lack of object-oriented databases' market success to their failure to gain a foothold in the computer-aided-design (CAD) industry.

Accessing Data as Entities

Stonebraker isn't sanguine about the long-term prospects for SQL, which he characterizes as a language that "started out as a simple standard and [grew] into a huge thing, with layer upon layer of junk."

However, he likes Ruby on Rails' incorporation of data access in the language itself—a la LINQ—and implementation of the entity-relationship model as the ADO.NET Entity Framework does. Stonebraker says:

C++ and C# are really big, hard languages with all kinds of stuff in them. I’m a huge fan of little languages, such as PHP and Python. Look at a language such as Ruby on Rails. It has been extended to have database capabilities built into the language. You don’t make a call out to SQL; you say, “for E in employee do” and language constructs and variables are used for database access. ...

Let’s look at Ruby on Rails again. It does not look like SQL. If you do clean extensions of interesting languages, those aren’t SQL and they look nothing like SQL. So I think SQL could well go away. More generally, Ruby on Rails implements an entity-relationship model and then basically compiles it down into ODBC. It papers over ODBC with a clean entity-relationship language embedding.

I don't agree with Stonebraker's description of Ruby on Rails compiling the E-R model to ODBC, which—along with JDBC—he characterizes as "the worst interfaces on the planet. I mean, they are so ugly, you wouldn’t wish them on your worst enemy." However, I share his disdain for ODBC, which is two generations older than SqlClient.

In reality, Stonebraker is describing the Entity Framework and LINQ to Entities.

Although LINQ has an SQL-like syntax, I certainly prefer it to the following Ruby on Rails examples:

class BlogComment < ActiveRecord::Base
  belongs_to :blog_post
end

and

class BlogPost < ActiveRecord::Base
  has_many :blog_comments, :order => "date"
end

or

class BlogPost < ActiveRecord::Base
  has_many :blog_comments, :order => "date",
  :dependent => true
end

Even LINQ's chained method syntax is easier to read than the preceding snippets from Exploring Ruby on Rails. Another benefit of the EF, as well as LINQ to SQL, is that you aren't locked into the ActiveRecord model.

Low-Latency Streaming

Another issue that Stonebraker raises is the issue of latency when processing streaming data, especially in the financial markets. Financial data feeds "are going through the roof" and "legacy infrastructures weren’t built for sub-millisecond latency, which is what everyone is moving toward." Stonebraker observes:

Until recently, everyone was using composite feeds from companies such as Reuters and Bloomberg. These feeds, however, have latency, measured in hundreds of milliseconds, from when the tick actually happens until you get it from one of the composite-feed vendors.

Direct feeds from the exchanges are much faster. Composite feeds have too much latency for the current requirements of electronic trading, so people are getting rid of them in favor of direct feeds.

They are also starting to collocate computers next to the exchanges, again, just to knock down latency. Anything you can do to reduce latency is viewed as a competitive advantage.

Let’s say you have an architecture where you process the data from the wire and then use your favorite messaging middleware to send it to the next machine, where you clean the data. People just line up software architectures with a bunch of steps, often on separate machines, and often on separate processes. And they just get clobbered by latency. ...

If I want to be able to read and write a data element in less than a millisecond, there is no possible way that I can do that from an application program to any one of the elephant databases, because you have to do a process switch, a message to get into their systems. You’ve got to have an embedded database, or you lose.

In the stream processing market, the only kinds of databases that make any sense are ones that are embedded. With all the other types, the latency is just too high.

Of course the same requirement holds true for processing digital sensor streams. These are the types of applications that might benefit from LINQ's expression tree approach. Oren Novotny's LINQ to Streams (SLinq) and Aaron Erickson's LINQ to Expressions (MetaLinq) are interesting steps in this direction. Stonebraker's comment about the reduced latency of embedded databases adds support for my contention that LINQ to SQL should support SQL Server Compact Edition (SSCE).

Technorati tags: Entity Framework, Entity Data Model, LINQ to Entities, LINQ, Ruby on Rails, ActiveRecord, LINQ to Streams, LINQ to Expressions