LINQ Changes from Orcas Beta 1 to VS 2008 Beta 2

There's a substantial amount of blogging going on about new features and changes to LINQ to Objects and its domain-specific versions in Visual Studio 2008 Beta 2. This post is intended as a central anthology of changes between the two betas with links to the original sources.

LINQ to Objects (Generic LINQ)

NEW VB LINQ-Related Keywords and Features

VB 9.0 gets the missing LINQ keywords promised by Amanda Silver and noted (except Into) in my May 15, 2007 Changes Coming to LINQ for SQL post:

• Join
• Group By
• Group Join
• Into
• Skip [While]
• Take [While]
• Aggregates (Sum, Min, Max, etc.)
• Let

VB 9.0 now supports the following LINQ-related features mentioned (except closures and mutable anonymous types) in the preceding post:

• Nullable shortcut syntax (int?, Double?)
• Lambda functions (the VB team calls them "inline functions")
• Partial methods
• Closures
• Ternary If operator
• Mutable anonymous types when using New With { ... } syntax *

* C# anonymous types are immutable, as are conventional VB anonymous types created by the LINQ Select clause and and implied Select clauses. You can make members of mutable VB anonymous types immutable by specifying their Key value, as in New With { a.Key, b.Key }. For more information on VB's new mutable anonymous types, see Paul Vick's Mutable and immutable anonymous types, and keys post of May 11, 2007.

VB's Group Join and Group By statements require Into Group rather than C#'s Into identifier expression, as reported my 7/28/2007 Strange Change to VB Join and Group By Expressions in VS2008 Beta 2 post.

VB MVP Bill McCarthy reports in a comment to Strange Change to VB Join and Group By Expressions in VS2008 Beta 2 that VB also supports Group Join identifier1 ... Into identifier2 = Group and Group By's Group prefix1 By prefix1.identifier1 Into identifier2 = Group expressions.

There's a bug in VB's implementation of the IEnumerable.Count() method. The <Extension> attribute doesn't override the default Count() operator for a List(Of T) in Beta 2. The following C# code works fine (OrderList is 100 Northwind order elements in an List<Order>):

int countUsOrders = OrderList.Count(c => c.ShipCountry == "USA");

The following VB statement won't compile:

Dim CountUsOrders2 As Integer = OrderList.Count(Function(c) c.ShipCountry = "USA")

It throws a "'Public ReadOnly Property Count() As Integer' has no parameters and its return type cannot be indexed" error. These two statements compile and run as expected:

Dim CountUsOrders1 As Integer = OrderList.Where(Function(c) c.ShipCountry = "USA").Count()

Dim CountUsOrders As Long = OrderList.LongCount(Function(c) c.ShipCountry = "USA")

See my Strange Behavior of VB 9.0 Count() Operator with Predicate LINQ Project General forum post and bug report 291116.

Update 8/9/2007: Jonathan Aneja, a program manager on the VB Team says in response to the bug report:

In VB Properties shadow-by-name, so this is actually by design. While it makes this case a bit inconvenient, you can work around it by calling LongCount, or by using non-Extension method syntax and calling Enumerable.Count(OrderList, ...).

It's unfortunate that VB programmers must special-case the most popular of built-in aggregate operators. (Timothy Ng provides a more detailed response to my forum message.) 

Mike Taulty notes in MMmmmmm. VB 9.0 in Beta 2 the new VB-specific query expression syntax for Skip and Take. He says:

There's an awful lot going on with VB in 9.0 - truly great stuff. Makes me feel like converting! [Emphasis added.]

Non-VB LINQ Issues

IQueryable has been refactored into IQueryable and IQueryProvider (see below.)

No Change: Joe Albahari reports that the NotSupportedException: Aggregate operators are not supported in projections bug he found in Beta 1 is still present in Beta 2, according to the Bug - LINQ to SQL aggregations in projections thread in the LINQ Project General forum.

The ToLookup() standard query operator returns the ILookup<TKey, TSource> or ILookup<TKey, TElement> interface, not the expected Lookup<TKey, TSource> or Lookup<TKey, TElement> type. C# requires an explicit cast to Lookup<,>; VB performs an implicit cast. See Jomo Fisher's response to my ToLookup() Operator Doesn't Behave As Expected in C# Query Expression message in the LINQ Project General forum. (This behavior might not have changed in Beta 2.)

Matthieu Mezil reports in a comment to this post:

System.Data.Entity.dll becomes System.Data.DataSetExtensions.dll with Beta 2. So if you used DataSet with Orcas Beta 1, you must change [the] reference in your project.

Off topic, Bill has a fix for the VB and C# code samples that won't open by double-clicking the .sln file to change "Visual Studio Codename Orcas" to "Visual Studio 2008" in the subdirectories of the folder you run the code from.

Update 7/31/2007: Added VB's new mutable anonymous types. Update 8/2/2007: Updated and moved Matt Warren's IQueryProvider series, added Mike Taulty's salute to VB 9.0, and my question about the ToLookup() standard query operator. Update 8/6/2007: Added VB 9.0 IEnumerable.Count() bug and Matt Warren's part 5. Update 8/8/2007: Added System.Data.Entity.dll change to DataSetExtensions.dll from Matthieu Mezil. Update 8/9/2007: Categorized LINQ to Objects issues and updated Microsoft response to VB Count() issue.

IQueryable, IQueryProvider, and Expression Trees

Matt Warren, principal architect in the C# group and LINQ to SQL protagonist, advises in his LINQ: Building an IQueryable Provider - Part I post of July 30, 2007 that IQueryable has been refactored into IQueryable and IQueryProvider in Beta 2. This is a breaking change for all third-party LINQ query providers.

On the other hand, current and erstwhile third-party LINQ query provider writers will finally get some help from Matt's informal documentation series. (Near the end of the comments, Frans Bouma says he's starting work on a query provider for his LLBLGenPro object/relational mapping (O/RM) tool.)

Matt's LINQ: Building an IQueryable Provider - Part II shows you how to create a very simplified version of the LINQ to SQL provider that translates a LINQ query into SQL command text and and translates the result of executing the command into objects. Matt also provides (by popular demand) a public implementation of the internal ExpressionVisitor class from the System.Linq.Expressions namespace. (Jomo Fisher says in an update to his May 23, 2007 Dealing with Linq’s Immutable Expression Trees post that Matt's implementation "is pretty close to the one we use internally.") Frans Bouma comments, "[The] design is flawed, and the root cause is the fact that it's a combination of concerns: a query and an enumerable resultset" that include a dependency on the database engine. (Updated 7/31/2007)

Matt continues by adding support (or a fix) in LINQ: Building an IQueryable Provider - Part III for translating local variables, such as:

string city = "London";
var query = db.Customers.Where(c => c.City == city);

This turns out to be more of an exercise than expected because local variables in expression trees turn into a compiler-generated class to hold the local variable. (You've seen these compiler generated classes if you've examined query expressions with Reflector.)

So Matt provides the code for an Evaluator class, which he says "exposes a static method ‘PartialEval’ that you can call to evaluate these sub-trees in your expression, leaving only constant nodes with actual values in their place." Nice work!

Not willing to leave well enough alone, Matt muddies the waters today with LINQ: Building an IQueryable Provider - Part IV, in which he implements an improved Select translator with a new ColumnProjector class and a visitor that builds an SQL SELECT clause. What could be next? And where's Frans Bouma? (Update 8/3/2007: See Frans Bouma's new comment to this post.)

Matt comes through with yet another refactoring exercise to solve member access (projection) problems with method calls that reverse the order of Where and Select and add a member that doesn't correspond to a column. This requires a new set of expression node types to abstract SQL (TableExpression, ColumnExpression, and SelectExpression) and LINQ (ProjectionExpression.) There's much more rework in LINQ: Building an IQueryable Provider - Part V, but fortunately Matt includes a zip file of the source code to date (Query5.zip), which compiles under Beta 2 and runs a simple demo query in the console. Matt says, "[I]t works no matter how many Select’s or Where’s I add, no matter how complex I make each projection." At this point in the game a flow diagram would be a big help. (Update 8/6/2007)

The djini must be out of the bottle because Matt, Luca Bolognese, and Charlie Calvert produced a 40-minute "LINQ [to SQL] Pipeline" video and released it to SoapBox yesterday afternoon. The first 15+ minutes is devoted to diagramming the translation of LINQ queries to T-SQL text strings. Other topics include diagramming the SubmitChanges operation, and reminiscing about the history of LINQ to SQL and its ObjectSpaces roots. Matt and Luca have interesting posts about their relationship, LINQ, LINQ to SQL and ObjectSpaces. (P.S. The ObjectSpaces PDC alpha preview was code-named "Orca." Object Spaces was in Whidbey Beta 1 and removed from Whidbey Beta 2. For the tragic tale of ObjectSpaces demise at the hands of WinFS, see my Microsoft Bites the Bullet: WinFS Bites the Dust post of June 26, 2007.) (Update 8/7/2007)

Part 6 uncovers another defect in the now-adolescent application: Subqueries in projections throw exceptions. In LINQ: Building an IQueryable Provider - Part VI, Matt takes on the case where projection nodes appear in selector expressions with nested queries. He solves the problem with "lots of extra little queries," which require MARS (MultipleActiveResultSets=true in the connect string) so the Query6 implementation won't work with SQL Server 2000. Version 6's source code gets the DataContext.Log property so you don't need Profiler to check the T-SQL emitted. (Update 8/10/2007)

LINQ to SQL

Joe Albahari reports that EntitySets still don't support IQueryable and Jomo Fisher says this will continue to be the case in RTM, according to the August 6, 2007 Why does EntitySet no longer implement IQueryable? and the August 15 What do others think of the crippled new EntitySet?post in the LINQ Project General forum. (EntitySets supported IQueryable in the LINQ May 2006 CTP.) Lack of support for IQueryable reduces composability of queries over EntitySets. Update 8/15/2007.

Creating an SQL Server database with SQLMetal.exe now generates a user instance (local .mdf file) instead of attaching the database to a SQL Server instance. (See Jay Hickerson's answer to Joe Rattz in the In Beta2, SQLMetal no longer attaches the database thread of the LINQ Project General forum. Update 8/7/2007.

Beta 2 requires attaching every object individually to the DataContext. Beta 1 and earlier attached the specified object and all reachable associated objects. (See Matt Warren's answer in the Attach, SubmitChanges doesn't save child objects thread of the LINQ Project General forum. Update 8/10/2007.)

VB MVP Jim Wooley contributes his July 30, 2007 list in LINQ to Sql changes from Orcas Beta 1 to Visual Studio 2008 Beta 2.

Jim Wooley also observed in the LINQ to SQL Entities as WCF DataContracts post in the LINQ Project General forum that Beta 2 removes the capability to set custom attributes for WCF in the LINQ to SQL Designer, but Dinesh Kulkarni says in an answer:

You can specify that you want [WCF] DataContract/DataMember attributes in designer or SqlMetal.

In designer, just right click on the design surface and specify "Unidirectional" in property grid for "Serialization Mode".

For SqlMetal, the /serialization:Unidirectional does the same trick.

Jim reports that the expected name change of the DataShape object to DataLoadOptions mentioned in messages in the LINQ Project General forum has occurred. Note that DataContext.Shape property names changes to DataContext.LoadOptionsShape also. (Updated 8/1/2007)

Mike Taulty adds more detail about the changes in his VS 2008 Beta 2 - LINQ to SQL post of July 30, 2007. 

Dinesh Kulkarni's July 26, 2007 LINQ to SQL (fka DLinq): What's new in Orcas beta2 post (unearthed by Julie Lerman) describes 15 LINQ to SQL improvements and new features. Dinesh would have better said "SQLMetal.exe supports SQL Server CE" in item 7 and I the same in the post's title, because the LINQ to SQL Designer doesn't work with SSCE in Beta 2 and won't in RTM. (See LINQ to SQL to Support SQL Server Compact Edition.)

According to a post by Dinesh in the LINQ Project General Forum:

1. GetQueryText() is superseded by GetCommand() which gives you the SqlCommand(). This allows programmatic access in addition to getting the text.
2. GetChangeText() is superseded by GetChangeSet() which gives you access to the the set of changed objects. Again this allows programmatic manipulation including complete takeover of SQL command generation or additional/alternate actions.

Dinesh's July 29, 2007 Orcas beta2: what's where (directories I mean)? post clarifies the location and provision of LINQ to SQL-related files and features:

So here is where you can find LINQ to SQL related files.

• New .NET Framework 3.5 DLLs (green bits) are in: Program Files\Reference Assemblies\Microsoft\Framework\v3.5\
• SqlMetal.exe is in Program Files\Microsoft SDKs\Windows\V6.0A\Bin\

Another source of confusion - designer vs. SqlMetal

• The visual designer for generating mapped LINQ to SQL classes is a part of Visual Studio 2008. You need to install some SKU of Visual Studio to use it.
• SqlMetal.exe is a command line tool that is a part of the Windows SDK and lives in the same directory as similar command-line tools like Xsd.exe

Scott Guthrie's LINQ to SQL (Part 5 - Binding UI using the ASP:LinqDataSource Control) post of July 16, 2007 is a tutorial for connecting the new LINQDataSource control for ASP.NET to LINQ to SQL's graphical object/relation mapping (O/RM) tool and taking advantage of LINQ to SQL as a data access layer (DAL) with paging, validation, and other business-related features. Andres Aguiar has an interesting commentary on the LinqDataSource control and its incestuous relationship with LINQ to SQL in his July 30, 2007 Digging into LinQDataSource post.

Scott also posted a tutorial for using the LINQ to SQL Debug Visualizer. He says:

The LINQ to SQL Debug Visualizer isn't built-in to VS 2008 - instead it is an add-in that you need to download to use.  You can download a copy of it here.

The SQL Debug Visualizer is included in the CSharpSamples.zip file located in the \Program Files\Microsoft Visual Studio 9.0\Samples\1033 folder. However, the ReadMe.html page's recommended deployment folder for the DLL file is incorrect. Scott's recommended location is \Program Files\Microsoft Visual Studio 9.0\Common7\Packages\Debugger\Visualizers\, which is correct. That folder contains the DataSetVisualizer.dll file, but requires Vista users to have Administrator privileges. Here's a link to my bug report on the issue. Microsoft says either location is OK, but Documents ... didn't work for me. (Update 8/4/2007)

Dan Wahlin's Video: First Look at Visual Studio .NET 2008 and the LinqDataSource shows you how to use the LINQ to SQL Classes template with the LINQDataSource and common databound controls.

Jim Wooley reports in his August 6, 2007 Use the new LINQ "Contains" extension method for the SQL "IN" clause post that LINQ to SQL Beta 2 translates LINQ queries that use the "reverse mode" of IEnumerable<T>'s Contains() extension method into an T-SQL IN() clause. For example, the following LINQ to SQL query: 

Dim strStates As String() = New String() {"CA", "OR", "WA"}
Dim States = From St In strStates Select St
Dim WestCoastCustomers = dc.Customers.Where(Function(c) States.Contains(c.Region))

Generates a T-SQL string for a single query with the equivalent of an IN('CA', 'OR', 'WA') clause. (Update 8/7/2007)

The Union/union operator bug is fixed (see Changes Coming to LINQ for SQL.) This bug was in the way the LINQ to SQL expression tree handled translation of literal column names, not in LINQ's standard query operator.

LINQ to XML

I haven't seen any news on this topic so far from the Microsoft XML Team's WebLog or Eric White's Blog.

Hopefully, Ralf Lämmel will update his LINQ to XSD preview for Beta 2 shortly.

LINQ to DataSet

Daniel Moth, a Microsoft UK developer evangelist and .NET CF expert, offers a July 27, 2007 LINQ to DataSet post that describes the change in the location of the LINQ to DataSet extensions from System.Data.Entity.dll to System.Data.DataSetExtensions.dll. Daniel also provides class diagrams which illustrate the six infrastructure-supporting types, as well as the three extension methods to aid working with typed and untyped DataSets: DataTableExtensions, TypedTableBaseExtensions and DataRowExtensions.

Update 8/9/2007: Added link to Daniel Moth's page and changed DataSets to DataSet.

I'll update this post as I uncover more changes myself or from others. Please add a comment if you have and changes to contribute.

Technorati tags: LINQ, LINQ to SQL, LINQ to XML, LINQ to DataSets, VS 2008 Beta 2, VB 9.0, C# 3.0, LinqDataSource