LINQ and Entity Framework Posts for 9/14/2007+

Changes in Store by RTM for LINQ to SQL's WinForm Databinding with Stored Procedures

My Upgrade to LINQ to SQL's WinForm Databinding with Stored Procedures Scheduled for RTM of September 19, 2007 describes the change from the ISingleResult<T> to the DataQuery<TEntity> type returned to the DataContext object by stored procedures that retrieve data. This means that populating DataContexts with stored procs will be closer to on par with dynamic SQL when Microsoft releases Visual Studio 2008. You'll probably hear more about this change from Dinesh Kulkarni in the near future. In the meantime, I'm anxious to hear from Scott Guthrie how the change will affect the LinqDataSource.

You can expect updates to (or a rewrite of) this and my earlier Problems Using Stored Procedures for LINQ to SQL Data Retrieval post shortly. The first update to this post, an added "DataQuery<T> and Table<TEntity> vs. ISingleResult<T> and List<T> Collections" section, is ready now. (Update 9/19/2007 17:00 PDT.)

Beth Massi and Charlie Calvert Talk VB 9.0 and C# 3.0 on .NET Rocks!

Carl Franklin and Richard Campbell interview Beth Massi, The Visual Basic Team's evangelist extradinoire, and Charlie Calvert of the C# Community in Charlie Calvert and Beth Massi on VB.NET, Orcas, and the FoxPro Knitting Guild, show #274. (Update #1 9/20/2007.)

Scott Guthrie on Using C#'s (New?) null Coalescing (??) Operator with LINQ to XML

Scott Guthrie's 9/19/2007 The C# ?? null coalescing operator (and using it with LINQ) post describes how to prevent exceptions in apps that use LINQ to XML when they encounter a null value type in an explicitly-typed XML element.

The original title of the post was "The New C# ?? null coalescing operator (and using it with LINQ)" and Scott originally started began this post with this sentence:

One of the subtle (but cool) new language features in the VS 2008 version of C# is the ?? "null coalescing" operator. [Emphasis added]

By the time I got around to adding this item, someone appeared to have let Scott know that the ?? operator arrived with VS 2005 in C# 2.0 and the underlined words were excised. Here's feed-squirrel.com's copy of the original post.

VBA has the Nz (null-to-zero) function (Nz(variant, [valueifnull]) to return 0 or a string when a Variant variable contained Nothing. Nz was very useful to replace holes in crosstab queries caused by lack of data for a cell. As I recall, an early version wasn't safe for scripting but was subsequently fixed. VB 6 didn't support this function and it's missing from VB.NET. (Update #1 9/20/2007.)

Tim Anderson posted on 9/19/2007 an interview with Scott, Scott Guthrie on .NET futures, conducted at the MIX07 UK conference held in London on September 11-12, 2007. Tim's September 11, 2007 Silverlight at Mix07 UK post mentions in his "Web service support and LINQ" section:

Silverlight will support JSON, WCF and SOAP. It will also include LINQ, with the possibility of creating custom LINQ data providers - Guthrie mentioned possibilities like a LINQ provider for Amazon’s S3 service.

Mike Tulty Repeats Disconnected Concurrency Tests with the Entity Framework and WCF

Following up on his Disconnected LINQ to SQL post of September 14, 2007, which tested optimistic concurrency management with LINQ to SQL in a Windows Communication Framework (WCF environment), his September 19, 2007 Disconnected LINQ to Entities post repeats the process with the Entity Framework (EF).

Mike finds that the EF complicates optimistic concurrency management (OCM) by:

1. Requiring invocation of the SetModifiedProperty("PropertyName") method on each property when the service doesn't save original values supplied by the client. (Julie Lerman reported this issue previously in her Exploring EntityKeys, Web Services and Serialization a little further post of September 2, 2007.)

2. Looping "around the properties of the entity, use reflection to compare the original and current value [using reflection]. If the values differ we copy the property value from the 'new' entity to the 'original' entity" the service saves original values supplied by the client.

It seems to me that, unlike LINQ to SQL, EF isn't very OCM-friendly. No wonder EF disables optimistic concurrency by default and it must be turned on explicitly, as Mike also mentions in his Entity Framework - Object Services Level. Concurrency. post of August 27, 2007.(Update 9/19/2007.)

Sort the Whole Resultset, Not Just the Page, When Using the LinqDataSource's Server-Side Paging Feature

After I got my ASP.NET/LinqDataSource/GridView test harness paging correctly with stored procedures for the primary LINQ to SQL resultset, EntityRefs and EntitySets—with some assistance from Matt Warren—it became clear that I had more work to do to make sorting work properly. Dynamic ORDER BY Clauses for ASP.NET Server-Side Paging with Stored Procedures of 9/18/2007 is a T-SQL-filled, illustrated chronicle of what's necessary to emulate with a stored procedure the server-side paging and sorting features of dynamic T-SQL. It's not pretty, but the test harness is finished. Now I can get back to finishing the WinForm stored procedure test harness. (Update 9/18/2007.)

Dare Obasanjo Analyzes Proposed Changes to Astoria Data Formats

Dare's Microsoft REST APIs: Astoria = Web3S + Atom - RDF of 9/18/2007 analyzes how Microsoft's Web3S (Web Structured, Schema’d & Searchable) specification fits into the RESTful protocol picture in general and the Astoria Web service in particular. The article's title derives from Pablo Castro's Astoria Design: payload formats post of 9/7/2007 , which announced support for a POX (plain old XML) format would be dropped in favor of Web3S, the APP (Atom Publishing Protocol) and JSON (JavaScript Object Notation) formats would remain, and RDF would be abandoned.

Dare includes links to his two prior posts about Astoria—Google Base Data API vs. Astoria: Two Approaches to SQL-like Queries in a RESTful Protocol—and Web3S—Web3S: A RESTful Protocol for Accessing Windows Live Services. If you're interested in an early, independent review of Astoria, check out Leonard Richardson's June 1, 2007 How's My Driving? #1: Microsoft Astoria post.

Dare concludes his post with this question:

For some reason, I'm not surprised about this decision. I do wonder if dropping RDF will actually bring to light some closet RDF supporters who'd love to see supported in Astoria?

A Google blog search on "Microsoft Astoria RDF" for the past month returned 288 hits but few complained about the demise of RDF as an astoria data format. Mike Bergman's Astoria is Whistling Past the Graveyard to Irrelevance post is one of the few. (Update 9/20/2007.)

Astoria Bits Updated for VS 2008 and Entity Framework Beta 2

Pablo Castro's Astoria CTP Refresh for Visual Studio 2008 Beta 2! post of September 17, 2007 announces updated (not upgraded) release that's compatible with Visual Studio 2008 Beta 2, Entity Framework Beta 2 and the EDM Tools August CTP. According to Pablo Castro, the Microsoft Codename "Astoria" - Sept 2007 Community Technology Preview offers no new features other than those required for compatibility with current beta and CTP bits.

Notice the the "Project" prefix has been dropped from the download headline as the result of Astoria emerging from the incubation project cocoon. Strangely, there have been only 59 posts in the Project Codename: Astoria forum since it opened on May 3, 2007.

So far, the Project Jasper team hasn't announced plans for a release compatible with VS 2008 Beta 2 and EF/EDM Tools Beta 2/August CTP. Andrew Conrad said in a September 13, 2007 Jasper using IronPython and VS 2008 August SDK post to the Project Codename: Jasper forum:

It is actually a bit more work to get Jasper working on Beta2 because of a number of breaking changes to the underlying ADO.Net Entity Framework from Beta 1 to Beta 2. As soon as I have a update with some solid plans, I will make it public.

We actually would love to get the community involved more with Project Jasper. Right now we are very interested in any/ all feedback folks might have.  We are in the middle of the planning process trying to figure out what to do in this space going forward – so feedback is invaluable.

Andy would be more likely "to get the community involved more with Project Jasper" if the team updated their bits to Beta 2. The forum has received a total of only 4 posts since it opened on May 2, 2007. (Update 9/18/2007.)

LINQ and WCF MSDN "Masterclass" Stealth Webcasts

Mike Benkovich, a Microsoft developer evangelist stationed in Minneapolis is 75% of the way through a series of four "Framework Masterclass" presentations on LINQ: 

• Framework Masterclass: LINQ to Entities (Level 200) - 9/24/2007
• Framework Masterclass: LINQ to XML (Level 200) - 9/17/2007
• Framework Masterclass: LINQ to SQL (Level 200) - 9/10/2007
• Framework Masterclass: Working with LINQ (Level 200) - 9/4/2007

He also has four Windows Communication Foundation presentations scheduled: Framework Masterclass:

• Framework Masterclass; Working with WCF - 10/1/2007
• Framework Masterclass: WCF Management & Diagnostics - 10/15/2007
• Framework Masterclass: WCF Custom Channels - 10/22/2007
• Framework Masterclass: WCF Performance & Deployment - 10/29/2007

I'm surprised that there hasn't been more publicity for his LiveMeeting presentations, all of which are about an hour in length. Mike posts the code from his presentations on his MSDN BenkoBlog.

Julie Lerman on Heterogeneous Joins with LINQ

Using LINQ to query Outlook emails joined with SQL data of September 16, 2007 matches simulated email messages from fictitious AdventureWorks customers and assigns them to the appropriate sales representative. In an excellent example of query composability, she defines Selects emails and contacts from Outlook folders and custSalesPerson from the AWCustomers object and performs a three-way Join to generate a "SenderName from CompanyName: Subject [for] SalesPerson" message.

An interesting alternative to the more common LINQ to XML joined with LINQ to SQL objects demo. (Update 9/17/2007.)

Danny Simmons Simulates a Multi-User Dungeon with Entity Framework

Danny's Concepts Part I: Getting an entity model up and running of September 15, 2007 describes an operational Multi-User Dungeon (MUD) game that he plans to use in "a series of blog posts I'd like to write explaining various Entity Framework (EF) concepts, particularly those related directly to writing code using the framework." The first episode includes a simplified Entity Data Model (EDM) diagram and offers links to its CSDL, MSL, and SSDL files. The CSDL file describes the scalar properties of Actor, Item, Room and Exit entities.

Danny discusses the following three EF concepts in depth:

1. Storing your metadata in resource files. I've always considered CSDL, MSL, and SSDL files to be free-standing so they can be edited without recompiling the application, similar to app.config files for connection strings. However, I suppose there is benefit to being able to deploy your client app as an Application.exe and app.config file pair.
2. Connection strings. If you accept the default connection string name in the EDM Designer, its codegen creates a parameterless constructor (ctor) for you. (LINQ to SQL creates a parameterless ctor, too, but it seems to disappear if you edit the construction string. The LinqDataSource needs a parameterless ctor. I'm checking further on this.)
3. Context creation and lifetimes. Danny's client is a WinForms app and assumes long-lived ObjectContexts. He also plans to explore Web services and Web site apps. Based on the problems I've encountered to date with stored procedures for LINQ to SQL data retrieval (see below), I'm sure he'll use dynamic SQL.

He concludes with sample code for console projects to:

Consolidate connection string building, but also creating a common, static entry point to the context, and automatically maintaining a separate context for each thread by putting the context into thread-local storage [and] prints out a list of all the rooms and items in the DPMud database but does so from two threads running concurrently.

Sounds to me like an interesting series. (Update 9/17/2007.)

Update 9/20/2007: Danny notes that storing metadata in resource files causes problems with Visual Studio's unit testing feature in his 9/19/2007 More about embedding metadata in resources... post. The problem arises because VS's unit testing feature uses unmanaged code!

Update: 1-1/2 Problems with Stored Procedures for LINQ to SQL Data Retrieval Now Solved

Many DBAs will require developers to use (DBA-approved) stored procedures to retrieve persisted entities from—as well as to persist them to—relational tables. There's been little in the way of Microsoft documentation or blog posts on how to use stored procedures for entity retrieval other than Scott Guthrie's LINQ to SQL (Part 6 - Retrieving Data Using Stored Procedures) post of August 16, 2007. Using the ASP.NET LinqDataSource control complicates the data retrieval issue with stored procedures for server-side paging of databound GridView and other grid-like controls. Stored procedures also interfere with populating entity associations.

My Problems Using Stored Procedures for LINQ to SQL Data Retrieval (initial of September 8, 2007, updated 9/14/2007 and 9/15/2007) pointed out the problems with server-side paging by handling the LinqDataSource's Selecting event and lazy-loading or eager-loading entity associations. The updated post shows how to solve the problem with inability to display page 2 or higher with a paging stored procedure (set the AutoPage property value to False.)

Input from Matt Warren showed how to override EntityNameLoad methods to substitute stored procedures for dynamic T-SQL for populating 1:many (1:n) associations, which is half the problem. However, you must force eager loading by using LoadWith expressions to invoke the overridden method. There doesn't appear to be a way to lazy-load 1:n associations, so this approach solves only 1/4 of the problem.

Matt posted this comment on 9/17/2007:

Using stored procedures to load associations does actually work with lazy loading. You don't need to use LoadWith.

Update 9/17/2007: Matt's right, as usual, so I changed the heading from 1-1/4 to 1-1/2 of the problems solved. There must have been cockpit error with my ASP.NET test harness that indicated my overriding function wasn't being invoked when performing a count of total Order_Details for the 10 displayed rows. The EntityNameLoad methods for populating 1:many (1:n) associations do work as expected.

There are still no docs or sample code for eager-loading or lazy-loading many:1 (m:1) associations, which is the remaining half of the problem.

Update 9/15/2007: Julie Lerman comments in her LINQ to SQL and stored procedures post about my "obsessive persistence" in tracking down these issues.

Enable LINQ to Query Non-Generic Collections

Fabrice Marguerie's Querying non-generic collections with LINQ to Objects post of 9/15/2007 shows you how to use the LINQ Cast<T> operator to enable use of other Standard Query Operators with ArrayList or other non-generic collections.

Installing the Project Jasper May 2007 CTP Permanently Clobbers the EDM Designer

Running the installer that you download from the Jasper : MIX 2007 CTP page after installing the EF Tools August 2007 CTP causes VS 2008 to be unable to load the EDM Designer bits (Microsoft.Data.Entity.Design.Package.MicrosoftDataEntityDesignPackage, Microsoft.Data.Entity.Design.Package, Version 1.0.0.0 ... .)

Uninstalling Jasper CTP and removing and reinstalling the tools doesn't solve the problem. I posted a question about this problem in the ADO.NET Orcas forum.

Microsoft's Jeff Derstadt said in answer to my question:

The Jasper CTP is not supported for VS 2008 Beta 2. The installer included a patch for an earlier VS 2008 CTP where the EDM tools would not work properly. You will likely need to reinstall VS 2008 Beta 2 to remove this patch.

Andy Conrad in a reply to my comment asking if he had tried it said:

Just to be clear - the sample works with Jasper running over Beta 1 of Orcas.  The problem exists when installing Jasper over Entity Framework Beta 2.  We are currently investigating a solution for this.

Uninstalling and reinstalling VS 2008 Beta 2 solved the problem.

I wonder how many folks are running Orcas Beta 1.

[Repeated from the last update to LINQ and Entity Framework Posts for 9/7/2008+ because of its importance.]

Mapping Stored Procedures to the Entity Data Model

Mike Dodaro, the ADO.NET team's programmer-writer, posted How to Map Stored Procedures using the ADO.NET Entity Framework on September 14, 2007. This article shows how to manually modify the SSDL, CSDL and MSL sections of the *.edmx file to accommodate stored procedures for entity data retrieval. His example uses the SalesOrderDetail table of the AdventureWorks Sales Model, which is an Entity Data Model (EDM) implementation based on the AdventureWorks sample database.

Hopefully, Mike will provide more insight into code for populating association/navigation entity properties with stored procedures.

SqlDependency Notifications from LINQ to SQL-Generated Queries

Ryan Dunn, co-author of the The .NET Developers Guide to Directory Services Programming, provides sample code to enable LINQ to SQL to participate with SQL Notification Services in his September 14, 2007 Using SQLDependency objects with LINQ post. His code creates a SQL dependency context, which Ryan says is very similar to a transaction context. Mike Pizzo, who pointed out the approach to Ryan, can probably provide more implementation details.

Update 9/15/2007: Julie Lerman provides more implementation information about the CallContext class in her Query Notifications and LINQ to SQL - Well I'll be, you *can* do it (with caveats) post.

Frans Bouma Continues LINQ to LLBLGen Pro Series

Frans Bouma, the developer of the LLBLGen Pro commercial O/R mapping tool and principal antagonist of LINQ to SQL, has started a project to adapt LINQ to LLBLGen Pro (or vice versa). His first posts are here:

1. Developing Linq to LLBLGen Pro, Day 0 (September 11, 2007)
2. Developing Linq to LLBLGen Pro, Day 1 (September 12, 2007)
3. Developing Linq to LLBLGen Pro, Day 2 (September 14, 2007)
4. Developing Linq to LLBLGen Pro, day 3 (September 19, 2007)

Thomas Wagner likens Frans' new series to A diary of LINQ pain.

Be sure to read the comments to the first post.

[Continued from LINQ and Entity Framework Posts for 9/7/2008+ ]

Julie Lerman and Entities in Web Pages

Julie's Entities, Viewstate and postback post of September 14, 2007 analyzes binary serialization and deserialization of entities to and from the ViewState object. She notes that managing optimistic concurrency is up to you because, as Mike notes in his LINQ to SQL analysis below, the ...Context object (DataContext or ObjectContext) holds the original values.

She also points to her Serializing entities for updates across web services thread in the ADO.NET Orcas form. The thread contains detailed responses from Danny Simmons that sound remarkably like those from Matt Warren about the corresponding issues with LINQ to SQL.

Mike Taulty Tackles Concurrency Issues with LINQ to SQL and WCF

Mike's Disconnected LINQ to SQL post of September 14, 2007 dissects managing optimistic concurrency with LINQ to SQL. His two code examples assume that LINQ to SQL's DataContext resides in a Windows Communication Foundation (WCF) physical tier that processes create, retrieve, update and delete (CRUD) operations with BasicHttpBinding (SOAP 1.1 transport) to a proxy client. One example does CUD operations with no concurrency checks; the other implements optimistic concurrency with new and original values stored on the server. He provides full SQL Server Profiler traces on the service side.

The LINQ to SQL team promised a sample implementation similar to this some time ago but haven't delivered so far.

Mike cites Rick Strahl's LINQ to SQL and Disconnected Entities Follow-up post of August 15, 2007 as a source of information on managing concurrency with detached entities, that is entities with no associated DataContext. I prefer the term detached, because LINQ to SQL's is a disconnected technology; the DataContext maintains data state between successive connections to SQL Server.

XML Schema Inference Add-In for VB 9.0

The VB Team announced on September 14, 2007 a new schema inference tool add-in for Visual Basic 9.0. The tool is especially important for users of VB's LINQ to XML literal XML feature, because you can add XML schema sets to a project in three ways:

1. Specify an XML file location
2. Specify an URL on the web that returns XML
3. Paste or type an XML document directly

Adding a schema to your XML document enables IntelliSense.

Scott Hanselman gives the add-in a rave review in his same-day Clever: XML To Schema InferenceWizard For Visual Studio 2008 post. For an early look as Microsoft's Schema Inference Engine, see my November 11, 2002 "Generate XSD Schemas by Inference" article for the long-departed XML & Web Services magazine.

Note: I'm still waiting for Ralf Lammel's successor to update LINQ to XSD to the Beta 2 LINQ implementation. Ralf has accepted a full professorship at the University of Koblenz and I wish him the best of luck in his new position. Ralf's last post with a LINQ tie-in was his More than a revision on “MapReduce revisited” of April 11, 2007, which include a C# 3.0, LINQ-style abstraction for MapReduce computations. 

A Compendium of Julie Lerman's Entity Framework Beta 2 Posts

Mike Taulty's Entity Framework Posts item of September 13, 2007 has links to all 13 of Julie Lerman's posts about Entity Framework Beta 2 and EDM Designer August 2007 CTP. Links to these articles also appear in previous OakLeaf posts, but having all links in one place is handy. Mike says her posts are "great stuff" and I agree.

First Appearance of LINQ for Java Clone

Anders Norås introduced Quaere, a.k.a language-integrated queries for Java, on September 10, 2007 at the JavaZone conference in Oslo, Norway, according to his Introducing Quaere - Language integrated queryies for Java post the next day. A snapshot of the source code for Quaere's first implementation, Quaere for Objects, is available as Quaere Snapshot.zip at http://andersnoras.com/files/folders/code/entry281.aspx.

Later posts explain the derivation of Quarere (9/12/2007, the Latin root of the English word query, imperative form of quaerere - to question or ask), Answering questions about Quaere (9/13/2007) and Solving Puzzles with Quaere (9/14/2007). The last post uses Quaere to solve the same puzzle Microsoft's Luke Hoban solved with LINQ in his Using LINQ to solve puzzles post.

Technorati Tags: LINQ to Entities, LINQ to SQL, SqlDependency, LINQ to XML, LINQ to XSD, LINQ, Entity Framework, Entity Data Model, EDM Designer, O/R Mapping, Object/Relational Mapping, Project Jasper, Project Astoria, WCF, Windows Communication Framework, VB 9.0, Quaere