Jon Udell: LINQ 101

InfoWorld's Jon Udell continues his investigation of LINQ and XLinq features with the C# implementation by demonstrating "a three-way join across an XML data source and two CLR objects." He concludes:

When I said that LINQ turns query inside out, I wasn't kidding. There is a lot here to think about and explore.

--rj Technorati: LINQ XLinq

More on Object-XML and Object-Relational Impedance Mismatch

In a very early (late 1950's) avatar, I was a studio and transmitter engineer for Berkeley's listener-sponsored FM station, KPFA and later majored in electrical engineering at UC Berkeley. Thus my initial reaction to the term "impedance mismatch" was from the audio and RF environments where mismatches cause serious problems, such as standing waves in antenna transmission lines. The advent of object-oriented programming has added a new meaning to the term. Wikipedia's Object-SQL Impedance Mismatch topic and Scott Ambler's The Object-Relational Impedance Mismatch essay provide an introduction to the usage in object/relational mapping (O/RM) scenarios. Dare Obasanjo's The Impedence Mismatch between W3C XML Schema and the CLR post discusses XML serialization of object state and conversion of XML to objects. The latter topic includes a reference to Programming with Circles, Triangles and Rectangles by Erik Meijer, et al. Following are additional links to other posts about impedance mismatch between objects, XML documents, and relational data from the CORBA, Java and LINQ perspective: Middleware Matters: Obsessed with the impedance mismatch—IONA Technologies' Steve Vinoski discusses mismatch issues between objects and XML, with a reference to COmega:

Basically, unless I'm real busy doing something else, my thoughts invariably turn to the impedance mismatch problem. I spend a fair bit of time searching for what others are doing to solve it, and I know about Comega and some of the domain-specific language ideas, but finding relevant work is hard because AFAIK there's no crisply defined category for such work.

Oracle Developer JAVA MONKEY: Advances in Java's Relationship to Data—Rene Bonvanie discusses "Understanding new and existing features of J2EE and EJB for working with data" and quantifies the programming effort to overcome impedance mismatch between objects and relational data:

With the ease of use associated with JDBC and SQLJ, however, comes what is frequently called impedance mismatch. Both cater directly to the relational database—working in rows and columns. Java, on the other hand, is naturally object-oriented and is designed to help developers think in classes and business objects. Additional work, estimated as high as up to 40 percent of the initial development effort, may be spent resolving this mismatch. [Emphasis added.]

Abstraction Impedance: Hidden Barriers and Explicit Boundaries—Dion Hinchcliffe (Sphere of Influence) qualifies impedance with "abstraction" and distinguishes "structural" and "behavioral" abstraction impedance:

The premise of the LINQ project, and many others like it, is the following: Now that the object-oriented paradigm has become established as the predominate model of abstraction in software development, developers have been incurring a large amount of overhead in mapping it to other abstractions, specifically XML and relational databases. ... Abstraction impedance isn't just a developer productivity tax. It leads to subtle and pernicious defects in software, contortions of design, and severely degraded performance. All of these lead to low quality software that doesn't deliver the expected value to the customer. ... So, I find the LINQ effort to be admirable and extremely interesting and I'm eager to see how effective it is. Solving this problem well may end up being one of the bigger advances in software development this decade.

Stay tuned for additional links about the LINQ Project, O/R or O/X mapping, and impedance mismatch.

--rj

Technorati: LINQ DLinq XLinq COmega

Delphi to Adopt LINQ Technology

eWeek's Darryl K. Taft writes in "Borland Plans to Support MS LINQ in Delphi Platform," published September 17, 2005:

Microsoft announced its Language Integrated Query Project earlier this month, and already third parties are talking about extending their technology to support the software giant's heralded new technology. In a Weblog post, Borland Software Corp.'s chief scientist Danny Thorpe talked about how Borland's Delphi programming environment can take advantage of LINQ (Language Integrated Query) to enable developers to more easily integrate and manipulate data in building Delphi-based applications.

ISV support for and comment on LINQ is growing. Danny's post relates primarily to LINQ's SQL-like syntax for querying data sources exposed by DLinq. (Note: Anders Hejlsberg held Danny's position at Borland before moving to Microsoft.) --rj Technorati: LINQ DLinq

CyrusN's C# 3.0 and LINQ Tutorials

Cyrus Najmabadi, a developer on Microsoft's C# team has started a series of tutorials on C# 3.0 and LINQ:

• So what's the deal with this whole C# 3.0 / LINQ thingy?—Background on the reasons for introducing LINQ and simple syntax examples.
• So what's the deal with this whole C# 3.0 / LINQ thingy? (Part 2)—LINQ query syntax and sequence.
• LINQ specifiqs - var—Implicitly Typed Local Variables (a.k.a. var).

These posts, which expand on earlier Microsoft white papers, are likely to be of equal interest to C# and VB programmers. I'll update the list as Cyrus adds new posts. --rj Technorati: LINQ

Comparing C# 3.0 and VB 9.0 LINQ Syntax

Prior to LINQ's introduction, differences between the C# and VB (.NET) languages were almost exclusively syntatical. The .NET Framework's classes and the CLR's MSIL unified the capabilities of programs written with C# and VB. If the release version of Visual Studio "Orcas" includes LINQ and its DLinq and XLinq APIs, much of the everyday data-management and XML-processing syntax of C# 3.0 and VB 9.0 source code will diverge dramatically. .NET developers will need to evaluate the language's query operator set and query effectiveness before deciding between C# 3.0 or VB 9.0 for writing data-intensive .NET 3.0 applications or components.

.NET developers usually base their choice of language on their programming background. As a general rule C/C++/Java programmers, who are likely to have learned these languages while attending a college or university, choose C#. VB (COM) developers, most of whom learned the language by reading books and attending conferences or training classes, adopt VB .NET. The VS team has identified the following three Windows programmer personae:

• Mort, the opportunistic developer, likes to create quick-working solutions for immediate problems, focuses on productivity, and learns new language skills as needed. Morts commonly create ad hoc, line-of-business applications and components with VB.
• Elvis, the pragmatic programmer, likes to create long-lasting solutions that address the problem domain, and learns while working on the solution. Elvises commonly use C# to develop reusable components and enjoy implementing advanced .NET features, such as remoting and the Windows Communication Framework (WCF or Indigo).
• Einstein, the paranoid programmer, likes to create the most efficient solution to a given problem, and typically learn in advance before working on the solution. Einsteins usually have a C/C++ background and a master's degree or higher, and some write Windows device drivers.

The preceding descriptions were adapted from Nikhil Kothari's blog entry. Wesner Moise offers similar descriptions for the three personae, and Rockford Lhotka provides a detailed analysis of Mort as a business programmer.

Microsoft's VB and C# teams diverge on LINQ syntax. VB 9.0 query comprehensions support SQL-like operators, while C# keywords have a decided XQuery FLWOR (for, let, where, order by, return) flavor. (Wes's "LINQ Syntax" item offers a sample FLWOR expression and its C# query counterpart.)

Compare this preliminary list of VB 9.0 query keywords from Paul Vick's PDC 2005 presentation:

• Project: Select <expr> From <expr> In <expr>
• Filter: Where <expr>
• Test: Any(<expr>), All(<expr>)
• Join: <expr> [[Left Right] Outer] Join <expr> On <expr>
• Group: Group By <expr>
• Aggregate: Count(<expr>), Sum(<expr>), Min(<expr>), Max(<expr>), Avg(<expr>)
• Partition: Top(<expr>);
• Set: Distinct, Union, Intersect, Except
• Order: Order By <expr> [[Asc] Desc]

with this much longer list of C# 3.0 query operators from the LINQ Project overview document, which is missing the From and In operators:

• OfType: Filter based on type affiliation
• Select/SelectMany: Project based on transform function
• Where: Filter based on predicate function
• Count: Count based on optional predicate function
• All/Any: Universal/Existential quantification based on predicate function
• First/FirstOrDefault: Access initial member based on optional predicate function
• ElementAt: Access member at specified position
• Take/Skip: Access members before/after specified position (paging)
• TakeWhile/SkipUntil: Access members before/after predicate function is satisfied
• GroupBy: Partition based on key extraction function
• ToDictionary: Create key/value dictionary based on key extraction function
• OrderBy/ThenBy: Sort in ascending order based on key extraction function and optional comparison function
• OrderByDescending/ThenByDescending: Sort in descending order based on key extraction function and optional comparison function
• Reverse: Reverse the order of a sequence
• Fold: Aggregate value over multiple values based on aggregation function
• Min/Max/Sum/Average: Numeric aggregation functions
• Distinct: Filter duplicate members
• Except: Filter elements that are members of specified set
• Intersect: Filter elements that are not members of specified set
• Union: Combine distinct members from two sets
• Concat: Concatenate the values of two sequences
• ToArray/ToList: Buffer results of query in array or List
• Range: Create a sequence of numbers in a range
• Repeat: Create a sequence of multiple copies of a given value

In addition to the query comprehension operator (keyword) differences between VB and C#, C# reverses the sequence of SQL's SELECT and FROM clauses. The following VB expression:

expr = Select s.ToUpper() From s In names Where s.Length = 5 Order By s

translates to this C# expression:

expr = from s in names where s.Length == 5 orderby s select s.ToUpper() ;

Paul Vick addresses the query comprehension operator sequence issue from the VB Team's standpoint—following the SQL convention simplifies usage because VB programmers (Morts) are likely to be conversant with SQL. However, substitution of In for As departs from SQL convention. I would have no problem with either

expr = Select s.ToUpper() From names As s Where s.Length = 5 Order By s

or

expr = From s In names Where s.Length = 5 Order By s Select s.ToUpper()

but I'd prefer the latter construction if it improved performance, reduced resource consumption, or is the only sequence that would support IntelliSense fully. Implementing IntelliSense for the previous example's Select list appears to me to be impossible in advance of defining the data source with the From ... In clause.

Wesner Moise makes the following observations in his VB 9 item of September 16, 2005:

The difference in focus between the C# and VB team is becoming much more evident: The VB team is opting for greater relevance and convenience in the short term, while the C# team is trying to avoid creating long-term baggage. The end result is that Visual Basic is definitely going to be much more fun to program. [Emphasis added.]

While the C# [team] has decided to include a few choice SQL keywords (select, from, where, orderby and groupby) and no XML literals, the VB team has opted to include the full range of keywords available in a SQL query as well as support for embedded XML with schema validation.

Paul Vick confirms Wes's conclusion from the XLinq perspective in his "Integrating XML: Playing with Fire?" item. However, DLinq users undoubtedly will want to see Having, the In() predicate and other commonly-used ANSI SQL reserved words added to VB's query comprehension operators list.

The upshot: VB 9.0 will be firmly in the Mort camp and C# 3.0 will continue to appeal primarily to the Elvis and Einstein crowd. Whether the two sets of query comprehension operators will have identical capabilities with both the DLinq and XLinq APIs remains to be seen.

--rj

Update: 10/11/2005: Interact Software, Ltd's Ian Griffiths explains the significance of and demonstrates C# 3.0 expression trees in his "C# 3.0 and LINQ - Expression Trees" post.

Technorati: LINQ XLinq VB 9.0 C# 3.0

Amanda Silver, Paul Vick, and Erik Meijer VB 2005 Interview

Carl Franklin interviews the VBTeam's Amanda Silver and Paul Vick, plus the SQL Server group's Erik Meijer (a.k.a., VBTV's "Head in a Box") for the second half of the September 26, 2005 .NET Rocks show. The interview doesn't deal with LINQ in depth—consider it a sales pitch for VB 2005 with a low-level introduction to forthcoming LINQ features. The interview contains only about 10 minutes of LINQ-related conversation. Following is a log for the VB-related topics of the full WMV file: 00:24:27 - New VB 2005 language features 00:32:20 - LINQ introduction by Amanada and Paul 00:39:20 - Commercial break 00:41:45 - More LINQ and DLinq features 00:44:45 - Erik Meijer (on XLinq) 00:49:20 - More VB 2005 language features 01:04:11 - New features deferred to VB 9.0 from VB 8.0 (400 to 500 in the queue) 01:09:00 - Bill Gates' influence on VB 2005 and successors 01:11:00 - VB generics 01:14:40 - VB Express, programmer education, and hobbyist programmers 01:17:25 - Links to resources 01:20:00 - Coaching Anders Hejlsberg for a VB demonstration 01:24:00 - End Carl recorded the interview on September 14, 2005 at PDC 2005. --rj Technorati: LINQ DLinq XLinq

XLinq Presentation at the XML 2005 Conference

Erik Meijer will present "XLinq: XML Programming Refactored (The Return of the Monoids)" at Atlanta's XML 2005 Conference on Wednesday, November 16, 2005. Note that the conference program lists the presentation title as "XMP [sic] Programming Refactored (The Return of the Monoids)" because the XLinq name was under NDA until PDC 2005; "XMP" is an obvious typo. The paper, by Erik and Brian Beckman*, provides a concise description of the objectives and implementation of the XLinq API for LINQ. XLinq replaces the document-oriented XML DOM with an element-oriented API that enabled constructing elements and attributes independently of a containing document. The paper is unique because all but one of its examples use VB 9.0 rather than C# 3.0. VB 9.0 is required to implement XML literals and axis members. XML literals enable definining elements and attribute with VB 9.0 code similar to the following:

Dim BillTo =
<billTo country="US" >
  <name>Robert Smith</name>
  <street>8 Oak Avenue</street>
  <city>Old Town</city>
  <state>PA</state>
  <zip>95819</zip>
</billTo>

Notice the lack of VB's line-continuation characters, which aren't required or permitted in XML-literal statements. Whitespace formatting is preserved and XLinq overloads the ToString method to simplify display of formatted documents. (Oleg Tkachenko has some reservations about functional construction of XML trees and VB literal XML.) Axis members substitute dot syntax for XPath axis specifiers. For example, BillTo.street returns all street child elements from the BillTo XElement object. Read Erik's paper to learn more about VB 9.0's XLinq implementation. --rj Technorati: LINQ XLinq * Brian Beckman was one of the original developers of Microsoft's XLANG—a notation for the specification of message exchange behavior among participating web services—and is the author of the "Physics of Racing" series.

Jon Udell: Microsoft driving toward .Net unity

The InfoWorld Test Center's lead analyst, columnist, and blogger-in-chief says: "LINQ is an extraordinary innovation that turns traditional query inside out. But the bigger story from PDC 2005 is that the .Net vision of unifying many balkanized disciplines within the Microsoft ecosystem is finally becoming a reality." Read Jon's September 21, 2005 "Strategic Developer" column. Not to be outdone by his colleague, Tom Yeager—InfoWorld Test Center's chief technologist—devotes this week's "Ahead of the Curve" column to LINQ. --rj Technorati: LINQ

LINQ Interview with Anders Hejlsberg and Paul Vick

InfoWorld columnists Jon Udell and Tom Yeager interviewed Anders Hejlsberg and Paul Vick at PDC 2005. Here's a link to the 27:41 Podcast, "A conversation with with Anders Hejlsberg and Paul Vick about LINQ," which also includes transcripts of selected comments by Anders. Jon's item also includes a link to "The LINQ Project: .NET Language Integrated Query" MSDN white paper by Anders Hejlsberg and Don Box. The white paper emphasizes C# 3.0, but includes a few VB 9.0 examples. Paul Vick at 13:40 says, "One of the great advantages [of VB 9.0 is that] you learn one language. It can be C# or Visual Basic, but you learn one language; you don't have to go out and learn SQL. A lot of our developers do know SQL because they've been forced to, but now you don't actually have to do that." Microsoft initially hyped CLR integration with SQL Server 2005 as a means to eliminate Visual Studio developers' need to be proficient in [T-]SQL, as mentioned in my recent "When To Use SQLCLR, And When Not To" article. Substituting a knowledge of VB 9.0 query comprehensions for [T-]SQL proficiency is equally impractical in the real world. No matter what programming language or platform you choose, a working knowledge of SQL is essential for developers of data-intensive applications or components. --rj Technorati: LINQ

Cure Data Type "Impedance Mismatch" With LINQ

This article from Fawcette Technical Publications' .NETInsight newsletter describes how to install and test the C# 2.0 and VB 8.0 implementations of the LINQ Project, DLink and XLink. Although this post from the VBTeam's blog states that the initial PDC 2005 Technology Preview (TP) bits work with the current VS 2005 release candidate, which is available for download from MSDN, I've found the following issues on various VS 2005 versions:

• Neither the VB or C# preview run correctly on the Beta 2 version (build 8.0.50215.44) mentioned as required in the Readme document for the C# version. This build is running here under Windows 2003 Server SP-1.
• The SQLMetal.exe command-line utility to generate C# entity classes for SQL Server databases and DLinq throws an unhandled exception—Invalid object name 'Dbo.customers'—when run under Beta 2 (build 8.0.50215.44).
• The C# preview doesn't add LINQ templates or integrate correctly under the VS 2005 RC IDE after running Install Visual Studio IDE Support.cmd. Attempts to edit any C# LINQ source code result in an instantaneous VS 2005 shutdown and restart. This build is also running here under Windows 2003 Server SP-1.
• The only VS build that appears to me to work correctly with the Technology Preview is Beta 2 build 8.0.50727.7 (a.k.a., the July CTP), which I'm running under Windows XP SP-2.
• The C# SampleQueries project throws 60 compilation errors, regardless of the VS 2005 build under which it runs.

I've also determined that DLinq support for VB projects in the initial Technology Preview is non-existent; installing the VB TP bits doesn't install DLinq.dll to provide the System.Data.DLinq classes. My article quotes the VBTeam's Amanda Silver: "For example, we've been working on XML integration pretty fiercely, meanwhile, we're [sic] don't have support in this initial version for DLinq." You can, however, emulate some VB DLinq features by executing simple LINQ Select ... From ... In ... Where ... Order By queries against in-memory instances of hand-coded entity classes that you populate with the ADO.NET 2.0 SqlDataReader.GetObject(objRow) method. --rj Technorati: LINQ Update: Here's a lightly edited question and answer regarding future C# 3.0 LINQ compatibility with the VS 2005 RC and RTM versions from the September 22, 2005 C# 3.0 chat: Q: May we expect a release of LINQ that works with the RC and/or RTM of .NET 2.0 in the near term? A: Yes. We are very actively working on it right now and would like to get it out as soon as we can.

Michael Rys' Personal Take on XLinq and XQuery

Michael Rys has posted his view of the relationship between XLinq's XPath-based query language and XQuery here. He's also posted links to 11 new XQuery Working Drafts dated September 15, 2005. Nine of the 11 documents are intended to evoke response to the Last Call comments. Candidate Recommendations might arrive the the near future (at last!) --rj Techorati: LINQ XQuery

Erik Meijer Selects VB as His "Language of Choice"

Erik Meijer recently posted a "Visual Basic and LINQ" guest entry to the Lambda the Ultimate programming-languages Weblog in which he stated that Visual Basic "has become his programming language of choice." This is no faint praise from a "language wonk" who's one of the major contributors to the LINQ Project and the Haskell99 functional programming language. Here's Erik's brief description of what VB 9.0 will offer to LINQ programmers that isn't likely to be available with C# 3.0:

Visual Basic adds some further enhancements to leverage the fact that it allows static typing where possible and dynamic typing where necessary in the form of relaxed delegates, improved nullable support, dynamic identifiers (makes writing meta-circular interpreters a breeze) and last but not least dynamic interfaces, or as I like to refer to them [as] strong duck typing (compare to simplified qualified types/type classes).

These enhancements enable VB 9.0 to support "XML Literals" and "Late-Bound XML." Watch for code examples in future OakLeaf blog items. Dare Obasanjo also weighs in on XLinq and Visual Basic 9 with a quote from the Microsoft XML Team's Mike Champion about Erik's contributions to X#, Xen, and COmega (Cω). On the software patent front, Erik is the primary inventor of two pre-grant publications—both dated August 18, 2005—of U.S. patent applications assigned to Microsoft Corp. that relate to Cω:

• 20050183071 Interaction with nested and non-nested streams
• 20050183058 Fast flattening of nested streams

Cω returns data rows from SQL-style queries as flattened streams, which “represent ordered homogeneous collections of zero or more values.” Streams are related to the .NET Framework 2.0's generic IEnumerable<T> interface. Each row of the stream is an anonymous struct (a tuple), which is defined by the Cω enhancements to C# 1.1. If a row is hierarchical (i.e., contains other rows), the runtime flattens the stream as its read from the data source, which can be a database or XML document.

--rj

Technorati: LINQ COmega

Update 9/19/2005: According to Microsoft's Matt Warren, Erik won Universal Studio’s Fear Factor—Live! award, which is no small achievement.

Update 9/28/2005: Erik presented "Why VB is my Favorite Language" at the JAOO 2005 Aarhus (Denmark) conferrence on September 27. Here's the abstract:

Some people have compared type systems with a dial; however in most languages that dial is either completely off (dynamically typed languages) or completely on (statically typed languages). Visual Basic is unique in that it allows the dial to cover the whole spectrum; it supports static typing where possible and allows dynamic typing where necessary. In this talk we will show some of the present and future support of mixed static and dynamic typing in Visual Basic.

--rj

Bill Gates Podcast on the LINQ Project

Infoworld's Jon Udell recorded a half-hour podcast interview with Bill Gates on September 13, 2005. The interview includes a serveral-minute discussion on the LINQ Project that starts at 3:33. You can read the transcript of the interview here. --rj

Technorati: LINQ

Erik Meijer on LINQ as a 6GL Language Feature

Erik Meijer is one of the primary contributors to Microsoft Research's Cω (C-Omega) language and the forthcoming C# 3.0 and VB 9.0 versions for .NET 3.0 in "Orcas". He's a co-author of several papers about extensions to a general-purpose programming language (C#) that reduce the "impedance mismatch" of relational-to-object and XML-to-object mapping and traditional query languates, such as SQL, XSLT, and XQuery. Here are links to four Microsoft Research papers that will interest developers of data-intensive .NET applications:

• "Unifying Objects, Tables and Documents” by Erik Meijer, Wolfram Schulte and Gavin Bierman
• "Programming with Rectangles, Triangles, and Circles" by Erik Meijer, Wolfram Schulte and Gavin Bierman presented at the XML 2003 Conference
• "The Essence of Data Access in Cω" by Erik Meijer, Wolfram Schulte and Gavin Bierman (rejected by the ACM's Symposium on Priniciples of Programming Languages - POPL 2005)
• "XML Types for C#" by Erik Meijer and Wolfram Schulte (the original Bill Gates ThinkWeek presentation on what ultimately became Cω.)

You can also watch (and listen to) Erik's entertaining 1H:22M-long "Data Integration in Programming Languages" Producer for PowerPoint 2002 presentation and check out Erik's c.v. on his Microsoft Research home page. Erik will present three papers that relate to Cω, C# and VB topics at the September 2005 JAOO Conference in Aarhus, Denmark. JAOO claims to be "the Premier European Conference on Software Engineering, Methods and Best Practices." Here's Erik's latest bio from the conference's site:

Erik Meijer is an architect in the Microsoft SQL [S]erver division where he currently works together with the Microsoft Visual C# and the Microsoft Visual Basic language design teams on data integration in programming languages. Prior to joining Microsoft he was an associate professor at Utrecht University and adjunct professor at the Oregon Graduate Institute. Erik is one of the designers of the standard functional programming language Haskell98 and more recently the C-Omega language.

Following is the abstract of Erik's "6GL features languages to come" JAOO 2005 presentation. The JAOO 2005 folks' definition of 6th Generation Languages is: "6GLs seek to go beyond scripting to full blown application languages which are easier to use and more productive than Java/C# languages or generators."

Moving forward, the dominant costs for IT projects are people costs more than hardware costs. Meeting business needs revolves around improving developer productivity. This provides great opportunities for language designers and compiler writers to concentrate on providing more powerful high-level programming abstractions rather than than on low-level optimizations. This desire for more abstraction has lead to a plethora of new domain specific programming languages for manipulating data including XML with XSLT and XQuery, objects with OQL, and relational data with SQL. We argue that instead of creating such specialized languages, general purpose languages such as C# and Visual Basic should be extended with query capabilities. This allows programmers to query any form of data, using a broad notion of "collection".

We will discuss various advancements in Visual Basic towards simplifying the development of data intensive applications. In particular we will concentrate on ways to bridge the impedance mismatches between objects, relation data, and XML and the importance of dynamism in this trend. [Emphasis added.]

It's especially interesting that Erik will emphasize VB extensions in this presentation, because all his preceding papers have been devoted to the C# extensions added by Cω. However, VB certainly qualifies as a "full blown application [language] which [is] easier to use and more productive than Java/C# languages or generators." [Emphasis added.]

It surprises me that Anders Hejlsberg receives all—or at least the majority—of the credit for LINQ development. As Wesner Moise notes in his September 14, 2005 LINQ post,"Visual Basic seems to have introduced more extensive changes to the language than C#. Many of the COmega-inspired features seem to have migrated into Visual Basic instead of C#; one wonders whether this was due in no small part to Erik Meijer’s close relationship to the VB team." --rj

P.S.: Channel9 has a 12-minute video of Gavin Bierman, Erik's co-author on several papers, discussing data query languages in Cω. Gavin is a member of the Programming Principals and Tools Group of Microsoft Research in Cambridge, U.K. Gavin has prepared a MSDN Webcast about Cω that also describes other research projects going on at MSR Cambridge.

--rj

Technorati: LINQ COmega

WinFS and the Language Integrated Query (LINQ) Project

Microsoft's pre-PDC release to MSDN subscribers of WinFS Beta 1 bits was a surprise to the trade press, as well as most .NET developers. InfoWorld's "Surprise! Microsoft's WinFS beta arrives" article by Eric Knorr appeared on August 31, 2005 and briefly describes the first WinFS tools for developers. Paul Thurrott's "Windows Storage Foundation (WinFS) Preview" provides a historical context for WinFS—begining with the Object File System (OFS, codenamed Cairo) that Microsoft once promised for Windows NT. Quentin Clark, Microsoft's director of program management for WinFS, briefly describes reasons for early release of the WinFS Beta and the relationship between WinFS and IQF in his initial post—Shipping WinFS Beta 1—on the WinFS team's blog. Here's the early version of the WinFS/LINQ story from Quentin's post:

Another piece of feedback was concerns over the WinFS api’s being a different data access pattern than our existing managed code data acess APIs. Further, that our APIs were not aligned with broader data platform needs like OR mapping. This was a big one, and as you explore the SDK and the APIs, you will see the beginnings of how we will be addressing this. We are in the process of building-out the next version of ADO.NET to have new features that provide a data model, object-relational mapping, and flexible query infrastructure. The new data model is about entities, and the WinFS data model of Item types is built on that model. Looking through our SDK and code samples you will see how Items are composed of underlying entities. OR mapping is a big requirement - WinFS is a very prescribed mapping (defining a type in WinFS generates both the underlying storage schema and the partial class to program to that type). But the real-world has lots of requirements for flexible mapping – to existing data, to existing objects, etc. On query, many of you have heard about Anders Hejlsberg’s work on Language Integrated Query – and the new ADO.NET functionality will plug directly underneath so that you can use the new query patterns on any entity data, including of course now WinFS Items.

Thus, it's likely that you'll be able to integrate WinFS and LINQ code in the .NET projects you create with Visual Studio "Orcas." PDC 2005 has four WinFS breakout sessions. Here's the description of DAT312—"WinFS" and ADO.NET: Future Directions for Data Access Scenarios (Samuel Druker)—that contains references to ObjectSpaces and "integrated query functionality:"

The next generation of ADO.NET incorporates innovations that integrate Entities, Relationships, Associations, a seamless CLR object programming surface, and flexible storage mapping for programming to data at all tiers. These technologies address the needs for "WinFS" data modeling and programming scenarios on the desktop and server, as well as the requirements from the ObjectSpaces project. This session discusses how ADO.NET entities will form the core for data integration, enabling the interoperability of applications and modules by providing a high-level framework for programming with data that includes integrated query functionality.

Following are links to the other three WinFS-specific sessions—DAT200 Future Directions for Data-Driven Applications: Storage, Applications, API’s (David Campbell); DAT209 "WinFS" Future Directions: An Overview (Shishir Mehrotra); and DAT310 "WinFS" Future Directions: Building Data Centric Applications Using Windows Presentation Foundation ("Avalon") and Windows Forms (Ramesh Nagarajan). Robert Scoble's one-hour Channel 9 video with members of the WinFS team includes a demonstration of some WinFS features and applications. The demo starts at about 11:00:00 into the program. As to the ultimate release of WinFS, Paul Thurrott bases the following timeline on "internal Microsoft documentation:"

WinFS Beta 1 will be followed by at least one Community Technology Preview (CTP) release, which is currently due on February 15, 2006. Then, on May 1, 2006, Microsoft is scheduled to release WinFS Beta 2. Beta 3 is currently scheduled for November 15, 2006, with a Beta 3 Refresh release expected in April 2007. WinFS is currently scheduled for RTM in Q3 2007, well after Longhorn Server is released.

--rj Technorati: LINQ, COmega, ADO.NET vNext, WinFS

The Language Integrated Query (LINQ) Project

Two features once promised for Visual Studio 2005 are conspicuous by their absence in VS 2005 Beta 2 Community Technical Preview drops:

• An object-relational mapping API and toolkit for ADO.NET 2.0 called ObjectSpaces
• An XQuery 1.0 implementation for the .NET Framework 2.0

Searching descriptions of PDC 2005 sessions for the keyword "query" indicates that the forthcoming Languate Integrated Query (LINQ) for .NET Project (nee Integrated Query Framework (IQF)) will resurrect in the VS Orcas release at least some of the promised features of the original .NET 2.0 implementations of ObjectSpaces and the XQuery 1.0 processor. (Copies of session descriptions are at the end of this post.) Recent MSDN articles and Microsoft patent applications add evidence of major changes in store for VB 9.0 and C# 3.0 in the next VS release.

Note: Technorati category links in the preceding text are experimental

ObjectSpaces Support in VS 2005 Beta 1 Orca (apparently an acronym for Object-Relational Component Architecture) was a technical preview of ObjectSpaces that first appeared in a PDC 2001 ADO.NET presentation and was promised as a Web release for VS .NET 1.0. ObjectSpaces introduced OPath, a proprietary object query language. I spent a substantial amount of time testing the PDC's technical preview bits. I don't believe there were subsequent Web release(s). Luca Bolognese's "DAT410 NET Framework: Developing Applications Using ADO.Net ObjectSpaces" presentation at PDC 2003 set the stage for ObjectSpaces' inclusion in the VS Whidbey release. Luca demonstrated a VS add-in tool to simplify the O-R mapping process, which required a set of three XML configuration (schema description) files for the object (Osd.xml), relational (Rsd.xml), and mapping (Msd.xml) layers.

Luca promised future support for generics with syntax similar to: Dim osQuery As New _ ObjectQuery(Of Customer)("Region='WA'","Orders") Dim osReader As ObjectReader(Of Customer) = _ ObjectSpace.GetObjectReader(osQuery). The final bulletpoint on Luca's "Conclusions" slide was: "It is included in the next version of Visual Studio." Peter Provost blogged the presentation and Paul Wilson's "ADO.NET v2.0: ObjectSpaces Delivers an O/R Mapper" article has sample code from the PDC alpha version. Dino Esposito's February 2004 MSDN article, "A First Look at ObjectSpaces in Visual Studio 2005," provides additional architectural details and sample code. The Whidbey alpha included a System.Data.ObjectSpaces namespace. Luca reappeared at Tech*Ed 2004 with the "DAT 301: ObjectSpaces" presentation, which included more advanced examples of OPath query syntax. Microsoft originally planned to use XPath as the query language but found that virgules (/) turned off developers, who preferred periods (.) as separators. Thus typical OPath query expressions against Northwind-based Orders and Details objects might have been: Orders.Details.Quantity > 50

Orders[ShippedDate > RequiredDate]

Orders[Freight > 1000].Details[Quantity > 30]

At the end of the presentation, Luca announced that ObjectSpaces had acquired a dependency on WinFS and thus was postponed to the Longhorn/Orcas timeframe. (He appeared close to tears as he made the announcement.) The dependence of the long-delayed Microsoft Business Framework (MBF) API on ObjectSpaces as its object persistence platform probably was a contributing factor to the demise of ObjectSpaces in VS 2005. Subsequently, Microsoft announced that WinFS had been excised from the planned Longhorn release version, now renamed Vista.

Here are three published Microsoft patent applications that include Luca Bolognese as an inventor and relate to Object Spaces:

• Mapping architecture for arbitrary data models (20050050068)
• System and method for presenting a query expressed in terms of an object model (20040078359)
• System and method providing API interface between XML and SQL while interacting with a managed object environment (20030236859)

Matt Warren's "ObjectSpaces:There and Back Again" and Dinesh Kulkarni's "ObjectSpaces -> DLinq" blog entries throw additional light on ObjectSpace's transition from mapping files to LINQ and DLinq.

XQuery 1.0 Support in .NET Framework 2.0 Beta 1

About three years ago, Microsoft sponsored an XQuery demonstration site that included an XQuery engine (Microsoft.Xml.XQuery) for .NET 1.0. This early XQuery implementation didn't support all XQuery keywords, constructs, or use cases, but it provided a simple route for .NET developers to become familiar with XQuery 1.0 and XPath 2.0 syntax.

Beta 1 of the .NET Framework 2.0 included a System.Xml.Query namespace, which provided an XQueryCommand class (the XQuery processor) to implement a subset of XQuery 1.0 for XML documents. Mark Fussell's March 2004 MSDN article, "What's New in System.Xml for Visual Studio 2005 and the .NET Framework 2.0 Release," has a section named "The XQuery Language," which includes a simple C# example of an XQuery FLWOR (for, let, where, order by, and return) expression:

using (XmlWriter writer = XmlWriter.Create("output.xml ")) { XQueryCommand xq = new XQueryCommand(); string query = "" + "{ for $s in /bookstore/book " + "where $s/@genre='autobiography' " + "return $s/title }" + ""; xq.Compile(query); xq.Execute("books.xml", new XmlUrlResolver(), writer); }

Notice that the XQuery processor compiles the FLWOR query and then executes the compiled version against the books.xml file to write the output.xml document. Output.xml contains the title subelements of book elements whose genre attribute value is autobiography. (After XQuery appeared in the Whidbey bits, Microsoft turned off the XQuery demo site.)

Mark's article also included a section titled "Universal Type Support and Conversion," which described changes to the the XmlReader, XmlWriter, and XPathNavigator classes required to support the XQuery 1.0 and XPath 2.0 Data Model with XML schemas and CLR types. Dare Obasanjo's January 2005 MSDN article, "An Overview of Cω: Integrating XML into Popular Programming Languages," describes constructs in Microsoft Research's Cω (C-Omega) programming language that "...bridge the gap between Relational, Object, and XML data access by creating a type system that is a combination of all three data models." Although Dare's article discusses Cω features in a C# context, it's reasonable to expect that VB in Orcas will gain similar type system additions and programming constructs.

Whidbey Beta 2 removed the XQueryCommand class for the reasons cited in Microsoft's brief "Status of XQuery in the .NET Framework 2.0" white paper. The Microsoft XML Team's Mike Champion and Arpan Desai weigh in on what appears to have been a traumatic decision. Michael Rys, program manager for SQL Server’s XML Technologies, explains why SQL Server's xml data type uses XQuery 1.0 despite the XML team's arguments against it's inclusion in the .NET Framework 2.0.

The XML Team's white paper suggests that VS 2005 "... customers continue to use XSLT and XPath to address their XML filtering and transformation scenarios." The System.Xml.Xsl.XslCompiledTransform object represents an XSLT processor that supports XSLT 1.0 syntax. Following is C# sample code from VS 2005's online help for XslCompiledTransform:

// Create the XslCompiledTransform object. XslCompiledTransform xslt = new XslCompiledTransform(); // Compile the style sheet. xslt.Load("output.xsl"); // Execute the transform and output the results to a file. xslt.Transform("books.xml", "books.html");

The XslCompiledTransform.Load method, which compiles the specified stylesheet, corresponds to the XQueryCommand.Compile method and the XslCompiledTransform.Transform method approximates the XQueryCommand.Execute method.

References to LINQ and XLinq Predecessors

Arpan Desai, program manager for XSLT and XQuery in Whidbey, says "...we came up with a new query architecture to support both XSLT and XQuery." Michael Brundage, author of XQuery: The XML Query Language, is an inventor of the Common Query Runtime (CQR). CQR "...features compilers for each XML query language (such as XSLT and XQuery), a common intermediate format (QIL [Query Intermediate Language]), and multiple backend execution engines (over XML and SQL).

Following are Microsoft's U.S. and international patent applications related to IQF (from Michael Brundage, all filed in 2003):

• 304064.1 Common Query Runtime System and API (20040260691)
• 303845.1 Query Optimizer System and Method (20050004892)
• 301638.1 Query Intermediate Language Method and System (20040267760)

Numbers in parenthesis are U.S. Patent Office published application document numbers with links to the applications' full text and drawings. The Patent Office published the applications in 2004 and 2005. The patent applications reference XQuery and OPath as XML query languages and provide examples of relational queries using the Northwind database's tables.

The ".NET Language" prefix indicates to me (and others, such as Wesner Moise) that implementations will be specific to VB or C#. As an example, VB and C# appear to share anonymous types, but only Paul Vick's PDC 2005 session mentions query comprehensions and object initializers.

Soumitra Sengupta's "Changing the Game" entry to the XML Team's Weblog provides additional insight on the effect of LINQ and XLinq on the future of Microsoft's XML technologies.

PDC 2005 Sessions on LINQ and Related Topics

Links to individual sessions at http://commnet.microsoftpdc.com/content/sessions.aspx don't appear to be possible, so following are copied descriptions of LINQ-related sessions at PDC 2005. InfoWorld published on August 25, 2005 an article by Paul Krill that refers to the LINQ sessions. A similar article by Darryl K. Taft in the August 26 issue of eWeek.com describes a purported FoxPro-LINQ connection.

The .NET Language Integrated Query Framework: An Overview Speaker(s): Anders Hejlsberg Modern applications operate on data in several different forms: Relational tables, XML documents, and in-memory objects. Each of these domains can have profound differences in semantics, data types, and capabilities, and much of the complexity in today's applications is the result of these mismatches. The future "Orcas" release of Visual Studio aims to unify the programming models through integrated query capabilities in C# and Visual Basic, a strongly typed data access framework, and an innovative API for manipulating and querying XML. This session introduces each of these areas and walks through how they are related.

Using the .NET Language Integrated Query Framework with Relational Data Speaker(s): Luca Bolognese Database-centric applications have traditionally had to rely on two distinct programming languages: one for the database and one for the application. This session introduces future advances Microsoft is making for the "Orcas" release of Visual Studio in programming languages and frameworks to help integrate relational data and queries with C# and Visual Basic. These advances enable developers to express queries and updates in terms of their local programming language without sacrificing the server-side execution model of today's high-performance SQL-based approaches. Using these advances, database queries that previously were stored as opaque strings now benefit from static type checking, CLR metadata, design-time type inference, and of course IntelliSense.

Using the .NET Language Integrated Query Framework with XML Data Speaker(s): Dave Remy One of the key challenges to working with XML data has been the impedance mismatch between XML and programming languages. This session introduces future advances Microsoft is making for the "Orcas" release of Visual Studio in programming languages and frameworks to help integrate XML and queries with C# and Visual Basic. The advances include a framework for navigating, querying, and transforming XML that is both easier to use and more efficient than current XML programming techniques. This framework marries the capabilities of XPath, XQuery, and the DOM with the language integrated query framework planned for C# and Visual Basic.

Visual Basic: Future Directions in Language Innovation Speaker(s): Amanda Silver, Paul Vick Visual Basic 9.0 will offer radical improvements in its ability to work with data in all its forms: as objects, as XML, as relational data. Join the language architects for a detailed discussion of features such as query comprehensions, object initializers and anonymous types that enable querying data in a more flexible, natural way than ever before. Also, get a glimpse into the future of dynamic programming in VB with coverage of new features intended to radically simplify working with dynamically typed data on the .NET platform.

C#: Future Directions in Language Innovation from Anders Hejlsberg Speaker(s): Anders Hejlsberg Join Anders Hejlsberg, Distinguished Engineer and chief architect of the C# language, for an in-depth walkthrough of the new language features in C# 3.0. Understand how features like extension methods, lambda expressions, type inference, and anonymous types make it possible to create powerful APIs for expressing queries and interacting with objects, XML, and databases in a strongly typed, natural way. It is suggested that you attend "The .NET Language Integrated Query Framework: An Overview" before attending this session.

--rj

Technorati tags: LINQ, ADO.NET vNext, ObjectSpaces

The Yukon Quintet (with Apologies to Lawrence Durrell and Robert Service)

Fawcette Technical Publications (FTPOnline) published on August 26, 2005 the last of my five recent Visual Studio Magazine columns about new features in SQL Server 2005, the database formerly known as Yukon. Following are links and brief descriptions of (decks for) the columns in ascending date sequence:

• Exploit Yukon's XML Data Type. Take advantage of SQL Server 2005’s new native XML data type to add XML columns to a table, populate and index the columns, and understand basic XQuery syntax.
• Migrate to SQL Server 2005 Express Edition. Take advantage of new SQL Server 2005 features, GUI administration, and XCopy or ClickOnce deployment with SQL Express and Express Manager.
• Encrypt and Decrypt Data in Yukon. SQL Server 2005's engine-based, data-encryption functions protect confidential information from unauthorized access and disclosure, and offer native key management as a bonus.
• Take Advantage of New T-SQL Features. New T-SQL keywords implement TRY...CATCH error handling, pivot aggregate values to create crosstab tables, rank and partition rowsets, generate temporary in-memory tables with common table expressions, and more.
• Create Interoperable Native Web Services. Yukon lets you take advantage of the new kernel-mode Http.sys listener and eliminates the need for IIS to process ASMX files or SQLXML 3.0 templates that implement Web services.

Each article has a downloadable Visual Studio 2005 Visual Basic project that demonstrates the features described in the article. Click the Get The Code link in the Article Tools group on the first page of each article to download the project's ZIP file. Enjoy. --rj