Comparing C# 3.0 and VB 9.0 LINQ Syntax

Prior to LINQ's introduction, differences between the C# and VB (.NET) languages were almost exclusively syntatical. The .NET Framework's classes and the CLR's MSIL unified the capabilities of programs written with C# and VB. If the release version of Visual Studio "Orcas" includes LINQ and its DLinq and XLinq APIs, much of the everyday data-management and XML-processing syntax of C# 3.0 and VB 9.0 source code will diverge dramatically. .NET developers will need to evaluate the language's query operator set and query effectiveness before deciding between C# 3.0 or VB 9.0 for writing data-intensive .NET 3.0 applications or components.

.NET developers usually base their choice of language on their programming background. As a general rule C/C++/Java programmers, who are likely to have learned these languages while attending a college or university, choose C#. VB (COM) developers, most of whom learned the language by reading books and attending conferences or training classes, adopt VB .NET. The VS team has identified the following three Windows programmer personae:

• Mort, the opportunistic developer, likes to create quick-working solutions for immediate problems, focuses on productivity, and learns new language skills as needed. Morts commonly create ad hoc, line-of-business applications and components with VB.
• Elvis, the pragmatic programmer, likes to create long-lasting solutions that address the problem domain, and learns while working on the solution. Elvises commonly use C# to develop reusable components and enjoy implementing advanced .NET features, such as remoting and the Windows Communication Framework (WCF or Indigo).
• Einstein, the paranoid programmer, likes to create the most efficient solution to a given problem, and typically learn in advance before working on the solution. Einsteins usually have a C/C++ background and a master's degree or higher, and some write Windows device drivers.

The preceding descriptions were adapted from Nikhil Kothari's blog entry. Wesner Moise offers similar descriptions for the three personae, and Rockford Lhotka provides a detailed analysis of Mort as a business programmer.

Microsoft's VB and C# teams diverge on LINQ syntax. VB 9.0 query comprehensions support SQL-like operators, while C# keywords have a decided XQuery FLWOR (for, let, where, order by, return) flavor. (Wes's "LINQ Syntax" item offers a sample FLWOR expression and its C# query counterpart.)

Compare this preliminary list of VB 9.0 query keywords from Paul Vick's PDC 2005 presentation:

• Project: Select <expr> From <expr> In <expr>
• Filter: Where <expr>
• Test: Any(<expr>), All(<expr>)
• Join: <expr> [[Left Right] Outer] Join <expr> On <expr>
• Group: Group By <expr>
• Aggregate: Count(<expr>), Sum(<expr>), Min(<expr>), Max(<expr>), Avg(<expr>)
• Partition: Top(<expr>);
• Set: Distinct, Union, Intersect, Except
• Order: Order By <expr> [[Asc] Desc]

with this much longer list of C# 3.0 query operators from the LINQ Project overview document, which is missing the From and In operators:

• OfType: Filter based on type affiliation
• Select/SelectMany: Project based on transform function
• Where: Filter based on predicate function
• Count: Count based on optional predicate function
• All/Any: Universal/Existential quantification based on predicate function
• First/FirstOrDefault: Access initial member based on optional predicate function
• ElementAt: Access member at specified position
• Take/Skip: Access members before/after specified position (paging)
• TakeWhile/SkipUntil: Access members before/after predicate function is satisfied
• GroupBy: Partition based on key extraction function
• ToDictionary: Create key/value dictionary based on key extraction function
• OrderBy/ThenBy: Sort in ascending order based on key extraction function and optional comparison function
• OrderByDescending/ThenByDescending: Sort in descending order based on key extraction function and optional comparison function
• Reverse: Reverse the order of a sequence
• Fold: Aggregate value over multiple values based on aggregation function
• Min/Max/Sum/Average: Numeric aggregation functions
• Distinct: Filter duplicate members
• Except: Filter elements that are members of specified set
• Intersect: Filter elements that are not members of specified set
• Union: Combine distinct members from two sets
• Concat: Concatenate the values of two sequences
• ToArray/ToList: Buffer results of query in array or List
• Range: Create a sequence of numbers in a range
• Repeat: Create a sequence of multiple copies of a given value

In addition to the query comprehension operator (keyword) differences between VB and C#, C# reverses the sequence of SQL's SELECT and FROM clauses. The following VB expression:

expr = Select s.ToUpper() From s In names Where s.Length = 5 Order By s

translates to this C# expression:

expr = from s in names where s.Length == 5 orderby s select s.ToUpper() ;

Paul Vick addresses the query comprehension operator sequence issue from the VB Team's standpoint—following the SQL convention simplifies usage because VB programmers (Morts) are likely to be conversant with SQL. However, substitution of In for As departs from SQL convention. I would have no problem with either

expr = Select s.ToUpper() From names As s Where s.Length = 5 Order By s

or

expr = From s In names Where s.Length = 5 Order By s Select s.ToUpper()

but I'd prefer the latter construction if it improved performance, reduced resource consumption, or is the only sequence that would support IntelliSense fully. Implementing IntelliSense for the previous example's Select list appears to me to be impossible in advance of defining the data source with the From ... In clause.

Wesner Moise makes the following observations in his VB 9 item of September 16, 2005:

The difference in focus between the C# and VB team is becoming much more evident: The VB team is opting for greater relevance and convenience in the short term, while the C# team is trying to avoid creating long-term baggage. The end result is that Visual Basic is definitely going to be much more fun to program. [Emphasis added.]

While the C# [team] has decided to include a few choice SQL keywords (select, from, where, orderby and groupby) and no XML literals, the VB team has opted to include the full range of keywords available in a SQL query as well as support for embedded XML with schema validation.

Paul Vick confirms Wes's conclusion from the XLinq perspective in his "Integrating XML: Playing with Fire?" item. However, DLinq users undoubtedly will want to see Having, the In() predicate and other commonly-used ANSI SQL reserved words added to VB's query comprehension operators list.

The upshot: VB 9.0 will be firmly in the Mort camp and C# 3.0 will continue to appeal primarily to the Elvis and Einstein crowd. Whether the two sets of query comprehension operators will have identical capabilities with both the DLinq and XLinq APIs remains to be seen.

--rj

Update: 10/11/2005: Interact Software, Ltd's Ian Griffiths explains the significance of and demonstrates C# 3.0 expression trees in his "C# 3.0 and LINQ - Expression Trees" post.

Technorati: LINQ XLinq VB 9.0 C# 3.0

Amanda Silver, Paul Vick, and Erik Meijer VB 2005 Interview

Carl Franklin interviews the VBTeam's Amanda Silver and Paul Vick, plus the SQL Server group's Erik Meijer (a.k.a., VBTV's "Head in a Box") for the second half of the September 26, 2005 .NET Rocks show. The interview doesn't deal with LINQ in depth—consider it a sales pitch for VB 2005 with a low-level introduction to forthcoming LINQ features. The interview contains only about 10 minutes of LINQ-related conversation. Following is a log for the VB-related topics of the full WMV file: 00:24:27 - New VB 2005 language features 00:32:20 - LINQ introduction by Amanada and Paul 00:39:20 - Commercial break 00:41:45 - More LINQ and DLinq features 00:44:45 - Erik Meijer (on XLinq) 00:49:20 - More VB 2005 language features 01:04:11 - New features deferred to VB 9.0 from VB 8.0 (400 to 500 in the queue) 01:09:00 - Bill Gates' influence on VB 2005 and successors 01:11:00 - VB generics 01:14:40 - VB Express, programmer education, and hobbyist programmers 01:17:25 - Links to resources 01:20:00 - Coaching Anders Hejlsberg for a VB demonstration 01:24:00 - End Carl recorded the interview on September 14, 2005 at PDC 2005. --rj Technorati: LINQ DLinq XLinq

XLinq Presentation at the XML 2005 Conference

Erik Meijer will present "XLinq: XML Programming Refactored (The Return of the Monoids)" at Atlanta's XML 2005 Conference on Wednesday, November 16, 2005. Note that the conference program lists the presentation title as "XMP [sic] Programming Refactored (The Return of the Monoids)" because the XLinq name was under NDA until PDC 2005; "XMP" is an obvious typo. The paper, by Erik and Brian Beckman*, provides a concise description of the objectives and implementation of the XLinq API for LINQ. XLinq replaces the document-oriented XML DOM with an element-oriented API that enabled constructing elements and attributes independently of a containing document. The paper is unique because all but one of its examples use VB 9.0 rather than C# 3.0. VB 9.0 is required to implement XML literals and axis members. XML literals enable definining elements and attribute with VB 9.0 code similar to the following:

Dim BillTo =
<billTo country="US" >
  <name>Robert Smith</name>
  <street>8 Oak Avenue</street>
  <city>Old Town</city>
  <state>PA</state>
  <zip>95819</zip>
</billTo>

Notice the lack of VB's line-continuation characters, which aren't required or permitted in XML-literal statements. Whitespace formatting is preserved and XLinq overloads the ToString method to simplify display of formatted documents. (Oleg Tkachenko has some reservations about functional construction of XML trees and VB literal XML.) Axis members substitute dot syntax for XPath axis specifiers. For example, BillTo.street returns all street child elements from the BillTo XElement object. Read Erik's paper to learn more about VB 9.0's XLinq implementation. --rj Technorati: LINQ XLinq * Brian Beckman was one of the original developers of Microsoft's XLANG—a notation for the specification of message exchange behavior among participating web services—and is the author of the "Physics of Racing" series.

Jon Udell: Microsoft driving toward .Net unity

The InfoWorld Test Center's lead analyst, columnist, and blogger-in-chief says: "LINQ is an extraordinary innovation that turns traditional query inside out. But the bigger story from PDC 2005 is that the .Net vision of unifying many balkanized disciplines within the Microsoft ecosystem is finally becoming a reality." Read Jon's September 21, 2005 "Strategic Developer" column. Not to be outdone by his colleague, Tom Yeager—InfoWorld Test Center's chief technologist—devotes this week's "Ahead of the Curve" column to LINQ. --rj Technorati: LINQ

LINQ Interview with Anders Hejlsberg and Paul Vick

InfoWorld columnists Jon Udell and Tom Yeager interviewed Anders Hejlsberg and Paul Vick at PDC 2005. Here's a link to the 27:41 Podcast, "A conversation with with Anders Hejlsberg and Paul Vick about LINQ," which also includes transcripts of selected comments by Anders. Jon's item also includes a link to "The LINQ Project: .NET Language Integrated Query" MSDN white paper by Anders Hejlsberg and Don Box. The white paper emphasizes C# 3.0, but includes a few VB 9.0 examples. Paul Vick at 13:40 says, "One of the great advantages [of VB 9.0 is that] you learn one language. It can be C# or Visual Basic, but you learn one language; you don't have to go out and learn SQL. A lot of our developers do know SQL because they've been forced to, but now you don't actually have to do that." Microsoft initially hyped CLR integration with SQL Server 2005 as a means to eliminate Visual Studio developers' need to be proficient in [T-]SQL, as mentioned in my recent "When To Use SQLCLR, And When Not To" article. Substituting a knowledge of VB 9.0 query comprehensions for [T-]SQL proficiency is equally impractical in the real world. No matter what programming language or platform you choose, a working knowledge of SQL is essential for developers of data-intensive applications or components. --rj Technorati: LINQ

Cure Data Type "Impedance Mismatch" With LINQ

This article from Fawcette Technical Publications' .NETInsight newsletter describes how to install and test the C# 2.0 and VB 8.0 implementations of the LINQ Project, DLink and XLink. Although this post from the VBTeam's blog states that the initial PDC 2005 Technology Preview (TP) bits work with the current VS 2005 release candidate, which is available for download from MSDN, I've found the following issues on various VS 2005 versions:

• Neither the VB or C# preview run correctly on the Beta 2 version (build 8.0.50215.44) mentioned as required in the Readme document for the C# version. This build is running here under Windows 2003 Server SP-1.
• The SQLMetal.exe command-line utility to generate C# entity classes for SQL Server databases and DLinq throws an unhandled exception—Invalid object name 'Dbo.customers'—when run under Beta 2 (build 8.0.50215.44).
• The C# preview doesn't add LINQ templates or integrate correctly under the VS 2005 RC IDE after running Install Visual Studio IDE Support.cmd. Attempts to edit any C# LINQ source code result in an instantaneous VS 2005 shutdown and restart. This build is also running here under Windows 2003 Server SP-1.
• The only VS build that appears to me to work correctly with the Technology Preview is Beta 2 build 8.0.50727.7 (a.k.a., the July CTP), which I'm running under Windows XP SP-2.
• The C# SampleQueries project throws 60 compilation errors, regardless of the VS 2005 build under which it runs.

I've also determined that DLinq support for VB projects in the initial Technology Preview is non-existent; installing the VB TP bits doesn't install DLinq.dll to provide the System.Data.DLinq classes. My article quotes the VBTeam's Amanda Silver: "For example, we've been working on XML integration pretty fiercely, meanwhile, we're [sic] don't have support in this initial version for DLinq." You can, however, emulate some VB DLinq features by executing simple LINQ Select ... From ... In ... Where ... Order By queries against in-memory instances of hand-coded entity classes that you populate with the ADO.NET 2.0 SqlDataReader.GetObject(objRow) method. --rj Technorati: LINQ Update: Here's a lightly edited question and answer regarding future C# 3.0 LINQ compatibility with the VS 2005 RC and RTM versions from the September 22, 2005 C# 3.0 chat: Q: May we expect a release of LINQ that works with the RC and/or RTM of .NET 2.0 in the near term? A: Yes. We are very actively working on it right now and would like to get it out as soon as we can.

Michael Rys' Personal Take on XLinq and XQuery

Michael Rys has posted his view of the relationship between XLinq's XPath-based query language and XQuery here. He's also posted links to 11 new XQuery Working Drafts dated September 15, 2005. Nine of the 11 documents are intended to evoke response to the Last Call comments. Candidate Recommendations might arrive the the near future (at last!) --rj Techorati: LINQ XQuery

Erik Meijer Selects VB as His "Language of Choice"

Erik Meijer recently posted a "Visual Basic and LINQ" guest entry to the Lambda the Ultimate programming-languages Weblog in which he stated that Visual Basic "has become his programming language of choice." This is no faint praise from a "language wonk" who's one of the major contributors to the LINQ Project and the Haskell99 functional programming language. Here's Erik's brief description of what VB 9.0 will offer to LINQ programmers that isn't likely to be available with C# 3.0:

Visual Basic adds some further enhancements to leverage the fact that it allows static typing where possible and dynamic typing where necessary in the form of relaxed delegates, improved nullable support, dynamic identifiers (makes writing meta-circular interpreters a breeze) and last but not least dynamic interfaces, or as I like to refer to them [as] strong duck typing (compare to simplified qualified types/type classes).

These enhancements enable VB 9.0 to support "XML Literals" and "Late-Bound XML." Watch for code examples in future OakLeaf blog items. Dare Obasanjo also weighs in on XLinq and Visual Basic 9 with a quote from the Microsoft XML Team's Mike Champion about Erik's contributions to X#, Xen, and COmega (Cω). On the software patent front, Erik is the primary inventor of two pre-grant publications—both dated August 18, 2005—of U.S. patent applications assigned to Microsoft Corp. that relate to Cω:

• 20050183071 Interaction with nested and non-nested streams
• 20050183058 Fast flattening of nested streams

Cω returns data rows from SQL-style queries as flattened streams, which “represent ordered homogeneous collections of zero or more values.” Streams are related to the .NET Framework 2.0's generic IEnumerable<T> interface. Each row of the stream is an anonymous struct (a tuple), which is defined by the Cω enhancements to C# 1.1. If a row is hierarchical (i.e., contains other rows), the runtime flattens the stream as its read from the data source, which can be a database or XML document.

--rj

Technorati: LINQ COmega

Update 9/19/2005: According to Microsoft's Matt Warren, Erik won Universal Studio’s Fear Factor—Live! award, which is no small achievement.

Update 9/28/2005: Erik presented "Why VB is my Favorite Language" at the JAOO 2005 Aarhus (Denmark) conferrence on September 27. Here's the abstract:

Some people have compared type systems with a dial; however in most languages that dial is either completely off (dynamically typed languages) or completely on (statically typed languages). Visual Basic is unique in that it allows the dial to cover the whole spectrum; it supports static typing where possible and allows dynamic typing where necessary. In this talk we will show some of the present and future support of mixed static and dynamic typing in Visual Basic.

--rj

Bill Gates Podcast on the LINQ Project

Infoworld's Jon Udell recorded a half-hour podcast interview with Bill Gates on September 13, 2005. The interview includes a serveral-minute discussion on the LINQ Project that starts at 3:33. You can read the transcript of the interview here. --rj

Technorati: LINQ

Erik Meijer on LINQ as a 6GL Language Feature

Erik Meijer is one of the primary contributors to Microsoft Research's Cω (C-Omega) language and the forthcoming C# 3.0 and VB 9.0 versions for .NET 3.0 in "Orcas". He's a co-author of several papers about extensions to a general-purpose programming language (C#) that reduce the "impedance mismatch" of relational-to-object and XML-to-object mapping and traditional query languates, such as SQL, XSLT, and XQuery. Here are links to four Microsoft Research papers that will interest developers of data-intensive .NET applications:

• "Unifying Objects, Tables and Documents” by Erik Meijer, Wolfram Schulte and Gavin Bierman
• "Programming with Rectangles, Triangles, and Circles" by Erik Meijer, Wolfram Schulte and Gavin Bierman presented at the XML 2003 Conference
• "The Essence of Data Access in Cω" by Erik Meijer, Wolfram Schulte and Gavin Bierman (rejected by the ACM's Symposium on Priniciples of Programming Languages - POPL 2005)
• "XML Types for C#" by Erik Meijer and Wolfram Schulte (the original Bill Gates ThinkWeek presentation on what ultimately became Cω.)

You can also watch (and listen to) Erik's entertaining 1H:22M-long "Data Integration in Programming Languages" Producer for PowerPoint 2002 presentation and check out Erik's c.v. on his Microsoft Research home page. Erik will present three papers that relate to Cω, C# and VB topics at the September 2005 JAOO Conference in Aarhus, Denmark. JAOO claims to be "the Premier European Conference on Software Engineering, Methods and Best Practices." Here's Erik's latest bio from the conference's site:

Erik Meijer is an architect in the Microsoft SQL [S]erver division where he currently works together with the Microsoft Visual C# and the Microsoft Visual Basic language design teams on data integration in programming languages. Prior to joining Microsoft he was an associate professor at Utrecht University and adjunct professor at the Oregon Graduate Institute. Erik is one of the designers of the standard functional programming language Haskell98 and more recently the C-Omega language.

Following is the abstract of Erik's "6GL features languages to come" JAOO 2005 presentation. The JAOO 2005 folks' definition of 6th Generation Languages is: "6GLs seek to go beyond scripting to full blown application languages which are easier to use and more productive than Java/C# languages or generators."

Moving forward, the dominant costs for IT projects are people costs more than hardware costs. Meeting business needs revolves around improving developer productivity. This provides great opportunities for language designers and compiler writers to concentrate on providing more powerful high-level programming abstractions rather than than on low-level optimizations. This desire for more abstraction has lead to a plethora of new domain specific programming languages for manipulating data including XML with XSLT and XQuery, objects with OQL, and relational data with SQL. We argue that instead of creating such specialized languages, general purpose languages such as C# and Visual Basic should be extended with query capabilities. This allows programmers to query any form of data, using a broad notion of "collection".

We will discuss various advancements in Visual Basic towards simplifying the development of data intensive applications. In particular we will concentrate on ways to bridge the impedance mismatches between objects, relation data, and XML and the importance of dynamism in this trend. [Emphasis added.]

It's especially interesting that Erik will emphasize VB extensions in this presentation, because all his preceding papers have been devoted to the C# extensions added by Cω. However, VB certainly qualifies as a "full blown application [language] which [is] easier to use and more productive than Java/C# languages or generators." [Emphasis added.]

It surprises me that Anders Hejlsberg receives all—or at least the majority—of the credit for LINQ development. As Wesner Moise notes in his September 14, 2005 LINQ post,"Visual Basic seems to have introduced more extensive changes to the language than C#. Many of the COmega-inspired features seem to have migrated into Visual Basic instead of C#; one wonders whether this was due in no small part to Erik Meijer’s close relationship to the VB team." --rj

P.S.: Channel9 has a 12-minute video of Gavin Bierman, Erik's co-author on several papers, discussing data query languages in Cω. Gavin is a member of the Programming Principals and Tools Group of Microsoft Research in Cambridge, U.K. Gavin has prepared a MSDN Webcast about Cω that also describes other research projects going on at MSR Cambridge.

--rj

Technorati: LINQ COmega

WinFS and the Language Integrated Query (LINQ) Project

Microsoft's pre-PDC release to MSDN subscribers of WinFS Beta 1 bits was a surprise to the trade press, as well as most .NET developers. InfoWorld's "Surprise! Microsoft's WinFS beta arrives" article by Eric Knorr appeared on August 31, 2005 and briefly describes the first WinFS tools for developers. Paul Thurrott's "Windows Storage Foundation (WinFS) Preview" provides a historical context for WinFS—begining with the Object File System (OFS, codenamed Cairo) that Microsoft once promised for Windows NT. Quentin Clark, Microsoft's director of program management for WinFS, briefly describes reasons for early release of the WinFS Beta and the relationship between WinFS and IQF in his initial post—Shipping WinFS Beta 1—on the WinFS team's blog. Here's the early version of the WinFS/LINQ story from Quentin's post:

Another piece of feedback was concerns over the WinFS api’s being a different data access pattern than our existing managed code data acess APIs. Further, that our APIs were not aligned with broader data platform needs like OR mapping. This was a big one, and as you explore the SDK and the APIs, you will see the beginnings of how we will be addressing this. We are in the process of building-out the next version of ADO.NET to have new features that provide a data model, object-relational mapping, and flexible query infrastructure. The new data model is about entities, and the WinFS data model of Item types is built on that model. Looking through our SDK and code samples you will see how Items are composed of underlying entities. OR mapping is a big requirement - WinFS is a very prescribed mapping (defining a type in WinFS generates both the underlying storage schema and the partial class to program to that type). But the real-world has lots of requirements for flexible mapping – to existing data, to existing objects, etc. On query, many of you have heard about Anders Hejlsberg’s work on Language Integrated Query – and the new ADO.NET functionality will plug directly underneath so that you can use the new query patterns on any entity data, including of course now WinFS Items.

Thus, it's likely that you'll be able to integrate WinFS and LINQ code in the .NET projects you create with Visual Studio "Orcas." PDC 2005 has four WinFS breakout sessions. Here's the description of DAT312—"WinFS" and ADO.NET: Future Directions for Data Access Scenarios (Samuel Druker)—that contains references to ObjectSpaces and "integrated query functionality:"

The next generation of ADO.NET incorporates innovations that integrate Entities, Relationships, Associations, a seamless CLR object programming surface, and flexible storage mapping for programming to data at all tiers. These technologies address the needs for "WinFS" data modeling and programming scenarios on the desktop and server, as well as the requirements from the ObjectSpaces project. This session discusses how ADO.NET entities will form the core for data integration, enabling the interoperability of applications and modules by providing a high-level framework for programming with data that includes integrated query functionality.

Following are links to the other three WinFS-specific sessions—DAT200 Future Directions for Data-Driven Applications: Storage, Applications, API’s (David Campbell); DAT209 "WinFS" Future Directions: An Overview (Shishir Mehrotra); and DAT310 "WinFS" Future Directions: Building Data Centric Applications Using Windows Presentation Foundation ("Avalon") and Windows Forms (Ramesh Nagarajan). Robert Scoble's one-hour Channel 9 video with members of the WinFS team includes a demonstration of some WinFS features and applications. The demo starts at about 11:00:00 into the program. As to the ultimate release of WinFS, Paul Thurrott bases the following timeline on "internal Microsoft documentation:"

WinFS Beta 1 will be followed by at least one Community Technology Preview (CTP) release, which is currently due on February 15, 2006. Then, on May 1, 2006, Microsoft is scheduled to release WinFS Beta 2. Beta 3 is currently scheduled for November 15, 2006, with a Beta 3 Refresh release expected in April 2007. WinFS is currently scheduled for RTM in Q3 2007, well after Longhorn Server is released.

--rj Technorati: LINQ, COmega, ADO.NET vNext, WinFS

The Language Integrated Query (LINQ) Project

Two features once promised for Visual Studio 2005 are conspicuous by their absence in VS 2005 Beta 2 Community Technical Preview drops:

• An object-relational mapping API and toolkit for ADO.NET 2.0 called ObjectSpaces
• An XQuery 1.0 implementation for the .NET Framework 2.0

Searching descriptions of PDC 2005 sessions for the keyword "query" indicates that the forthcoming Languate Integrated Query (LINQ) for .NET Project (nee Integrated Query Framework (IQF)) will resurrect in the VS Orcas release at least some of the promised features of the original .NET 2.0 implementations of ObjectSpaces and the XQuery 1.0 processor. (Copies of session descriptions are at the end of this post.) Recent MSDN articles and Microsoft patent applications add evidence of major changes in store for VB 9.0 and C# 3.0 in the next VS release.

Note: Technorati category links in the preceding text are experimental

ObjectSpaces Support in VS 2005 Beta 1 Orca (apparently an acronym for Object-Relational Component Architecture) was a technical preview of ObjectSpaces that first appeared in a PDC 2001 ADO.NET presentation and was promised as a Web release for VS .NET 1.0. ObjectSpaces introduced OPath, a proprietary object query language. I spent a substantial amount of time testing the PDC's technical preview bits. I don't believe there were subsequent Web release(s). Luca Bolognese's "DAT410 NET Framework: Developing Applications Using ADO.Net ObjectSpaces" presentation at PDC 2003 set the stage for ObjectSpaces' inclusion in the VS Whidbey release. Luca demonstrated a VS add-in tool to simplify the O-R mapping process, which required a set of three XML configuration (schema description) files for the object (Osd.xml), relational (Rsd.xml), and mapping (Msd.xml) layers.

Luca promised future support for generics with syntax similar to: Dim osQuery As New _ ObjectQuery(Of Customer)("Region='WA'","Orders") Dim osReader As ObjectReader(Of Customer) = _ ObjectSpace.GetObjectReader(osQuery). The final bulletpoint on Luca's "Conclusions" slide was: "It is included in the next version of Visual Studio." Peter Provost blogged the presentation and Paul Wilson's "ADO.NET v2.0: ObjectSpaces Delivers an O/R Mapper" article has sample code from the PDC alpha version. Dino Esposito's February 2004 MSDN article, "A First Look at ObjectSpaces in Visual Studio 2005," provides additional architectural details and sample code. The Whidbey alpha included a System.Data.ObjectSpaces namespace. Luca reappeared at Tech*Ed 2004 with the "DAT 301: ObjectSpaces" presentation, which included more advanced examples of OPath query syntax. Microsoft originally planned to use XPath as the query language but found that virgules (/) turned off developers, who preferred periods (.) as separators. Thus typical OPath query expressions against Northwind-based Orders and Details objects might have been: Orders.Details.Quantity > 50

Orders[ShippedDate > RequiredDate]

Orders[Freight > 1000].Details[Quantity > 30]

At the end of the presentation, Luca announced that ObjectSpaces had acquired a dependency on WinFS and thus was postponed to the Longhorn/Orcas timeframe. (He appeared close to tears as he made the announcement.) The dependence of the long-delayed Microsoft Business Framework (MBF) API on ObjectSpaces as its object persistence platform probably was a contributing factor to the demise of ObjectSpaces in VS 2005. Subsequently, Microsoft announced that WinFS had been excised from the planned Longhorn release version, now renamed Vista.

Here are three published Microsoft patent applications that include Luca Bolognese as an inventor and relate to Object Spaces:

• Mapping architecture for arbitrary data models (20050050068)
• System and method for presenting a query expressed in terms of an object model (20040078359)
• System and method providing API interface between XML and SQL while interacting with a managed object environment (20030236859)

Matt Warren's "ObjectSpaces:There and Back Again" and Dinesh Kulkarni's "ObjectSpaces -> DLinq" blog entries throw additional light on ObjectSpace's transition from mapping files to LINQ and DLinq.

XQuery 1.0 Support in .NET Framework 2.0 Beta 1

About three years ago, Microsoft sponsored an XQuery demonstration site that included an XQuery engine (Microsoft.Xml.XQuery) for .NET 1.0. This early XQuery implementation didn't support all XQuery keywords, constructs, or use cases, but it provided a simple route for .NET developers to become familiar with XQuery 1.0 and XPath 2.0 syntax.

Beta 1 of the .NET Framework 2.0 included a System.Xml.Query namespace, which provided an XQueryCommand class (the XQuery processor) to implement a subset of XQuery 1.0 for XML documents. Mark Fussell's March 2004 MSDN article, "What's New in System.Xml for Visual Studio 2005 and the .NET Framework 2.0 Release," has a section named "The XQuery Language," which includes a simple C# example of an XQuery FLWOR (for, let, where, order by, and return) expression:

using (XmlWriter writer = XmlWriter.Create("output.xml ")) { XQueryCommand xq = new XQueryCommand(); string query = "" + "{ for $s in /bookstore/book " + "where $s/@genre='autobiography' " + "return $s/title }" + ""; xq.Compile(query); xq.Execute("books.xml", new XmlUrlResolver(), writer); }

Notice that the XQuery processor compiles the FLWOR query and then executes the compiled version against the books.xml file to write the output.xml document. Output.xml contains the title subelements of book elements whose genre attribute value is autobiography. (After XQuery appeared in the Whidbey bits, Microsoft turned off the XQuery demo site.)

Mark's article also included a section titled "Universal Type Support and Conversion," which described changes to the the XmlReader, XmlWriter, and XPathNavigator classes required to support the XQuery 1.0 and XPath 2.0 Data Model with XML schemas and CLR types. Dare Obasanjo's January 2005 MSDN article, "An Overview of Cω: Integrating XML into Popular Programming Languages," describes constructs in Microsoft Research's Cω (C-Omega) programming language that "...bridge the gap between Relational, Object, and XML data access by creating a type system that is a combination of all three data models." Although Dare's article discusses Cω features in a C# context, it's reasonable to expect that VB in Orcas will gain similar type system additions and programming constructs.

Whidbey Beta 2 removed the XQueryCommand class for the reasons cited in Microsoft's brief "Status of XQuery in the .NET Framework 2.0" white paper. The Microsoft XML Team's Mike Champion and Arpan Desai weigh in on what appears to have been a traumatic decision. Michael Rys, program manager for SQL Server’s XML Technologies, explains why SQL Server's xml data type uses XQuery 1.0 despite the XML team's arguments against it's inclusion in the .NET Framework 2.0.

The XML Team's white paper suggests that VS 2005 "... customers continue to use XSLT and XPath to address their XML filtering and transformation scenarios." The System.Xml.Xsl.XslCompiledTransform object represents an XSLT processor that supports XSLT 1.0 syntax. Following is C# sample code from VS 2005's online help for XslCompiledTransform:

// Create the XslCompiledTransform object. XslCompiledTransform xslt = new XslCompiledTransform(); // Compile the style sheet. xslt.Load("output.xsl"); // Execute the transform and output the results to a file. xslt.Transform("books.xml", "books.html");

The XslCompiledTransform.Load method, which compiles the specified stylesheet, corresponds to the XQueryCommand.Compile method and the XslCompiledTransform.Transform method approximates the XQueryCommand.Execute method.

References to LINQ and XLinq Predecessors

Arpan Desai, program manager for XSLT and XQuery in Whidbey, says "...we came up with a new query architecture to support both XSLT and XQuery." Michael Brundage, author of XQuery: The XML Query Language, is an inventor of the Common Query Runtime (CQR). CQR "...features compilers for each XML query language (such as XSLT and XQuery), a common intermediate format (QIL [Query Intermediate Language]), and multiple backend execution engines (over XML and SQL).

Following are Microsoft's U.S. and international patent applications related to IQF (from Michael Brundage, all filed in 2003):

• 304064.1 Common Query Runtime System and API (20040260691)
• 303845.1 Query Optimizer System and Method (20050004892)
• 301638.1 Query Intermediate Language Method and System (20040267760)

Numbers in parenthesis are U.S. Patent Office published application document numbers with links to the applications' full text and drawings. The Patent Office published the applications in 2004 and 2005. The patent applications reference XQuery and OPath as XML query languages and provide examples of relational queries using the Northwind database's tables.

The ".NET Language" prefix indicates to me (and others, such as Wesner Moise) that implementations will be specific to VB or C#. As an example, VB and C# appear to share anonymous types, but only Paul Vick's PDC 2005 session mentions query comprehensions and object initializers.

Soumitra Sengupta's "Changing the Game" entry to the XML Team's Weblog provides additional insight on the effect of LINQ and XLinq on the future of Microsoft's XML technologies.

PDC 2005 Sessions on LINQ and Related Topics

Links to individual sessions at http://commnet.microsoftpdc.com/content/sessions.aspx don't appear to be possible, so following are copied descriptions of LINQ-related sessions at PDC 2005. InfoWorld published on August 25, 2005 an article by Paul Krill that refers to the LINQ sessions. A similar article by Darryl K. Taft in the August 26 issue of eWeek.com describes a purported FoxPro-LINQ connection.

The .NET Language Integrated Query Framework: An Overview Speaker(s): Anders Hejlsberg Modern applications operate on data in several different forms: Relational tables, XML documents, and in-memory objects. Each of these domains can have profound differences in semantics, data types, and capabilities, and much of the complexity in today's applications is the result of these mismatches. The future "Orcas" release of Visual Studio aims to unify the programming models through integrated query capabilities in C# and Visual Basic, a strongly typed data access framework, and an innovative API for manipulating and querying XML. This session introduces each of these areas and walks through how they are related.

Using the .NET Language Integrated Query Framework with Relational Data Speaker(s): Luca Bolognese Database-centric applications have traditionally had to rely on two distinct programming languages: one for the database and one for the application. This session introduces future advances Microsoft is making for the "Orcas" release of Visual Studio in programming languages and frameworks to help integrate relational data and queries with C# and Visual Basic. These advances enable developers to express queries and updates in terms of their local programming language without sacrificing the server-side execution model of today's high-performance SQL-based approaches. Using these advances, database queries that previously were stored as opaque strings now benefit from static type checking, CLR metadata, design-time type inference, and of course IntelliSense.

Using the .NET Language Integrated Query Framework with XML Data Speaker(s): Dave Remy One of the key challenges to working with XML data has been the impedance mismatch between XML and programming languages. This session introduces future advances Microsoft is making for the "Orcas" release of Visual Studio in programming languages and frameworks to help integrate XML and queries with C# and Visual Basic. The advances include a framework for navigating, querying, and transforming XML that is both easier to use and more efficient than current XML programming techniques. This framework marries the capabilities of XPath, XQuery, and the DOM with the language integrated query framework planned for C# and Visual Basic.

Visual Basic: Future Directions in Language Innovation Speaker(s): Amanda Silver, Paul Vick Visual Basic 9.0 will offer radical improvements in its ability to work with data in all its forms: as objects, as XML, as relational data. Join the language architects for a detailed discussion of features such as query comprehensions, object initializers and anonymous types that enable querying data in a more flexible, natural way than ever before. Also, get a glimpse into the future of dynamic programming in VB with coverage of new features intended to radically simplify working with dynamically typed data on the .NET platform.

C#: Future Directions in Language Innovation from Anders Hejlsberg Speaker(s): Anders Hejlsberg Join Anders Hejlsberg, Distinguished Engineer and chief architect of the C# language, for an in-depth walkthrough of the new language features in C# 3.0. Understand how features like extension methods, lambda expressions, type inference, and anonymous types make it possible to create powerful APIs for expressing queries and interacting with objects, XML, and databases in a strongly typed, natural way. It is suggested that you attend "The .NET Language Integrated Query Framework: An Overview" before attending this session.

--rj

Technorati tags: LINQ, ADO.NET vNext, ObjectSpaces

The Yukon Quintet (with Apologies to Lawrence Durrell and Robert Service)

Fawcette Technical Publications (FTPOnline) published on August 26, 2005 the last of my five recent Visual Studio Magazine columns about new features in SQL Server 2005, the database formerly known as Yukon. Following are links and brief descriptions of (decks for) the columns in ascending date sequence:

• Exploit Yukon's XML Data Type. Take advantage of SQL Server 2005’s new native XML data type to add XML columns to a table, populate and index the columns, and understand basic XQuery syntax.
• Migrate to SQL Server 2005 Express Edition. Take advantage of new SQL Server 2005 features, GUI administration, and XCopy or ClickOnce deployment with SQL Express and Express Manager.
• Encrypt and Decrypt Data in Yukon. SQL Server 2005's engine-based, data-encryption functions protect confidential information from unauthorized access and disclosure, and offer native key management as a bonus.
• Take Advantage of New T-SQL Features. New T-SQL keywords implement TRY...CATCH error handling, pivot aggregate values to create crosstab tables, rank and partition rowsets, generate temporary in-memory tables with common table expressions, and more.
• Create Interoperable Native Web Services. Yukon lets you take advantage of the new kernel-mode Http.sys listener and eliminates the need for IIS to process ASMX files or SQLXML 3.0 templates that implement Web services.

Each article has a downloadable Visual Studio 2005 Visual Basic project that demonstrates the features described in the article. Click the Get The Code link in the Article Tools group on the first page of each article to download the project's ZIP file. Enjoy. --rj

Is XQuery Right for Scalable Databases?

Microsoft SQL Server 2005's new xml data type enables an alternative approach to accessing and modifying hierarchical—rather than relational—data. The primary data access objective for the xml data type is to return structured information from semi-structured XML content. XQuery 1.0, still a W3C Working Draft, is the XML query language implementation for SQL Server's xml data type and basis of the future second edition of SQL/XML-2003 for ANSI SQL-n. The SQL Server team has increased XQuery's utility by adding proprietary methods to update content of xml columns. You can learn more about Microsoft's XQuery implementation in SQL Server 2005 and download an interactive XQuery VB .NET 2005 Windows form project from my "Exploit Yukon's XML Data Type" Visual Studio Magazine article. The article also describes Microsoft's approach to typing xml columns with XML Schema collections and indexing nodes to improve XQuery performance. My tests against SQL Server 2005 beta versions indicate that Microsoft's XQuery implementation meets the requirements of most XQuery use cases and delivers acceptable peformance with a few simuated clients executing simultaneous queries. Hopefully, I'll be able to set up larger size scalability tests with the November 2005's RTM version . XQuery isn't yet an industry "open standard," but the language can stake a claim to maturity. The first XQuery 1.0 Working Draft was posted by the W3C on 15 February 2001, thus XQuery has had a gestation period almost as long as SQL Server 2005. InfoWorld's Jon Udell has posted an interview with Jonathan Robie, one of the editors of the W3C's Working Draft, that recounts the roots of XQuery in earlier XML query languages, such as Object Query Language (OQL), XQL (XML Query Language), XML-QL, and Quilt. Jon's post includes a link to a brief summary of the Robie interview in InfoWorld's "Innovation is alive and well in 2005" special report. Adding Processing Logic to the Database Engine vs. Scalability In his "Database Requirements in the Age of Scalable Services" keynote address to O'Reilly's MySQL Users Conference, held April 18-25, 2005 in Santa Clara, California, Google's Adam Bosworth stated, "If it takes four years, you've overdone the thing," referring to XQuery 1.0. Adam's XQuery condemnation could be deemed applicable to SQL Server 2005 and even to Microsoft Access 1.0, if you include the development time spent on it's ill-fated predecessor (code-named "Omega.") Adam was in charge of the design and development of Access 1.0 during his career at Microsoft. Adam is a champion of simple, highly-scalable queries for Web searches. He says Google has the "dumbest query language in the world ... just type in words. ... It would be silly for MySQL to support XQuery. ... Centralizing [processing] logic in the database ... is not going to give you scale." The same issue applies to executing .NET 2.0 CLR stored procedures within the SQL Server 2005 engine's process, publishing native Web services that use an engine-provided SOAP stack, or defining triggers and user-defined types with C# or VB .NET code. Scaling up databases by adding processors increases hardware and licensing costs faster than scaling out middle tiers by adding Web or application servers. What's missing at this point are independent (third-party) quantitative cost/performance comparisons for engine-based versus middle-tier processing of business rules and other logical—rather than set-based—processing operations. The results aren't likely to become widely known until large organizations start piloting XQuery information retrieval and in-process CLR database objects for real or simulated production environments. Federated Content vs. Centralized Data Domains Google's requirement is high scalability to achieve fast response times for simple full-text queries that return low-granularity (page-level) result sets from caches. Blog searches require increasing granularity to the item level and, presumably, enabling creation or modification date/time criteria. According to Adam Bosworth, "RSS and ATOM will become the lingua franca of data access. ... ATOM was formed by a consortium of bloggers." He observes that RSS/ATOM queries must operate at the item level and won't support joins because joins don't scale; this results in much lower-tech queries than XQuery. The queries aren't intended for the Semantic Web with the Resource Description Framework (RDF) and its top-down taxonomies and schemas. Adam claims that RDF has failed, as demonstrated by the fact that RSS 1.0 was an RDF grammer, RSS 2.0 was not, and RSS 2.0 "won in a walk." Despite the enormous amount of content in the form of Web pages and the rapid increase in the number and size of Weblogs (federated content domains), it's a safe bet that the majority of content that qualifies as useful information is stored in restricted-access databases (centralized data domains). These databases are primarily relational and very-fine grained; relatively few include searchable full-text XML content or XML documents shredded to relations and indexed attributes (tables and columns). As XML documents or fragments increasingly find their way into columns of the xml data type, the ability to write efficient XQuery expressions that extract and transform fine-grained semi-structured content (i.e., XML Infosets) will become a skill required of professional database developers. However, XQuery execution cost—and thus scalability—will remain an open question until all major RDBMS vendors offer implementations that have at least roughly equivalent feature sets. --rj

New Incentives Loom for Data(base) Encryption [Updated]

SQL Server 2005's native data encryption features gain new importance as U.S. congressmen and state lawmakers curry favor with constituents alarmed by Internet identity theft. Another 40 million compromised credit card numbers and "security codes" add fuel to calls for protection of consumers' private financial information.

[Update 1/26/2006] The U.S. Federal Trade Commission (FTC) has levied a US$10 million fine on ChoicePoint for violations of the Fair Credit Reporting Act (FCRA). The FTC also expects ChoicePoint to establish a US$5 million "trust fund for individuals who might have become victims of identity theft as a result of the breach." Senators Charles E. Schumer (D-NY) and Bill Nelson (D-FL) introduced S. 768, the "Comprehensive Identity Theft Prevention Act," on April 12, 2005. Subsequently, S. 768 gained Sen. Mark Dayton (D-MN) and Sen. Edward M. Kennedy (D-MA) as cosponsors. According to the Commercial Law League of America (CLLA), the Act would establish an Office of Identity Theft within the Federal Trade Commission and would provide the FTC with broad authority to prevent identity theft and establish limitations on businesses that collect, maintain, sell or transfer sensitive personal information of individuals. The FTC would have civil jurisdiction over all commercial organizations that collect, maintain, sell or transfer sensitive personal information. Sensitive personal information includes an individual's:

• Social security number
• Driver's license number or state identification number
• Bank or investment account number
• Credit or debit card number
• Certain medical information
• Payment history
• Other information specified by the FTC

Unauthorized disclosure of sensitive personal information could result in civil penalties up to $1,000 per violation, depending on the nature of the violation, such as failure to meet "reasonable standards" for data protection.

A recent New York Times article, "The scramble to protect personal data," mentioned the newly-proposed act in conjunction with CitiGroup's loss of a box of unencrypted backup tapes of CitiFinancial records that contained sensitive personal information (names, addresses, Social Security numbers, and account numbers) for about four million customers. If the act were in effect before the loss, CitiGroup's liability could be as much as $4 billion. A June 10, 2005 InfoWorld editorial, "Another week, another few million confidential records lost," provides more details on CitiGroup's lost backup tapes and the 3.9 million notices being sent. The tapes were lost in transit from a New Jersey datacenter to a credit bureau in Texas. According to the article, "Citibank made it clear in its statement that the company had plans to begin encrypting their credit bureau information." Computer security expert Bruce Schneier weighed in with a different take on the CitiGroup loss. Schneier is concerned that the California Information Practices Act (S.B. 1386), which requires entities to notify persons whose unencrypted Social Security, state identification, driver's license, bank account, or credit card numbers have been subject to unauthorized access, will lead to reduced press coverage of personal information theft and loss incidents. A compromise of 100,000 customer records doesn't give rise to a major news story after earlier (and repeated) reports that millions of personal records have been lost or stolen. Update: June 18, 2005: In a record-breaking security breach, MasterCard International reported on June 17, 2005 that about 40 million credit and debit card account numbers and security codes might have been stolen from CardSystems Solutions, a MasterCard processor. About 20 million were Visa cards and 13.9 were MasterCharge; the remainder were American Express and Discover cards. A New York Times article quoted a MasterCard spokesperson: "[A]n infiltrator had managed to place a computer code or script on the CardSystems network that made it possible to extract information." The Times article explained that "MasterCard said its investigation found that CardSystems, in violation of MasterCard's rules, was storing cardholders' account numbers and security codes on its computer systems. That information, MasterCard said, was supposed to be transferred to the bank handling the merchants' transactions but not retained by CardSystems." It's not clear from the article, but the term "security codes" might be the three-digit card validation code that's printed on the back of MasterCharge, Visa, Discover, JCB, or Diner's Club cards or four-digit code for American Express cards. The card validation code usually is required for on-line transactions. A San Francisco Chronicle article states: "Neither MasterCard nor Visa would say what was lacking in the firm's security, except to say it was out of compliance with their minimum security standards. But experts say that in order for a hacker to steal and use the information, it could not have been encrypted, a basic step that is required by the card companies' standards." The article goes on to quote Gartner analyst Avivah Litan: "They [MasterCard] weren't actively monitoring compliance. It wouldn't take that much to send an auditor to see if that data is encrypted or not." Neither Visa nor MasterCard require encrypting stored credit-card data. For example, here's a link to the Payment Card Industry (PCI) Security Standard on the Visa site, and a link to a simplified version on the MasterCharge site. MasterCard paraphrases PCI requirement 3 as "Protect stored transaction data. Keep transaction storage to a minimum and never store sensitive authentication data after authorization. Take precautions to make stored transaction data unreadable through encryption or some other secure and robust approach." [Emphasis added.] Following is the text of the Visa U.S.A. Cardholder Information Security Program (CISP) Frequently-Asked Questions #7: "7. Are there alternatives to encrypting stored data? Stored cardholder data should be rendered unreadable according to requirement 3 of the PCI Security Audit Procedures document. If encryption, truncation, or another comparable approach cannot be used, encryption options should continue to be investigated as the technology is rapidly evolving. In the interim, while encryption solutions are being investigated, stored data must be strongly protected by compensating controls. These compensating controls should be considered as part of the compliance validation process. [Emphasis added.] An example of compensating controls for encryption of stored data is complex network segmentation that may include the following: • Internal firewalls that specifically protect the database • TCP wrappers or firewall on the database to specifically limit who can connect to the database • Separation of the corporate internal network on a different network segment from production, fire- walled away from database servers." MasterCard's Electronic Commerce Best Practices for Acquirers classifies encryption of stored data as a "best practice," not a requirement. (The original title of this document was "Electronic Commerce Requirements and Best Practices for Acquirers.") MasterCard processors (a.k.a. "acquirers") "themselves do not need to go through the SDP compliance process but they must manage the SDP process for their merchants and service providers." [SDP is an abbreviation for MasterCard's Site Data Protection process for merchants.] A TransactionWorld page compares Visa's CISP and MasterCharge's SDP as of February 2004. [Note that an issuer whose account data was exposed to possible compromise as a result of this event has a right to claim reimbursement for costs related to reissuance of cards and monitoring of potentially compromised accounts that remain open. For any given account, the issuer may request reimbursement of up to US$25.00 for each reissued card, or up to US$5.00 for each monitored account without reissuance. In theory, CardSystems could be liable for as much as US$1 billion in issuer charges if all 40 million cards were reissued.] End of June 18, 2005 Update.

Update, July 1, 2005: Infoworld's Ephraim Schwarz concludes that Visa and MasterCard weren't enforcing their own security requirements. "Frank Smith, vice president of the technology strategy group at Capgemini, said, 'They don’t supply due diligence to the whole system.' Gartner’s [Avivah] Litan said, 'They have everything in place; they just don’t enforce it.' Paul Stamp, security analyst at Forrester Research, said 'The processes were not properly enforced.' [John] Pescatore [a Gartner security analyst] said that the standards 'have been pure eyewash. No enforcement.' End of July 1, 2005 Update:

June 20, 2005 Update: CardSystems' CEO John Perry admitted to the New York Times (free registration required) that his company should not have stored the data that was compromised. CardSystems was using the stored transaction data for "research purposes." The Times article states: "Under rules established by Visa and MasterCard, processors are not allowed to retain cardholder information including names, account numbers, expiration dates and security codes after a transaction is handled." The article confirms that "security codes" refers to the three-digit or four-digit codes printed on the back of the credit card, which are specifically embargoed from storage by merchants or processors. End of June 20, 2005 Update.

Update July 22, 2005: Visa USA announced on July 18, 2005 that it will no longer allow CardSystems to process Visa transactions. Visa will allow banks that use CardSystems to handle merchant transactions until the end of October 2005. The New York Times also reported on July 19, 2005 that American Express will terminate its relationship with CardSystems at the end of October 2005. In congressional testimony on July 21, 2005, CardSystems' CEO John Perry said the firm faces "immediate extinction" and blamed former Cable & Wireless auditors for practices that led to the Visa termination. End of July 22, 2005 update.

H.R.1653 (a.k.a. "Safeguarding Americans From Exporting Identification Data Act" or the "SAFE-ID Act") is entitled "To prohibit the transfer of personal information to any person outside the United States, without notice and consent, and for other purposes." H.R.1653 expands S.768's list of sensitive personal information to include the following:

• Name
• Postal address
• Financial information
• Medical records
• Date of birth
• Phone number
• E-mail address
• Social Security number
• Mother's maiden name
• Password
• State identification information
• Driver's license number
• Personal tax information
• Any consumer transactional or experiential information relating to the person

Consumers must take explicit action ("opt-out") to prevent U.S. firms possessing the personal information from transmitting it to "any foreign affiliate or subcontractor located in a country that is a country with adequate privacy protection." Opt-in is required for countries that don't have "adequate privacy protection." Most privacy advocates would place the U.S. in the latter category. After the CardSystems debacle, ordinary U.S. citizens undboubtedly would do the same. However, it's not so clear that any U.S. government agency would have the courage to so categorize this country. Violation of the Act would be treated as a violation of a rule defining an unfair or deceptive act or practice prescribed under section 18(a)(1)(B) of the Federal Trade Commission Act (15 U.S.C. 57a(a)(1)(B)). Persons could bring a state court action "to recover for actual monetary loss from such a violation, or to receive $10,000 in damages for each such violation, whichever is greater." Related—but more restricted—proposed acts would "regulate information brokers and protect individual rights with respect to personally identifiable information" (H.R.1080 and S.500) and "strengthen the authority of the Federal Government to protect individuals from certain acts and practices in the sale and purchase of Social Security numbers" (H.R.1078). H.R. 1080 would permit persons to bring a state court action "to recover for actual monetary loss from such a violation, or to receive $1,000 in damages for each such violation, whichever is greater." H.R. 1080 doesn't designate a default damage amount. The probability of enactment of any of these proposals in the current congress is low, at best. Few proposed consumer privacy acts have significant bi-partisan support (read "Republican cosponsors"). Heavy-handed and well-funded lobbyists will attempt to kill the proposals in committee. But grass-roots concern with identity theft is growing, so there's hope for some future form of federal protection for "personally identifiable information" beyond that currently provided by HIPAA. Update: June 18, 2005: The Gramm-Leach-Bliley act of 1999 (15 U. S.C. § 6801 et seq., GLBA) includes provisions that are purported to protect consumers' non-public personally identifiable financial information (NPI) by restricting its transfer from financial institutions to non-affiliated third parties. Currently, GLBA applies only to financial institutions that provide services to consumers, such as Visa and MasterCard—but not processors/acquirers like CardSystems. What's worse, consumers must proactively (affirmatively) opt-out of the third-party information-sharing process. Adoption of the "opt-out" method (versus the more commonly accepted "opt-in" approach that applies to health-related information) was the subject of an intense lobbying campaign by the financial industry in general and credit-card issuers in particular. It's not known how many, if any, of the holders of the 40 million compromised credit cards had exercised their "opt-out" rights. A relatively simple (but very unlikely) method of minimizing exposure of NPI to third parties is amendment of GLBA to change "opt-out" to "opt-in" and include credit-card processors and any other organizations in the processing chain as "financial institutions." End of June 18, 2005 Update. --rj P.S. Bruce Schneier points out in his recent "U.S. Medical Privacy Law Gutted" post that a new ruling by the U.S. Justice Department "sharply limits the government's ability to prosecute people for criminal violations" of the HIPAA privacy regulations. Criminal penalties, the department said, apply to insurers, doctors, hospitals and other providers—but not necessarily their employees or outsiders who steal personal health data.

A Business Intelligence Demo with Real-World Data

The Gartner Group attributes much of the 10.3 percent growth of the total relational database market from 2003 to 2004 to new business intelligence features—data analysis, warehousing and reporting—as well as the weakening US dollar. Gartner's May 2005 "No Clear Winner in Overall RDBMS Market Share Race" report gave IBM 34.1%, Oracle 33.7%, and Microsoft 20% of the total market. Teradata, Sybase, and others claimed 2.9%, 2.3% and 6.6%, respectively. Microsoft's market share increase from 18.7% in 2003 is surprising when you consider that prospective licensees were then anticipating Yukon to release to manufacturing in mid-2004, not November 2005. eWeek magazine's Lisa Vass quotes Microsoft director of SQL Server product management, Tom Rizzo: "BI is a tremendous growth driver for us, especially Reporting Services, which we've seen a ton of customers buying and deploying. That's why we invested so heavily in BI technologies across SQL Server. ... We put a down payment many years ago, and now it's paying off in terms of revenue growth." Unlike its major database competitors, Microsoft includes business intelligence integration, development, reporting and management features in the basic license fee for all editions of SQL Server 2000 (except MSDE) and SQL Server 2005 (except Express and Workgroup editions, which do include Reporting Services.) SQL Server 2005 has a raft of new and improved business intelligence (BI) features. But the standard sample online transaction processing (OLTP) and data warehouse (DW) databases are based on the relational AdventureWorks sample database. Demonstrating the performance of Integration Services (SSIS, formerly Data Transformation Services. DTS) extract, transformation, and loading (ETL) features isn't practical with tables that contain only a few hundred or thousand rows. You need partitions containing multimillion-row dimension and fact tables to emulate the BI systems of, for example, nationwide or multi-national retailers. The table-size issue with performance testing is similar to that I describe in my FTP Online articles about SQL Server 2005's xml data type and data encryption features. Microsoft's Project REAL is "a reference implementation of a business intelligence (BI) system using real large-scale data from a real customer." Phase 1 of the project used source data from a large electronics retailer. Phase 2's "real customer" is Barnes & Noble, the largest U.S. bookseller, who contributed the masked source data and BI scenario. Barnes & Noble have about 40,000 employees and 800 stores in the U.S.

According to the Technical Overview for Phase 2, Project REAL's goal "is to discover the best practices for creating BI systems with SQL Server 2005 and to build a system that exhibits as many of those best practices as we can. This project is not just a demo—we are creating this system for ongoing operation. It is a complete system, including daily incremental updates of the data, large multiuser workloads, and system monitoring."

The Technical Overview is the first in a promised series of articles that will be based on the B&N source data and analytical model. Hopefully, the Project REAL team will provide dynamic, online demonstrations of simulated ad hoc and preprogrammed BI reports. Making Project REAL's warehoused data and Reporting Services accessible to developers by Web service methods similar to those for TerraServer or MapPoint maps would be a major SQL Server 2005 marketing coup.

Hey, Tom Rizzo—are you listening?

--rj

Microsoft Promises RSS 2.0 Support in Longhorn

Steve Ballmer reportedly considers RSS "a little too simple," but damnation by faint praise by Microsoft's CEO hasn't prevented the Longhorn team from climbing on the RSS bandwagon. Dean Hachamovitch, general manager for Longhorn browsing and RSS announced in a June 24, 2005 keynote speech at the Gnomedex 5.0 conference that Longhorn will include support for RSS with a Common RSS Feed List and Data Store, plus an RSS Platform Feed Engine. Hachamovitch also proposed a specification for an RSS extension to support ordered lists. Despite its release under a Creative Commons Share-Alike license, the "Simple List Extensions Specification" generated an extraordinary amount of third-party blogging and comment activity, much of which was uninformed. The next day, a Longhorn RSS Team Blog opened with a post from Sean Lyndersay, senior program manager on Microsoft's RSS team and majordomo of the MSDN IEBlog. Developer feedback is handled by Channel9 Longhorn RSS and SimpleListExtensions WIKIs. Unfortunately, the RSS in Longhorn announcement occurred about three weeks after the publication date of my "Longhorn Redux" FTPOnline article, which concentrated on consumer features and IIS 7.0. It's good to see a feature added to—instead of removed or borrowed from—Longhorn for a change. --rj