WinFS and the Language Integrated Query (LINQ) Project

Microsoft's pre-PDC release to MSDN subscribers of WinFS Beta 1 bits was a surprise to the trade press, as well as most .NET developers. InfoWorld's "Surprise! Microsoft's WinFS beta arrives" article by Eric Knorr appeared on August 31, 2005 and briefly describes the first WinFS tools for developers. Paul Thurrott's "Windows Storage Foundation (WinFS) Preview" provides a historical context for WinFS—begining with the Object File System (OFS, codenamed Cairo) that Microsoft once promised for Windows NT. Quentin Clark, Microsoft's director of program management for WinFS, briefly describes reasons for early release of the WinFS Beta and the relationship between WinFS and IQF in his initial post—Shipping WinFS Beta 1—on the WinFS team's blog. Here's the early version of the WinFS/LINQ story from Quentin's post:

Another piece of feedback was concerns over the WinFS api’s being a different data access pattern than our existing managed code data acess APIs. Further, that our APIs were not aligned with broader data platform needs like OR mapping. This was a big one, and as you explore the SDK and the APIs, you will see the beginnings of how we will be addressing this. We are in the process of building-out the next version of ADO.NET to have new features that provide a data model, object-relational mapping, and flexible query infrastructure. The new data model is about entities, and the WinFS data model of Item types is built on that model. Looking through our SDK and code samples you will see how Items are composed of underlying entities. OR mapping is a big requirement - WinFS is a very prescribed mapping (defining a type in WinFS generates both the underlying storage schema and the partial class to program to that type). But the real-world has lots of requirements for flexible mapping – to existing data, to existing objects, etc. On query, many of you have heard about Anders Hejlsberg’s work on Language Integrated Query – and the new ADO.NET functionality will plug directly underneath so that you can use the new query patterns on any entity data, including of course now WinFS Items.

Thus, it's likely that you'll be able to integrate WinFS and LINQ code in the .NET projects you create with Visual Studio "Orcas." PDC 2005 has four WinFS breakout sessions. Here's the description of DAT312—"WinFS" and ADO.NET: Future Directions for Data Access Scenarios (Samuel Druker)—that contains references to ObjectSpaces and "integrated query functionality:"

The next generation of ADO.NET incorporates innovations that integrate Entities, Relationships, Associations, a seamless CLR object programming surface, and flexible storage mapping for programming to data at all tiers. These technologies address the needs for "WinFS" data modeling and programming scenarios on the desktop and server, as well as the requirements from the ObjectSpaces project. This session discusses how ADO.NET entities will form the core for data integration, enabling the interoperability of applications and modules by providing a high-level framework for programming with data that includes integrated query functionality.

Following are links to the other three WinFS-specific sessions—DAT200 Future Directions for Data-Driven Applications: Storage, Applications, API’s (David Campbell); DAT209 "WinFS" Future Directions: An Overview (Shishir Mehrotra); and DAT310 "WinFS" Future Directions: Building Data Centric Applications Using Windows Presentation Foundation ("Avalon") and Windows Forms (Ramesh Nagarajan). Robert Scoble's one-hour Channel 9 video with members of the WinFS team includes a demonstration of some WinFS features and applications. The demo starts at about 11:00:00 into the program. As to the ultimate release of WinFS, Paul Thurrott bases the following timeline on "internal Microsoft documentation:"

WinFS Beta 1 will be followed by at least one Community Technology Preview (CTP) release, which is currently due on February 15, 2006. Then, on May 1, 2006, Microsoft is scheduled to release WinFS Beta 2. Beta 3 is currently scheduled for November 15, 2006, with a Beta 3 Refresh release expected in April 2007. WinFS is currently scheduled for RTM in Q3 2007, well after Longhorn Server is released.

--rj Technorati: LINQ, COmega, ADO.NET vNext, WinFS

The Language Integrated Query (LINQ) Project

Two features once promised for Visual Studio 2005 are conspicuous by their absence in VS 2005 Beta 2 Community Technical Preview drops:

• An object-relational mapping API and toolkit for ADO.NET 2.0 called ObjectSpaces
• An XQuery 1.0 implementation for the .NET Framework 2.0

Searching descriptions of PDC 2005 sessions for the keyword "query" indicates that the forthcoming Languate Integrated Query (LINQ) for .NET Project (nee Integrated Query Framework (IQF)) will resurrect in the VS Orcas release at least some of the promised features of the original .NET 2.0 implementations of ObjectSpaces and the XQuery 1.0 processor. (Copies of session descriptions are at the end of this post.) Recent MSDN articles and Microsoft patent applications add evidence of major changes in store for VB 9.0 and C# 3.0 in the next VS release.

Note: Technorati category links in the preceding text are experimental

ObjectSpaces Support in VS 2005 Beta 1 Orca (apparently an acronym for Object-Relational Component Architecture) was a technical preview of ObjectSpaces that first appeared in a PDC 2001 ADO.NET presentation and was promised as a Web release for VS .NET 1.0. ObjectSpaces introduced OPath, a proprietary object query language. I spent a substantial amount of time testing the PDC's technical preview bits. I don't believe there were subsequent Web release(s). Luca Bolognese's "DAT410 NET Framework: Developing Applications Using ADO.Net ObjectSpaces" presentation at PDC 2003 set the stage for ObjectSpaces' inclusion in the VS Whidbey release. Luca demonstrated a VS add-in tool to simplify the O-R mapping process, which required a set of three XML configuration (schema description) files for the object (Osd.xml), relational (Rsd.xml), and mapping (Msd.xml) layers.

Luca promised future support for generics with syntax similar to: Dim osQuery As New _ ObjectQuery(Of Customer)("Region='WA'","Orders") Dim osReader As ObjectReader(Of Customer) = _ ObjectSpace.GetObjectReader(osQuery). The final bulletpoint on Luca's "Conclusions" slide was: "It is included in the next version of Visual Studio." Peter Provost blogged the presentation and Paul Wilson's "ADO.NET v2.0: ObjectSpaces Delivers an O/R Mapper" article has sample code from the PDC alpha version. Dino Esposito's February 2004 MSDN article, "A First Look at ObjectSpaces in Visual Studio 2005," provides additional architectural details and sample code. The Whidbey alpha included a System.Data.ObjectSpaces namespace. Luca reappeared at Tech*Ed 2004 with the "DAT 301: ObjectSpaces" presentation, which included more advanced examples of OPath query syntax. Microsoft originally planned to use XPath as the query language but found that virgules (/) turned off developers, who preferred periods (.) as separators. Thus typical OPath query expressions against Northwind-based Orders and Details objects might have been: Orders.Details.Quantity > 50

Orders[ShippedDate > RequiredDate]

Orders[Freight > 1000].Details[Quantity > 30]

At the end of the presentation, Luca announced that ObjectSpaces had acquired a dependency on WinFS and thus was postponed to the Longhorn/Orcas timeframe. (He appeared close to tears as he made the announcement.) The dependence of the long-delayed Microsoft Business Framework (MBF) API on ObjectSpaces as its object persistence platform probably was a contributing factor to the demise of ObjectSpaces in VS 2005. Subsequently, Microsoft announced that WinFS had been excised from the planned Longhorn release version, now renamed Vista.

Here are three published Microsoft patent applications that include Luca Bolognese as an inventor and relate to Object Spaces:

• Mapping architecture for arbitrary data models (20050050068)
• System and method for presenting a query expressed in terms of an object model (20040078359)
• System and method providing API interface between XML and SQL while interacting with a managed object environment (20030236859)

Matt Warren's "ObjectSpaces:There and Back Again" and Dinesh Kulkarni's "ObjectSpaces -> DLinq" blog entries throw additional light on ObjectSpace's transition from mapping files to LINQ and DLinq.

XQuery 1.0 Support in .NET Framework 2.0 Beta 1

About three years ago, Microsoft sponsored an XQuery demonstration site that included an XQuery engine (Microsoft.Xml.XQuery) for .NET 1.0. This early XQuery implementation didn't support all XQuery keywords, constructs, or use cases, but it provided a simple route for .NET developers to become familiar with XQuery 1.0 and XPath 2.0 syntax.

Beta 1 of the .NET Framework 2.0 included a System.Xml.Query namespace, which provided an XQueryCommand class (the XQuery processor) to implement a subset of XQuery 1.0 for XML documents. Mark Fussell's March 2004 MSDN article, "What's New in System.Xml for Visual Studio 2005 and the .NET Framework 2.0 Release," has a section named "The XQuery Language," which includes a simple C# example of an XQuery FLWOR (for, let, where, order by, and return) expression:

using (XmlWriter writer = XmlWriter.Create("output.xml ")) { XQueryCommand xq = new XQueryCommand(); string query = "" + "{ for $s in /bookstore/book " + "where $s/@genre='autobiography' " + "return $s/title }" + ""; xq.Compile(query); xq.Execute("books.xml", new XmlUrlResolver(), writer); }

Notice that the XQuery processor compiles the FLWOR query and then executes the compiled version against the books.xml file to write the output.xml document. Output.xml contains the title subelements of book elements whose genre attribute value is autobiography. (After XQuery appeared in the Whidbey bits, Microsoft turned off the XQuery demo site.)

Mark's article also included a section titled "Universal Type Support and Conversion," which described changes to the the XmlReader, XmlWriter, and XPathNavigator classes required to support the XQuery 1.0 and XPath 2.0 Data Model with XML schemas and CLR types. Dare Obasanjo's January 2005 MSDN article, "An Overview of Cω: Integrating XML into Popular Programming Languages," describes constructs in Microsoft Research's Cω (C-Omega) programming language that "...bridge the gap between Relational, Object, and XML data access by creating a type system that is a combination of all three data models." Although Dare's article discusses Cω features in a C# context, it's reasonable to expect that VB in Orcas will gain similar type system additions and programming constructs.

Whidbey Beta 2 removed the XQueryCommand class for the reasons cited in Microsoft's brief "Status of XQuery in the .NET Framework 2.0" white paper. The Microsoft XML Team's Mike Champion and Arpan Desai weigh in on what appears to have been a traumatic decision. Michael Rys, program manager for SQL Server’s XML Technologies, explains why SQL Server's xml data type uses XQuery 1.0 despite the XML team's arguments against it's inclusion in the .NET Framework 2.0.

The XML Team's white paper suggests that VS 2005 "... customers continue to use XSLT and XPath to address their XML filtering and transformation scenarios." The System.Xml.Xsl.XslCompiledTransform object represents an XSLT processor that supports XSLT 1.0 syntax. Following is C# sample code from VS 2005's online help for XslCompiledTransform:

// Create the XslCompiledTransform object. XslCompiledTransform xslt = new XslCompiledTransform(); // Compile the style sheet. xslt.Load("output.xsl"); // Execute the transform and output the results to a file. xslt.Transform("books.xml", "books.html");

The XslCompiledTransform.Load method, which compiles the specified stylesheet, corresponds to the XQueryCommand.Compile method and the XslCompiledTransform.Transform method approximates the XQueryCommand.Execute method.

References to LINQ and XLinq Predecessors

Arpan Desai, program manager for XSLT and XQuery in Whidbey, says "...we came up with a new query architecture to support both XSLT and XQuery." Michael Brundage, author of XQuery: The XML Query Language, is an inventor of the Common Query Runtime (CQR). CQR "...features compilers for each XML query language (such as XSLT and XQuery), a common intermediate format (QIL [Query Intermediate Language]), and multiple backend execution engines (over XML and SQL).

Following are Microsoft's U.S. and international patent applications related to IQF (from Michael Brundage, all filed in 2003):

• 304064.1 Common Query Runtime System and API (20040260691)
• 303845.1 Query Optimizer System and Method (20050004892)
• 301638.1 Query Intermediate Language Method and System (20040267760)

Numbers in parenthesis are U.S. Patent Office published application document numbers with links to the applications' full text and drawings. The Patent Office published the applications in 2004 and 2005. The patent applications reference XQuery and OPath as XML query languages and provide examples of relational queries using the Northwind database's tables.

The ".NET Language" prefix indicates to me (and others, such as Wesner Moise) that implementations will be specific to VB or C#. As an example, VB and C# appear to share anonymous types, but only Paul Vick's PDC 2005 session mentions query comprehensions and object initializers.

Soumitra Sengupta's "Changing the Game" entry to the XML Team's Weblog provides additional insight on the effect of LINQ and XLinq on the future of Microsoft's XML technologies.

PDC 2005 Sessions on LINQ and Related Topics

Links to individual sessions at http://commnet.microsoftpdc.com/content/sessions.aspx don't appear to be possible, so following are copied descriptions of LINQ-related sessions at PDC 2005. InfoWorld published on August 25, 2005 an article by Paul Krill that refers to the LINQ sessions. A similar article by Darryl K. Taft in the August 26 issue of eWeek.com describes a purported FoxPro-LINQ connection.

The .NET Language Integrated Query Framework: An Overview Speaker(s): Anders Hejlsberg Modern applications operate on data in several different forms: Relational tables, XML documents, and in-memory objects. Each of these domains can have profound differences in semantics, data types, and capabilities, and much of the complexity in today's applications is the result of these mismatches. The future "Orcas" release of Visual Studio aims to unify the programming models through integrated query capabilities in C# and Visual Basic, a strongly typed data access framework, and an innovative API for manipulating and querying XML. This session introduces each of these areas and walks through how they are related.

Using the .NET Language Integrated Query Framework with Relational Data Speaker(s): Luca Bolognese Database-centric applications have traditionally had to rely on two distinct programming languages: one for the database and one for the application. This session introduces future advances Microsoft is making for the "Orcas" release of Visual Studio in programming languages and frameworks to help integrate relational data and queries with C# and Visual Basic. These advances enable developers to express queries and updates in terms of their local programming language without sacrificing the server-side execution model of today's high-performance SQL-based approaches. Using these advances, database queries that previously were stored as opaque strings now benefit from static type checking, CLR metadata, design-time type inference, and of course IntelliSense.

Using the .NET Language Integrated Query Framework with XML Data Speaker(s): Dave Remy One of the key challenges to working with XML data has been the impedance mismatch between XML and programming languages. This session introduces future advances Microsoft is making for the "Orcas" release of Visual Studio in programming languages and frameworks to help integrate XML and queries with C# and Visual Basic. The advances include a framework for navigating, querying, and transforming XML that is both easier to use and more efficient than current XML programming techniques. This framework marries the capabilities of XPath, XQuery, and the DOM with the language integrated query framework planned for C# and Visual Basic.

Visual Basic: Future Directions in Language Innovation Speaker(s): Amanda Silver, Paul Vick Visual Basic 9.0 will offer radical improvements in its ability to work with data in all its forms: as objects, as XML, as relational data. Join the language architects for a detailed discussion of features such as query comprehensions, object initializers and anonymous types that enable querying data in a more flexible, natural way than ever before. Also, get a glimpse into the future of dynamic programming in VB with coverage of new features intended to radically simplify working with dynamically typed data on the .NET platform.

C#: Future Directions in Language Innovation from Anders Hejlsberg Speaker(s): Anders Hejlsberg Join Anders Hejlsberg, Distinguished Engineer and chief architect of the C# language, for an in-depth walkthrough of the new language features in C# 3.0. Understand how features like extension methods, lambda expressions, type inference, and anonymous types make it possible to create powerful APIs for expressing queries and interacting with objects, XML, and databases in a strongly typed, natural way. It is suggested that you attend "The .NET Language Integrated Query Framework: An Overview" before attending this session.

--rj

Technorati tags: LINQ, ADO.NET vNext, ObjectSpaces

The Yukon Quintet (with Apologies to Lawrence Durrell and Robert Service)

Fawcette Technical Publications (FTPOnline) published on August 26, 2005 the last of my five recent Visual Studio Magazine columns about new features in SQL Server 2005, the database formerly known as Yukon. Following are links and brief descriptions of (decks for) the columns in ascending date sequence:

• Exploit Yukon's XML Data Type. Take advantage of SQL Server 2005’s new native XML data type to add XML columns to a table, populate and index the columns, and understand basic XQuery syntax.
• Migrate to SQL Server 2005 Express Edition. Take advantage of new SQL Server 2005 features, GUI administration, and XCopy or ClickOnce deployment with SQL Express and Express Manager.
• Encrypt and Decrypt Data in Yukon. SQL Server 2005's engine-based, data-encryption functions protect confidential information from unauthorized access and disclosure, and offer native key management as a bonus.
• Take Advantage of New T-SQL Features. New T-SQL keywords implement TRY...CATCH error handling, pivot aggregate values to create crosstab tables, rank and partition rowsets, generate temporary in-memory tables with common table expressions, and more.
• Create Interoperable Native Web Services. Yukon lets you take advantage of the new kernel-mode Http.sys listener and eliminates the need for IIS to process ASMX files or SQLXML 3.0 templates that implement Web services.

Each article has a downloadable Visual Studio 2005 Visual Basic project that demonstrates the features described in the article. Click the Get The Code link in the Article Tools group on the first page of each article to download the project's ZIP file. Enjoy. --rj

Is XQuery Right for Scalable Databases?

Microsoft SQL Server 2005's new xml data type enables an alternative approach to accessing and modifying hierarchical—rather than relational—data. The primary data access objective for the xml data type is to return structured information from semi-structured XML content. XQuery 1.0, still a W3C Working Draft, is the XML query language implementation for SQL Server's xml data type and basis of the future second edition of SQL/XML-2003 for ANSI SQL-n. The SQL Server team has increased XQuery's utility by adding proprietary methods to update content of xml columns. You can learn more about Microsoft's XQuery implementation in SQL Server 2005 and download an interactive XQuery VB .NET 2005 Windows form project from my "Exploit Yukon's XML Data Type" Visual Studio Magazine article. The article also describes Microsoft's approach to typing xml columns with XML Schema collections and indexing nodes to improve XQuery performance. My tests against SQL Server 2005 beta versions indicate that Microsoft's XQuery implementation meets the requirements of most XQuery use cases and delivers acceptable peformance with a few simuated clients executing simultaneous queries. Hopefully, I'll be able to set up larger size scalability tests with the November 2005's RTM version . XQuery isn't yet an industry "open standard," but the language can stake a claim to maturity. The first XQuery 1.0 Working Draft was posted by the W3C on 15 February 2001, thus XQuery has had a gestation period almost as long as SQL Server 2005. InfoWorld's Jon Udell has posted an interview with Jonathan Robie, one of the editors of the W3C's Working Draft, that recounts the roots of XQuery in earlier XML query languages, such as Object Query Language (OQL), XQL (XML Query Language), XML-QL, and Quilt. Jon's post includes a link to a brief summary of the Robie interview in InfoWorld's "Innovation is alive and well in 2005" special report. Adding Processing Logic to the Database Engine vs. Scalability In his "Database Requirements in the Age of Scalable Services" keynote address to O'Reilly's MySQL Users Conference, held April 18-25, 2005 in Santa Clara, California, Google's Adam Bosworth stated, "If it takes four years, you've overdone the thing," referring to XQuery 1.0. Adam's XQuery condemnation could be deemed applicable to SQL Server 2005 and even to Microsoft Access 1.0, if you include the development time spent on it's ill-fated predecessor (code-named "Omega.") Adam was in charge of the design and development of Access 1.0 during his career at Microsoft. Adam is a champion of simple, highly-scalable queries for Web searches. He says Google has the "dumbest query language in the world ... just type in words. ... It would be silly for MySQL to support XQuery. ... Centralizing [processing] logic in the database ... is not going to give you scale." The same issue applies to executing .NET 2.0 CLR stored procedures within the SQL Server 2005 engine's process, publishing native Web services that use an engine-provided SOAP stack, or defining triggers and user-defined types with C# or VB .NET code. Scaling up databases by adding processors increases hardware and licensing costs faster than scaling out middle tiers by adding Web or application servers. What's missing at this point are independent (third-party) quantitative cost/performance comparisons for engine-based versus middle-tier processing of business rules and other logical—rather than set-based—processing operations. The results aren't likely to become widely known until large organizations start piloting XQuery information retrieval and in-process CLR database objects for real or simulated production environments. Federated Content vs. Centralized Data Domains Google's requirement is high scalability to achieve fast response times for simple full-text queries that return low-granularity (page-level) result sets from caches. Blog searches require increasing granularity to the item level and, presumably, enabling creation or modification date/time criteria. According to Adam Bosworth, "RSS and ATOM will become the lingua franca of data access. ... ATOM was formed by a consortium of bloggers." He observes that RSS/ATOM queries must operate at the item level and won't support joins because joins don't scale; this results in much lower-tech queries than XQuery. The queries aren't intended for the Semantic Web with the Resource Description Framework (RDF) and its top-down taxonomies and schemas. Adam claims that RDF has failed, as demonstrated by the fact that RSS 1.0 was an RDF grammer, RSS 2.0 was not, and RSS 2.0 "won in a walk." Despite the enormous amount of content in the form of Web pages and the rapid increase in the number and size of Weblogs (federated content domains), it's a safe bet that the majority of content that qualifies as useful information is stored in restricted-access databases (centralized data domains). These databases are primarily relational and very-fine grained; relatively few include searchable full-text XML content or XML documents shredded to relations and indexed attributes (tables and columns). As XML documents or fragments increasingly find their way into columns of the xml data type, the ability to write efficient XQuery expressions that extract and transform fine-grained semi-structured content (i.e., XML Infosets) will become a skill required of professional database developers. However, XQuery execution cost—and thus scalability—will remain an open question until all major RDBMS vendors offer implementations that have at least roughly equivalent feature sets. --rj

New Incentives Loom for Data(base) Encryption [Updated]

SQL Server 2005's native data encryption features gain new importance as U.S. congressmen and state lawmakers curry favor with constituents alarmed by Internet identity theft. Another 40 million compromised credit card numbers and "security codes" add fuel to calls for protection of consumers' private financial information.

[Update 1/26/2006] The U.S. Federal Trade Commission (FTC) has levied a US$10 million fine on ChoicePoint for violations of the Fair Credit Reporting Act (FCRA). The FTC also expects ChoicePoint to establish a US$5 million "trust fund for individuals who might have become victims of identity theft as a result of the breach." Senators Charles E. Schumer (D-NY) and Bill Nelson (D-FL) introduced S. 768, the "Comprehensive Identity Theft Prevention Act," on April 12, 2005. Subsequently, S. 768 gained Sen. Mark Dayton (D-MN) and Sen. Edward M. Kennedy (D-MA) as cosponsors. According to the Commercial Law League of America (CLLA), the Act would establish an Office of Identity Theft within the Federal Trade Commission and would provide the FTC with broad authority to prevent identity theft and establish limitations on businesses that collect, maintain, sell or transfer sensitive personal information of individuals. The FTC would have civil jurisdiction over all commercial organizations that collect, maintain, sell or transfer sensitive personal information. Sensitive personal information includes an individual's:

• Social security number
• Driver's license number or state identification number
• Bank or investment account number
• Credit or debit card number
• Certain medical information
• Payment history
• Other information specified by the FTC

Unauthorized disclosure of sensitive personal information could result in civil penalties up to $1,000 per violation, depending on the nature of the violation, such as failure to meet "reasonable standards" for data protection.

A recent New York Times article, "The scramble to protect personal data," mentioned the newly-proposed act in conjunction with CitiGroup's loss of a box of unencrypted backup tapes of CitiFinancial records that contained sensitive personal information (names, addresses, Social Security numbers, and account numbers) for about four million customers. If the act were in effect before the loss, CitiGroup's liability could be as much as $4 billion. A June 10, 2005 InfoWorld editorial, "Another week, another few million confidential records lost," provides more details on CitiGroup's lost backup tapes and the 3.9 million notices being sent. The tapes were lost in transit from a New Jersey datacenter to a credit bureau in Texas. According to the article, "Citibank made it clear in its statement that the company had plans to begin encrypting their credit bureau information." Computer security expert Bruce Schneier weighed in with a different take on the CitiGroup loss. Schneier is concerned that the California Information Practices Act (S.B. 1386), which requires entities to notify persons whose unencrypted Social Security, state identification, driver's license, bank account, or credit card numbers have been subject to unauthorized access, will lead to reduced press coverage of personal information theft and loss incidents. A compromise of 100,000 customer records doesn't give rise to a major news story after earlier (and repeated) reports that millions of personal records have been lost or stolen. Update: June 18, 2005: In a record-breaking security breach, MasterCard International reported on June 17, 2005 that about 40 million credit and debit card account numbers and security codes might have been stolen from CardSystems Solutions, a MasterCard processor. About 20 million were Visa cards and 13.9 were MasterCharge; the remainder were American Express and Discover cards. A New York Times article quoted a MasterCard spokesperson: "[A]n infiltrator had managed to place a computer code or script on the CardSystems network that made it possible to extract information." The Times article explained that "MasterCard said its investigation found that CardSystems, in violation of MasterCard's rules, was storing cardholders' account numbers and security codes on its computer systems. That information, MasterCard said, was supposed to be transferred to the bank handling the merchants' transactions but not retained by CardSystems." It's not clear from the article, but the term "security codes" might be the three-digit card validation code that's printed on the back of MasterCharge, Visa, Discover, JCB, or Diner's Club cards or four-digit code for American Express cards. The card validation code usually is required for on-line transactions. A San Francisco Chronicle article states: "Neither MasterCard nor Visa would say what was lacking in the firm's security, except to say it was out of compliance with their minimum security standards. But experts say that in order for a hacker to steal and use the information, it could not have been encrypted, a basic step that is required by the card companies' standards." The article goes on to quote Gartner analyst Avivah Litan: "They [MasterCard] weren't actively monitoring compliance. It wouldn't take that much to send an auditor to see if that data is encrypted or not." Neither Visa nor MasterCard require encrypting stored credit-card data. For example, here's a link to the Payment Card Industry (PCI) Security Standard on the Visa site, and a link to a simplified version on the MasterCharge site. MasterCard paraphrases PCI requirement 3 as "Protect stored transaction data. Keep transaction storage to a minimum and never store sensitive authentication data after authorization. Take precautions to make stored transaction data unreadable through encryption or some other secure and robust approach." [Emphasis added.] Following is the text of the Visa U.S.A. Cardholder Information Security Program (CISP) Frequently-Asked Questions #7: "7. Are there alternatives to encrypting stored data? Stored cardholder data should be rendered unreadable according to requirement 3 of the PCI Security Audit Procedures document. If encryption, truncation, or another comparable approach cannot be used, encryption options should continue to be investigated as the technology is rapidly evolving. In the interim, while encryption solutions are being investigated, stored data must be strongly protected by compensating controls. These compensating controls should be considered as part of the compliance validation process. [Emphasis added.] An example of compensating controls for encryption of stored data is complex network segmentation that may include the following: • Internal firewalls that specifically protect the database • TCP wrappers or firewall on the database to specifically limit who can connect to the database • Separation of the corporate internal network on a different network segment from production, fire- walled away from database servers." MasterCard's Electronic Commerce Best Practices for Acquirers classifies encryption of stored data as a "best practice," not a requirement. (The original title of this document was "Electronic Commerce Requirements and Best Practices for Acquirers.") MasterCard processors (a.k.a. "acquirers") "themselves do not need to go through the SDP compliance process but they must manage the SDP process for their merchants and service providers." [SDP is an abbreviation for MasterCard's Site Data Protection process for merchants.] A TransactionWorld page compares Visa's CISP and MasterCharge's SDP as of February 2004. [Note that an issuer whose account data was exposed to possible compromise as a result of this event has a right to claim reimbursement for costs related to reissuance of cards and monitoring of potentially compromised accounts that remain open. For any given account, the issuer may request reimbursement of up to US$25.00 for each reissued card, or up to US$5.00 for each monitored account without reissuance. In theory, CardSystems could be liable for as much as US$1 billion in issuer charges if all 40 million cards were reissued.] End of June 18, 2005 Update.

Update, July 1, 2005: Infoworld's Ephraim Schwarz concludes that Visa and MasterCard weren't enforcing their own security requirements. "Frank Smith, vice president of the technology strategy group at Capgemini, said, 'They don’t supply due diligence to the whole system.' Gartner’s [Avivah] Litan said, 'They have everything in place; they just don’t enforce it.' Paul Stamp, security analyst at Forrester Research, said 'The processes were not properly enforced.' [John] Pescatore [a Gartner security analyst] said that the standards 'have been pure eyewash. No enforcement.' End of July 1, 2005 Update:

June 20, 2005 Update: CardSystems' CEO John Perry admitted to the New York Times (free registration required) that his company should not have stored the data that was compromised. CardSystems was using the stored transaction data for "research purposes." The Times article states: "Under rules established by Visa and MasterCard, processors are not allowed to retain cardholder information including names, account numbers, expiration dates and security codes after a transaction is handled." The article confirms that "security codes" refers to the three-digit or four-digit codes printed on the back of the credit card, which are specifically embargoed from storage by merchants or processors. End of June 20, 2005 Update.

Update July 22, 2005: Visa USA announced on July 18, 2005 that it will no longer allow CardSystems to process Visa transactions. Visa will allow banks that use CardSystems to handle merchant transactions until the end of October 2005. The New York Times also reported on July 19, 2005 that American Express will terminate its relationship with CardSystems at the end of October 2005. In congressional testimony on July 21, 2005, CardSystems' CEO John Perry said the firm faces "immediate extinction" and blamed former Cable & Wireless auditors for practices that led to the Visa termination. End of July 22, 2005 update.

H.R.1653 (a.k.a. "Safeguarding Americans From Exporting Identification Data Act" or the "SAFE-ID Act") is entitled "To prohibit the transfer of personal information to any person outside the United States, without notice and consent, and for other purposes." H.R.1653 expands S.768's list of sensitive personal information to include the following:

• Name
• Postal address
• Financial information
• Medical records
• Date of birth
• Phone number
• E-mail address
• Social Security number
• Mother's maiden name
• Password
• State identification information
• Driver's license number
• Personal tax information
• Any consumer transactional or experiential information relating to the person

Consumers must take explicit action ("opt-out") to prevent U.S. firms possessing the personal information from transmitting it to "any foreign affiliate or subcontractor located in a country that is a country with adequate privacy protection." Opt-in is required for countries that don't have "adequate privacy protection." Most privacy advocates would place the U.S. in the latter category. After the CardSystems debacle, ordinary U.S. citizens undboubtedly would do the same. However, it's not so clear that any U.S. government agency would have the courage to so categorize this country. Violation of the Act would be treated as a violation of a rule defining an unfair or deceptive act or practice prescribed under section 18(a)(1)(B) of the Federal Trade Commission Act (15 U.S.C. 57a(a)(1)(B)). Persons could bring a state court action "to recover for actual monetary loss from such a violation, or to receive $10,000 in damages for each such violation, whichever is greater." Related—but more restricted—proposed acts would "regulate information brokers and protect individual rights with respect to personally identifiable information" (H.R.1080 and S.500) and "strengthen the authority of the Federal Government to protect individuals from certain acts and practices in the sale and purchase of Social Security numbers" (H.R.1078). H.R. 1080 would permit persons to bring a state court action "to recover for actual monetary loss from such a violation, or to receive $1,000 in damages for each such violation, whichever is greater." H.R. 1080 doesn't designate a default damage amount. The probability of enactment of any of these proposals in the current congress is low, at best. Few proposed consumer privacy acts have significant bi-partisan support (read "Republican cosponsors"). Heavy-handed and well-funded lobbyists will attempt to kill the proposals in committee. But grass-roots concern with identity theft is growing, so there's hope for some future form of federal protection for "personally identifiable information" beyond that currently provided by HIPAA. Update: June 18, 2005: The Gramm-Leach-Bliley act of 1999 (15 U. S.C. § 6801 et seq., GLBA) includes provisions that are purported to protect consumers' non-public personally identifiable financial information (NPI) by restricting its transfer from financial institutions to non-affiliated third parties. Currently, GLBA applies only to financial institutions that provide services to consumers, such as Visa and MasterCard—but not processors/acquirers like CardSystems. What's worse, consumers must proactively (affirmatively) opt-out of the third-party information-sharing process. Adoption of the "opt-out" method (versus the more commonly accepted "opt-in" approach that applies to health-related information) was the subject of an intense lobbying campaign by the financial industry in general and credit-card issuers in particular. It's not known how many, if any, of the holders of the 40 million compromised credit cards had exercised their "opt-out" rights. A relatively simple (but very unlikely) method of minimizing exposure of NPI to third parties is amendment of GLBA to change "opt-out" to "opt-in" and include credit-card processors and any other organizations in the processing chain as "financial institutions." End of June 18, 2005 Update. --rj P.S. Bruce Schneier points out in his recent "U.S. Medical Privacy Law Gutted" post that a new ruling by the U.S. Justice Department "sharply limits the government's ability to prosecute people for criminal violations" of the HIPAA privacy regulations. Criminal penalties, the department said, apply to insurers, doctors, hospitals and other providers—but not necessarily their employees or outsiders who steal personal health data.

A Business Intelligence Demo with Real-World Data

The Gartner Group attributes much of the 10.3 percent growth of the total relational database market from 2003 to 2004 to new business intelligence features—data analysis, warehousing and reporting—as well as the weakening US dollar. Gartner's May 2005 "No Clear Winner in Overall RDBMS Market Share Race" report gave IBM 34.1%, Oracle 33.7%, and Microsoft 20% of the total market. Teradata, Sybase, and others claimed 2.9%, 2.3% and 6.6%, respectively. Microsoft's market share increase from 18.7% in 2003 is surprising when you consider that prospective licensees were then anticipating Yukon to release to manufacturing in mid-2004, not November 2005. eWeek magazine's Lisa Vass quotes Microsoft director of SQL Server product management, Tom Rizzo: "BI is a tremendous growth driver for us, especially Reporting Services, which we've seen a ton of customers buying and deploying. That's why we invested so heavily in BI technologies across SQL Server. ... We put a down payment many years ago, and now it's paying off in terms of revenue growth." Unlike its major database competitors, Microsoft includes business intelligence integration, development, reporting and management features in the basic license fee for all editions of SQL Server 2000 (except MSDE) and SQL Server 2005 (except Express and Workgroup editions, which do include Reporting Services.) SQL Server 2005 has a raft of new and improved business intelligence (BI) features. But the standard sample online transaction processing (OLTP) and data warehouse (DW) databases are based on the relational AdventureWorks sample database. Demonstrating the performance of Integration Services (SSIS, formerly Data Transformation Services. DTS) extract, transformation, and loading (ETL) features isn't practical with tables that contain only a few hundred or thousand rows. You need partitions containing multimillion-row dimension and fact tables to emulate the BI systems of, for example, nationwide or multi-national retailers. The table-size issue with performance testing is similar to that I describe in my FTP Online articles about SQL Server 2005's xml data type and data encryption features. Microsoft's Project REAL is "a reference implementation of a business intelligence (BI) system using real large-scale data from a real customer." Phase 1 of the project used source data from a large electronics retailer. Phase 2's "real customer" is Barnes & Noble, the largest U.S. bookseller, who contributed the masked source data and BI scenario. Barnes & Noble have about 40,000 employees and 800 stores in the U.S.

According to the Technical Overview for Phase 2, Project REAL's goal "is to discover the best practices for creating BI systems with SQL Server 2005 and to build a system that exhibits as many of those best practices as we can. This project is not just a demo—we are creating this system for ongoing operation. It is a complete system, including daily incremental updates of the data, large multiuser workloads, and system monitoring."

The Technical Overview is the first in a promised series of articles that will be based on the B&N source data and analytical model. Hopefully, the Project REAL team will provide dynamic, online demonstrations of simulated ad hoc and preprogrammed BI reports. Making Project REAL's warehoused data and Reporting Services accessible to developers by Web service methods similar to those for TerraServer or MapPoint maps would be a major SQL Server 2005 marketing coup.

Hey, Tom Rizzo—are you listening?

--rj

Microsoft Promises RSS 2.0 Support in Longhorn

Steve Ballmer reportedly considers RSS "a little too simple," but damnation by faint praise by Microsoft's CEO hasn't prevented the Longhorn team from climbing on the RSS bandwagon. Dean Hachamovitch, general manager for Longhorn browsing and RSS announced in a June 24, 2005 keynote speech at the Gnomedex 5.0 conference that Longhorn will include support for RSS with a Common RSS Feed List and Data Store, plus an RSS Platform Feed Engine. Hachamovitch also proposed a specification for an RSS extension to support ordered lists. Despite its release under a Creative Commons Share-Alike license, the "Simple List Extensions Specification" generated an extraordinary amount of third-party blogging and comment activity, much of which was uninformed. The next day, a Longhorn RSS Team Blog opened with a post from Sean Lyndersay, senior program manager on Microsoft's RSS team and majordomo of the MSDN IEBlog. Developer feedback is handled by Channel9 Longhorn RSS and SimpleListExtensions WIKIs. Unfortunately, the RSS in Longhorn announcement occurred about three weeks after the publication date of my "Longhorn Redux" FTPOnline article, which concentrated on consumer features and IIS 7.0. It's good to see a feature added to—instead of removed or borrowed from—Longhorn for a change. --rj

Article: ChoicePoint and "Garbage In, Garbage Out"

Baseline magazine's Web site offers a full-length feature article, "ChoicePoint: Blur," which investigates inaccuracies in sensitive personal information distributed by data brokers, in this case ChoicePoint of Alpharetta, Georgia. ChoicePoint sold personal information—such as addresses, Social Security and driver's license numbers, and birth dates and locations—of about 145,000 individuals to Nigerian criminals posing as legitimate businesses. [Update 1/26/2006] The U.S. Federal Trade Commission (FTC) has levied a US$10 million fine on ChoicePoint for violations of the Fair Credit Reporting Act (FCRA). The FTC also expects ChoicePoint to establish a US$5 million "trust fund for individuals who might have become victims of identity theft as a result of the breach." The focus of the Baseline article however, isn't identity theft. Instead, the authors concentrate on the erroneous data that ChoicePoint supplies to its customers—legitimate or otherwise—and the company's failure to even attempt to verify the accuracy of third-party personal data it acquires and publishes. The Electronic Privacy Information Center (EPIC) has been pursuing details of government use of CheckPoint data since filing a Freedom of Information Act request in 2001. EPIC wants the FTC to regulate all CheckPoint data that contains sensitive personal information under the Fair Credit Reporting Act (FCRA), which gives consumers the right to view their records and correct erroneous information. Bruce Schneier weighs in with an observation that persons whose personal information was improperly disclosed by ChoicePoint to identity thieves would not have occurred were it not for California's S.B. 1386. S.B. 1386, the California Information Practices Act, requires data brokers and other organizations to report improper disclosure of unencrypted personal data on California residents. --rj P.S. eWeek magazine's "Garbage In, Garbage Out of Control" article by Linda Voss raises the issue of users drawing incorrect conclusions from bad data, erroneous correlations, improper analytics, flawed visualizations, or all four. If the users are local law-enforcement personnel, as was the case for two examples that the Baseline article describes, the military, or federal anti-terrorist agencies, incorrect conclusions drawn from database applications can have serious consequences.

Tech*Ed 2005 Show Daily Articles from FTPOnline

If you subscribe to Fawcette Technical Publications' .NETInsight newsletter, you've received three "Show Daily" newsletters that contain links to news from and analyses of Tech*Ed 2005 keynote speeches and breakout sessions. If not, following are links to articles from the three "Show Daily" newsletters, and here's a link to the Preferences page where you can sign up to receive any or all of the FTP newsletters. Day One - Monday Visual Studio 2005 to Launch November 7 Microsoft VP Paul Flessner said today at Tech*Ed that Visual Studio 2005, SQL Server 2005, and BizTalk Server 2006 will launch internationally in a series of events the week of November 7. .NET 2.0 Gets Faster The .NET 2.0 beta outperforms .NET 1.1 by 20 to 40 percent, said Microsoft CEO Steve Ballmer during his opening keynote at this week's Tech*Ed conference. Longhorn Redux The next version of Windows will differ dramatically from its PDC 2003 preview version. Roger Jennings gauges the impact that Longhorn and its back-ported Avalon and Indigo components will have on VS developers. Ballmer: "Look out, Rational." Microsoft CEO declares that Visual Studio 2005 is nearly ready and that its Team System will challenge IBM/Rational head on, says Jim Fawcette in his blog. Where Microsoft Stands With Security Microsoft shows its commitment to security with recent releases such as Windows Server Update Services (WSUS) and Service Pack 1 for Windows Server 2003 with its Security Configuration Wizard. Ballmer Touts .NET Adoption, Office Dev Tools In his Tech*Ed keynote, Microsoft CEO Steve Ballmer quoted the obligatory .NET momentum statistics, but left some questions unanswered, says Jeff Hadfield. Day Two - Tuesday Revisiting Whidbey, Yukon, and Beyond This week's Tech*Ed is in many ways a major milestone for Microsoft's product strategy, especially for its server and tools products. Peter O'Kelly provides an overview and analysis of the latest developments. SQL Server June 2005 CTP Now Available The newest SQL Server 2005 preview version, the June Community Technical Preview, is now available for download. This CTP is the first preview version to be publicly available. Put the 'Smart' in Smart Client Building applications for multiple platforms requires more consideration from developers than ever before. Find out how to make your apps "smarter." Is C# the Only Language that Matters? One promise of .NET has always been that the language you use is up to you. It's a nice theory, says Patrick Meader, but the perception in many circles is that C# is the only language that matters. Ballmer: .NET Outperforms WebSphere, Again .NET 2.0 ups the stakes as Ballmer claims 200 percent better performance than WebSphere, says Jim Fawcette in a blog post. Infrastructure Takes Center Stage Building a solid IT infrastructure is essential to maintaining communications in your organization. Here's a sampling of key products that vendors are showing off at Tech*Ed 2005. Day Three - Wednesday Yukon Tempts Database Developers Long-awaited SQL Server 2005 offers Visual Studio 2005 developers more than just CLR integration. ADO.NET 2.0 opens the door to native data encryption, a new XML data type, and many added T-SQL features. CLR in SQL Server Actually Has a Benefit! During his Tech*Ed keynote, Microsoft VP Paul Flessner briefly mentioned a benefit of having CLR support built into SQL Server 2005, says Jim Fawcette. Connect Systems With Indigo Tech*Ed's Connected Systems track presentations propose the forthcoming Indigo messaging bus as the service-oriented successor to ASMX Web services, Web Services Enhancements (WSE), .NET Remoting, MSMQ, COM+, and, lest we forget, Global XML Architecture (GXA). New Dev Tools Integrate With VS 2005 Vendors at Tech*Ed showcase new development tools, including products that help you build database apps, enhance your app's presentation layer, design Web apps more easily, and more. Admin Benefits of SQL Server 2005 Any database or system administrator will be glad to move on from SQL Server 2000, say Danielle Ruest and Nelson Ruest. Get the scoop on some of the improvements in SQL Server 2005. --rj

Powerful Stuff: WS-*, Single Sign-On, and InfoCard

It's risky to speculate on the nature of the "powerful stuff" that Steve Ballmer mentioned during a Q&A session at the recent TiEcon 2005 conference:

"We are working on more existing powerful stuff around XML Web services that will address many issues beyond RSS."

However, a likely "powerful stuff" candidate is Microsoft's InfoCard initiative for personal digital identity management and Web-based single-sign-on (SSO) in the forthcoming Windows Longhorn client OS. The first public demonstration of InfoCard occurred in May at the Digital ID World 2005 conference in San Francisco. Microsoft more or less simultaneounsly published Kim Cameron's "The Laws of Identity" white paper and a more extensive "Microsoft's Vision for an Identity Metasystem" article. (Kim Cameron is Microsoft's Identity and Access Architect, and publishes the Identity Weblog). The "Pre-Release Software Code Named “Avalon” and “Indigo” Beta1 RC" download, which appeared on May 23, 2005, runs on Windows XP SP-2 or Windows 2003 Server. This download provides the Indigo runtime infrastrastructure for InfoCard Beta 1. (The only references to InfoCard Beta 1appear in the press release and the main Longhorn Developer page's link to the RC, which also has a link to the Release Notes.) Running the Indigo setup program installs the .NET Framework 2.0 April CTP (Beta 2) version. You also can download and install an updated WinFX SDK as an ISO image from a link on the download page. The runtime and SDK compatible with Visual Studio 2005 Beta 2. Johannes Ernst's Blog provides an independent overview of InfoCard and describes its reliance on the WS-* stack. According to Johannes, InfoCard employs the following WS-* members and related specs:

• SOAP [1.2]
• WS-Addressing
• WS-MetadataExchange
• WS-Policy
• WS-Security
• WS-SecurityPolicy
• WS-Transfer
• WS-Trust
• XML Signature
• XML Encryption
• SAML
• WS-Federation (?, unclear)

[Note that Indigo bindings for WS-* support use SOAP 1.2, which results in Web services that don't meet WS-I Basic Profile 1.1. As Tim Ewald observes, many organizations require that all Web services they publish or consume to claim BP-1.1 conformance.] If processing InfoCard identities requires implementation of the eight WS-* specs from the above list, support for SAML, and the Indigo messaging infrastructure, is InfoCard destined for HailStorm's fate? At this point, only WS-Security is an official OASIS specification; the remaining members are at varying points in the standards process. So far, InfoCard appears to me to be another example of the overly complex "everything at once" syndrome that doomed HailStorm. The preceding Indigo and InfoCard Beta 1 RC release followed a May 13, 2005 joint publication by Microsoft and Sun Micrososystems of the Web Single Sign-On Interoperability Profile and Web Single Sign-On Metadata Exchange Protocol (WSSOMEX) specifications. These specs provide a mechanism for integrating WS-* and Liberty Alliance identity management of Web-based single sign-on technologies. WSSOMEX represents Sun's first—if tentative—committment to the WS-* standards beyond WS-Security. The press release, transcript of remarks by Steve Ballmer and Scott McNealy's comments, and related links are here. WSSOMEX is the first concrete result of the 10-year Sun-Microsoft technical collaboration agreement of April 2004. Paul Madsen posted an early analysis of WSSOMEX and WS-MetadataExchange:

WSSOME[X] defines how WS-MetadataExchange can be used to determine which Single Sign-On protocol suites (SAML 1.1, ID-FF 1.2, SAML 2.0, WS-Federation, etc) your partner is capable of supporting so that the two of you can actually do something interesting (like enabling SSO for your customers, employees, etc). WS-MetadataExchange defines a SOAP-based request/response protocol. Fundamentally, one provider says to the other 'tell me what you can do'. If the returned list includes something that the asking provider can also [do], then we have an intersection of capabilities and we're off to the races. If [there's] no intersection, [there's] no way forward.

Sun's Hubert Le Van Gong posted a response to Paul's post and added his own initial InfoCard analysis and a follow-up in response to Kim Cameron's comments. InfoWorld's Jon Udell also weighed in with a post about Web single-sign-on with client-side certificates, a much simpler technology that never caught on, versus InfoCard. In fairness to InfoCard, the Liberty Alliance lists a large number of "Liberty-Enabled Products," but, according to Web service analyst Ron Schmelzer, "[T]here are still very few products, if any, that implement Liberty Alliance on the desktop client, and so Microsoft has a distinct advantage."

RSS and Web Services: Keep It Simple, Steve.

Amit Malhotra's May 19, 2005 post on therssweblog includes a transcript of his unscheduled Q&A session with Microsoft's Steve Ballmer at the TiEcon 2005 conference in Santa Clara:

Q1. How important is RSS? A fad, important, huge or will [it] replace the Web/HTML dominance of the internet? A1. We believe RSS is important and will be around for a while, but it is not going to change the world. It is a little too simple; that is also the reason everyone’s using it. We are working on more existing powerful stuff around XML Web services ... that will address many issues beyond RSS. RSS will be around, but whatever we are working next will be cooler and more prevelant. [Emphasis added] Having said that, there are groups in MS that believe RSS has the potential to change everything and many future technolog[ies] will be built around RSS, the internal debate goes on.

Q2. How do you, or do you, see Google/Blogger and similar tools being a threat to MS Office dominance? A2. Not at all, people will always need Office for the complexity of tasks they perform and, as such, Google’s offerings in the strain of Gmail/Blogger will not replace it. We think it will be part of what we offer in future versions of Office. Besides the next release of MS Office will have the tools to publish blogs as part of its collaborative tools; watch for them.

Amit's preceding transcript, to which I've made minor edits and emphasized text, generated a brief flurry of reaction by other well-known bloggers, including Microsoft's Robert Scoble, who left a comment to a related posting on Steve Rubel's Micropersuasion PR blog. A reader named Bud left the following comment that's a propos my preceding post on the issue of overcomplicating Web services and SOA:

When I talk to people who are doing web services and show them RSS, they say something like, "Hey, we could achieve a heck of a lot with just that. We should just implement it. Web services is so complicated and requires so much effort just to work." And, these are fairly sophisticated developers and architects. Simple, easy to implement technologies where developers can quickly do it and people can immediately see the value are going to win the day. That's things like AJAX as well as RSS.

Steve Ballmer's first answer indicates that Microsoft has a Web service-based replacement in mind for RSS that has a higher "coolness quotient," but I question whether any Web service API or toolkit can come close to competing with RSS's current prevelence in Web developer mindshare. [The assumption is that preceding references to RSS also encompass ATOM.] My bookshelf (as well as Don Box's) has a copy of the .NET My Services Specification (517 pp.) from the 2001 Microsoft Professional Developer Conference (PDC). .NET My Services, better known by its "HailStorm" codename, was intended to evelope XML-based identity, address-book, Web site favorites, calendar, travel and much other personal data with a digital wallet in a Web services wrapper. The essence of HailStorm, in the words of Mark Lucovsky, then Microsoft's Distinguished Engineer and Chief Software Architect of .NET My Services, was:

HailStorm embraced the idea of decoupling the data from the application. The idea was to allow a variety of applications to process and manipulate your calendar data, your address book, your email, your favorite web sites, your travel preferences and itineraries, etc. This is not a new, novel idea, but was certainly something that was important and core to the system. Simple applications that we were trying to enable included the ability to overlay your personal calendar with the calendar of your favorite band, or your favorite sports team, or your spouse, etc. We wanted to enable a unified "address book" where your contacts could be used across applications written by any vendor.

The HailStorm announcement aroused a call-to-arms among Microsoft's competitors to prevent Passport from becoming the world's default identity management and authentication protocol. The Liberty Alliance, championed by Sun Microsystems, proposed a "federated identity management system" and successfully thwarted serious consideration of Passport and HailStorm technology by IT management. Another issue with HailStorm was that client applications weren't really "simple." For example, the introductory chapter for the .NET Calendar service specificaton was 68 pages long. Lucovsky subsequently moved on to Google and recently posted a comparison of the core HailStorm concepts with those of RSS 2.0 and Atom. Mark cites the common core concepts as:

• Network Centric, Extensible Data Model, for Everyday Data
• Data Decoupled from Applications
• Anytime, Anyplace, and from Any Device Access
• Identity Centric Data Access

A major difference I see is that HailStorm proposed to standardize a wide range of data structures and methods for reading and updating personal information. HailStorm relied on the proprietary .NET Passport service (now "Passport Network") for authentication, which Microsoft's Kim Cameron now admits is out of context for authentication by non-Microsoft sites. As one commentor observed, HailStorm proposed to do "everything at once." RSS 2.0 and ATOM concentrate on public content sydication and have proven very successful at their assigned tasks, while only the successor of .NET My Alerts—SQL Server Notification Services—has achieved any semblance of industry adoption.

Microsoft Web Services DevChannel Opens at FTPOnline

Fawcette Technical Publications recently added a new Microsoft Web Services DevChannel to the FTPOnline portals collection. The DevChannel consists of links to Microsoft white papers, artlcles from FTP print and online publications, video clips from FTP conferences and MSDN TV, plus related resources. Service-Oriented Architecture (SOA) gets the primary emphasis, as expected, but many articles include sample C# and VB .NET Web service code. My "Build Real-Time Web Images" article from the August 2004 issue of Visual Studio Magazine was the lead link for the initial DevChannel page. The article describes how to write VB. NET 2003 Windows form clients for Microsoft's SQL Server-driven TerraService and MapPoint Web services. TerraService is an example of a set of freely-accessible Web services that enable .NET developers to display tiled aerial/satellite photographic images of most of the earth, and USGS topographic map images of North America. MapPoint Web services render bitmaps from vector-based maps of the U.S. and many other countries. Using MapPoint Web services requires obtaining a commercial or developer license and executing a simple username/password authentication request prior to invoking the desired Web method.

Update 6/14/2005: InfoWorld's Jon Udell commented on developers use of GoogleMaps versus TerraServer:

"Years ago an early reviewer of Visual Basic 1.0 (Steve Gibson, I think) said that VB increased the software developer's leverage by an order of magnitude. That was true, but you can't keep going back to the same well. In a column on Google Maps I wrote:

Developers haven't flocked to TerraServer. What's Google's secret? Web DNA and no Windows tax. Responding in email, Jim Gray reminded me that TerraServer does offer SOAP interfaces [1, 2]. And yet those interfaces demonstrably have not inspired a flurry of innovation. Why not? Microsoft is obliged to portray the Web-based user experience as a dead end that can never be improved, and Windows as the only way forward. So it's going to be Visual Basic and client/server all over again: Windows applications will control the user experience; servers will dish out the data; developers will connect the dots."

For a preview of the TerraService and MapPoint Web service client projects, check out the illustrated online help page on the OakLeaf Web site.

Like OakLeaf's public CFR Web services, the TerraService and MapPoint Web services take full advantage of Visual Studio's Add Web Reference Wizard to autogenerate .NET 1.0+ C# or VB .NET Web service proxy classes. Simplified programming of data-driven Web services and the capability to autogenerate Web service client proxy classes were Microsoft's primary marketing topics for .NET 1.0. VS 2005's new built-in Web server simplifies development and publication of basic SQAP 1.1 and 1.2 Web services, but offers few other Web service enhancements. SQL Server 2005 boasts the capability to host native (in-process) SOAP Web services that substitute Windows Server 2003's and Windows XP's built-in HTTP.sys driver for IIS and VS-generated .asmx files. Promotion of the WS-Security and other WS-* specifications, Web Services Enhancements (WSE) 2.0+, and the forthcoming Indigo message bus appears to have increased the FUD factor surrounding SOA in general and Web services in particular. This uncertainty has reduced IT management and developer interest in implementing basic .NET and Java Web services as the first step in demonstrating the practicality of enterprise-level SOA to corporate and line-of-business management. Perhaps the Web services developer community would be better served by more publicly accessible, data-intensive, real-world Web services and fully implemented client examples, such as the TerraService and MapPoint projects, rather than white papers that describe esoteric SOAP headers and emerging messaging "standards." RSS 2.0 and ATOM adhere to the Keep It Simple, Stupid (KISS) principal, which is the secret to their current success in content syndication. --rj