The Xanadu Dream

Links are the fundamental building blocks of the web. And every time I click on one, I can't help recalling the odd visionary who came up with the original idea of clickable links in text, aka hypertext, in 1963 -- Ted Nelson.

Ted Nelson is, shall we say, a character. He has gone on record many times with the four maxims that guide his life. He isn't shy about sharing them with anyone he meets:

1. most people are fools
2. most authority is malignant
3. God does not exist
4. everything is wrong

And that, in a nutshell, is pretty much everything you need to know about Ted Nelson. He is the archetypal borderline autistic, non-conformist, free-thinking technologist. Any resemblance between Ted and your average programmer is, I'm sure, completely coincidental. That's why his story is so fascinating to me. It hits close to home.

Like most programmers, Ted's reach often exceeded his grasp. Ted's vision of hypertext was far more grandiose than the motley assortment of links that is today's web. His vision was (and is) a bit of computer history lore that has become the stuff of legend: Project Xanadu.

Xanadu, a global hypertext publishing system, is the longest-running vaporware story in the history of the computer industry. It has been in development for more than 30 years. This long gestation period may not put it in the same category as the Great Wall of China, which was under construction for most of the 16th century and still failed to foil invaders, but, given the relative youth of commercial computing, Xanadu has set a record of futility that will be difficult for other companies to surpass. The fact that Nelson has had only since about 1960 to build his reputation as the king of unsuccessful software development makes Xanadu interesting for another reason: the project's failure (or, viewed more optimistically, its long-delayed success) coincides almost exactly with the birth of hacker culture. Xanadu's manic and highly publicized swerves from triumph to bankruptcy show a side of hackerdom that is as important, perhaps, as tales of billion-dollar companies born in garages.

Among people who consider themselves insiders, Nelson's Xanadu is sometimes treated as a joke, but this is superficial. Nelson's writing and presentations inspired some of the most visionary computer programmers, managers, and executives - including Autodesk Inc. founder John Walker - to pour millions of dollars and years of effort into the project. Xanadu was meant to be a universal library, a worldwide hypertext publishing tool, a system to resolve copyright disputes, and a meritocratic forum for discussion and debate. By putting all information within reach of all people, Xanadu was meant to eliminate scientific ignorance and cure political misunderstandings. And, on the very hackerish assumption that global catastrophes are caused by ignorance, stupidity, and communication failures, Xanadu was supposed to save the world.

The above text is excerpted from the definitive 1995 Wired article on Project Xanadu, which is still as electrifying to read today as it was then. The hubris and sheer scale of the Xanadu dream are at turns both inspiring and desperately, hopelessly out of touch.

Xanadu has 17 rules defining its behavior. As you're reading through this list, ask yourself how many of these rules are satisfied by the current world wide web, even in part:

1. Every Xanadu server is uniquely and securely identified.
2. Every Xanadu server can be operated independently or in a network.
3. Every user is uniquely and securely identified.
4. Every user can search, retrieve, create and store documents.
5. Every document can consist of any number of parts each of which may be of any data type.
6. Every document can contain links of any type including virtual copies ("transclusions") to any other document in the system accessible to its owner.
7. Links are visible and can be followed from all endpoints.
8. Permission to link to a document is explicitly granted by the act of publication.
9. Every document can contain a royalty mechanism at any desired degree of granularity to ensure payment on any portion accessed, including virtual copies ("transclusions") of all or part of the document.
10. Every document is uniquely and securely identified.
11. Every document can have secure access controls.
12. Every document can be rapidly searched, stored and retrieved without user knowledge of where it is physically stored.
13. Every document is automatically moved to physical storage appropriate to its frequency of access from any given location.
14. Every document is automatically stored redundantly to maintain availability even in case of a disaster.
15. Every Xanadu service provider can charge their users at any rate they choose for the storage, retrieval and publishing of documents.
16. Every transaction is secure and auditable only by the parties to that transaction.
17. The Xanadu client-server communication protocol is an openly published standard. Third-party software development and integration is encouraged.

It is instructive, then, to consider the primary ways in which the modern web is functionally broken:

• Link rot. The odds of a hyperlink working are inversely proportional to the age of that hyperlink. Old links frequently break, because the server hosting the content disappears for any number of reasons over time. I've gotten to the point where I dread clicking on links from old web pages, because the per-click success rate is so abysmally low.
• Every website has unique usernames and passwords. There is almost no reliable centralized form of identity on the internet, and those few that do exist are often poisoned by explicit commercial affiliation, such as Facebook Connect and Microsoft Passport. This is why curated anonymous, lightweight participation dominates the net -- best illustrated by Wikipedia.
• No redundancy. If content is driven offline by temporary high traffic levels or, worse, catastrophic data loss, there may not be any way to recover that content. I know that Digg has services which auto-mirror highly rated links because Digg traffic can be so toxic to the destination links. (Ironically, all duggmirror.com links redirect to amazon.com now, which illustrates how ephemeral all this stuff tends to be.) I suppose if you're lucky the wayback machine will eventually pick up a historical copy of the content, or the Google cache will hold a copy for some unknown amount of time while the site is offline. You'd probably have better odds praying for missing content to reappear.

Let's put aside the more ambitious parts of Project Xanadu for the moment. The current world wide web does basically one thing: simple, stupid, mindless hyperlinks. But even that alone was enough to build a functional and useful internet for the world. And Google was able to build a zillion dollar algorithm out of discovering the relationship between those dumb hyperlinks.

All that, when the most fundamental building block of the web, the hyperlink, barely works at all. Hyperlinks are fraught with peril and pitfalls even under the best of conditions. The current state of hyperlinking is almost literally the stupidest thing we could build that works. Frankly, the current system sucks beyond belief, as Ted himself notes:

HTML is precisely what we were trying to prevent -- ever-breaking links, links going outward only, quotes you can't follow to their origins, no version management, no rights management.

The next time you're about to embark on a grandiose, glorious software Xanadu Dream of your own, take Dare's advice.

The bottom line is that a lot of the time it's OK to create a solution that solves 80% of the problem. Always remember that shipping is a feature.

Consider the reality of what's actually possible, what people can understand, and what us all too human programmers can practically implement. It might not be the Xanadu you dreamed of -- heck, it might even suck -- but it'll at least have a fighting chance of existing in reality rather than fantasy.