Preserving The Internet... and Everything Else

April 2, 2012

In Preserving Our Digital Pre-History I nominated Jason Scott to be our generation's digital historian in residence. It looks like a few people must have agreed with me, because in March 2011, he officially became an archivist at the Internet Archive.

Jason recently invited me to visit the Internet Archive office in nearby San Francisco. The building alone is amazing; when you imagine the place where they store the entire freaking Internet, this enormous former Christian Science church seems … well, about right.

It's got a built in evangelical aura of mission, with new and old computer equipment strewn like religious totems throughout.

Doesn't it look a bit like the place where we worship servers, with Jason Scott presiding over the invisible, omnipresent online flock? It's all that and so much more. Maybe the religious context is appropriate, because I always thought the Internet Archive's mission – to create a permanent copy of every Internet page ever created, as it existed at the time – was audacious bordering on impossible. You'd need to be a true believer to even consider the possibility.

The Internet Archive is about the only ally we have in the fight against pernicious and pervasive linkrot all over the Internet. When I go back and review old Coding Horror blog entries I wrote in 2007, it's astonishing just how many of the links in those posts are now, after five years, gone. I've lost count of all the times I've used the Wayback Machine to retrieve historical Internet pages I once linked to that are now permanently offline – pages that would have otherwise been lost forever.

The Internet Archive is a service so essential that its founding is bound to be looked back on with the fondness and respect that people now have for the public libraries seeded by Andrew Carnegie a century ago … Digitized information, especially on the Internet, has such rapid turnover these days that total loss is the norm. Civilization is developing severe amnesia as a result; indeed it may have become too amnesiac already to notice the problem properly. The Internet Archive is the beginning of a cure – the beginning of complete, detailed, accessible, searchable memory for society, and not just scholars this time, but everyone.

— Stewart Brand

Without the Internet Archive, the Internet would have no memory. As the world's foremost expert on backups I cannot emphasize enough how significant the Internet Archive is to the world, to any average citizen of the Internet who needs to source an old hyperlink. Yes, maybe it is just the world's largest and most open hard drive, but nobody else is doing this important work that I know of.

Let's Archive Atoms, Too

While what I wrote above is in no way untrue, it is only a small part of the Internet Archive's mission today. Where I always thought of the Internet Archive as, well, an archive of the bits on the Internet, they have long since broadened the scope of their efforts to include stuff made of filthy, dirty, nasty atoms. Stuff that was never on the Internet in the first place.

The Internet Archive isn't merely archiving the Internet any more, they are attempting to archive everything.

• Open Library – one web page for every book ever published.
• Live Music Archive – every live concert ever recorded, free for non-commercial use.
• Physical Archive – one physical copy of every book ever published.
• Moving Image Archive – every free movie, film, or video ever recorded.
• Text Archive – with 1.6 million scanned books already online, and a thousand more scanned in every day.

All of this, in addition to boring mundane stuff like taking snapshots of the entire Internet every so often. That's going to take, uh … a lot of hard drives. I snapped a picture of a giant pile of 3 TB drives waiting to be installed in one of the storage rooms.

The Internet Archive is a big organization now, with 30 employees in the main San Francisco office you're seeing above, and 200 staff all over the world. With a mission of such overwhelming scope and scale, they're going to need all the help they can get.

The Internet Archive Needs You

The Internet Archive is a non-profit organization, so you could certainly donate money. If your company does charitable donations and cares at all about the Internet, or free online access to human knowledge, I'd strongly encourage them to donate to the Internet Archive as well. I made sure that Stack Exchange donated every year.

But more than money, what the Internet Archive needs these days is … your stuff. I'll let Jason explain exactly what he's looking for:

I'm trying to acquire as much in the way of obscure video, obscure magazines, unusual pamphlets and printed items of a computer nature or even of things like sci-fi, zines – anything that wouldn't normally find itself inside most libraries. Hence my computer magazines collection – tens of thousands of issues in there. I'd love to get my hands on more.

Also as mentioned, I love, love, love shareware CDs. Those are the most bang for the buck with regards to data and history that I want to get my hands on.

Being the obsessiveconscientious geeks that I know you are, I bet you have a collection of geeky stuff exactly like that somewhere in your home. If so, the best way you can help is to send it in as a contribution! Email jscott@archive.org about what you have, and if you're worried about rejection, don't be:

There's seriously nothing we don't want. I don't question. I take it in, I put it in items. I am voracious. Omnivorous. I don't say no.

The Internet Archive has an impossible mission on an immense scale. It is an unprecedented kind of open source archiving, not driven by Google or Microsoft or some other commercial entity with ulterior motives, but a non-profit organization motivated by nothing more than the obvious common good of building a massive digital Library of Alexandria to preserve our history for future generations. Let's do our part to help support the important work of the Internet Archive in whatever way we can.