Brewster Kahle - For the love of media and metadata! Update

A lot of valuable broadcast material is just about to thrown out in Hilversum

Important broadcast material documenting the 20th century, which has cost millions of Euros to make, is rotting in vaults. Worse still, as the economic crisis in Europe worsens, valuable collections are being destroyed. This is complete madness when, in other parts of the world, archivists like Brewster Kahle have come up with practical solutions. Of course, not everything can be preserved. But dumping the lot is destroying heritage which cannot be replaced.

The archivists on the West Coast of the US have a path to preserving heritage for next generations. And I love people who not only have big ideas, but also execute on them. Leo Laporte interviewed Brewster this past week. It's a great interview because Brewster has refined his story of why this is all so important. And he is now getting the backing of many libraries who realise that the old models are breaking down. Libraries provide access to knowledge in coherent ways. And although we often think that "everything is on the Internet", that is just not true. The US Library of Congress has a collection of 26 million pieces. The Internet Archive is collecting about 1 billion items a week. They have around 10 Petabytes (10 thousand Terabytes) which they store inside a former church.

I see the BBC's been to the archive too. There is a clip from Alan Yentob on YouTube.

Scanning books to digitize the content @the Internet Archive

Petabyes of Servers - with backups in Amsterdam it seems

Starting to preserve the physical books...

Few people know that the Internet Archive has also been taping US TV channels off the air since 2009 and coming up with a great search engine to find quotes (like they do on the Daily Show).

West Coast of Europe Connection

Recently had the privilege of talking with Brewster on his visit to Amsterdam to talk with info.nl.

Part of the Internet Archive is also backed up on Petabyte servers in Amsterdam

Brewster and the team on a recent visit to Amsterdam

Love the sticker on his MacBook. 10 Petabytes filed so far....

Call to action - Help and advice needed.

I have compiled and curated an archive of the Media Network programmes I made between 1980-2000. It is about 200 hours of audio material documenting the (inernational) broadcast media for the last half of the 20th century. We did quite a few documentaries on broadcasting before, during and after the Second World War. I am wondering how to transfer this on-line collection to the Internet Archive, together with the metadata that goes with it. I notice that that in general, people on the production side of things are awful at writing useful metadata so that others can find their great productions. That's why I rewrote the summaries. Without that material, it doesn't make sense. I don't get the 10 million downloads a month that the Internet Archive can boast, but having 5000 unique downloads a month of material which is 20 years old still shows it has some value. By all means explore. You can get in touch through www.jonathanmarks.com or About me.

Update: interesting stats from this article in the Guardian today.

Kahle, a computer scientist who made a fortune in the 1990s with tech ventures, including Alexa Internet, dreamed of a Great Library of Alexandria 2.0 since he studied at MIT. The archive's first headquarters was in the nearby Presidio district. In 2009 it moved into a former Christian Science church on Funston Avenue; its pillars and facade evoke antiquity.

About 50 staff work here and another 100 work elsewhere in the bay area and in 32 scanning centres, usually in libraries, around the world. The centres digitise books, microfilm and regular film. Automation proved imprecise so it is done manually, each worker processing 800 to 1000 pages per hour. This labour means material such as Boston's John Adams Library, the Hoover archive and the 1930 US census are now online and free. Institutions such as government agencies, libraries and universities, many outside the US, pay modest fees for special requests.

The archive has also stored 750,000 actual books at a nearby climate-controlled storage unit, a literary equivalent of the Svalbard global seed vault. There is space for another 780,000.

Engineers "crawl" the world's top million websites, capturing and storing pages which link to other pages which are captured and stored. Every three months they start over, because the list of top million sites constantly changes. An average web page lasts 75 days. In 2009, they raced against the clock to save as much as they could of the web-hosting service GeoCities, before Yahoo shut it down. If the owner of a defunct website prefers that the pages remain dead, he or she can ask the archive to remove them, requests that are almost always granted.

Engineers also collect news from more than 60 TV stations worldwide and YouTube videos, selecting the latter according to Twitter mentions. "It's not perfect but tweets give us an idea of what people consider important," said Alexis Rossi, the web collections manager. She estimated that the 10bn URLs saved each every three month cycle represented – very, very roughly – about a 10th of the internet's output:

Activating Archives for Intelligent Discussion of the Future

Found this after following a link on James Cridland's excellent blog.

What a thoughtful piece by Trevor Dann about radio archives. He gave it at the great NextRad.io conference last year, but I see it hasn't had the reach it deserves.

It's true that there seems to be paradox in radio that there is a love of playing old records but the speech in between has to be contemporary...the latest news and weather. Putting old speech clips out is seen as less important than having a fresh new statement that no-one has yet heard.

Most radio archives across Europe are in a rather sorry state. The tapes may be physically in bad condition. More often than not it;s the catalogue that is incomplete, so you can't find stuff. Then the fastest way to get something done is to the find someone and ring them up rather than digging out the context yourself.. So because it takes too much time to find things (Youtube is often faster) I see many broadcasters wondering what to do next - and whether they should be keeping all this stuff. Rooms full of tapes have been chucked out on the grounds that no-one is interested. Yet when you work out what it has cost to make this material, the cost of making a coherent collection pales by comparison.

As an aside, I remember calculating that at Radio Netherlands we were putting about 9 million Euros worth of material into the radio archive each year - without a real plan of why we doing this. I'm glad we did. The archive is infact provides access to Dutch society in foreign languages for the second part of the 20th century. Now that the RNW building is going to be cleared for new tenant, I wonder what will happen to that archive.

But back to Trevor Dann. He discusses the idea of a Peoples' Archive, where the public broadcaster (in the case the BBC) would declare an amnesty on the tapes recorded at home by listeners and announce what they were looking for to complete their own archives. There is a time element here. Cassettes and home-burned CD's don't last more than a couple of decades, especially if they are not stored under the right conditions (too hot or too much sunlight). I note that some of the CD's I made in 1995 are becoming unplayable on computer's though they still seem to work on CD players.

Trveor's point that using the archive is generally frowned upon by station management is true. Repeating a programme was often seen as a lazy way out of a scheduling problem, yet a scheduled repeat of something makes a lot of sense and audiences like it. I have been releasing time travel editions of Media Network, a show I used to produce and present on Radio Netherlands. They are left, as broadcast, so you can understand the context of what is being said. But there is clearly a market for dipping into this kind of archive and putting ideas from the past into a modern context in order to have a serious discussion about the future.

I note that the excellent series that Trevor has produced on the Sounds of the 20th Century has a page on BBC Radio 2'website - but it could be so much more.

Update: Trevor kindly responded with an update this morning: You’ll be pleased to know that the People’s Archive, now re-named The Listeners’ Archive, will be launched shortly as part of the BBC’s 90th anniversary celebrations. The announcement will be made after the Olympics.

Lessons Learned: We need to start thinking about more creative uses of archive material. History has a horrible habit of repeating itself. The media, in general, is poor at putting discussions of the past into a useful modern context. Now that search technologies are improving, creative use of archives needs a rethink.

Brilliant Attack on Vinyl

I understand that some broadcast archives are being cleaned up rather drastically (either closed or chucked out) as the economy bites and current leaders don't see the purpose in keeping their heritage. On the other hand, I thought the BBC's idea to donate a collection of is Caribbean service recordings to the University of the West Indies was a good move to preserve material that had been made with public money. I rather like what artists have been doing with vinyl in Spain, though I guess archivists will find the video too painful to watch.