Compatibility: the challenge for digital archiving

'5 1/4 floppy disk' by Rae Allen
under a CC license

Today I've spent a good couple of hours in migrating some 15-year old e-mails of mine from a legacy e-mail client to Thunderbird. It wasn't a difficult process but it did need a bit of research to figure the steps needed to do the job and to try a couple of suggested alternatives. Last week I had a (shorter) adventure in getting text from Word 2.0 and Wordperfect documents. Maybe 15 or 20 years is too much time for the digital world but that won't stop me from mentioning - again - the challenges of forward/backwards compatibility in media and formats.

(Sigh)


Whoever has been using a computer for a fair amount of time is probably aware of the advice to backup their data. He/she may not be actually following that or may not even know how to do it but he/she is very likely to have heard the advice.

There are plenty of reasons to backup one's data. The main one, of course, is security against data loss due to:

• hardware failure (e.g., hard drive damage)
• disaster of any kind
• user error (e.g., file deleted + trash can emptied + free space wiped, file overwritten, etc.)
• malicious act (e.g., file destroyed by malware of any kind), etc.

For the enterprise environment, backup is (supposed to be) a must. In certain countries, the backup of specific corporate data is mandated by law. Regardless of that, corporate backup tends to be more comprehensive, maintaining data versions, multiple copies, distribution of copies across different media and locations, ideally both on-site and off-site, etc.

Corporations that depend on their data or need to keep a digital archive, inevitably, have dedicated infrastructure and people to take care of their backup needs.

Individuals, though, normally have much less. Yes, there is plenty of software that can take backups both free and commercial. Also, most OSes have some kind of in-house backup-restore utility. However, their user-friendliness and their compatibility across different platforms or, even, major OS versions is not guaranteed.

Even if a user chooses to stick to the same backup solution (which could be something as simple as a plain file copy from one disk to another) there is the challenge of the medium suitability and durability. Anyone who has been using a PC for more than 10 years is likely to have used floppy disks and/or ZIP drives and/or CDs and/or DVDs and/or external hard drives and/or flash drives for their temporary or long term backup. The problem is that some of the aforementioned media are not readily supported by a modern PC, e.g., modern PCs have neither 5¼'' drives to read the old floppies, nor parallel ports to support the original ZIP drives.

In order to be on the safe side, a user keen on archiving should, from time-to-time, migrate data from one medium to another. This is a very tedious tasks, especially if a large number of storage media is involved but let's assume that it is reasonably feasible.

The ultimate challenge is compatibility across file formats and program versions. Common formats that adhere to widespread standards are normally on the clear. Image files, for instance, such as JPEG or GIF or BMP have a long history, so files created decades ago will be displayed by virtually all modern software. The opposite doesn't necessarily apply, i.e., newer versions isn't possible to be displayed by legacy software. When it comes to formats for files not-so-frequently exchanged, however, compatibility may be an issue. Take e-mail files, for instance. Different e-mail clients tend to store e-mail in different structures. Nowadays, where e-mail clients are part of the OS, things tend to be clearer, though a few years ago there was considerably higher fragmentation (e.g., different format for Eudora, Netscape/Unix, Outlook express, Outlook, Pegasus mail, etc.). In fact, today, a large portion of our e-mail stays in the cloud, which sort of solves the compatibility problem, although it introduces a different set of challenges.

Is there a bottom line to this? Well, not really. If one needs to have data from the past, one needs to either maintain legacy hardware and software (which may or may not be possible) or put the effort to migrate the data to newer formats and media. It sounds deceivingly simple, doesn't it?

(The following video is a talk of Chad Fowler from a Scala days conference regarding 'Legacy' in software development - it is a long, not well-lit, but interesting presentation.)

What is the must-have knowledge and how does one protect it?

'Book stack' by ginny
under a CC license

There was that Slashdot article on survival knowledge that got me thinking again. The question is: (i) what is the minimum knowledge that human civilisation should have in order to kick-start itself after a catastrophic event of some sort and (ii) how does one effectively preserve that knowledge?

There is even a book on 'rebooting civilization' and I'm sure that there plenty more works on that question, which - by the way - is not at all uncommon.

The second part of the question seems simpler. There is no ultimate backup medium and we already know that the internet is no safe bet. Modern technology is good and sleek but it can fail, too (plenty of personal experience on that front). So, really, modern non-magnetic storage media, such as DVD's, seem like a decent backup solution but they haven't been tested against time, yet. Magnetic media are now reliable for operation in the scale of 5-10 years but one shouldn't expect miracles. The 'cloud' could do better, since the storage equipment is maintained but then access to the stored data can't be guaranteed. For an digital storage medium and format there's the additional challenge of compatibility with future (or past) equipment.

I hate to admit that but as a storage medium, paper has served us reasonably well. Despite it being fragile, compostable, flammable, etc. Amazing, isn't it? And by some tricks we could store even more per page, even though that would make pages illegible to humans (for example, the QR code below, contains the first 2 paragraphs of this blog entry - and, yes, it can be printed smaller and still be readable by a smartphone).

To settle the argument, let's say that we use a combination of media and storage methods to be on the safe side. What should we put on those? The Survivor library that the article mentioned has an interesting selection of topics that range from 'hat making' and 'food', to 'anesthesia' and 'lithography'. Several state-of-the-art areas are missing (but that may be the point) and so do some well-established disciplines such as mathematics and physics, while with some topics, we could possibly do without.

Some have proposed keeping a copy of Wikipedia at a safe place. Yes, Wikipedia can be downloaded (its database dump, at least) and the size - so far - is said to be about 0.5 Tb (i.e., 512 Gb or about common 110 DVDs) with all the media files included.

There are also several physical and digital archives. Some specialised, other more general. The Internet Archive is an interesting approach, as it keeps snapshots of the various websites at various times. Not necessarily useful for the survival of mankind but interesting, anyway.

Another tricky bit that may not be apparent is that knowledge can only be effectively used by skilled people. So, not only do we need the knowledge but also a group of people with sufficient expertise to put that knowledge in good use. And then we need materials, resources, tools to allow for knowledge to be put into practice....

Hmmm.... Saving the civilisation seems to need a lot of thinking, after all :)

Stuff happens

'New Outhouse'
by andyarthur under
a CC license

Yes, it does.

I've been the proud owner of a humble Android tablet for quite some months now. Not the most mission-critical hardware I have but, still, a fully respectable gadget.

Yesterday, the YouTube app wanted to have it updated. Blindly, I obeyed. The new version kept crashing on start. Then another app crashed. And then another. And then the entire system froze. You know, the point where you restart the thing and hope for the best.

The restart took ages. A few things were missing but, most importantly, no app could start without crashing. They all had to be un-installed and re-installed. One-by-one. Yes, I tried clearing the dalvik cache before that. Yes, I double checked that there is enough space left. Yes, I got rid of things I didn't really need, just in case. Nothing really made a difference.

I ended up wiping the entire thing up and starting fresh with the initial factory version. Let me just say that this one is not a crystal clear process if you don't have a functional device. There is a start-up key combination that takes you to a pre-boot Android menu but that's not easy to assume. You can google for it, of course, and it comes up easily. But, still, you need access to the internet for that.

Getting things back to where they were (well, to approximately where they were before stuff happened) took about 3-4 hours and a good amount of cursing. Having survived that I can come up with a list of nice and bad things).

The nice things:

• Since most of the apps come from the Android market, Google may remember, right after the fresh installation, which ones the user bought and offer to install them (one by one) again. For some reason, that happens for some only for-pay apps.
• Google play remembers all the apps the user bought. So in the case Android doesn't remember the apps already bought, if the user searches for them, Google play will offer to install them without having to buy them again.
• E-mails, contacts, photos and anything that is stored on the Google cloud will be easily accessible after the wiping/ re-installation process.
• To be fair, the whole re-installation process is dramatically faster than what would be needed for a PC. Then again, I'm only talking about what I think was caused by a software glitch, not a hardware failure. Had that been the case, the only option for the tablet would have been a trip to the manufacturer's service (i.e., weeeeks of waiting).

The bad things:

• All the good things above can happen only if the user has access to his/her Google account. Being locked out or without internet access is very bad news in that case. In fact, internet access is really needed for things a user would normally do with a tablet.
• Google play won't remember the list of free apps the user has installed. Those apps have to be located and installed manually, one-by-one (unless the user kept a backup - there are special apps for that).
• Device settings, widget configuration, notes, calendar entries, documents and any other kind of files, etc. are all lost (unless the user kept a backup).

Moral lines of the story:

• Tablets occasionally fail. Take backups (yeah right!) both using cloud services and local media.
• If you rely on access to specific data or applications, have a backup device at hand that can hold and process the data you want. That could be your laptop/ netbook/ etc. As a bonus, you get something that will help you find your way in making the tablet operational again in case you get stuck (yes, Google is your friend but you need something to access it from.

That's all folks.

It wasn't in the plan to write about that. But I'm afraid that those pieces on gluten and crisis management will have to wait for a few more days. After all, stuff happens....