The digital archiving nightmare

[UstadKhanAli]

 

Of course, I mentioned the Akai 12 track because you seemed to be citing the equally ephemeral Syquest disk storage medium as some sort of exemplar of problems with digital archival. I would have told you then that such media was doomed to a short half-life of practical usability. (And don't even start me on graphic designers . As a web developer, it's a total love/hate relationship. I admire their arguably higher creativity -- and many are technologically quite adept -- but, overall, they are an ongoing, vexing challenge to work with. God love 'em.)

 

 

Well, I didn't know that. I mentioned Syquests because at the time, all graphic designers I knew seemed to be using these. So I was listing this, not as an archaic bizarre format, but one that appeared to be popular (unlike the Akai, which was doomed from the start!!!).

[UstadKhanAli]

It seems like we've argued this issue a little closer to a mutually visible common ground, so, with that in mind, I'm going to go address the minutes-ago arrival of a new example of that most ephemeral of digital devices, the 'free' mobile phone. UPS just brought the LG Optimus Android I'm replacing my 'old' Blackberry with. At almost two years and ticking, the BB -- which has been my favorite mobile so far, with only a few provisos -- has really been showing the limitations of its ancient design precepts...

 

Well, who knows, maybe there's some more robust storage medium that's around the corner. I don't know. I just know that with what we have right now, the best strategy for digital stuff in general is to keep several copies of back-up and to keep updating it several years to the most robust and popular medium. Sure, it's work, but if you care about something, then it should be worth it.

 

And if you're like EB and you're thinking, "Who the hell is gonna want this?", then all of that is really not very important.

 

I seem to get asked for reproductions of my old photos and things like that regularly, and I value my photos greatly, so I definitely want to keep trying to archive my digital photos as much as that's possible. It's important to me personally. But someone else may not value their photos, their audio, their documents, their whatever, and so the response of "who cares?" is equally valid.

[Ernest Buckley]

 

EB, I need to save for at least a while because I sometimes need to remix something or remix something for a client several years or more after the fact, and for that alone, I do need to think in terms of properly archiving mixes so that I can open it later (even if I or someone else no longer has some of the specific plug-ins or specific hardware used to create those sounds). Regardless, there are some considerations that come along with digitally archiving, and this is what the article is indicating.

 

 

I understand. Whenever I finish a project for a client, I give them all the files. Its up to them to archive. I keep a copy myself but its theirs at that point so... I understand the need to backup but seriously, I do ask myself all the time, what exactly is the point when the finished product is in hand?

[Jeff da Weasel]

 

I do ask myself all the time, what exactly is the point when the finished product is in hand?

 

 

Because today more than ever, the finished product isn't finished. Remasters, mastering for other media (like vinyl), use for soundtracks, commercials, games, and more dictate that the original master multitracks may require access and manipulation years down the road.

[Ernest Buckley]

 

I fully agree with you here. We don't have lacquer masters of records made 100 years ago, but they're still getting re-issued on CD. But the ego says "maybe some day someone will want to hear that old recording but with a new guitar part" or maybe when you're famous you'll want to re-release that first album mixed in a good studio this time, and there you go. It's certainly a good idea to keep backups (that's not an "archive" ) of work in progress, but when the project's done, heck, you probably have a closet full of CDs anyway.

 

 

Keyword: EGO

 

The EGO wants to preserve because what the EGO creates must be the end all, be all but the truth is, its all passing. None of this will be remembered 100 years from now. That bothers the EGO. No one wants to hear this.

[TimOBrien]

My first computer professor had a saying: "You don't have a REAL backup until it's on 3 different media in different places, preferably one copy in Aunt Sofie's bank vault in another city."

 

The chances of multiple storage all going bad on you are very very small. Even just keeping copies on different computers on your home network increases the chance you wont lose ALL your data. Besides keeping my important data (songs, photos, documents) on multiple computers I keep a USB drive backup and once or twice a year I archive to DVD that goes in a waterproof firesafe - yeah, I'm semi-paranoid about it but I live in Florida Hurricane Alley and have NEVER completely lost my data.

 

I wasn't kidding about Aunt Sofie - if you've got something REALLY important, ask your Aunt or Grandma in another town to put a copy of your disk in her safety deposit box and hold it for you...

 

As for longevity - I have CDs from the early 90s from back when 1x drives were $3500 and blank cds were $25 EACH and they're still good. I'm sure at some point everything will get bumped up to whatever replaces DVDs and Blue-Rays and it'll still be good.

 

Good Article: How to Choose Archival Disk Media

I've always used Taiyo-Yuden for important stuff and have never been let down.

[Kendrix]

As we speak I am dismantling my studio as we will be moving south.

I just packed a box full of DAT tapes used to backup sessions on an Akai 16bit HD ercorder from the mid-nineties.

So far this medium hasn't failed me. I have successfully accessed files created over 15 yrs ago.

Chances are im about done remixing all that ole stuff by now.

However, I am attached to it since it reperesents so many hours of my life.

 

So far my CDR backups have also performed flawlessly.

 

Im farily comfortable that the huge existing database of PCM material will keep this file format supported at least so as to allow for a transfer to a newer beast.

 

Beyond that I agree- what AM I saving it for?

 

Cheers

[blue2blue]

I understand. Whenever I finish a project for a client, I give them all the files. Its up to them to archive. I keep a copy myself but its theirs at that point so... I understand the need to backup but seriously, I do ask myself all the time, what exactly is the point when the finished product is in hand?

I deliver the files to them and tell them not to lose them -- but almost every time I've checked to see if they still have them, the response is something like, Oh, is that what those were? I think I must have them around here somewhere.

 

As a consequence, I always keep copies myself. But I don't tell them that until I have to pull their backsides out of the fire.

 

_______________________

 

 

Ken... yeah. I know the graphics guys were big on Syquest. And, really, there weren't that many other options that made sense for big data files.

 

I was only doing database work back then, so it wasn't really an issue for me, personally. In the 90s, I did momentarily succumb to those forgotten drives from Iomega (mine was a SCSI device)... hell, I can't even remember the name. They made the ephemerality of the ADAT tape cart medium look like the Pyramid of Cheops by comparison. It felt hinky from the gitgo, but one of my buddies who was a corpo-IT guy wanted to buy it from me for what I paid for it for some reason (maybe I got a deal) so I did and within a few months it had gone bad. He had some kind of in, sent it back under his corporate colors, got another. It went bad and he replaced it. The next one went bad and he replaced it but at some point, I think he just moved on.

 

I also had one of those tape cart based back-ups that I used for a few years but they never (as far as I know) upgraded to Win 95. I didn't jump into '95 myself, but I got tired of waiting for them.

 

 

I don't have any doubt that we will see considerably more robust storage evolving -- but when you have so much data that you need to be able to pack it in super tight, there will certainly be challenges. Right now, bucket brigading seems to be the best way to deal with the very real danger of device and media failure.

 

I learned back in the inglorious end days of the Ampex nameplate not to put all your eggs in one media maker's basket. I might still have a half-case of Ampex ADAT tapes that were all bad. After that, I switched to Maxell and TDK S-VHS and never had any more problems.

[blue2blue]

 

Keyword: EGO The EGO wants to preserve because what the EGO creates must be the end all, be all but the truth is, its all passing. None of this will be remembered 100 years from now. That bothers the EGO. No one wants to hear this.

 

 

As someone who frequently listens to music and watches movies made as long ago as the 1920's (almost 100 years ago), I would tend to disagree, in priniciple. Who is to say what kind of crazy, wild-eyed mofo is going to want to listen to your stuff then. Hell, maybe you will.

[MikeRivers]

 

My first computer professor had a saying: "You don't have a REAL backup until it's on 3 different media in different places, preferably one copy in Aunt Sofie's bank vault in another city."

 

Yeah, but if they're all on 8" floppy disks, how much trouble do you think you're going to go to in order to play them? This is why, no matter how good the plastic and die or silicon, you have to keep the media current enough so that it will be playable when you want it.

 

Fortunately there are people, both hobbyists and professionals, who are into preserving and using analog tape recorders so that at least for our lifetimes, any of the common formats will be playable. But do computer hobbyists get any pleasure out of getting data off old disk and tape drives? Maybe they do. I don't know.

[MikeRivers]

 

Because today more than ever, the finished product isn't finished. Remasters, mastering for other media (like vinyl), use for soundtracks, commercials, games, and more dictate that the original master multitracks may require access and manipulation years down the road.

 

That's a great point - re-purposing our music. But then, what's wrong with creating new music, or a new version of the old music, when you need it? Does it really save so much time or money to use a few of the original tracks? If your song is a hit in Russia, wouldn't it sell better if a real Russian singer recorded it (and paid you royalties)?

[MikeRivers]

 

I seem to get asked for reproductions of my old photos and things like that regularly, and I value my photos greatly, so I definitely want to keep trying to archive my digital photos as much as that's possible.

 

A friend of mine got a high class negative scanner and digitized about 5,000 of his photos over the past year. I thought it would be fun to "preserve" some of my photos from the 60s folk festivals when I used to be a pretty active photographer. I had carefully stored the negatives in the right kind of envelopes, dug them out of a cabinet in the basement, and found that most of the film was stuck to the envelopes.

 

I think I could probably save them by washing them, but I decided to just put them away and wait until I really needed them, and then take them to a professional. They'll probably be there when I die. But there are still mounted prints in a closet upstairs that will live on for a while longer.

[cooterbrown]

How ironic would it be if something like a consumer-affordable laquer mastering machine became the future of music archiving?

[Ronan Murphy]

 

I would have no idea whatsoever why a big artist who an engineer Ronan knows is working for has decided to archive all his material to analog tape -- you'd have to ask that big artist.

 

 

Quite simple actually. if some one finds those analog tapes in a vault somewhere 150 years from now, they will almost certainly be able to listen to the music in its entirely. It will probably have lost some high end and perhaps become a bit more distorted, but unless they is some massive reality altering blast of magnetic energy on the earth, the music will be there for many generations to come. It is far less likely to be true of almost all of the digital storage mediums we are using today.

[bp]

 

Dave Glasser at Airshow has a Beta deck that's opened up so that he can tweak the alignment and tension in order to get PCM tapes from the 1985-1995 era to play and he's had some success with that, but there's still, as they say, some assembly required.

 

 

I remember Dave from his early days at Bias in Springfield, VA. His "Beta" wasn't the consumer type. It was the old 1630 format.

[Philter]

 

Quite simple actually. if some one finds those analog tapes in a vault somewhere 150 years from now, they will almost certainly be able to listen to the music in its entirely. It will probably have lost some high end and perhaps become a bit more distorted, but unless they is some massive reality altering blast of magnetic energy on the earth, the music will be there for many generations to come. It is far less likely to be true of almost all of the digital storage mediums we are using today.

 

The analog tapes will be a sticky mess, the machines to play them will be run down antiques with no replacement parts available, and the digital copies will sound the same as the day they were made, assuming whatever organization responsible for the backups practiced due diligence.

 

But if you're looking to make a time capsule and bury behind town hall, by all means bury an analog tape. I just wouldn't consider that a realistic form of "backing up." Inherent in the concept of secure back ups is some kind of archival oversight and activity, which is exactly what the Library of Congress is dealing with.

[bp]

Almost all SACD releases were either done from old analog masters (recorded and mixed analog) or from high sample rate digital recordings. 44.1/16bit masters would never benefit from the higher quality SACD format. I hope that all analog to digital archiving is done with the finest tech available and that playback/transfer hardware is retained and preserved with every bit as much care as is taken with storage of the original recordings and backups.

[Anderton]

 

All the Linda Cohen classical guitar albums I produced during the 70s were re-issued at some point, and some of her later ones too.

 

I still get inquiries from labels (a couple a year, actually) that want to get decent copies of material my band did in the 60s for re-issues.

 

And thankfully no one's noticed, but those soundtracks on the HC show coverage videos are often music I did many years ago, but without vocals and other identifying characteristics

 

So, Mike Rivers might ask, why not just record new music? Because it's 2AM in Anaheim and I have six videos to upload before I go to sleep. I copy a bunch of old stuff over to a DVD-ROM before I go to a trade show, and I have plenty of potential soundtrack music.

 

Go green - recycle!

[blue2blue]

Quite simple actually. if some one finds those analog tapes in a vault somewhere 150 years from now, they will almost certainly be able to listen to the music in its entirely. It will probably have lost some high end and perhaps become a bit more distorted, but unless they is some massive reality altering blast of magnetic energy on the earth, the music will be there for many generations to come. It is far less likely to be true of almost all of the digital storage mediums we are using today.

Well, I certainly hope so (on the former).

 

With regard to the digital storage media, stamped CDs would appear to have a fair degree of longevity. The main problem as I understand it is that the non metallic materials will deteriorate and eventually make the stamped substrate unreliably readable through the clear media. But if the data on those stamped CDs is valuable enough -- and our great grandkids aren't living in a sort of new dark ages (or heaven forbid worse), I strongly suspect that someone will have developed ways of reliably removing the stamped substrate and getting reliable reads off it.

 

I'm betting that -- barring the kind of global catastrophe that actually does seem to be brewing in our ecosystem or some other, less likely but equally calamitous series of events -- that we will have excellent copies of the Beatles work.

 

The problem is that people have to want to preserve the music, movies, or other archived works.

 

And if they want to preserve it, I think it's rather absurd for us to sit here in 2010 and presume that we as a technological society will still be facing the same archival dilemmas in 2050.

 

 

But even if you had a fool proof archival system, you'd still need to have some assurance that the desire to preserve a given set of works will be there and that the archive documentation will be preserved and remain matched to the archival media.

 

Look at the great master recordings and first strikes of films that have been lost or even willfully destroyed -- sometimes in the hope of driving up the box office at a subsequent remake of a pioneering, would-have-been classic film. You have to have the method to faithfully preserve and document, the will to pursue it, and the discipline to maintain it.

[blue2blue]

Almost all SACD releases were either done from old analog masters (recorded and mixed analog) or from high sample rate digital recordings. 44.1/16bit masters would never benefit from the higher quality SACD format. I hope that all analog to digital archiving is done with the finest tech available and that playback/transfer hardware is retained and preserved with every bit as much care as is taken with storage of the original recordings and backups.

 

Well, I think one has to be wary of some of the claims made for some of these technologies -- or you end up over-investing in flawed or limited technologies. (Like Syquest drives. )

 

Converter design guru Dan Lavry, responding to someone in this thread at his site wrote:

DSD is not a higher sample rate format. DSD is 64fs at 1 bit or 2.8224 Mbits/sec. A PCM operating at say 88.2KHz and 24 bits is 2.117 Mbits /sec. At 96KHz 24 bits you have around 2.3 Mbits /sec, so they are comparable. In fact most of the AD

[Ernest Buckley]

 

, the music will be there for many generations to come. It is far less likely to be true of almost all of the digital storage mediums we are using today.

 

 

Actually, I think we`re about due for one around 2013. No kidding.

[Ernest Buckley]

 

So far my CDR backups have also performed flawlessly. Im farily comfortable that the huge existing database of PCM material will keep this file format supported at least so as to allow for a transfer to a newer beast. Beyond that I agree- what AM I saving it for? Cheers

 

 

Thats what I was saying before about the EGO. We get attached to our own stuff and so much importance on it but its only because of our own time invested and personality we put into a project. Aside from that, it doesn`t really mean much. I know its hard to hear this because I too have had issues with it but long term, people have really busy lives and songs come and go. Generations pass and no matter how great a song was, it loses its appeal after a while. Yes, certain songs will always be "treasured" but chances of some unheard of songwriter being stumbled upon in 100 years and then the urgency to remix and listen to those tracks... nah.... its not happening. Its a big EGO trip.

[MikeRivers]

 

How ironic would it be if something like a consumer-affordable laquer mastering machine became the future of music archiving?

 

A few years ago, a local archivist, Steve Simolian, wrote a gag April Fool article for the Library of Congress in which he described a new and highly reliable archive medium. About half way through, it became clear that he was talking about the 78 RPM phonograph record.

 

There used to be consumer disk recorders back when the "record player" was a piece of living room furniture. It had a second tone arm with a cutting stylus, and recorded either on aluminum or lacquer coated blanks. Few (either the machines or recordings) survived, but I know someone who has set up to play them and make a fresh copy on another medium.

 

Vestrax makes, or at least recently made, a disk cutter that was designed for DJs. I think it cost about $2,000.

[MikeRivers]

 

Quite simple actually. if some one finds those analog tapes in a vault somewhere 150 years from now, they will almost certainly be able to listen to the music in its entirely.

 

150 years from now, all the Ampex, Studer, Otari, MCI, etc. machines will probably be part of the environment. But assuming that you know that you have a magnetic tape recording of an audio program, with some assumptions (like you have a pretty good idea of what it should sound like, which, admittedly, may be difficult with some music unless you can find some examples on other media that are still playable) you could reverse engineer and build a machine to play it. You know the size, you can determine the track format, the speed, and required frequency response. What more do you need other than a good machinist, a good electronics engineer, and some good taste?

 

Due to the nature of things computer, I doubt that there will be sufficient documentation surviving to build a CD player. And it would be highly unlikely that one would be able to reverse engineer a CD player from a CD even if it survived the elements for 100 years.

 

For a fiction example, read the short story "Time Shards" by Gregory Benford. It's only half a buck, and it has one of my favorite lines in it: "We should have bought the Hewlett-Packard."

[MikeRivers]

 

I remember Dave from his early days at Bias in Springfield, VA. His "Beta" wasn't the consumer type. It was the old 1630 format.

 

Different machine. The 1630 was the standard format for sending a master to a pressing plant back in that day. The Beta setup came along later when he started getting mixes in that format for mastering.