Gettin' geeky wid it

I have two computers running. The first, my 17-inch mac laptop, is connected to our firewire tape deck and a giant 500GB hard drive and is digitizing some of the 120 hours of footage we have shot so far. We have two of these big hard drives now, bringing our hard-drive total to nearly two terrabytes.

Video is incredibly, incredibly storage hungry. Each hour-long interview takes up about 15 gigabytes. And just a three years ago, when we started this project, hard drives cost a lot of money, so we could only afford a grand total of 250 gigabyes. So we had to digitize our beautiful footage at low-resolution.

Let me try that again: we have 120 hours of video for this documentary. 120 hours X 15 gigabytes = 1800 gigabytes, or 1.8 terrabytes. In the editing room you want to be able to see and access all of it, so what were we going to do?

Luckily, Final Cut Pro (the video editing software I use) has a low-resolution setting. You can digitize your video footage at a lower rate that cuts the size by about a factor of ten.

Unfortunately, a bi-product of this lower resolution means that the video footage looks fuzzy and is 1/2 the size. So I've been cutting together our film by watching it on a frame about the size of a credit card.

But thanks to rapidly declining hard drive costs (here's a great chart that shows the initial cost per megabyte in 1956 [$10,000] and the cost in 2004 [per GIGABYTE, $1.15]. Should be noted that the cost has approximately halved since then) we were able to purchase our terrabyte last month. So, at long last, the reason for this post: I'm finally able to start digitizing everything at full-resolution, and will soon see all our interview subjects and Fermilab's incredible environment in crisp detail and at full size.

I bring all this up because we've been lucky enough to have access to quite a bit of vintage fermilab footage, and in an interview in the 1980s Leon Lederman was discussing how he imagined advances in computer technology would mean faster processors, more storage space, and therefore more ability to analyze particle collisions. There are literally millions of collisions between protons and anti-protons that take place in the giant donut-shaped detectors at Fermilab. The computer systems from a few decades ago recorded information about them on tape, and (I'm surmising here) could most likely only pay attention to a percentage of the collisions that happened (the equivalent of fuzzy, half-sized images). Now, with the banks and banks of computers that are orders of magnitude faster and with vast, practically unlimited quantities of hard-drive space, the detectors can "look at" many more collisions (this is called the "trigger," and is one of the things Ben Kilminster works with) and determine in a split zillionth of a second which ones are boring and ordinary and can be ignored, which ones are possibly interesting (a few of which should be saved and looked at later), and which ones are "golden collisions," as Lederman said and need to be saved and studied in great detail. As I said, all this happens MILLIONS of times per second. Only possible with the incredible speed of computers and the drastic reductions in cost that computer equipment has seen in the last couple of decades. It's been estimated that CERN is going to generate one dvd of data per second. Per second! That's 4.7 gigabytes per second, minute after minute, hour after hour, day after day. Just not possible a couple of decades ago. Want 500 gigabytes of hard drive space in 1989? It's going to cost you. It cost about $800 for 20 MEGAbytes back then: 500 GB would be over... wait for it... 16 million dollars. Ha! Now, you can plunk down less than $200 bucks. As John Conway pointed out to us, a huge factor in the amazing advances in high energy physics is a result of simple economics (and Moore's Law). Stuff is cheaper now.

So, I'm digitizing in full-resolution, and Ben Kilminster is looking at amazing amounts of good data in his work with the trigger, and you're reading this blog split seconds after I post it. All thanks to our friends in Silicon Valley. Makes even more funny the (purported) statement from Thomas Watson, Sr., president of IBM from the 1920s through the 1950s, that "there is a world market for maybe five computers." If that were the case, I'd be a heck of a lot more handy with film, razor blades, and adhesive; Ben Kilminster would be spooling through miles of magnetic tape; and you would have no idea our film exists.

OK, so it doesn't exist YET...