Data Management for Photos and Videos

January 22, 2016  •  Leave a Comment


I'm sure I covered this once or twice before in the Blog but the subject comes up so frequently it's always good to revisit it. Actually I see my Data Management now as a fluid situation that changes as new technology and storage services are brought to the market. I've spent the better part of two weeks helping some friends build 3 file servers, complete with RAID arrays for backup. It went a lot better than a similar project a few years back but not without some hiccups. Now that we have it all working for the most part, I'll share the Why, What and How of it all. 

Even seasoned photo and video professionals sometimes take data management for granted. Trust me...I've blown up my share of computers and hard drives over the years. Major companies do not, and if you want to see how serious they take it, check out my second favourite company's data management system/compound.


HOW

I'm definitely not "Google" big but I have to ensure that my client data is protected for the duration of our contract period. Being sued is not fun.

My Adobe Lightroom CC indicates that I have a catalogue of just over 32,000 files taken in 2015...after curation, culling, sorting etc. Whatever you want to call that. I have other files which are not in the catalogue but are older personal and client files stored on DVD's, CD-ROMS and an large number of Hard Disk Drives. Essentially, all that I've done since 2008.

The diagram above is my content creation eco-system

  • STEP ONE: Import photo/video data from the cameras into my workstation computer running Adobe Lightroom CC where they are named, keyworded and stored on the MAIN STORAGE drive. That main storage drive is a 12 Terabyte RAID 1 unit which makes two identical copies of all my files. Should one of the drives in the array fail, it rebuilds itself once I replace it with a new unit.Very safe... Once all the data is safely transferred the images and video are sorted and culled down to a final number of shots that will be kept. I delete duplicates/similar, over/under exposed, out of focus and otherwise unsuitable shots. Once i'm down to the permanent shoot collection i'll transfer the ones I want to work on to the WORKING FILES drive, which is a RAID 0 unit built strictly for speed. There is no redundancy so if one of the 2 drives in it fail, I have to start over from scratch from the last backup. This drive is great for working on video projects because it's fast and can keep up with scrubbing, especially with 4K and higher source footage. 

 

  • STEP TWO: Each time a project file on the WORKING FILES unit is altered, the system backs it up to the MAIN STORAGE unit. At most i'll lose a day's work if something goes wrong. This ensures that I always have a backup of the original image/video and an edited version in case of any failure in the system.

 

  • STEP THREE: TIME MACHINE backs up my workstation computer to it's own designated storage drive. I only keep applications and System files on the computer's main storage. In fact, that's one of the reasons I switched to the Adobe Creative subscription service. I can download Photoshop & Lightroom on any compatible computer and log in with my USER ID and password. If the computer crashes I can download them again once it's been restored. My documents such as letters, invoices etc. are stored in my cloud service (Dropbox, Google Drive and Microsoft). I do this mostly so I can access those documents no matter what computer, tablet or Smartphone I'm using.

 

For those new to this stuff...

RAID 0: 2 or more Hard Disk Drives (HDD) used together as a single storage unit built for fast data transfer speeds. There is no redundancy so when it crashes all the data is lost. Usually used in situations where performance is more important than data safety.

RAID 1: 2 or more Hard Disk Drives (HDD) used together as a single drive with the total storage capacity equalling the smallest of the two. For instance, if you have a 1TB and a 2TB drive together in this configuration your total storage capacity will be 1TB. Both drives are an exact duplicate of the other so if any one of them fails, the other retains a full copy of your data. You pop out the failed disk and replace it with a working unit and the system automatically fills it in with a exact copy of the original working disk.

TIME MACHINE: ...Is an Apple Inc. construct for it's Mac products but Microsoft and other operating system applications have similar tools. Essentially it backs up whatever you want it to periodically to a dedicated storage device of your choosing.

So that's the "HOW"...here is the "WHY"

Images and Video are memory...in the simplest terms. It doesn't matter if it's work related or personal, you just want to save it for as long as needed or wanted. I've had the misfortune of knowing people who have literally lost everything they own in house fires. You can always make more money, buy a new house and fill it up with stuff, but those images saved on your computer or in a printed album are gone forever unless they are backed up somewhere else. Professional work for clients or customers share the same fate...except nobody will sue you for losing your personal files, clients/customers can and in our litigious society you can spend all the money you have defending yourself.

From a professional standpoint you should have a backup system, period.

My clients enjoy a 7 month period from the date of a job delivery in which they can call me up and get copies of their images and/or videos. After that point it's best efforts only. I spell that out at the time of my engagement to ensure there is no misunderstanding. Clients have called me up several weeks after I've given them a USB, CD-ROM, DVD-ROM or Hard Disk Drive with their files, saying they deleted or lost it. It's good for them to know that within 7 months if some misfortune should befall them, I have a copy.

The reason I limit it to 7 months is this...any longer and I become a defacto "Cloud Service" for my clients. A service I'm not getting paid for and could possibly be liable for in the event of failure. I have neither the time nor resources to run a Cloud Storage Service for anyone except myself.

I make an entry in my Calendar from the date of delivery that reminds me in 7 months from the day.

Then I go through that particular shoot and select files that I want to keep and delete the rest. This might sound draconian and maybe it is. Some people like to keep every image they shoot and more power to you. When you shoot upwards of 50,000 image per year the costs of storage add up pretty fast. Megapixels are increasing for cameras, not declining. A 16 Megapixel file can easily balloon to 600 Megabytes with post processing and editing in Photoshop. Do the math...50,000 X 600MB adds up to a lot of storage hassles. If I had your money I would keep every single one for posterity. Until then I'm an aggressive delete technician.

Your situation may be different and may not require the extensive arrangements I have. There are simple solutions...

Short of that you can always build your own.

 

 

 


Comments

No comments posted.
Loading...