# Which Storage Devices ???



## Dymonds (May 17, 2011)

Calling for all those in the know for advice on what is the best storage medium. Have just bought a MacBook Pro 17" and when discussing options for storage it became apparent that none of the external hard drives types last any longer than c. 7-10 yrs. Surely with all the investment in DSLR's there is something out there that will last.
Also wanting to know what I can use out in the field (given my Mac isn't exactly portable)


----------



## bvukich (May 17, 2011)

Get a NAS, minimum of 4 drives, RAID 10, replace it every 3-5 years, migrate data.


----------



## bvukich (May 17, 2011)

bvukich said:


> Get a NAS, minimum of 4 drives, RAID 10, replace it every 3-5 years, migrate data.



Now that I think about it, RAID 10 is overkill, RAID 5 or 6 is probably more appropriate.


----------



## Admin US West (May 18, 2011)

I do not know of any technology that will last forever. 

Hard drives wear out and must be replaced. I just upgraded my 4 drive NAS to a 6 drive one. I've had the old one about 7 or 8 years(running 24/7). It still works fine and has never lost any data. However, after that long, NAS speeds have increased a lot. The old one would do 23-25mb/sec at the very most, and usually a lot less. The new one commonly runs 110 mb/sec or a little faster for large video files, but small files still transfer much slower, about 55-60mb/sec.

Six 2TB hard drives cost much less than 4 250GB hard Drives 7 years ago. In another 7 years, SSD's might have dropped enough in price to be practical for large volume storage. I use a 128GB SSD as a Windows7 Boot drive all my data is on a conventional hard drive and backed up to the NAS every 2 hours. I have two PC's and my laptop with SSD drives now, and I'm planning on doing one more.


----------



## neuroanatomist (May 18, 2011)

Dymonds said:


> Calling for all those in the know for advice on what is the best storage medium. Have just bought a MacBook Pro 17" and when discussing options for storage it became apparent that none of the external hard drives types last any longer than c. 7-10 yrs. Surely with all the investment in DSLR's there is something out there that will last.
> Also wanting to know what I can use out in the field (given my Mac isn't exactly portable)



I use LaCie external drives with FW800- one at home and at work - for backing up my RAW images. If you have the newest MBP with Thunderbolt, data transfer will be mugh faster with a Thunderbolt external drive. Every so often, as drive capacities increase and costs decrease (and RAW files get bigger), I get larger drives. I also use a LaCie FW800 bus-powered drive for additional portable storage.

For truly portable storage, consider something like the Epson P-7000.


----------



## bvukich (May 18, 2011)

neuroanatomist said:


> For truly portable storage, consider something like the Epson P-7000.



Wow, that is one of the coolest things I've ever seen. A tad pricey, but may be worth it.

http://www.epson.com/cgi-bin/Store/jsp/Product.do?sku=B31B192002


----------



## hutjeflut (May 18, 2011)

looks nice indeed to bad its more expencive then the average laptop


----------



## dstppy (May 18, 2011)

You're asking two questions here . . . portable backup, and reliability of backup.

*Hardware failure etc:*
Nothing is forever; Raid CAN mitigate the issue of unplanned replacement . . . but you'll still have to replace it.

*For the field:*
Lots of suggestions here, an actually portable laptop with an SD card slot comes to mind (unless you're using CF etc).

*Cheapest, laziest solution for long-term backup:*
Every 2 years, buy whatever $100 will get you on a drive (2TB USB on a deal now) . . . and keep 2 copies, then an extra on your current PC.

If you're asking about store it and forget it, nothing is fail proof.  Accidental Overwrite is your worst enemy . . . yourself. In the end, nothing replaces proper backup procedures, so the true solution of no data loss is to get a software solution that allows it to be schedules and does not overwrite copies outside of how comfortable you are with it, and store a copy off-site. 

I don't even follow these procedures, so _see, cheapest/laziest_.


----------



## bvukich (May 18, 2011)

dstppy said:


> If you're asking about store it and forget it, nothing is fail proof.  Accidental Overwrite is your worst enemy . . . yourself. In the end, nothing replaces proper backup procedures, so the true solution of no data loss is to get a software solution that allows it to be schedules and does not overwrite copies outside of how comfortable you are with it, and store a copy off-site.



That can't be emphasized enough. RAID is not backup. It's protection from hardware failure.

I would use all the following data protection strategies that you can afford:

*RAID:* Protection against hardware faults. Sunk cost that I would consider mandatory.
*Snapshots:* Near instant recovery of human "Oops". Virtually free (performance and cost) on any decent NAS.
*Local Backup:* Quick recovery from major failures. Costs media (drives or tape), and time to manage.
*Offsite Backup:* Slow recovery from catastrophic failures (e.g., house burns down). Can be a second copy of your local backups that rotate offsite, or an online service (dropbox, amazon s3, etc.). If it's physical media, there are professional services that will come an pick up your backups every week, or you can just put them at a friends house, preferably outside of range of a single tornado/hurricane/earthquake/flood/godzilla.

Remember, just like you only have to brush the teeth you want to keep; you only have to backup the bits you want to keep.


----------



## distant.star (May 18, 2011)

I just came across this:

http://www.peachpit.com/articles/article.aspx?p=1670192

While focused on overall computer setup for LR, it goes into a long discussion of hard drives and hard drive setups, including RAID. It may be useful.


----------



## motorhead (May 18, 2011)

Technology is advancing so rapidly that no solution is guaranteed to last even ten years. I've used CD's in the past and provided we retain the ability to read them in the future they are as bulletproof as anything else. By that I mean CD's have problems of their own, as does everything else.

I am currently using a dual RAID external hard drive system. But all I can offer is to keep you wits about you as things change and export your saved files to a newer system in good time.

It is a concern for us all. I read a while ago that if the human race was wiped out overnight, in 10,000 years there would be nothing left to indicate we were ever here with the possible exception of stone carvings. Electronic records would vanish in milliseconds, never mind thousands of years.


----------



## Admin US West (May 18, 2011)

bvukich said:


> dstppy said:
> 
> 
> > If you're asking about store it and forget it, nothing is fail proof.  Accidental Overwrite is your worst enemy . . . yourself. In the end, nothing replaces proper backup procedures, so the true solution of no data loss is to get a software solution that allows it to be schedules and does not overwrite copies outside of how comfortable you are with it, and store a copy off-site.
> ...



Yes. You paid a lot to buy that camera and lenses, so your photos are valuable. Now, you find that keeping them costs a bit as well. There are lots of strategies for backups, usually a combination is the most reliable such as both local and remote backups. I


----------



## steven63 (May 18, 2011)

I haven't tried it yet but has anyone considered an online backup service such as Carbonite?


----------



## sb (May 18, 2011)

Hey Dymonds, every drive eventually fails, so your best bet is to refresh your hard drives every couple of years.
I also think RAID storage is an overkill for most people. You have to think about what's acceptable "down time" for you. If the answer is zero - then you need RAID. Note that, like somebody already said, RAID is not a backup solution. RAID is simply a solution to minimize downtime in case failure does happen. Also, a lot of people buy 2 identical drives at the same time when they decide to implement raid, and because identical drives will age in parallel, simultaneous (or nearly simultaneous) failures cannot be completely ruled out either.

If you're ok with your images being unavailable for a couple of hours, then you'll be ok with simply having a bunch of back up drives on a secondary computer (i.e. desktop) without RAID. Just keep refreshing those drives every few years, and you will minimize the chance of failure. Hard drives are so cheap these days anyway, dishing out a couple of hundred bucks every couple of years is not too bad.

I have a fairly simple setup:

2 desktop computers, each with a bunch of hard drives, and my own mirroring batch script which I run on demand (whenever I deem it necessary). That way, I'm not wasting power by running a massive redundancy setup. Ideally, I should have an off site backup place too in case my house burns down, but for now, I'm ok with taking that risk.

As far as backup in the field is concerned, those portable media backup solutions are great - they take memory cards and backup with one click. Far better than bringing a laptop into the mix.


----------



## Dymonds (May 18, 2011)

Appreciate all your comments. Gives me a sound starting point to investigate this. I don't need the instant access to the photos so will look at some decent external hard drives. Understand technology is a contsant moving beast.

I note that Canon have a portable storage device, M80-Media Storage Device. Looks interesting but pricy. Any one used this


----------



## pdirestajr (May 18, 2011)

I recently got so freaked out about how in this digital era there aren't really any tangible records anymore- so I went out and bought a used EOS-3 to shoot some slides! Also a cheap way ($200) to get a full frame camera (I'm shooting on a 7D). I'm also getting more into printing those special images.


----------



## Mike Ca (May 19, 2011)

steven63 said:


> I haven't tried it yet but has anyone considered an online backup service such as Carbonite?



I have considered Carbonite and other online backup services, but when you get into the details, way too expensive.

Carbonite has $59 per year per PC with unlimited data storage, but only for the internal hard drive of your PC or Mac. I have all my photo data on an external (usb) raid 1 unit. So for external drive backup, you have to buy the small business package. For the small bus package, if you store 300-500GB it costs $250 a month!!!

I've found most of the online backup companies don't support external hard drives in their low price packages, and they get petty expensive for the amount of data photographers shooting RAW can generate.

Definitely need to do a better job of deleting my non-keepers.


----------



## leGreve (May 19, 2011)

You're going to want to get he Lacie Little Big with Thunderbolt once it arrives in 2 months or so... anything else would be pointless and outdated.

I bought the 15" mbp, removed the stock HD, installed a new one, then I took the stock drive and put it in an IcyBox and wupti a free external drive. It will serve me decently till the Lacie arrives


----------



## dstppy (May 19, 2011)

*Comment:*

Off-site, web storage is a good idea with a few exceptions; the biggest of which is 'who do you trust?'*

Honestly, following standard sysadmin style backup is the way to go, and every week/big shoot, you drop all of your UNEDITED stuff on an "FTP"** server. I would say avoid 'solutions' -- as my boss always says, "I don't buy solutions, I buy hardware and software". Avoid anything 'image' related, because there is a tendency to compress or alter the images to do you a favor; hell, avoid a web interface.

* Who do you trust? I like my web host, but I certainly don't love them. They have had their billing information compromised twice in less than as many years, and I'm now shopping around for an alternative. Things to consider:
1) Do they patch the servers diligently? Do you have the ability to verify server versions?
2) Is billing and hosting separate? (Make sure you use different credentials)
3) What does their EULA say?
3a) Who owns YOUR stuff (most places have a line that says they do)?
3b) Do they guarantee that what you delete gets deleted?
4) Will they protect you?
4a) Will they turn over your account information willingly to 'the authorities' (don't forget, this includes people CLAIMING to be the authorities or you [social engineering])
4b) How do they handle 'takedown' notices -- if you're doing something web based with this.

**Why did I put FTP in quotes? Because I meant it more as a relationship style than actually endorsing the protocol; there may be faster/more secure ways at this point, I haven't investigated recently.

*Question:*
For those of you using off-site data storage, who and what do you use?


----------



## motorhead (May 19, 2011)

Dymonds,

Based on my own experience, locating a file on an external hard drive is only a couple of mouse clicks more than finding the file on the PC's own hard drive. Apart from the time taken to make those two clicks, it's no slower at all, so don't worry about it taking longer. One extra heart beat, maybe two and thats it.


----------



## scottkinfw (May 22, 2011)

For the field, I would recommend Hyperdrive UDMA. I purchased the unit, a separate hdd, and assembled them (no problem at all). 

The unit is self powered, and accepts common memory cards. You can buy some other accessories to increase functionality if desired.

I recently took it on safari to Tanzania, and every day downloaded cf & sd cards. It worked like a champ. It has a color screen to view pics and a histogram, etc.

Here are links: http://www.hypershop.com/HyperDrive-COLORSPACE-UDMA-Casing-Only-p/hdcsu-000.htm (cheapest way to do it, and it is easy to do)
On the go adapter: http://www.hypershop.com/COLORSPACE-UDMA-OTG-Sync-Adapter-Hard-Carry-Case-p/hdcsu-acs.htm (a bit confusing documentation, but uesful).

I have no financial ties to the company.

sek


----------



## autochrome (May 31, 2011)

Whatever you do, just don't be cheap on HDDs, and backup, and backup often, it will save you from a massive disappointment one day, that's guaranteed. If possible use RAID1, or if you can afford hardware RAID controllers, you need a 4 ports controller to have RAID5 mode, in which one of the hdds is a parity disk, so if any single hdd of the remaining fails you can reconstruct the missing image, but you need an 8 ports controller for a RAID6 mode (with 2 parity disks), which brings you an extra level of redundancy. People spend big amounts of money on cameras, L glass, devote their time to create good images, process them and that's fine, but here's nothing more frustrating than watching IO errors on the hdd and realizing a week, a month or more of work are gone, beyond repair. For the most time, you'll be asking yourself why on Earth are you doing weekly backups, spent a considerable amount of money on a RAID6 array, enclosure, enterprise quality hdds, until one day you'll suddenly find out why, and in that moment all that investment in time and money justifies itself.
This of course is somewhat relative, if the information you store is just a hobby or if it's work, these two situations might warrant different levels of investment.


----------



## Chewy734 (Jun 7, 2011)

If photography is your business, get a dedicated RAID server for your backups. It doesn't have to be hardware RAID-5 or 6, as we use software RAID-5 or 6 for a lot of mission critical stuff. If photography is your hobby, then RAID-1 should be good enough with 2 large HDs. It all depends on your requirements (speed, capacity, etc).


----------



## gmrza (Jun 7, 2011)

Chewy734 said:


> If photography is your business, get a dedicated RAID server for your backups. It doesn't have to be hardware RAID-5 or 6, as we use software RAID-5 or 6 for a lot of mission critical stuff. If photography is your hobby, then RAID-1 should be good enough with 2 large HDs. It all depends on your requirements (speed, capacity, etc).



There are two points here that a few people have discussed:
1) data protection (RAID is the most common technology to address this); and
2) backups.

The two should not be confused.
There is no one-size-fits all storage solution - we all need to analyse our needs, and build a solution that fits.

What is important to remember is that RAID will protect you from a single hard drive failure (or two hard drive failures with RAID6 / RAID-DP - if you are using Netapp storage).
RAID will not protect you from a NAS or controller failure. To do that you need to look at solutions like replication (e.g. replicate data between two NASes).

RAID also does not protect against human error - i.e. accidental file deletion, or against operating system failure - e.g. your system crashes due to power loss, and the file system is trashed when you try to boot up again.

Try to consider all the failure modes you need to protect against, and then architect a storage solution that fits your needs.

For instance, RAID is not for everyone. You may find that individual hard drives are large enough, and that replicating between 2 NASes protects you from drive and controller failure, while backing up to the cloud protects you from finger trouble...


----------



## prestonpalmer (Jun 8, 2011)

Ignore all these guys. If you want something that works, out of the box with extreme reliability and you can use your existing drives in, purchase a DROBO. There simply is no better product on the market for professional photographers. I own 4 of the USB3 DROBO S 5 bay models and have been 100% pleased with performance and reliability. Fantastic product. Checkout their website and watch some videos and check out the dual disk redundancy options. I have had drives fail and with a drobo it just isn't an issue. BTW. I have no financial gain from pitching drobo! I just LOVE their products!


----------



## prestonpalmer (Jun 8, 2011)

Get a DROBO


----------



## morphers82 (Jun 16, 2011)

There are some great options for storage on the go. There's a breakdown on this site http://bit.ly/lr3lqw of the different options


----------



## autochrome (Jun 17, 2011)

gmrza said:


> Chewy734 said:
> 
> 
> > If photography is your business, get a dedicated RAID server for your backups. It doesn't have to be hardware RAID-5 or 6, as we use software RAID-5 or 6 for a lot of mission critical stuff. If photography is your hobby, then RAID-1 should be good enough with 2 large HDs. It all depends on your requirements (speed, capacity, etc).
> ...



Overall that's the difference between a hobby and work. Clients don't want excuses, so you shell out for the battery backup modules hw RAID card, and you have a good UPS.
A journalling filesystem helps, and specially SGI XFS has a consistently high performance when dealing with series of large size files - it was developed by SGI for the visual effects workstations - all the compositing and post-production work was done on scanned 35mm film, which meant you ended up with imagine (the following is nothing abnormal), frame sequences of 5000 or 10000 16bpc TIFF files, or Kodak Cineon DPX files. Now mostly OpenEXR half-float, but still a very large number of frames of relatively the same size. Then you had to "record" the frame sequences into film again, but the all-digital workflow of today simplified that a lot. You still end up with vast sequences of large files for post-production and visual effects work, since that's the highend applications expect (mostly developed in-house, such as Nuke by the software department of Digital Domain, then marketed and sold, others not marketed at all, such as the compositor used by ILM, but they share an SGI IRIX ancestry, and porting them to Linux was relatively easy (and now some, such as Nuke, exist on OSX and Windows as well).
This just to point out that the choice of the file system is also a factor in the problem at hands. Some other filesystems, Reiser3 are great for very large series of small files, others are generalists, with average performance, such as Ext3 (i won't mention Ext2 since it doesn't has journalling), and there are other filesystems which record "snapshots" of the filesystem state at user-defined periods, i'm thinking mostly about Sun ZFS.
If the battery backup module of the hw RAID card, and UPS fail at the same time (unlikely), you still have the filesystem journal/snapshot, and for the most time you'll be able to savage the data without inconvenience. In some of these filesystems you might even place the filesystem journal/metadata in other hdds, mirrored, extra redundancy.

The backup plan, that's trickier and more expensive. Safe backups imply copies in different physical locations, tapes or hdds in fire proof vaults, and so on. There are companies that provide this type of service for critical data. It just depends what your definition of critical data is and how much you think it is worth. Granted this is overkill for the majority of people, but it's worth mentioning nevertheless.

I agree with your views, just wanted to add those details regarding filesystems.


----------

