# Best Methods For Long Term File Storage ??



## 1255 (Nov 10, 2012)

Hi all.

Would like to ask for your thoughts on best methods for long term file storage, meaning, external hard drives or DVD/Blue Ray? Or something else?

I had always thought that external hard drives were best, at the very least because who knows when we'd need to convert those DVD or Blue Rays to a new technology. 

Then I just read in another post that external hard drives corrupt files over time. Is this true? I had always thought that it was the process of dragging files and/or copy and pasting them over and over that impacted data bits and degraded the files. 

What do you think is the best method for long term file storage?

As always, thanks to all of you for your input. 

Much appreciated.


----------



## Drizzt321 (Nov 11, 2012)

Hard drives suffer from bit-rot (as do some other forms of storage). It can occur in a couple of cases, such as improper storage, flooding, etc, or cosmic rays, or other similar high energy events.

There are a couple of ways to mitigate this. First, keep a copy on several different storage devices, preferably of different types. This helps mitigate one of them going bad. If it's all hard drives, get ones with serial numbers which aren't right near each other, which can help mitigate if a particular lot of hard drives has issues (it happens, manufacturing isn't perfect).

Further, you can use a filesystem or storage mechanisms which provide a high level of error correction and detection. You can use something like Par2/Parchive which provides parity and checksuming which can detect, and often correct for bit-rot errors.

A good archiving solution will including multiple storage media, and proper storage of that media. In general, a high quality tape backup with proper storage tends to be the best, very long term archival mechanism and is still very popular and widely used in various enterprise backup/archive solutions.


----------



## 1255 (Nov 13, 2012)

thanks very much


----------



## Standard (Nov 13, 2012)

I simply archive all my files onto DVDs. If there are important files that I worry about, I make sure to archive them onto several copies for redundancy (I do this over a period of time) so as to avoid possible disc errors. All discs are then catalogued (with a cataloguing app for quick search and retrieval at a later time), then stored onto large 200+ disc binders and placed in metal file drawers. Simple as that. No issues thus far. And yes. I archive an extra copy of select important files onto Lacie 1+ TB external hard drives.


----------



## Iahcon (Nov 13, 2012)

I keep two copies on separate external hard drives.


----------



## neuroanatomist (Nov 13, 2012)

Iahcon said:


> I keep two copies on separate external hard drives.



As do I, and the two HDDs are stored in different physical locations (home and work). Those copies are made as soon as the images are processed (and until then, they're still on the CF cards - since getting the 1D X, I always have at least 2 copies of every image). Quarterly, I make a DVD backup that goes in a safe deposit box in a third location (different town than home or work).


----------



## florianbieler.de (Nov 13, 2012)

Got mine synced to a 2,5" external hdd which i use not that often.


----------



## Crapking (Nov 13, 2012)

What about on-line "cloud" storage sites which allow access without physically requiring the disk drive? 
Of course this is in addition to the physical backups, but I use Phanfare.com {with a reasonable annual fee}, which allows web-based access, including RAW storage (for an extra fee) for the really important files.


----------



## myocyte (Nov 13, 2012)

I've been meaning to sign up for this, but Amazon provides a great, affordable solution for cloud storage. If anyone has tried it, I'd be interested to hear about their experience.

http://aws.amazon.com/glacier/


----------



## sb (Nov 13, 2012)

neuroanatomist said:


> Iahcon said:
> 
> 
> > I keep two copies on separate external hard drives.
> ...



Same here with a small difference - I make a copy immediately after dumping the files on the primary computer. Processing is all in LR metadata anyway, and LR catalogues are also included in my backup script.

My backup script is a hand made batch file using the "Robocopy" command line utility. 

My back up drives reside on a second computer which is identical with all the same applications installed as the primary. That way if my primary fails, I can continue working uninterrupted. 

I don't use any optical media (DVD/Bluray) because they become unreadable after a few years. I also don't use external drives because for the price of a "network storage" I can build myself a fully functional second computer, so what's the point?


----------



## neuroanatomist (Nov 13, 2012)

sb said:


> I don't use any optical media (DVD/Bluray) because they become unreadable after a few years.



How many is 'a few'? I recently listened to The Cars _Greatest Hits_, a CD that I bought in 1985.


----------



## Drizzt321 (Nov 13, 2012)

neuroanatomist said:


> sb said:
> 
> 
> > I don't use any optical media (DVD/Bluray) because they become unreadable after a few years.
> ...



That's also a pressed CD, vs a burned CD/DVD. The material in a burnable CD/DVD is, generally, at least partially organic and over time degrades. Some of them are 'archival' quality, but I believe that's generally only guaranteed for only 40-50 years. Not sure, I'd have to look that number up, but I know it's not forever.


----------



## Emeyerphoto (Nov 13, 2012)

1. Dump on to external as soon as get home.
2. Modified images get saved on secondary external.
3. Backblaze backs them up onto the cloud, can recover them anywhere, and if necessary I can receive an external hard drive with my files on them for a fee.
4. Relax, since they are on the cloud and do not need to mess with safety deposit boxes, nuclear proof storage, or 10 external hard drives of the same data.


----------



## cayenne (Nov 13, 2012)

Lots of good posts here.
ONE thing that should stand out...redundancy, and multiple backup places and methods.

CDs/DVDs/BRs...all can degrade, and depend also on the quality of your starting materials...cheap will usually rot quickly.

I'm looking to build a fairly large, redundant NAS (Network Accessible Storage) system. I'm looking to use the freeNAS system:

http://www.freenas.org/features/feature-overview/category/features

Wikipedia:

http://en.wikipedia.org/wiki/FreeNAS

What makes this nice...is using the ZFS filesystem, which allows you to add different sized / speed drives over time...and can be hot swappable....and can lose 2 (or more depending on set up I believe) drives at the same time in the system, and still not lose data.

I just bought a special at newegg the other day, a barebones computer I'm about to assemble for about $168...8GB ram (which I'll likely upgrade to 16 for about $50 or so)...

Anyway, set that up with a 5 disk enclosure I bought awhile back to start with, and build it from there. In the future to have the comfort of redundancy *offiste* , in case of fire, flood, hurricane (I live in NOLA after all), I'm going to set up a similar system at my mom's house out of state...and that way, I can justify paying for her internet connectivity since it will be a 'business expense'.

Anyway, I'm getting off topic...but if you don't mind a little DIY spirit....take a look at the links and systems I've listed above. It is very high quality stuff, and can be turned into a highly reliable, flexible system that most any computer you own can connect to and backup, whether it be a Windows PC, or Linux box...or even OSX on a mac with time machine, which can often be finicky at times on its own right.

HTH,

cayenne


----------



## cliffwang (Nov 13, 2012)

Always keep your files duplicated. Any kind of media is fine. However, you have to ensure the files you backed up is always good. DVD is not my suggest. Nowadays external drive is very cheap, so you can get 3TB or bigger external drive to backup your files. Do not waste your money to buy too big external drive because hard drive price drops very fast.
Here is my setting.
* 6 2TB hard drives @ RAID 5 for master drive
* 1 3TB external drive for backing up important files
* Using MS SynToy to keep my files always updated.


----------



## Halfrack (Nov 13, 2012)

myocyte said:


> http://aws.amazon.com/glacier/



Glacier is an amazing service, literally, in the cost of backing up huge amounts of data 'just in case'. The catch is all in the amount of time it takes to upload all of what you currently have, depending on your internet connection. It'll cost you a fair amount to get the data back, but compared to 'never' it's damn cheap.

The dual drive solution is an easy step 1 for most folks. If you're looking for a cool tidbit, Sentry Safe makes a few models that have a USB2 pass thru - so the drive can be in the safe and yet online at the same time. The key to these 'data rated' safes is that the internal temp is much lower than other safes, and it's rated to how long it will keep the insides under 120 degrees.

Tape is a funny medium. It's amazing in that you don't have to worry about motors and stuff like with hard drives, but the drive itself is a huge expense. It is the only method to look at that has 20+ year archival bits - that and MO or UDO drives and media (small, expensive).

It comes down to why are you saving the images. 

Kids/family/vacations? Dual hard drives, maybe toss some onto USB flash memory cards (or old 1/2/4gb SD/CF cards - great way to justify faster/bigger/new cards). Lots of these are online as part of your smugmug/flickr/picasa/etc gallery right?

Paying work for clients? This is where you need to consider multiple mediums and methods. Online may not be acceptable, so you will want something that's plugged in and running (a NAS with redundancy (RAID) and a good filesystem (ZFS)), plus a copy that's not spinning. Depending on the price point charged, having it archived to tape may be a safe cya, or encrypt it and send it up to glacier.


----------



## neuroanatomist (Nov 13, 2012)

Drizzt321 said:


> neuroanatomist said:
> 
> 
> > sb said:
> ...



Makes sense, thanks!


----------



## Dylan777 (Nov 13, 2012)

cliffwang said:


> Always keep your files duplicated. Any kind of media is fine. However, you have to ensure the files you backed up is always good. DVD is not my suggest. Nowadays external drive is very cheap, so you can get 3TB or bigger external drive to backup your files. Do not waste your money to buy too big external drive because hard drive price drops very fast.
> Here is my setting.
> * 6 2TB hard drives @ RAID 5 for master drive
> * 1 3TB external drive for backing up important files
> * Using MS SynToy to keep my files always updated.



+1....I used to pickup 2-3TB drives, but I now I going for min 5TB. It's getting cheaper everyday though. Keep one at home and other at my in law house with monthly backup.


----------



## bkorcel (Nov 13, 2012)

More important is not the disk longevity but after 5 years when your DV burner stops working, you will buy a new one and there is a very high probability that it will not ready your DVD's. I've had instances where it can read sectors close to the spindle but as you move out, it gets out of spec.

I do not rely on CD or DVD media anymore. I keep my RAW and processed files duplicated on external USB 3TB drives and my processed files go out on the cloud. All originals get copied to at least two locations before I format the cards in the camera.

I've had an instance where a USB drive failed....because they do! Fortunately under warranty and I was able to robocopy the original onto the new drive.

Also note that magnetic media does not last forever either. Every few years you will need to dump the contents onto another disk, reformat, dump them back.

The cloud is my final last resource to prevent loss of processed images.





Drizzt321 said:


> neuroanatomist said:
> 
> 
> > sb said:
> ...


----------



## Halfrack (Nov 13, 2012)

Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written. A lot of times you can't tell if a file is corrupted until you attempt to open it. This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4). ZFS does a good job with this, but would only be used on a NAS type of setup.


----------



## neuroanatomist (Nov 13, 2012)

Drizzt321 said:


> ...I know it's not forever.



But what is? I've probably still got files stored on 5.25" floppy disks (and maybe an 8" or two, remember when things called floppy were actually floppy? ...dating myself again, I know). The data may be intact, but I have no way to read them...


----------



## ahab1372 (Nov 13, 2012)

neuroanatomist said:


> Drizzt321 said:
> 
> 
> > ...I know it's not forever.
> ...


Common problem, and a huge one for archives, libraries etc.
Every time we upgrade or dispose of hardware and/or software, we should be checking if there is data somewhere that we should be converting or copying. I'm not obliged to archive everything, and most files that I haven't looked at in a year or two, or was created with software that I cannot run anymore, are probably not worth keeping. But for pictures I might judge differently.


----------



## cliffwang (Nov 13, 2012)

Halfrack said:


> Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written. A lot of times you can't tell if a file is corrupted until you attempt to open it. This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4). ZFS does a good job with this, but would only be used on a NAS type of setup.



Too bad! ZFS does not apply on Windows system. That's the file system I really want for Windows environment. ReFS is the only solution for Windows system. I will try Windows Server 2012 next month after I return from my vacation.


----------



## Drizzt321 (Nov 13, 2012)

cliffwang said:


> Halfrack said:
> 
> 
> > Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written. A lot of times you can't tell if a file is corrupted until you attempt to open it. This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4). ZFS does a good job with this, but would only be used on a NAS type of setup.
> ...



You can still use tools like Par2 on Windows, which will generate the parity information alongside the regular file. This would let you get a higher level of confidence and reliability. You'll still need to scrub the files regularly to ensure the parity is still good, which ZFS has built in and you just cron it, as well as keep it on a separate, not always connected media, and then on a separate offsite location.

If you are going to have a NAS style machine at home for this (I'm guessing by your reference to Windows Server 2012), then why not have it running FreeBSD/FreeNAS or something similar that DOES support ZFS?


----------



## cliffwang (Nov 14, 2012)

Drizzt321 said:


> cliffwang said:
> 
> 
> > Halfrack said:
> ...


I used to have FreeNAS as my NAS. However, my wife complains I had too many systems. Thus, I have to reduce my toys. I have already phased out two systems. We still have few systems at home. My target is to have only one desktop(server) and two Windows 8 PRO tablets. If I build another server, I will get trouble with my wife. ;D


----------



## Drizzt321 (Nov 14, 2012)

cliffwang said:


> Drizzt321 said:
> 
> 
> > cliffwang said:
> ...



Hmm...maybe have one big honkin' machine that you run virtualized Windows on for your Windows needs? I've recently had the thought that maybe I'll do that for my Lightroom/PS needs...still haven't made up my mind on investigating that yet, but it's crossed my mind.


----------



## Promature (Nov 14, 2012)

I recommend Synology and WD Red Line drives. Synology is a great "NAS" and if you get a 2 bay solution, you can add a second drive later. Synology software is also much better thatn Windows Home Server 2008.
http://www.amazon.com/gp/product/B008JJLZ7G/ref=ox_sc_sfl_title_5?ie=UTF8&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B005YW7OLM/ref=ox_sc_sfl_title_6?ie=UTF8&smid=ATVPDKIKX0DER


----------



## kdw75 (Nov 14, 2012)

Because of changing media and technology I tend to favor cloud storage. I keep a copy on my computer and when my drive fills up I put in a larger one and put the old one in a drawer. This way I can always revert back to older drives if necessary. Then I also keep my stuff stored with a backup service since keeping it in that form doesn't rely on me having a particular computer or medium to read it. It is accessible any where I go with an internet connection. I probably should backup off site, but I haven't gone that far. I plan to get a RAID system at some point.


----------



## gmrza (Nov 14, 2012)

neuroanatomist said:


> Drizzt321 said:
> 
> 
> > neuroanatomist said:
> ...



A bit of reading I did into DVD/BluRay media recently basically indicated that there are no real standards for archival optical media, and quite honestly, your mileage may vary. - It is rather uncharted territory, and there seems to be no consensus as to how long a burned DVD or Bluray disk could last. (I have some CD-Rs which are around 20 years old, and still readable. - Is that pure luck? I have no idea.)

The bottom line is that for data backups you need to have a policy of copying to new media every few years. That however tends to happen by default as existing drives become too small and larger, cheaper drives hit the market.

To be heretical: the most reliable long-term storage medium for a photo still seems to be paper. Provided a print is stored under controlled conditions, you may achieve the rated life of around 200 years for archival paper. - I don't think any of us would be around to confirm. Of course, that does not answer the OP's question!


----------



## CharlieB (Nov 14, 2012)

I dump to primary HD on desktop. It automatically copies to secondary on my Windows server. The Windows server cross-pollinates with Linux server nightly. Done. Three copies, three machines, three hard drives.

CD... dont last, unless you do a slow burn on them. I've got two and three year old CD's... lots of problems with those.


----------



## dirtcastle (Nov 14, 2012)

If money is not an issue, I would go with a "cloud" solution. The main two problems with cloud storage is moving the files and host reliability. The issue of upload/download speed can be solved with a Fios or a fast cable connection. The reliability issue can be solved by having two cloud accounts and also have a local backup. Or you could do the double backup on your end and use a single cloud host.

While it's true that there will be some "bit rot" on large collections being stored over decades of time... unless you are the Smithsonian, I don't see why a little loss would be a major problem.

I think the main problem with long term storage is organization. By using a reliable, long term, cloud host, you can have a single storage solution with easy access and no barriers to organization. All existing "local" storage methods will have limits of both size and ability to organize.

If money is an issue... forget long term storage: carefully pick your favorites and buy cheap arrays. Reliable long term storage for large collections costs money; there's no way around it (at least, not that I know of).


----------



## Hillsilly (Nov 14, 2012)

I'm hedging my bets. I print my favourite photos at 5x7 or 8x12. I upload photos to an online site. I keep backups on multiple hard drives. I also have photos stored on the hard drives of a couple of computers. With film, I keep negatives in archival plastic sleeves. 

But what really worries me is that, at the rate I'm shooting, I'll have hundreds of thousands (maybe a million+?) images by the time I finish. Is anyone ever going to look at that many photos or care for such a collection?


----------



## papa-razzi (Nov 14, 2012)

Hillsilly said:


> But what really worries me is that, at the rate I'm shooting, I'll have hundreds of thousands (maybe a million+?) images by the time I finish. Is anyone ever going to look at that many photos or care for such a collection?



There will be data mining software that will recognize faces and other image content to pull up specific photos of people, events, etc.


----------



## cayenne (Nov 14, 2012)

papa-razzi said:


> Hillsilly said:
> 
> 
> > But what really worries me is that, at the rate I'm shooting, I'll have hundreds of thousands (maybe a million+?) images by the time I finish. Is anyone ever going to look at that many photos or care for such a collection?
> ...


You might wanna all read the TOS for the Cloud service you are using......THEY just might be using said data mining software using your images already for their own purposes.

I'm fairly sure three letter agencies are likely doing it quite prevalently...too large of a free data store for them to pass by.


----------



## Mt Spokane Photography (Nov 14, 2012)

What do you mean by "Long Term" Do you mean hands off as compared to periodic maintenance and renewal?

5yr?, 10yr?, 50 Yr?, forever?

There really are no 50 or 100 year hands off storage solution, too many issues can and will happen over that time, and you will likely have to move image files to different media every 10 years or so.
Solid State Memory - forget it
Hard Drives - better, but no
Online storage - no
Prints - possibly

Certainly, storing them online is a poor answer, companies go out of business, their systems fail, you forget to pay your bill, not a 50 + year solution at all. In about 2000, my hard drive crashed and everythinng was lost. I had a one year old backup on CD of my images, and recent backups on floppy disks.

The CD's had deteriorated in only one year and were unrecoverable, but I did recover most from the floppy drives. I then started making copies to hard drives as well as CD's and DVD's. I also switched to a raid system. Since then, I've never had a crash that lost data, but its a matter of when, not if.

I have a box full of 100 year old B&W prints and another box of negatives going back to 1940 of my ancestors. The color prints starting in the 1950's are faded away, and I have some color negatives from the 1940's that have faded as well. Some of the prints have been copied and sent to me by relatives, so the process of distributing images to relatives who might want to keep them seemed to work. I have scanned them and backed up the scans as well.

Current printing technology has been said to produce 100+ year life prints, so that might be the best backup method, prints and a fireproof box. Send copies to relatives as well. Many may still be arounnd in 100 years when the files have long been lost.


----------



## revup67 (Nov 14, 2012)

With respect to all of these ideas I wish to comment in that how are you backing up? Is it a straight copy without verification? Possibly a bad move. After using several different backup programs I found that Paragon's Hard Disk Manager Suite was an excellent product based on the what the product offers, their client base and tenure in the business and cost is a mere $50. There are other tools in the suite as well for partitioning, etc. You should be doing a data verification at the very least if using a copy method. Without verification you've got no way of knowing whether the file(s) were copied properly for a variety of reasons. Paragon will actually tell you if a drive is faulty as it checks the drive before backup. I do IT for a living and this actually happened at a client's. Ultimately I replaced her drive. In addition, I use Diskeeper. When files are created fragmentation takes place. In NTFS you have 4096 bytes in each cluster size. Each RAW or JPG or TIF (etc) can have numerous clusters which are tossed about the drive in no apparent order. When a file is called up to be read the drive spins (unless solid state) to assemble the file which creates heat, wear and tear, etc which all shorten the life of the drive. Solid state drives are not excluded from fragmentation of course. Diskeeper keeps your files defragged both in read and write mode and places your fave files and folders at the front of the drive (it does a lot more). It runs real time, small footprint and no performance impact. All HD drives on a single PC are kept at 0% fragmentation. You can run all kinds of reports on the drive. There are various indicators that will tell you the drives health as well. Images / Videos due to their size (IMHO) have the one of the biggest impacts on a drive's performance. I minimize my risks with both of these software tools and have yet to have an issue losing with my personal data or a client's. I've got a link to disk keeper on my site if anyone is interested drop me a note. I can get you a demo. PS they are now called Condusiv Technologies. Sorry if this sounded like an ad but its my personal experience and using these products daily. Been using Diskeeper for approx. 9 years and know it well. It's on every server I manage.


----------



## Drizzt321 (Nov 14, 2012)

revup67 said:


> With respect to all of these ideas I wish to comment in that how are you backing up? Is it a straight copy without verification? Possibly a bad move. After using several different backup programs I found that Paragon's Hard Disk Manager Suite was an excellent product based on the what the product offers, their client base and tenure in the business and cost is a mere $50. There are other tools in the suite as well for partitioning, etc. You should be doing a data verification at the very least if using a copy method. Without verification you've got no way of knowing whether the file(s) were copied properly for a variety of reasons. Paragon will actually tell you if a drive is faulty as it checks the drive before backup. I do IT for a living and this actually happened at a client's. Ultimately I replaced her drive. In addition, I use Diskeeper. When files are created fragmentation takes place. In NTFS you have 4096 bytes in each cluster size. Each RAW or JPG or TIF (etc) can have numerous clusters which are tossed about the drive in no apparent order. When a file is called up to be read the drive spins (unless solid state) to assemble the file which creates heat, wear and tear, etc which all shorten the life of the drive. Solid state drives are not excluded from fragmentation of course. Diskeeper keeps your files defragged both in read and write mode and places your fave files and folders at the front of the drive (it does a lot more). It runs real time, small footprint and no performance impact. All HD drives on a single PC are kept at 0% fragmentation. You can run all kinds of reports on the drive. There are various indicators that will tell you the drives health as well. Images / Videos due to their size (IMHO) have the one of the biggest impacts on a drive's performance. I minimize my risks with both of these software tools and have yet to have an issue losing with my personal data or a client's. I've got a link to disk keeper on my site if anyone is interested drop me a note. I can get you a demo. PS they are now called Condusiv Technologies. Sorry if this sounded like an ad but its my personal experience and using these products daily. Been using Diskeeper for approx. 9 years and know it well. It's on every server I manage.



I've used diskeeper in the past, worked fine. Generally if it's a single large disk that you simply dump photos onto without deleting, chances are you won't get much, if any, fragmentation. Paragon seems like simply a quality backup solution, but there's nothing inherently unique about it. Doing basic S.M.A.R.T. checks, there are a bunch of tools out there, and you can always use something like rsync and/or par2 parity blocks and checks which are F/OSS. However, Paragon is likely to do all that in a much easier to use, nicer to use, prettier package that will probably be well worth the cost since it seems fairly inexpensive.

That said, data at rest on a HDD is, over time, likely to experience bit-rot, which is where a proper backup solution will include some form of parity generation which can at worst detect a problem, and most of the time actually fix/recover the missing data. This you get automatically with a filesystem such as ZFS, but not NTFS, Fat32, Ext2/3 (not sure about Ext4, I don't think so). You can generate that information from my previously mentioned par2 generated parity blocks, or I'm sure there are other tools out there. Not sure if Paragon would do that or not, something to look for in the feature list anyway.


----------



## dirtcastle (Nov 14, 2012)

Mt Spokane Photography said:


> f
> 
> Certainly, storing them online is a poor answer, companies go out of business, their systems fail, you forget to pay your bill, not a 50 + year solution at all. In about 2000, my hard drive crashed and everythinng was lost. I had a one year old backup on CD of my images, and recent backups on floppy disks.



While it's true that many online solutions could be risky, I would trust the lifespan of Amazon over a consumer-grade hard drive. We've all had personal hard drives fail. I've also had hosted drives fail too. But the difference was that my online host had a backup system and restored everything without me paying any extra money or doing any extra work.

Ultimately, the best strategy is redundancy. For a home solution, it means arrayed drives (and a fireproof safe). But the level of cost and maintenance on such a system is formidable and you will be doing all of the work of maintenance, organizing, and syncing. And then, presumably, you will need to completely rebuild the system every 10 years or so. And without an associated online solution, you have only local redundancy.

For me, the big difference between a home solution and "hosted" solution is that the host will upgrade their equipment over the years. They will transfer your data and maintain backups. I would never suggest relying exclusively on a cloud host. Never. But let's recognize that storing data is what these companies do; and setting up an equivalent system at home is unrealistic for most photographers.

That said, as someone who has had hosted drives go down, I can understand people's wariness of using an online solution. And it's true that if the host goes down, those crusty backups will be the last line of defense. That was why, for super critical collections, I suggested having two online hosts.

It's all about redundancy and multiple points of failure. No single system will give a high level of reliability. All of these systems involve risk.


----------



## cayenne (Nov 14, 2012)

cliffwang said:


> Halfrack said:
> 
> 
> > Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written. A lot of times you can't tell if a file is corrupted until you attempt to open it. This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4). ZFS does a good job with this, but would only be used on a NAS type of setup.
> ...


Well, you know there _are_ other OSes out there besides Windows....some are VERY robust, etc.

You've heard the saying...the right tool for the right job.

If nothing else, go look at my posts about using freeNAS on this thread. It is close to turnkey...OS fits and boots from a USB thumb drive...has ZFS, and everything you'd need. 
Good tutorials written/video out there on the site....

A little DIY and learning a new thing...is never a bad thing.

I don't use windows much, unless a tool I need to use requires it...but I figure every OS has its uses....best not to get too dependent on any one, especially one owned by a company that can change its mind, direction or go out of business as your primary OS. 

You might get to start familiarizing yourself with Linux or other variants of Unix type OSes....they can really be your friend. Sure there is a learning curve.....but there is always a LC when learning any new, valuable skill, no?

;D

cayenne


----------



## cliffwang (Nov 14, 2012)

cayenne said:


> cliffwang said:
> 
> 
> > Halfrack said:
> ...


I used to use Linux system and built FreeNAS years ago. However, Windows systems are very stable nowadays. Now what I am looking for is a compromised solution between performance, simple, and few machines. What I might going to do later is building a Windows Server 2012 with ReFS system and run some VMs on it for different jobs. VHD will let me manage storage easily. I will play around and to see if I need to have an extra file server or not. Since my current file server is already 4 years old, I might build a FreeNAS next year if necessary.


----------



## Halfrack (Nov 14, 2012)

cayenne said:


> cliffwang said:
> 
> 
> > Halfrack said:
> ...



ZFS applies to Windows very well - it's over the network, same as Windows Server 2012 (you're not going to use 2012 as your desktop OS are you?). WinFS was supposed to be the end all, but it got dropped.

Too bad there are minimal if any options in the small form factor boxes that hold 4-6 drives. If you have been doing giant servers, your wife is correct in making you downsize - it's for your own good


----------



## etg9 (Nov 14, 2012)

NAS (Raid 5 or 6) to Cloud is what I safe is the safest bet. That covers corruption, accidental file damage, disk failure, enclosure failure, catastrophic damage, and regional damage. International damage would not be covered but if the US is gone then I don't need my pictures anymore. 

I have a Synology NAS at home and love it. It's small and quiet and has lots of options. I have it keep 5 snaps of any changes so I can revert if need be. There are cheaper things to buy, but their are tradeoffs and this one fit my needs. Pick a provider with a good long term track record and replace the NAS every 4-5 years

After that data gets sent off to the cloud for an offsite backup. I use Amazon Glacier now but have jumped around and another service may fit your needs better. I copy compressed JPEG only to Skydrive as a complete last ditch about every quarter. 

My photo backup is currently about 500GB (I don't keep too many raw) so that's only $5/mo. To get it all back out of there would be about $60, for DR this is cheap and Amazon is a good trusted name.


----------



## TrumpetPower! (Nov 14, 2012)

Lots of questionable advice, here.

First, do not _rely_ on Teh Cloud, in any form. If you don't have physical control over your data, you don't own it, and whoever _does_ own it can do anything they want to it and you've got very little recourse. That could include deleting it, peeking inside it, or even sharing it with the world.

If you're not overly worried about your cloud provider accidentally or intentionally sharing your data with the world, it can make a nice additional supplement to your data archiving strategy, but only as a "if everything else I'm actually relying upon goes tits-up, I'd hopefully still be able to get to it in the cloud" sort of last gasp hope.

With that out of the way, the only reliable method is to continually keep all your data readily available and online and part of your regular backup routine. As others have pointed out, old media die in lots of different ways. If you copy all your old media to whatever you're using today, you don't care if the old stuff dies for whatever reason, and you're also confident that you've got a valid copy. There's no worry that your several-year-old DVDs might be starting to delaminate, or that your Zip drives will have the click-o-death, or whatever.

Yes, that means you need bigger hard drives today, but the good news is that hard drives are dirt cheap compared to whatever you spent on your old media. A single DVD doesn't even store as much as a typical CF card. A hundred bucks gets you a hard drive that holds the equivalent of a few hundred DVDs. When that drive fills up, get another.

The simplest and most reliable backup method these days is to get three times as much disk space as you need. Disk(s) 1 are where you keep everything. Put a copy of everything on disk(s) 2. Every week (or month or whatever), take disk(s) 2 offsite to your bank deposit vault or your parent's place or somewhere you trust and exchange it for disk(s) 3, which you bring back with you and start treating as you used to do with disk(s) 2. The next week, do the swap again.

Also worth investigating, depending on your performance needs and your desire for tidyness, are RAID arrays. Be careful; many commonly-used RAID modes actually put you _more_ at risk for data loss than a single hard drive, meaning you need that much more redundancy in your backups to compensate. Safer RAID modes eat up more disk space. Duh! But you only want to think about RAID if a single disk isn't big and / or fast enough to hold all your stuff, and you should then think of the RAID array as a single disk that happens to have some extra moving parts.

IOSafe also makes near-indestructible hard drives: fireproof, waterproof, crushproof. They're more expensive than a regular hard drive, but very reasonably priced. If you're on a Mac, just get one (or three or however many you need) and point TimeMachine to it (them), and the only reason you'd need an offsite backup is if you're worried about theft.

Cheers,

b&


----------



## squarebox (Nov 15, 2012)

Amazon glacier starts to get expensive as you add more data. 500GB of storage seems to work out to $5 USD a month. I use 1and1.com as they hook me up with webspace, email addresses, and UNLIMITED webspace for just $7/month. Just ftp all my pictures up there. The only limitation is you can't have more than 250,000 files.


----------



## Leadfingers (Nov 15, 2012)

Lots of VERY questionable advice here in my opinion.

By far the easiest solution is to get a raid5 NAS box and use it as your primary storage medium. On a regular schedule you can make a redundant copy to an external drive and store it somewhere else.

When I hear stories of people backing up to DVD, the first thing I always think is, oh my god, how much free time do you have? I have 140 gigs of pictures and I'm sure that's a lot less than a true pro. Backing that up to DVD would require 28 disks for crying out loud. 

Drobo makes a pretty clever solution for all this.


As to the suggestions for FreeNAS...I haven't used it in two or three years, but when I did, I thout it was a debacle. It was far too complicated and required far too much user configuration. I think this was back in the version 7 days...has it improved since then?


----------



## dirtcastle (Nov 15, 2012)

Anyone know how various stock image companies store their data?


----------



## Mt Spokane Photography (Nov 15, 2012)

While it's true that many online solutions could be risky, I would trust the lifespan of Amazon over a consumer-grade hard drive. 

\
You are aware that some lost everything in the recent Amazon crash?? Some lost their files when their online photo services folded. Online storage has not really been proven to be a reliable system that you can upload to and expect to have your data 50 years from now. Nor is a hard drive, raid system, SSD, memory card, DVD, or CD.
You must actively keep multiple backups and move data as media becomes obsolete. That was my point, nothing is as reliable as those 100 year old prints that many of my cousins have in a shoebox in their closet. Some may be lost, some may burn up, but there seem to always be someone who managed to save them. I guess thats a case of many many backup copies to a media that does not lost the images after 50 or 100 years.
http://www.businessinsider.com/amazon-lost-data-2011-4

My Grandfather from around 1905. You had to sit very still then, no fast lenses or shutters.







My Grandmother born in 1886





The point is that we still do not really have media that can match the longevity of the old prints from 100 years or more ago, except to make and store prints.


----------



## brianleighty (Nov 15, 2012)

You do have a point but that outage was with EC2 not glacier. I'm not sure what the promised durability for EC2 is but glacier is very high due to the fact that it's stored in multiple facilities and multiple hard drives within that facility. I have not had faith to offer retaining data to my clients but Amazon's Glacier looks promising and relatively affordable. I'll of course keep a local hard drive with the data on it. But as you say, that's not enough. With Glacier, I can feel secure that even if my house burns down I'm ok. And I don't have to worry about trying to put data in other locations other than on Glacier.


----------



## dirtcastle (Nov 15, 2012)

@ MSP

I totally agree with you on the facts. Cloud storage has risks. Local storage has risks. And print is a great format, with a long shelf life.

With regards to "prints as backup storage", I think it boils down to volume. The problem with print storage is that it has scaling issues. For smaller collections (or people with lots of money), print is a great way to preserve and backup photos. But for most shooters, print doesn't work as a primary method of cold storage. It requires one to have both a digital storage system and an analog system. It's just too much work/money for most of us.

Now, having said all that... for selected images, having prints makes a lot of sense.


----------



## JerryB (Nov 15, 2012)

It might be the only effective way to save our photos is to print them out. Hard drives fail, DVD's, and CD's only last so long, and who knows if any of these will exist in a few years. But prints will still be there. 
Many companies use RAID array's which consist of multiple drives to backup their files. Some like, credit card companies, have a backup of their array's located halfway around the world.


----------



## tron (Nov 15, 2012)

Someone goes on a bus and validates two tickets instead of one! So a second person approaches him and asks him why he has done that.
First person answers: "In case one of the tickets is lost". But then the second person says: "Well, in that case you may as well lose both tickets".
First person answers: "I also have a monthly card" ;D

Well this joke reflects my opinion for the use of hard drives. I store my photos in many copies. I have not yet filled another disk to put it in a safe box though...


----------



## bkorcel (Nov 15, 2012)

Now that you mention it, Many safes now are coming with integrated power and USB hubs so you can lock up your hard drives or small servers inside to protect from theft. Some are even fire rated.



tron said:


> Someone goes on a bus and validates two tickets instead of one! So a second person approaches him and asks him why he has done that.
> First person answers: "In case one of the tickets is lost". But then the second person says: "Well, in that case you may as well lose both tickets".
> First person answers: "I also have a monthly card" ;D
> 
> Well this joke reflects my opinion for the use of hard drives. I store my photos in many copies. I have not yet filled another disk to put it in a safe box though...


----------



## cayenne (Nov 15, 2012)

Leadfingers said:


> Lots of VERY questionable advice here in my opinion.
> 
> By far the easiest solution is to get a raid5 NAS box and use it as your primary storage medium. On a regular schedule you can make a redundant copy to an external drive and store it somewhere else.
> 
> ...



freeNAS looks pretty straightforward from what I'm seeing.

I'll try to report back when I get the box constructed and running....

It is version 8.x.x something now.....

C


----------



## brianleighty (Nov 15, 2012)

For those that use Glacier, have you actually had to pull any data off? Reason I ask is their retrieval charge is astronomical if you don't spread it out correctly. Without some special software to this, I think this ones dead in the water for me now


----------



## etg9 (Nov 15, 2012)

As an IT guy who designs DR (just a part of what I do) for a living and like I said earlier in this thread a working copy on site, a back up on site and a backup offsite and in many locations as possible is the safest bet. 

I never told anyone to use the cloud as a single backup method. Tape is a poor backup solution for long term, as are DVD's but for different reasons. DVD's tend to get scratched or wear out, tapes no one has that model tape drive 10 years later and they are not backwards compatible. Single external hard drives die all the time. 

Keep you data yourself, on a NAS. Use the cloud as a backup to that. Very easy to manage and very hard to kill. 

Print is a bad idea for archival, I'm happy MSP's photos are around. a lot of people's aren't: fires, floods, tornadoes, earthquakes, burglars, or just getting lost have claimed lots of photos. They also take lots of room and are hard to move about and organize. This may work as a last ditch backup of very important family photos.

People telling you to keep backups in bank boxes or at parents houses tend to live close by, no geographic dispersion. that big earthquake/flood/fire may very well hit them too. 




TrumpetPower! said:


> do not _rely_ on Teh Cloud, in any form. If you don't have physical control over your data, you don't own it, and whoever _does_ own it can do anything they want to it and you've got very little recourse. That could include deleting it, peeking inside it, or even sharing it with the world.
> 
> Disk(s) 1 are where you keep everything. Put a copy of everything on disk(s) 2. Every week (or month or whatever), take disk(s) 2 offsite to your bank deposit vault or your parent's place or somewhere you trust and exchange it for disk(s) 3, which you bring back with you and start treating as you used to do with disk(s) 2. The next week, do the swap again.
> 
> ...



The Cloud, read the TOS for the provider and make sure it fits your needs. don't make the cloud (or anything) your single backup source.

The disk swap is a ton of work, hopefully the last backups were good and hopefully the disk didn't fail. when you fill up one disk you get to buy 3 more. don't do this. 

RAID (other than 0-1) also protects against hardware failure with an extra hot spare (5/6) or array (10/50). A NAS also doesn't have to be a single drive, the space can be logically separated.

or any other natural disaster.


----------



## JaxPhotographer (Nov 15, 2012)

I currently use RAID 1 (mirroring) on 2x2TB Drives in my primary computer just for image storage. I have SSD's for my OS and primary Apps and non-redundant HDD's for non-critical files.

I keep an active 3 TB External connected for nightly scripted backup of the RAID 1 files.

I keep another 3 TB External at work that I bring home once a month to backup and store as my off-site.

I replace all my drives every 2-3 years to reduce the risk of aging drive failures.

In the event I have any failure in either drive in the RAID 1 array, the other is still fully functional. I perform an immediate backup to the external, grab a spare 2TB spare, and swap the problem drive out and allow it to automatically rebuild the mirror image on the new drive.

Perhaps at some point I may consider cloud but have found that my current system has served me well and survived a hard drive failure once already. If I am able to increase client level work, then I will probably make the move to cloud storage as my off-site method.


----------



## gmrza (Nov 15, 2012)

Leadfingers said:


> Lots of VERY questionable advice here in my opinion.
> 
> By far the easiest solution is to get a raid5 NAS box and use it as your primary storage medium. On a regular schedule you can make a redundant copy to an external drive and store it somewhere else.



One point to take into account especially with RAID5 or RAID6 - you probably want to make use of enterprise drives (not desktop drives) to be sure that you get the reliability you want.


----------



## bycostello (Nov 15, 2012)

off site for sure...


----------



## el bouv (Nov 15, 2012)

You could also consider Thecus N7700 series.

Refer to http://www.thecus.com/product.php?PROD_ID=14

It basically seems to be Freenas with a reasonable management GUI in a 7 slot box with support for both Apple and PC and iSCSI et al..

Supports RAID 5 and 6 and now has a 10GE interface.

I run a 7x2TB and a 7x3 TB setup which yields 8TB and 12 TB formatted data space. I prefer RAID 6 as it can take more than 12 hours to rebuild a failed array. This puts a RAID 5 setup at risk during the rebuild window.

The Thecus does only have 1 power supply, so there is no redundancy and hence a single point of failure, but it has been good enough for SOHO use. In three years I have not had any failures except for drives.

I prefer normal Seagate drives, not the more expensive Enterprise quality, they last between 14000 and 18000 hours, that is between 1.5 and 2 years, and the price is OK. In my setup WD has had recurring failures in the first 1000 hours. 

Swopping drives are easy. The system is hot pluggable so you simply unplug the drive carrier, loosen the four screws that hold the failed drive, replace drive, refasten and plug back. The system detects the new drive and functions in Degraded mode until fully rebuilt. No real impact on user experience.


----------



## PeterJ (Nov 15, 2012)

Lots of talk of RAID but I hope everyone makes multiple backups in different locations, I'm pulling this number out of my backside but I suspect RAID protects against under 50% of real-life data loss. That includes viruses, application errors, user error, fires, lightning strikes and theft etc. Can also make recovery harder if you can't replace your EOL RAID controller with something compatible.

Personally I'd only use hardware RAID for performance or where 100% uptime was needed, otherwise spend the extra cash on extra drives for backup and leave a few you only rotate once a year or so in case you need to recover files from an earlier point.


----------



## AmbientLight (Nov 15, 2012)

RAID is nice enough to protect against disk errors, but it does not protect at all against errors of the RAID controller, which can be damaging to multiple disks in one instant.

I therefore recommend using RAID only for performance optimization and using separate disks/disk storage systems stored ideally at multiple locations.


----------



## bkorcel (Nov 15, 2012)

I agree. RAID makes sense for certain applications but it's still a single point of failure, flood, fire, theft. You still need to have a copy off site somewhere else. In this case RAID is not necessary and frankly a waste of money. You can save your money and buy a bunch of 3TB USB drives to replicate copies to.

I suppose you could REALLY go out on limb and buy two RAID systems, one to keep offsite at work or a relatives house.



PeterJ said:


> Lots of talk of RAID but I hope everyone makes multiple backups in different locations, I'm pulling this number out of my backside but I suspect RAID protects against under 50% of real-life data loss. That includes viruses, application errors, user error, fires, lightning strikes and theft etc. Can also make recovery harder if you can't replace your EOL RAID controller with something compatible.
> 
> Personally I'd only use hardware RAID for performance or where 100% uptime was needed, otherwise spend the extra cash on extra drives for backup and leave a few you only rotate once a year or so in case you need to recover files from an earlier point.


----------



## aprotosimaki (Nov 15, 2012)

I think it is important to distinguish between data availability and data integrity, since some of this discussion is conflating these two principles. RAID, for example, provides data availability but it does not provide data integrity as does, for example, off site backups. 

Think about it this way. If you build a mirrored pair of disks and copy all of your photographs to the raid set, you are still at risk of data loss; it is easy to accidentally delete a photograph and the raid set will not enable you to recover it because the data will be deleted from both disks. 

If you want both data availability and data integrity then implement RAID (raid 1, raid 5, etc) *and* then off site backups (off site here defined as media that is not in the same location as your raid set). 

In short, you need to back up your data even if you are using RAID. It is a serious error to think that raid provides data integrity. 

In my view it would be better to simply not use raid but copy your data daily to different media since data integrity is probably more useful for most people, but of course not all. This is what I do on a regular basis. I have one disk holding my photos and two other disks containing backups of that same data (I am paranoid). If my main disk fails, I am guaranteed of not losing any data and I perform incremental and full backups using these two disks.


----------



## etg9 (Nov 15, 2012)

el bouv said:


> You could also consider Thecus N7700 series.
> 
> It basically seems to be Freenas with a reasonable management GUI in a 7 slot box with support for both Apple and PC and iSCSI et al..
> 
> ...



Thecus is another well regarded brand. I believe the box has version control to guard against what people are talking about with accidental deletions, viruses, etc.

The snaps of the RAID I was talking about before was exactly this and what I have recommended. They work just like Time Machine or VSS if you are familiar with either of those. 

One of the things I don't like about hard drive pushing is that it's not automagic. You forget to back up to disk after a big download and 36 hours later your back at your computer to find a failed disk that data is gone. My NAS doesn't forget, my computer might bug out and install windows updates in the middle of the day when I'm gone but the operation isn't computer based so backups don't stop. If your backup spans more than one disk you're in more drives to your rotation. I have 7TB of files that I would like to take care of, not all of it goes up to the cloud but I would like to keep it around if I can and that would mean 9 single 3TB hard drives to do active/passive/offsite plus rotating them out so offsite is fresh every 2 weeks or month. too much work for my tastes.

Do you use the 10GBE link? with what switch? I've been thinking about picking up a 24 port dell for home use as it's one of the cheaper ones.


----------



## PeterJ (Nov 15, 2012)

etg9 said:


> The snaps of the RAID I was talking about before was exactly this and what I have recommended. They work just like Time Machine or VSS if you are familiar with either of those.


Sure ZFS solves that but it sounds like because of the volume of data you sync over a local network so it's physically close?


----------



## etg9 (Nov 15, 2012)

PeterJ said:


> etg9 said:
> 
> 
> > The snaps of the RAID I was talking about before was exactly this and what I have recommended. They work just like Time Machine or VSS if you are familiar with either of those.
> ...



Correct, and the out to the Amazon cloud for my offsite.


----------



## Ellen Schmidtee (Nov 15, 2012)

A few guidelines:

* No backup is forever, so refresh. Hard disks, DVDs, etc have a finite life time. At some stage it would corrupt, so you have no choice except making a fresh copy.

* Keep an off-line copy. An on-line disk exposes the files to viruses, and might be damaged due to power surge.

* Keep an off-site copy. A disk in your apartment might be damaged in fire, earth quake, flood, etc.

I stopped backing up to DVDs, because copying >100GB of data takes a long time, and distributing the work over time (e.g. start disk 1 before going to sleep, start disk 2 before going to work, start disk 3 before going to sleep, ...) makes tracking tedious.

BR would make it a lot easier, but around here both the BR-drives and the media are expensive to the point it's cheaper & faster to use hard disks.

I rent a locker in my gym, and keep a hard disk there. I go there often enough to rotate it.


----------



## thexfile (Nov 15, 2012)

Backup your files to Blu-ray. 25GB or 50GB discs. 

Technology will change before the discs ever go bad.


----------



## tron (Nov 15, 2012)

Ellen Schmidtee said:


> I rent a locker in my gym, and keep a hard disk there. I go there often enough to rotate it.


That way you keep fit too ;D


----------



## friedmud (Nov 15, 2012)

Currently I just use Apple time machine to back up to an external hard drive AND a Flickr Pro account where I upload full resolution versions of all my photos. In the case of catastrophe I could recover my photos from Flickr (albeit not the RAWs)

A good option is to use with Time Machine with two drives. Keep one at a friend's house and swap them out every now and again.


----------



## SilverSnake (Nov 15, 2012)

squarebox said:


> I use 1and1.com as they hook me up with webspace, email addresses, and UNLIMITED webspace for just $7/month. Just ftp all my pictures up there. The only limitation is you can't have more than 250,000 files.



Now this is a really bad idea. Yes, they give you what they call "UNLIMITED webspace", but it's limited to 50GB and gets increased once per day if you near the cap. They offer no guarantee that your data will be available and they perform no form of backup other than their "two geographically separated datacenters".

They also quite clearly say that if they terminate your account, for any reason, they will just toss all your data without giving you a chance to fetch them. Add to that that they expressively forbid you to use their service as a backup service, you are actively breaking the agreement doing so and if / once caught it'll all be gone anyway.

[quote author="1and1 ToS"]3.1.8.

You are responsible for backing up Your Data on your own computer. 1&1 does not warrant or otherwise guarantee that it will back up your data or that data which has been backed up can be retrieved, and will not be responsible for any archiving or backup of Your Data. If any of Your Data is damaged, deleted, lost or corrupted in any way, or becomes otherwise unavailable due to termination or suspension of your account pursuant to this Agreement, 1&1 will have no obligation or liability to you.

8.16.

You shall at all times use Web Site Space exclusively as a conventional Web Site. You shall not use the Web Site Space or Your Services in any way which may result in an excessive load on the 1&1 Equipment, including but not limited to installing or running web proxies, *using your allotted space as online backup or storage*, or mirroring mass downloads.[/quote]

You are much better off using a proper backup service, there's plenty of them out there and they can be had with unlimited storage for 5US$/month if you pay monthly and almost half that if you pay a year at a go. Don't be clever and exploit web hosting solutions, 1and1 isn't alone in having things like the above in their agreements.

Source for the above ToS quote: http://www.1and1.com/Gtc?__lf=Static&linkOrigin=linux-web-hosting&linkId=ft.nav.tandc


----------



## dawgfanjeff (Nov 15, 2012)

We don't really have a firm set of requirements here, and obviously that affects the advice. 

Archival 
I can assume that we want to protect against fire/flood, theft. For these you need offsite backups, or a good safe. Cloud offers automated backup while you're sleeping, but slow and not for free, on the upside, they handle all the messy RAID stuff for you. CD/DVD/BD sound great but are extremely time consuming and tedious, and you still have to be disciplined to physically take them someplace. 
Local NAS. On local network, so GB backups are fast. You can run them on demand, and/or scheduled. AND you don't have to worry about paying for bandwidth used. Downside, not as turnkey as some would like. I have a netgear readynas ultra 2. Two disks, mirrored (RAID 1). Best for redundancy, not for performance. Like other NAS solutions, it supports lots of other features, has an ftp server, DLNA, etc.., etc...lots more. Not saying its the best one in the world, but it's what I went with. I use robocopy scripts to copy data to it. 

Note, for scripts, onsite backups are fine, but it needs to be additive, NOT with a sync, else you run the risk of replicating the accidental deletion (d'oh!) or simply overwriting non corrupt data with corrupt data. If you are using robocopy, that means you want /e /s but NOT /mir. 

Retrieval/Media
How fast do you need to get the data back? An hour? Same day? A month?
For local workstation crash, accidental deletion. NAS gets the data back, fast and free, but of course there is upfront cost. Cloud gets the data back, ondemand, but SLOW, and might be subject to data consumption charges. CD/DVD/BD...gets the data back, but slow b/c you have to go get it. Tape I discount entirely. Too tedious, expensive and quite subject to obsolescence. If you'd archived with top of the line SCSI tape system 10 yrs ago, you'd be kicking yourself today trying to use it to restore data. 
I don't have a solution for this one either...yrs ago a few of us talked bout hosting ftp sites for one another at each other's houses, but that is hardly permanent/reliable either, plus the bandwidth hogging issue. I am thinking about 7zip ing all of it (with verify switch) and dumping it to USB3 removable storage and putting it in a safe at home every so often. Of course, neither USB nor sata is future proof, but its the closest thing we have right now, and it's cheap/scriptable.

One note about defragging...it's on by default in Win7, and presumably, Win8. Not sure diskkeeper is adding any value. In Windows server, you can schedule it with defrag.exe.


----------



## etg9 (Nov 15, 2012)

dawgfanjeff said:


> ... I have a netgear readynas ultra 2...



ReadyNAS was developed by a company called Infrant, they were aquired by Netgear. Up until they were aquired they were considered one of the best quality NAS boxes you could get. They don't have the sweetheart status they used to but they are still a great product.

I moved from Infrant to Synology and have been extremely happy.

I also agree with most of what you said, it's hard to tell someone what the best thing to do is when we don't know what he is trying to accomplish.


----------



## And-Rew (Nov 15, 2012)

Taking the OP at its basic question - i work with 5 EHD's as follows:

I work with an iMAc so have a large EHD plugged into permenantly to be used for Time Machine back up - i really love this back up system.

All image processing is done via Adobe LR so, files are imported via LR with an untouched copy being backed up to a second EHD. This EHD is backed up to a 3rd EHD via Chronosynch.

So, that's the original files covered.

All processed images are held within the LR database with side files on the iMac, but once an image is ready to be used - a copy is exported to a 4th EHD which, you guessed it, is backed up to a 5th EHD via Chronosync.

EHD's 2 and 3 are stored 1 at home and 1 at a family members with the same applying for EHD's 4 & 5.

Cost of this solution is a whopping £400 for all 5 EHD's and a copy of Chronosync. Time Machine comes free with Mac OS X and all hard drives are of the 1Tb or 2Tb capacity. As i run out of capacity, i epxect i will upgrade to a pair of 4 or 5Tb EHD's (by the time i run out of space) for another £150 in a few years time. Yep - still doing it cheaply 

So, the real price of all this is some cheap (and always getting cheaper) EHD's and a bit of time on my part - but that time can not compete with the cost of losing all any of my files :'(


----------



## Mt Spokane Photography (Nov 15, 2012)

dirtcastle said:


> @ MSP
> 
> I totally agree with you on the facts. Cloud storage has risks. Local storage has risks. And print is a great format, with a long shelf life.
> 
> ...


I'm not suggesting prints as a solution for the reasons you mention, just pointing out that with digital media, we do not have a nice reliable long term solution similar to the store it in a shoebox method so our descendents can view images 100 years from now.
I'm hoping that someone steps up and creates a storage media that is reliable, its certainly possible, but only available to the technically astute, and who knows if anyone could read the media in 50 years. Its not happening because no one sees a market, or maybe there is no good known technical solution (I doubt that).


----------



## dawgfanjeff (Nov 15, 2012)

I don't love the printing idea; however...all those pictures will be the most valuable thing you have when the zom-pocalypse occurs! Except bullets of course ;D


----------



## dirtcastle (Nov 15, 2012)

Mt Spokane Photography said:


> dirtcastle said:
> 
> 
> > @ MSP
> ...



Agreed!


----------



## etg9 (Nov 15, 2012)

dirtcastle said:


> Mt Spokane Photography said:
> 
> 
> > dirtcastle said:
> ...



We just really have no idea, 100 years ago they probably weren't sure the prints would be around now as the first prints in the mid 1800's didn't hold up so well. RAW is a little funkier as it has to get processed when opened but JPEG is a much more developed and solid standard. I can imagine that will be around in 100 years as it's already been around for 20 and it doesn't seem like it's going anywhere.


----------



## thedge (Nov 15, 2012)

squarebox said:


> Amazon glacier starts to get expensive as you add more data. 500GB of storage seems to work out to $5 USD a month. I use 1and1.com as they hook me up with webspace, email addresses, and UNLIMITED webspace for just $7/month. Just ftp all my pictures up there. The only limitation is you can't have more than 250,000 files.



FYI, if you read the terms of service for 1&1 (and most or all webhosts), using the FTP space as a backup is not allowed. Them finding out is different, but it is against the ToS.


----------



## mws (Nov 15, 2012)

Not by any means a feasible solution, but I wonder how many sheets of paper I would need to print out all my photos in a series on 1s and 0s………...


----------



## tron (Nov 15, 2012)

mws said:


> Not by any means a feasible solution, but I wonder how many sheets of paper I would need to print out all my photos in a series on 1s and 0s………...


 ;D ;D (poor trees)


----------



## bycostello (Nov 16, 2012)

http://www.sandisk.com/products/usb/memory-vault/


----------



## Mt Spokane Photography (Nov 16, 2012)

My Great Great Grandfather who moved from a real house in Upstate New York to a sod house in Kansas in 1874. Note that the floor was below the surrounding ground. The first winter, it rained for weeks, so you can imagine the wet muddy mess inside.
This is from a newspaper clipping, I haven't been able to track down the actual print. Those sod houses were replaced with log houses as soon as possible. It looks like they were laying out a new house from the dug out ground and the string and stakes on the lower right. I have a diary telling the story of that move.
If we suceed in being able to save our photos for 120 years, they will be a far cry from this one. However, its entirely possible that the glass plates for this one are in a university library or a museum somewhere in Kansas.


----------



## bkorcel (Nov 16, 2012)

The biggest problem I see with this idea is that after 100 years no one will remember what USB is or have anything that will read it! You'll have a wonderful silver paper weight. Kind of like all of these 3 1/2" drink coasters I have now.




bycostello said:


> http://www.sandisk.com/products/usb/memory-vault/


----------



## Drizzt321 (Nov 16, 2012)

bkorcel said:


> The biggest problem I see with this idea is that after 100 years no one will remember what USB is or have anything that will read it! You'll have a wonderful silver paper weight. Kind of like all of these 3 1/2" drink coasters I have now.
> 
> 
> 
> ...



Even beyond that, the real information on how it keeps your data from bit-rot via cosmic rays and such is non-existent. What type of parity/crc protection is built in? Is it using SLC or MLC NAND flash? What kind of write durability does it have? It also doesn't have any provision for data recovery. If you have to send it off for data recovery, will they let you RMA it? This seems like mostly a marketing tool, although I believe it will _physically_ last more or less as long as they say. But even assuming USB50 is backwards compatible with this (doesn't even say if it's USB2 or USB3!), you need it to be more than physically intact, you need to know that cosmic rays, marginal soldering, etc won't have corrupted the data or rendered it impossible to read off certain NAND dies, or even that the controller will still function fine.


----------



## And-Rew (Nov 18, 2012)

Mt Spokane Photography said:


> dirtcastle said:
> 
> 
> > @ MSP
> ...


----------



## Ellen Schmidtee (Nov 18, 2012)

tron said:


> Ellen Schmidtee said:
> 
> 
> > I rent a locker in my gym, and keep a hard disk there. I go there often enough to rotate it.
> ...



I rented a locker because the monthly fee for the gym + locker is a lot cheaper bank safe, and has more opening hours than a bank, e.g. the bank is closed on weekends.

Then the gym owner told me the lockers are for people who actually excercise, so I started doing that as well. Life isn't easy...


----------



## thedge (Nov 22, 2012)

And to answer the OP's question...

To me as an IT guy with data backup OCD the current best solution for longer term file storage is ZFS. Its the only solution that is reliable and has integrated file checksumming to check for bit rot, corruption, memory errors, etc. Yes you can run third party programs to do checksumming on any file system, but ZFS does it on file access in addition to a full scan whenever you want.

Significant Benefits to ZFS:
-File checksumming to check for corruption
-Scrubbing your drives, which is a complete read of every single file to check it against its checksum for corruption
-Hardware agnostic and "port agnostic", meaning a pool of ZFS drives can be moved to any other computer that can read ZFS, plugged in and read, without worrying about correct order of the drives in SATA ports (as with most hardware RAID solutions) or cross vendor compatibility (aka, an Areca RAID array cant be read on an LSI card), with ZFS you only worry about the pool software version which is easy to work with (backwards compatible and pool version can be upgraded etc)
-Immediate detection of failing hard drives, cables, interfaces, etc, far earlier than most solutions so far in my experiences with failing hard drives
-Resilvering a pool (aka, rebuilding an array) is based on data on the array, not the entire thing, so a ZFS pool that has 20TB capacity but only 2TB of data only has to rebuild the 2TB of data when a drive fails vs hardware RAID which rebuilds the 20TB including 18TB of nothing
-Copy on write, which leads to (among many other awesome things) snapshots of the data, which is fantastic for back ups (oops, deleted that file, but can be copied out of the snapshot that was automatically taken 15 minutes ago, or the one from an hour ago, two hours ago, or a day ago, or a week ago, or a month ago, or a year ago)
-Filesystem is always consistent, there is no file system corruption if the power cord is yanked out or some other sudden failure
-ARC and L2ARC (Adaptive Read Cache and Level Two Adaptive Read Cache) caching, which means that a file that is read from ZFS will get stored in the servers RAM (ARC) or on a designated SSD for faster access in the future

I could go on but those are the most of the juiciest ones.

I have all my images and my Lightroom catalog stored on my NAS which is running OpenSolaris and ZFS. Yes there is a performance hit for having the catalog over the network, but with ZFS's ARC and L2ARC it is quite mitigated. If I power the server off (which clears out the ARC) and power it up, then open my catalog there is a noticeable slowdown as it is read from the disks. Then after that it is faster as the parts of the catalog that are being read regularly are sitting in the RAM on the server. Same with previews, its slow the first time a folders previews are loaded then fast after that, even after Lightroom is closed and reopened since they are still in ARC.

ZFS is not perfect, it does need some computer knowledge to make use of it. Its other downside is it is not well suited to running on crap old hardware. It needs a 64 bit CPU and 4GB RAM to start. More RAM is better. It wont work well on that old Celeron in the basement for example. But barring that, its pretty fantastic.

For the curious people, the specs on my ZFS NAS are:
Intel Core i3-2100 3.1Ghz
16GB ECC RAM
Supermicro X9SCL motherboard
Three M1015 cards, each with 8 SATA ports (yes, 24 ports total)
2x 16GB SSDs as mirrored boot drives (aka, RAID 1)
4x 300GB Raptor 10,000RPM drives in striped mirror (aka, RAID 10)
2x 50GB SSD as write cache drives
1x 120GB SSD as L2ARC read cache
8x 500GB Seagates in RAIDZ2 (aka, RAID 6)
8x 1TB WD Scorpios in RAIDZ2 (aka RAID 6)

Yes, a lot of disks. The four 300GB Raptors and two 50GB SSDs are storage for my VMWare server. The eight 500GB drives store my photos, Lightroom catalog, previews, documents, email, backups, etc. The eight 1TB Scorpios I bought as cheap refurbs to store my many TB of downloaded TV shows and such until drive prices come down and I can upgrade with 8-10 3TB drives.


----------

