# Bottleneck when opening images of CF card?



## V8Beast (Nov 4, 2013)

I'm planning out my next custom PC build, and want to narrow down potential bottlenecks. More specifically, when reading images directly off the CF card using a card reader, I'd like them to open up as quickly as possible. I'd imagine that this is determined by the read speed of the CF card itself, and the transfer rate of the USB interface. Opening up 200 or so RAW files at once seriously slows down my current system, so I'd like to cut down the wait time as much as possible. 

Dumb question #1: Does the amount of RAM factor into this equation at all?

Dumb question #2: Since the images aren't being copied into the hard drive in this scenario, does the HDD write speed matter for this specific issue? My guess would be no. 

I'm debating whether or not it's worth it to go with solid state drives for storage, or go with a standard HDD in a RAID 0 configuration to boost read/write speed. A 1 TB solid state drive still costs $600, and while they're blazing fast, that's still a lot of money.


----------



## vlad (Nov 4, 2013)

What do you mean by "opening" 200 RAWs at once? Do you mean importing them to your hard drive, or actually opening that many in Photoshop at once (seems extremely unlikely)? 

To import them faster, using a fast card and a fast (usb 3) card reader is your best bet. Amount of RAM won't make a noticeable difference, although with RAM it's always good to have enough to not worry about it.

SSDs versus HDDs for faster photo editing? I have experimented with keeping photos on an SSD, or keeping just my Lightroom catalog and previews on an SSD, and in my workflow (mostly Lightroom with a bit of Photoshop), I have not seen any performance boosts that would warrant major investment into SSD. The bottleneck with importing or exporting is usually with Lightroom itself: the disk and the CPU don't even come close to 100% utilization for these time-intensive tasks.

Also, I would strongly recommend against raid0, unless you have a great and reliable backup system that you can't wait to try out.


----------



## pwp (Nov 5, 2013)

I get very fast downloads from CF card to PC using either a USB3 Sandisk card reader or a blindingly fast internal SATA card reader. Obviously fast cards will help. 

I wouldn't be storing on an SSD... far too expensive and unnecessary. On my two current home-built PC's I have the OS (Win7) and programs on a modest sized 180Gb Intel SSD, and local storage on a 2Tb HDD, then archive network storage on a five-HDD Synology NAS. BTW, I wouldn't go with Win 8. It's a dog in the tradition of Vista and the dreaded ME. Win 7 is the new XP. 

If you were really needing maximum speed, you could keep a small folder on your SSD named "Current Project" and import your current job, do your work and then shunt it out again once you've done your post-pro. Photoshop scratch would also be on the SSD. 

Here's a good tip, buy your RAM from http://www.crucial.com/ If you give them the specs of your EXACT mobo, they identify & send you the most compatible memory possible for that board. 

-pw


----------



## timmy_650 (Nov 5, 2013)

What a few of my computer friends do is they put a smaller SSD drive in to put the operation system on it so that will run faster and they put a normal hard drive (2tb range).

So what I am guessing what you are asking; When you are transferring you Pictures to you computer, you are worried about transfer speed. So if you card reader is usb 3 you should be fine or if you build it in your computer (sata) you should be fine.


----------



## fugu82 (Nov 5, 2013)

+ 1 on the above comments. Buy a very fast USB3 reader, and don't go through a bus. 

I use a Lexar LRW300U and it's been great.


----------



## Drizzt321 (Nov 5, 2013)

A good quality USB3 reader (personally I love my Lexar Pro one) along with a quality spinning disk to hold all of the files, and then I store the Lightroom Catalog + Previews on an SSD. My OS and stuff lives there, and those don't take all that much more space. Plus with the LR5 Lossy Previews, I can keep the files elsewhere if I need to and just export later when I get home.


----------



## V8Beast (Nov 5, 2013)

Thanks for the great advice, fellas.



vlad said:


> What do you mean by "opening" 200 RAWs at once?



Yeah, that's what I meant 



> SSDs versus HDDs for faster photo editing? I have experimented with keeping photos on an SSD, or keeping just my Lightroom catalog and previews on an SSD, and in my workflow (mostly Lightroom with a bit of Photoshop), I have not seen any performance boosts that would warrant major investment into SSD. The bottleneck with importing or exporting is usually with Lightroom itself: the disk and the CPU don't even come close to 100% utilization for these time-intensive tasks.



Good to know!



> Also, I would strongly recommend against raid0, unless you have a great and reliable backup system that you can't wait to try out.



My plan, as others have suggested, was loading the OS and major software programs onto a small SSD. Then, I planned on utilizing a RAID 0 on a standard HDD for storage, and having a separate RAID 1 HDD to back up the two RAID 0 HDDs. If there is no real benefit in system performance from the RAID 0 setup, I will gladly forgo the added complexity of the RAID 0+1 arrangement. Instead, I'd probably just opt for a RAID 1 setup, and back that up onto another external HDD so I'd have triple redundancy of my images.


----------



## V8Beast (Nov 5, 2013)

pwp said:


> I wouldn't be storing on an SSD... far too expensive and unnecessary. On my two current home-built PC's I have the OS (Win7) and programs on a modest sized 180Gb Intel SSD, and local storage on a 2Tb HDD, then archive network storage on a five-HDD Synology NAS. BTW, I wouldn't go with Win 8. It's a dog in the tradition of Vista and the dreaded ME. Win 7 is the new XP.
> 
> -pw



That's a slick setup, even more so since it can be setup as a cloud. 

And yes, I will gladly pass on Windows 8. My wife's new laptop has it, and I hate it. I thought Microsoft got its act together after the Vista debacle, and Windows 7 was a refreshing improvement, and then it comes out with a rubbish OS like Windows 8. No thanks.


----------



## Mt Spokane Photography (Nov 5, 2013)

If you are ope3ning files, they will load into RAM, and opening 200 at once takes a lot of ram. A 30mb CR2 file uncompresses to more like 50mb, and 200 of them is 10GB. With overhead for the OS and Lightroom, you are going to gobble up 16GB of ram, I'd get 24 or 32.
Your lightroom catalog needs to be on a SSD if you want fast opening, its a gigantic file. I use a 512GB SSD for my OS, Lightroom Catalog, and all of my programs (256GB is plenty). My data files are on a 3TB SSD and a 12GB NAS.

SSD's and memory are cheap now , and a Samsung 256 GB SSD for a boot disk to hold programs and Lightroom Catalog as well as scratch disk for photoshop is usually plenty large, and costs less than $200.
You will get more speed by spending $400 on memory and a SSD over a upgrade to a $1000 processor.


----------



## Drizzt321 (Nov 5, 2013)

V8Beast said:


> My plan, as others have suggested, was loading the OS and major software programs onto a small SSD. Then, I planned on utilizing a RAID 0 on a standard HDD for storage, and having a separate RAID 1 HDD to back up the two RAID 0 HDDs. If there is no real benefit in system performance from the RAID 0 setup, I will gladly forgo the added complexity of the RAID 0+1 arrangement. Instead, I'd probably just opt for a RAID 1 setup, and back that up onto another external HDD so I'd have triple redundancy of my images.



Ok, so, you seem to have a mis-understanding of what RAID 0+1 (or 1+0, aka RAID10) is. It's doing a strip (RAID0) across mirrors (RAID1), or mirrors across strips. As opposed to having a separate RAID0 then a separate RAID1.

Plus, RAID is not a backup, regardless of if it's 1, 5, 6, 10, 50, etc. It's a means to higher availability and sometimes better performance.

If you're after performance, invest in a decent SSD. Otherwise, if you want storage space/reliability, go for RAID1 or RAID6 (RAID5 is relatively fragile). 0+1/1+0 is best if you want a better balance between performance & reliability.

I agree, backup to an external source (and then to another geographically distributed place) is the correct idea.

If I were you, I'd invest in a better CPU, and then do a RAID1/6 on your storage drive if you want enhanced reliability with a proper backup strategy. And either invest a significant amount of money into a proper RAID controller...or go for Windows software RAID. It'll be much more reliable and theoretically portable as opposed to whatever motherboard half-assed software RAID that they do.


----------



## pwp (Nov 5, 2013)

Mt Spokane Photography said:


> My data files are on a 3TB SSD and a 12GB NAS.


Awesome! You have a 3Tb SSD? Is this a typo or did you mean HDD?
The biggest SSD on the planet is 4Tb and costs an eye-watering $29,000
http://semiaccurate.com/2013/06/23/lsi-puts-out-a-4tb-pcie3-ssd/

-pw


----------



## V8Beast (Nov 5, 2013)

Drizzt321 said:


> Ok, so, you seem to have a mis-understanding of what RAID 0+1 (or 1+0, aka RAID10) is. It's doing a strip (RAID0) across mirrors (RAID1), or mirrors across strips. As opposed to having a separate RAID0 then a separate RAID1.



You're right. I read up some more about RAID configurations, and I have lots more research to do. RAID6 is completely over my head. I think I'll just stick with a standard HDD for storage and back up to an external drive


----------



## Drizzt321 (Nov 5, 2013)

V8Beast said:


> Drizzt321 said:
> 
> 
> > Ok, so, you seem to have a mis-understanding of what RAID 0+1 (or 1+0, aka RAID10) is. It's doing a strip (RAID0) across mirrors (RAID1), or mirrors across strips. As opposed to having a separate RAID0 then a separate RAID1.
> ...



Or RAID1 (mirror), that's pretty basic and easy and will help ensure high availability and avoid having to restore from a backup in the event of a single disk failure. Of course, still backup properly, but RAID1 is a viable part of that.


----------



## Halfrack (Nov 5, 2013)

OP needs to look at Photo Mechanic - http://www.camerabits.com/products/ - if you're going to open up lots of files directly, it renders faster than Lightroom, but it isn't a full on photo editor.

If you're going to open that many files at once, you are spending more money on getting top end CF cards - or import them onto a SSD and open from there. It adds the benefit of not deleting the only copy of an image.


----------



## RustyTheGeek (Nov 5, 2013)

OK, IT guy here. Sorry for the long post but I hope this advice helps.

I've read everything up to now and it all makes sense and I agree. So far, the info, comments, opinions, corrections, etc all add up and make sense. No need to repeat it all again.

My primary advice is that you take your time. Build it, install Win7Pro x64 OS (great OS choice at this time) to a decent SSD like maybe an Intel 520 series or Samsung 840 Pro series and then systematically install all the updates and find the most up to date drivers. Get a solid motherboard from ASUS, Intel or several other reputable companies. Don't buy cheapo motherboard+CPU deals. Buying the motherboard is usually my hardest decision. Make sure the BIOS is fully updated. Make sure the USB3 firmware is fully updated. Put your catalog and/or photoshop cache files on the SSD. Store the actual image files on a secondary large RAID protected volume. Don't buy cheapo RAM. Stay with Crucial, Micron, Kingston, PNY, G.Skill, Patriot, etc. The reputable motherboard manufacturers usually have lists of RAM they certify to work that they tested.

Once all that is done, do an image backup and start benchmark/stress testing the system with file transfers, benchmark utilities, etc. Make sure you are getting the performance of the CPU, RAM, Drives, Motherboard, Video, etc that you paid for.

Oh, and get a high quality power supply. Get a large tower case. (Antec 300 is a good choice.) You'll be adding drives, believe me. Get something that is at least 750W. Real watts, not cheapo watts. Something like a good Corsair unit. If you paid less than $100 for the power supply, it probably isn't good enough. Newegg has great informative IT geek reviews and will likely help you get a good unit for your needs.

Put this system and the display on a UPS with AVR. My Costco regularly carries the CyberPower 1350AVR for around $100. It's a great unit and it's on Newegg for around $145. Just get one with AVR. Direct power from the wall (the grid) isn't nice to your sensitive electronics. I have a good UPS on everything in my home that is electronic and worth more than $100. You know, like that $3500 plasma tv! It's not a guarantee against calamity but it's better than nothing.

Don't go overboard on the video card for a photo pc. Most of the performance benefit will be gained from the SSD, CPU, RAM, USB3, then video, etc. Spend the video money on a nice high end IPS display and calibration tool. Maybe get dual displays but one large high end IPS panel is better than two cheapo LCD panels. Put in plenty of storage in RAID 1 configurations for working data storage and individual internal and external drives for backups.

The hard part isn't just the hardware, it's setting up the software, getting an effective sync/backup system in place and then keeping up with it. You can't have too much backup. Consider an offsite solution as well, online or physically carrying external drives to another location or both. Online is good for data but not for recovering the system so you need a local image backup of at least the system drive.

Finally, don't use this nice fast photo pc to do everything. If you install a bunch of stuff like QuickBooks, Office, HP Printer software, huge antivirus security packages, etc, you will kill the performance. Just use it for photography. If you don't use it to surf porn, play online games or do your online banking, don't install antivirus or just install the perfectly fine Microsoft Antivirus. Seriously, you'll thank me later.


----------



## tolusina (Nov 5, 2013)

I simply cannot imagine why you would want to open a large collection of file directly from the memory card, maybe open a very few by file name as you've noted on your camera's info screen.
For a large quantity of files, copy them all to the PC first.
---
I'm also getting ready to build a new PC.
I'm figuring on two 256GB internal SSDs, multi layer BluRay discs and 1-2 TB hard drives for back up.
Backup hard drive(s), only one to start with, will slot into one of these.......
http://www.newegg.com/Product/Product.aspx?Item=N82E16817998185

I was planning on building in a card reader, didn't care for the number of negative reviews of the ones I found at NewEgg.
Then I found this set up which is Jeff Cable approved......
http://www.bhphotovideo.com/c/search?Ntt=Lexar+HR1+Hub&N=0&InitialSearch=yes&sts=ma&Top+Nav-Search=
Readers can be used individually or up to four can be mixed and/or matched into the hub, contents of up to four cards can be downloaded to your PC simultaneously. Even four of the fastest cards should not be able to saturate the USB3 bus.
---
Back to PC drive configuration.
I briefly considered some RAID possibilities, researched a bit and decided not.
Here's my plan...
Install one SSD, install Window 7, maybe partition the drive into halves for dual booting, 128GB should be more than plenty for my non-gaming OS and programs. My current XP with about 70 programs lives on a 50GB partition that still has 1/3 free space.
Windows up and running, second half of the primary drive left un-formatted, Shut down, connect second drive, reboot.
Use Windows Disk Management to format (and partition if desired) the second physical drive, create a single folder in the root of that fresh E(?) drive, call that folder something clever like eMyDocuments. 

In XP, I'm hoping Win 7 is similar, right click on the "My Documents" folder on the desktop, select >Properties.
In the Properties tab is a window where you can specify "My Documents" target location, type e:\eMyDocuments or simple select the browse tab. Windows should ask if you want to move/copy/migrate/whatever the contents of the current My Documents, choose YES!! My Photos, My Music and all those will move right over along with each folder's special properties.

Windows has now "symbolically linked" or "mapped" your "My Documents" to your second SSD, since the default save location for most programs is My Documents, those programs will now automatically save to you second drive.

The idea here is for the OS and programs to be on one physical drive, data on another, each can be backed up separately.

At this point, I sure don't know exactly how DPP and LightRoom will want to play with this set up, when offered options, I'll aim ALL data files to the second SSD.

The external drive dock will be for hot swapping in bare HDDs or SSDs as I choose for back ups.




.


----------



## vlad (Nov 5, 2013)

tolusina said:


> Here's my plan...
> Install one SSD, install Window 7, maybe partition the drive into halves for dual booting, 128GB should be more than plenty for my non-gaming OS and programs. My current XP with about 70 programs lives on a 50GB partition that still has 1/3 free space.
> Windows up and running, second half of the primary drive left un-formatted, Shut down, connect second drive, reboot.
> Use Windows Disk Management to format (and partition if desired) the second physical drive, create a single folder in the root of that fresh E(?) drive, call that folder something clever like eMyDocuments.
> ...



A word of warning. I built my current Win7 system such that the OS is on an SSD and the Users folder is on a 3TB Raid1 HDD. I used audit mode to set it up (google audit mode user profile for more info). The system works fine, but it won't allow an upgrade to Win8. It's just not a supported configuration. Which is a bummer, because I actually quite like Win8, and would love to upgrade this machine instead of having to reinstall.

If you are building a new Win7 or Win8 system and want good separation of your OS and data, just use the Libraries feature. You can create your documents folder on your external HD, or Raid1, or whatever, right click it, Include in Library, and you're set.


----------



## JPAZ (Nov 5, 2013)

Interesting. I, too, am working on a new machine. My plan is to have an SSD for the OS, Software and Lightroom Catalog as well as for the PS "scratch" drive (what we used to call virtual RAM). I presently have 2 1TB drives in a RAID 1 (mirror) configuration but I am planning on two 3 TB drives in the new machine. Over the years, the mirror RAID has saved my hide more than once, when a single HDD failed. All I needed to do was replace the faulty drive with a new one and the mirror rebuilt itself. 

But, the RAID setup does not replace a backup. If both drives fail or if the computer / motherboard fails or if the house burns down (you get the idea) you can lose everything even though the drive is self-duplicated. Being a big believer in the belt and suspenders approach, I also have a continuous backup of my photos to the "cloud" (I use Crashplan) and do periodic backups to an external HDD, as well. The initial synch of my very large collection of phot files to the cloud took a while, but now it runs continuously in the background, I'll continue this plan with my new PC.

I use Win 7 Pro 64 bit and will continue to do this. Be sure to get as much RAM as you can (at least 16 GB). Another thought.....don't skimp on the cooling. Heat is one of the biggest dangers to the internals of a computer. Fans are cheap and most cases give lots of options for cooling fans.


----------



## Drizzt321 (Nov 5, 2013)

JPAZ said:


> Interesting. I, too, am working on a new machine. My plan is to have an SSD for the OS, Software and Lightroom Catalog as well as for the PS "scratch" drive (what we used to call virtual RAM).



Are you sure you don't mean Virtual Memory? Where the OS automatically overflows to disk? Or are you referring to something specific in Photoshop where it'd use a specific drive for data?


----------



## JPAZ (Nov 5, 2013)

"Virtual RAM" and Virtual Memory are the same thing, I think. That is when the PC uses the hard drive as RAM because the physical RAM is otherwise occupied.


----------



## pwp (Nov 5, 2013)

RustyTheGeek said:


> OK, IT guy here. Sorry for the long post but I hope this advice helps.
> 
> _(trimmed)_
> 
> Finally, don't use this nice fast photo pc to do everything. If you install a bunch of stuff like QuickBooks, Office, HP Printer software, huge antivirus security packages, etc, you will kill the performance. Just use it for photography. If you don't use it to surf porn, play online games or do your online banking, don't install antivirus or just install the perfectly fine Microsoft Antivirus. Seriously, you'll thank me later.



Thanks RustyTheGeek, that's a very fine, generous post. Home PC builders should bookmark this one. 

Your final comment is an area that is too often overlooked. My photo PC's do just that...all that's installed are the required Adobe CC components, PhotoMechanic, Firefox, Microsoft Security Essentials and a handful of required utilities.

All the Office, MYOB, iTunes and basically unnecessary entertaining stuff is on a separate modestly powered machine, in this case an entry-level Mac Mini running OSX Mavericks and Windows 7 installed via VM Ware.

-pw


----------



## Drizzt321 (Nov 5, 2013)

JPAZ said:


> "Virtual RAM" and Virtual Memory are the same thing, I think. That is when the PC uses the hard drive as RAM because the physical RAM is otherwise occupied.



Ah, you were using the terms interchangeably. I've never really heard the term Virtual RAM associated with it since it's not the appropriate industry/technical term. At least not that I'm familiar with.


----------



## RustyTheGeek (Nov 6, 2013)

pwp said:


> RustyTheGeek said:
> 
> 
> > OK, IT guy here. Sorry for the long post but I hope this advice helps.
> ...



Thanks for the kind words *pwp*. Yes, these days hardware is cheap. People sometimes don't realize the benefits of having more than one physical system. Esp if they have kids!! Another option is running a separate VM system. I've been doing this for over 20 years and it still amazes me how poorly written most software is. It's actually much worse than it was even 10 years ago. Lazy or overworked programmers and unrealistic deadlines and/or not much talent. It's sad but this is just the poor state of affairs most software companies are in these days. Look at how long it has taken to get a version of Windows out in the past 10 years and then look what we ended up with! And Adobe regularly releases buggy code.

Most of the time it pays to keep things as simple as possible.


----------

