# ISO and long exposure issues:5Dmkiii



## jonjt (May 6, 2015)

Hello all,
I just purchased a 5Dmkiii, as an upgrade from a 7D. I tried doing my first long exposure night shot last night and came across some peculiar noise issues. If you take a look at the attached raw file, marked GWB.cr2, you will see there is significant rgb "snow" in the image. Some places in particular show pixels that seem to be stuck at 100% of their respective color values. I know the 5D is a noisy camera but, I expected it to be better than the 7D I came from. I've never seen this sort of issue with the 7D sensor, and I've taken quite a few long exposures at ISOs between 400 and 1000.

To investigate further, I took two shots with the lens cap on, for 450 seconds. As such, the exposure time is the same for the test shots as for the shot of the bridge in the GWB.cr2 file. One is at ISO100 and the other is at ISO800 (as you can see from the file names). The ISO100 file looks clean but the ISO800 file looks awful.

Just for comparison purposes, I took a shorter (30s) exposure at ISO800, with the lens cap on. The resulting image showed similar issues although, not to the same extent as in the attached pictures taken at ISO800. Short exposure images taken at higher ISOs, including ISO25600 do not show such behavior. They clean up quite well, in my opinion. To take the comparison 1 step further, I turned high ISO noise reduction on and tried again. The resulting body-cap-on images looked far, far less noisy. As such, it seems to be an issue that is positively correlated with high ISO, long exposure and disabled high ISO noise reduction, together.

Is this normal? I'm worried the camera sensor is malfunctioning. Or is it just me using poor technique?

If you are wondering, I didnt decrease the ISO because I didnt want to wait around for another 4 stops of light to hit the sensor. I assumed that ISO800 wouldn't cause any issues.

Particulars for the GWB.cr2 file:
70-200 f/2.8 mkii at 70mm
f/10
450s exposure
ISO800 for the bridge pic
5Dmkiii firmware 1.3.3
Lightroom 4.4. Imported directly from the card.
High ISO noise reduction and Long exposure Noise Reduction are both turned off.

The other two raw files (ISO100 and ISO800) were taken with the cap on the camera.

Files:
https://www.dropbox.com/s/mirtkfrc82oe9jq/GWB.CR2?dl=0
https://www.dropbox.com/s/i1a690fp98qf05p/BodyCapISO100.CR2?dl=0
https://www.dropbox.com/s/j74yvxbsx50z8ce/BodyCapISO800.CR2?dl=0

Help!





EDIT: THE ISO100 FILE WAS ACTUALLY TAKEN AT ISO50, SORRY FOR THE MISTAKE.


----------



## neuroanatomist (May 6, 2015)

I haven't looked at your images (posting from my photo) but, you say you're seeing noise in dark areas of long exposure images, and it gets worse as you raise the ISO. Totally unsurprising. Long exposures result in sensor heat, heat results in noise. There's a reason cameras used for astrophotography, and the cameras I use for long exposure fluorescent microscopy, have Peltier-cooled sensors. 

As an aside, hot pixels are pretty common. They can be dealt with by remapping (initiate a manual sensor cleaning with the lens cap or body cap on, let it run for a minute or so, then shut the camera off). Also, many RAW converters detect and remove them automatically.


----------



## jonjt (May 6, 2015)

Thanks for the reply.

So, it's not the fact that I have noise that is bothersome, it's the quantity and characteristics of noise that is. The luminance and color "noise granule" size is very large, much more so than I've seen in 5dmkiii shots a much higher ISOs and lower shutter speeds.. This is the case even in areas of the image that are well exposed (1 stop below being blown out). Also, there are multiple hot red and blue pixels in the image. And, worse yet, those hot pixels are not random. I was able to reproduce them in multiple exposures, at the same ISO and shutter speed eventhough the exposures were different.

I never saw such bad noise in my 7D so, i want to confirm that this isn't a sensor issue.


----------



## neuroanatomist (May 6, 2015)

RE the hot/stuck pixels, no they are not random - that's why they're called 'stuck'! Try the remapping procedure, or as I said if you shoot RAW your converter may just remove them.


----------



## jonjt (May 6, 2015)

It sounds like the behavior I'm seeing is normal so, I'll stop fretting. 

Long exposure noise reduction works pretty well so, if I have to shoot a long exposure at high ISO, I'll be sure to turn it on.

Thanks.


----------



## jrista (May 6, 2015)

The noise you are seeing is noise caused by dark current. I downloaded all of your images, and checked them out in PixInsight, an advanced astrophotography processing tool. 


As far as I can tell, you do not have any many pixels. Stuck pixels tend to be significantly brighter than the background noise in a dark frame all the time regardless of exposure time or thermal status. I did see a very small handful of such pixels...values around 3500 ADU when the background noise averaged around 513 ADU (ISO 800 dark).


The thing about dark current is it is an additional signal added to your image signal. As a signal, it includes it's own random (gaussian) noise. In a 450 second exposure at higher ambient temperatures, you could easily accumulate a dark current signal in the range of several hundred ADU. If your dark current was 300 ADU, it's noise would be SQRT(300) ADU, or 17.32. That is additional random noise, on top of read noise and photon shot noise, and separate from the increase in hot pixels (which are different from stuck pixels, as they change with exposure time and thermal factors.) 


None of this is unusual. The use of long exposure noise reduction (LENR) will take another 450s exposure after the image exposure, with the shutter closed, and subtract that from the image frame. That will usually remove hot pixels and stuck pixels, however it will actually also usually increase random noise, making the image noisier (even if the characteristic is better). 


If you wish to get the best results possible with your night imaging, you could take the astrophotography approach. Generate a master dark frame, and subtract that yourself, rather than using LENR. A master dark frame is generated by taking a bunch of frames, say 25 or 36, and stacking them together with an averaging algorithm. This single master dark could then be used to subtract the dark current and it's noise from your nighttime images. It can be reused, so long as the temperatures of your images is within a few degrees of your master dark, so you waste less time on-scene waiting for LENR to take an additional exposure after each and every frame. 

By averaging 25 dark frames together, you reduce the random noise by a factor of 5. Average 36, and you reduce it by a factor of 6. Average 100, you reduce it by a factor of 10. The lower the random noise in your master dark, the lower your random noise will be in your "calibrated" image frames. This will give you the best noise characteristics possible in your images, removing the hot and stuck pixels, the bias signal, as well as any dark signal offset.


----------



## jonjt (May 6, 2015)

jrista,
Thank you for the informative reply.

Could you tell me what the acronym ADU means?

When making a master dark frame, do I need to use the same exposure parameters as for the picture itself? So, for example, if I were to take a shot at ISO800 and 450s, would I then need to take X number of dark frames at the same exposure, for averaging? I'm assuming so but, I want to double check.

Finally, I'm assuming I can learn averaging and subtraction in GIMP but, I'm not certain that GIMP is the best tool for the job. Is there a astrophotography tool for averaging and subtraction that you would recommend?

Thank you.


----------



## jrista (May 7, 2015)

ADU just means analog-to-digital unit. It's just a term used to describe the digital value of the signal after it is converted by the ADC. Sometimes it's called a DN (digital number), same thing. 


Regarding dark frames, yes. Same exposure, ISO, and temperature. Temp doesn't need to match exactly, it's "good enough" to have them be close, say three degrees C, which is about 5.4 degrees F. If you just take a handful of dark frames after you are done with your long exposure night photography, that should be good enough. It might help to "warm the camera up" by taking some darks as well. Canon puts a rough temperature in the EXIF. It's not accurate enough to really gauge anything truly specific off of, but it's accurate enough that you can use it to gauge whether your in the right ballpark. If your lights and darks differ by about 6 F, your probably fine. 


Take a bunch of darks at the beginning, before "lights" (light frames, actual image exposures), to get the sensor temp up. Then take all your lights, and so long as you don't wait too long between light frames, the temperatures should remain consistent enough. Then take some more darks at the end. That should give you enough darks to make a decent master from.


GIMP is not the best tool for the job. Personally I use PixInsight, but it's pricey enough that I wouldn't use it unless you also do astrophotography. PixInsight has PHENOMENAL image "integration" (stacking and averaging and hot pixel rejection) tools, as it's a high end tool for astrophotographers. It also uses the VNG demosaicing algorithm, which handles Canon RAW data much better than AHD (which is what Lightroom, ACR, and most other RAW editors use by default with the exception of a couple.) You could use DeepSkyStacker, a free astro stacking tool that can create master darks. It also uses AHD, so it doesn't demosaic the data the best, but it is pretty much automated. It'll generate them as TIFF files. Technically speaking, if you put in an individual image exposure, it should actually also "calibrate" the image for you with the dark. Then you could just save the image out as a 16-bit TIFF, and work it in GIMP.


----------



## jrista (May 7, 2015)

I've done a little bit of processing on your image. (If you would prefer I not post these here, just let me know and I'll take them down.) I wanted to show you what was possible, so I brought your image and the ISO 800 dark frame into PixInsight. 


I used an advanced tool called PixelMath to subtract the dark frame from the image before the image was "debayered". This is just an easy way to calibrate an image with a dark frame. It was not done with a master dark, so the calibration is not ideal, but it was actually good enough that it removed most of the hot pixels, as you can see here:















These two images are before debayering (hence the black and white), and I doubled the size, so you can see the pixels. Subtraction of a single dark frame produced a much cleaner image just by itself. Subtraction with a master dark of say 10 frames (say 7 for warmup before starting your imaging, tossing two that don't match temp, and 5 at the end) would produce an even better result.


I then debayered the image with PixInsight's default settings. This uses a VNG (Variable Number Gradients) demosaicing algorithm, which as I mentioned before, handles Canon RAW data better than the AHD used by most RAW editors (I believe CaptureOne actually uses a derivative of VNG, FWIW). This version and it's crops are the "base" images. They are linear data that has been "stretched" with a tool in PixInsight called HistogramTransformation, and the color has been calibrated very basically to deal with the excessive green (which is not handled automatically by PixInsight like it is with other tools...no default tone curves have been applied, just some basic shifts in how much each color channels was stretched). This shows you the full dynamic range without any other edits:




















Note, there has _not _been any noise reduction on this image other than the subtraction of the dark. Goes to show you how good dark subtraction with long exposures really is, and this is basically what LENR will get you. Subtraction of a properly built master dark is even better. Subtraction with a full dark also removes the bias signal. In Canon cameras, that is actually a source of vertical banding if you have to lift shadow detail enough (as is necessary with this image). The use of VNG demosaicing avoids any other potential banding issues (demosaicing with AHD will actually likely cause banding to show up.) 


I then applied a brightening curve, just to show off the deeper shadows:




















I applied a proper contrast curve, to make the image pop more like a normal RAW editor would:




















And finally, I applied a moderate saturation curve:








Remember, the only noise reduction performed here was to subtract a dark frame. No other noise reduction of any kind was applied. Some of these 100% crops show random luminance and color noise...that can be taken care of with other tools, say Topaz DeNoise or Nik DeFine or Noise Ninja or whatever you prefer. But it's nice, clean, random noise, without the hot pixels, without the banding (although part of that is the VNG debayering, AHD will likely increase banding some.) 


Take some dark frames before each session. I'd say ~7, depending on how quickly the sensor heats up with the long exposures. You could star that an hour before blue hour (or whenever it is your going to do your imaging) with the cap on, and just let it rip. Then do your night photography. At the end of the night, before you pack your camera away, take another 5 dark frames. That will give you 10 darks, which is sufficient to generate a good master dark. An intervolometer helps here, as you can just set it and go. You can have the dark frame acquisition running while you drive to your location or whatever, and while you drive home, even.


----------



## Valvebounce (May 7, 2015)

Hi Jon. 
I don't know about the OP, but I found that most interesting, incredible difference from the dark frame, and soooo much good advice, especially the notion of taking the darks on the way out and way home, should save a lot of shivering, simple when you know how! 

Cheers, Graham.


----------



## AmselAdans (May 7, 2015)

The first post by jrista, which I actually enjoyed reading 

I am sure that there is a plethora of interesting posts by him buried here in the forum, but 90% of his postings I see (namely to those topics appearing at the front page) are debating the shortcomings of current technology.
Instead, I find it far more exciting to see the results Pros are able to squeeze out of current tech, such as it is done in this wonderfully illustrated post.


----------



## jonjt (May 7, 2015)

jrista,
Thanks for the posts. They are absolutely fantastic. I'll definitely incorporate those recommendations into my workflow in the future.


----------



## drmikeinpdx (May 7, 2015)

Very impressive post Jrista. Makes me want to take up astrophotography!


----------



## tron (May 7, 2015)

@jrista: Thank you very much for your technical posts. They are very educating.


----------



## jrista (May 8, 2015)

Glad everyone learned something. 

Here is one other option. Instead of debayering, it is also possible to use "superpixel" conversion, which simply takes 2x2 groups of pixels and converts all four into individual RGB output pixels. This doesn't do any interpolation, and gives you the full color quality and depth that the sensor is capable of delivering. It's the closest thing to a layered sensor you can get with bayer sensors, and since it is independently blending 2x2 matrices of pixels together, it improves noise.

The cost, however, is that you get half the resolution. Sigma, Foveon's proponent, argues that you never really had the greater resolution with bayer to start with. In terms of color, they are correct, however in terms of luminance (which is really where we get our sense of detail), they are wrong. Using superpixel conversion gives you a result somewhere in-between, and you can certainly upsample these images quite considerably for print if you need to, but maybe not quite as far as bayer interpolation. 

Anyway, here is another 100% crop, from a calibrated image, converted with superpixel blending rather than any form of interpolation (click to zoom it to the full 800x800 pixel size, same as the above images):






So, final set of images. I applied the same saturation curve in PixInsight, which produced this image:






Increasing saturation tends to exacerbate color noise a bit. Now, up till now, I have not applied any other form of noise reduction. Lightroom/ACR has some decent NR tools, but there are vastly more advanced and effective noise reduction routines out there than what Lightroom/ACR or other tools have to offer. PixInsight is packed full of incredibly advanced multi-scale wavelet or morphologic noise reduction routines that work wonders, as it's designed to reduce the excessively high noise levels you have to deal with in astrophotography. Using some of the more basic NR tools and basic NR masking techniques, I ran the above saturated image through ACDNR (Adaptive Contrast-Directed Noise Reduction), and applied unique settings to both the Luminance and Chrominance channels for best results (WARNING: These are maximum quality JPEGs without any optimization, in order to preserve the noise characteristics, which are usually lost with any amount of lossy compression...so these images are rather large in file size @ ~400KB each):

Before:





After:





Before:





After:





You may want to open each image in something with a dark background, like Photoshop, and stack them in layers that you can toggle on and off to see the difference.


----------



## jrista (May 8, 2015)

Alright, one more pass of another advanced noise reduction tool, and I'll really call it done. 

I used one of the more powerful noise reduction tools in PixInsight. Multiscale Median Transform, or MMT. This is a multi-scale wavelet noise reduction tool. It's highly configurable, and works best with properly attenuated. I extracted a luminance channel copy from the original image, then tweaked it to apply fairly strong attenuation of MMT. Without doing this, MMT will literally obliterate noise, gone, entirely...but will usually take a decent amount of detail with it. It will also apply the same to highlights as to shadows, however highlights have a higher SNR to start with, so we really want to apply it more heavily to the shadows than to the highlights. 

So, with the following mask:






I reduced noise further to produce this final image:






Here are the same close up crops:











Significant improvement in shadow noise levels, without any appreciable loss in detail. Perhaps someday more advanced noise reduction tools like this will find their way into programs like Lightroom, as they are extremely effective at performing significant NR on heavily lifted shadows, while leaving real detail largely untouched.


----------



## jonjt (May 12, 2015)

jrista,
Thanks for the excellent addition. Just for comparison purposes, what do you think of Topaz Denoise?


----------



## jrista (May 13, 2015)

jonjt said:


> jrista,
> Thanks for the excellent addition. Just for comparison purposes, what do you think of Topaz Denoise?



Topaz DeNoise is one of the best noise reduction tools out there. That said, it's intended to be used on otherwise fully processed images. It does great with banding reduction, and it can recover some DR. It is not side effect free, however...it WILL degrade your IQ to some degree. You probably couldn't get quite the results I've achieved here with DeNoise. You might be able to get fairly moderately close, but I still recommend subtracting a properly generated master dark.

It is very important to note that the master dark must be subtracted from the photographs BEFORE they have been debayered (demosaiced). Hot pixels spread out when they are debayered, as does all the rest of the random noise. With AHD noise can actually increase and take on a worse characteristic. As such, it is inadvisable to subtract a master dark from a debayered image. Any image you get out of Lightroom or load into Photoshop via ACR is going to be debayered, so dark subtraction will not really work.

Not many programs can properly subtract a dark. DSS (which is free) can, and it can generate the master dark for you. Iris (a very low level RAW editor) can. There might be a couple of other astro programs that can help you as well. Personally, I recommend PixInsight. If you do any kind of low signal strength imaging, be it just night photography at higher ISO or astrophotography, PixInsight is by far the most powerful tool on the market. It's about $250 (depends on the exchange rate), however for night photographers, it's lower level editing capabilities and advanced noise reduction tools are bar none the best available...as I hope you can see from the results above. 

I have started working on simple programs for astrophotography lately. DCRAW is the RAW processing library that PixInsight uses. I may at some point be able to create a simple command line tool that can perform basic pixel operations on images with DCRAW. Subtraction of bias and dark, division of flat, etc. Don't hold your breath for it, but it's something I've put on my list of things to write at some point (hopefully in the not too distant future.) There is a 45-day trial version of PixInsight that you could use to give it a whirl. There is also PixInsight LE, and older free version, although I am not quite sure how capable that version is (I fell in love with PixInsight about 10 days after trying the trial version, so I've never used the LE version.)


----------



## jonjt (May 13, 2015)

jrista,
My intended work flow would utilize Topaz denoise after the subtraction of the master dark. And so, I'm most interested in how well it performs on the luminescence and RGB noise, in comparison to the noise reduction tool you used. 

Your comment about how Topaz Denoise should be used on fully processed images raises another question. Should I be using it AFTER I apply any and all other PP steps/tools to the image? Currently, I apply lens correction, then use Topaz Denoise, then perform all other corrections/adjustments I need to perform. What do you think of that?

Thanks for the software recommendations. $250 is a bit much for me right now so, I may try one of the free alternatives.


----------



## jrista (May 13, 2015)

I am still curious how you intend to subtract your darks. It is pretty critical that be done on the non-debayered raw, otherwise you could end up with artifacts around hot pixels, which can sometimes be worse than the hot pixel themselves (i.e. funky ringing around the hot pixels, as hot pixels blend into their neighboring pixels during demosaicing, but with modern evaluative demosaicing algorithms, how they blend in the darks is not guaranteed to be the same as how they blend in the lights.) 

If you subtract the darks before debayering, then you should be fine with running DeNoise. I recommend running it last, as a wide range of processing steps can enhance noise. If you reduce noise mid-processing, then subsequent processing can increase your noise. Doing noise reduction at the end, you know how much noise you really have. Just keep in mind, DeNoise DOES blur detail. You can run it really mildly, and not have a very visible impact to your detail, but to run it more heavily (i.e. on par with what I did using PixInsight), your going to lose some detail.

That's the benefit of PixInsight. Everything almost all the tools are directly affected by masking, so you can use masks to attenuate and control the impact of NR, prevent it from affecting highlights, etc. Topaz filters are theoretically supposed to work the same way, but I've never been able to gain the same kind of control over NR with DeNoise as I have with PixInsight. DeNoise is trying to control the masking and everything for you, and how they combine their masks with any other masks you may have applied is a black box.


----------



## Aglet (May 14, 2015)

Thanks JRista, that's a good demo of what can be done with some really good processing tools and methods.


----------



## jonjt (May 14, 2015)

jrista said:


> I am still curious how you intend to subtract your darks. It is pretty critical that be done on the non-debayered raw, otherwise you could end up with artifacts around hot pixels, which can sometimes be worse than the hot pixel themselves (i.e. funky ringing around the hot pixels, as hot pixels blend into their neighboring pixels during demosaicing, but with modern evaluative demosaicing algorithms, how they blend in the darks is not guaranteed to be the same as how they blend in the lights.)
> 
> If you subtract the darks before debayering, then you should be fine with running DeNoise. I recommend running it last, as a wide range of processing steps can enhance noise. If you reduce noise mid-processing, then subsequent processing can increase your noise. Doing noise reduction at the end, you know how much noise you really have. Just keep in mind, DeNoise DOES blur detail. You can run it really mildly, and not have a very visible impact to your detail, but to run it more heavily (i.e. on par with what I did using PixInsight), your going to lose some detail.
> 
> That's the benefit of PixInsight. Everything almost all the tools are directly affected by masking, so you can use masks to attenuate and control the impact of NR, prevent it from affecting highlights, etc. Topaz filters are theoretically supposed to work the same way, but I've never been able to gain the same kind of control over NR with DeNoise as I have with PixInsight. DeNoise is trying to control the masking and everything for you, and how they combine their masks with any other masks you may have applied is a black box.



I wanted to use DSS for the subtraction of the master darks but, your comment about the noise performance of PixInsight has me wondering if the cost is justified. I also have not been able to master the mask capabilities of Denoise so, there might be an additional value add there.

Thanks for the comments.


----------



## jrista (May 14, 2015)

jonjt said:


> I wanted to use DSS for the subtraction of the master darks but, your comment about the noise performance of PixInsight has me wondering if the cost is justified. I also have not been able to master the mask capabilities of Denoise so, there might be an additional value add there.
> 
> Thanks for the comments.



Glad to be of help. If you do get PixInsight. It is an extremely powerful program, and it's built with an object-oriented UI. It's a rather different UI, so it won't look or behave like Photoshop or other common image editors. However do give it a chance. There is the 45 day free trial period....I strongly encourage you to take advantage of it.

A lot of things in PI are done by dragging instances of tools to instances of images to "apply" that tool. It seems unusual and inverted at first (at least, so I hear from other astrophotographers), but once you get the hang of it, it's a very fast and flexible way to work with your images. Also, previews are the most critical feature. Using previews, both static previews you can draw onto any image, as well as the real time preview, you can learn a lot of PixInsight's tools via trial-and-error. 

Learning PI's tools by trial and error is key to using it successfully. If you look around the web, you'll hear that PI has a steep learning curve. Taken as a whole, as a single monolithic application like Photoshop, this is true. It's about as complicated as Photoshop, and to learn it in it's entirety, it would take as much effort as it takes to learn Photoshop. The nice thing about PI is everything is done with independent, isolated tools, possibly with masks. If you take each tool as a whole little app to be learned in and of itself, learning PI becomes a much less daunting task.

For someone who does night photography, I think PI can certainly bring some new powerful ways to produce quality images to the table. In the end, I expect you will use PI for some things (calibration with darks, noise reduction), and PS for everything else (all the more artsy aspects of image processing...PI can export 16-bit TIFF, which you can then just bring into PS.) Both programs are complimentary to each other, use them both together, leveraging each one's strengths to get the most out of your imaging.

Also, you are always free to ask me for help when you need it.


----------



## jonjt (May 15, 2015)

Thanks for all the help!


----------



## johnnycash (May 16, 2015)

jrista said:


> The noise you are seeing is noise caused by dark current. I downloaded all of your images, and checked them out in PixInsight, an advanced astrophotography processing tool.
> 
> 
> As far as I can tell, you do not have any many pixels. Stuck pixels tend to be significantly brighter than the background noise in a dark frame all the time regardless of exposure time or thermal status. I did see a very small handful of such pixels...values around 3500 ADU when the background noise averaged around 513 ADU (ISO 800 dark).
> ...



Hi, I have checked your website and your astrophotography is amazing!!
http://jonrista.com/photography/astrophotography/deep-sky-nebula-and-galaxies/


----------



## privatebydesign (Jun 13, 2015)

I have been meaning to post to this thread for weeks! Hope it isn't completely buried.

Anyway, I also downloaded the RAW file and dark frame and in my opinion, whilst there is obviously a pixel level improvement from the Pixinsight, I just used Photoshop and Lightroom and think they did a very good job, certainly good enough for even the largest prints, at making the image a good one. 

The work that I did is very rudimentary and I am happy to elaborate if anybody is interested, but was simply process in LR, open as Smart Object, then open the dark frame as a Smart Object and bring it into the previous file as a layer, then choose blending mode 'subtract', play with a few settings and you are done.

Here are my crops as close to John's as possible. To be sure there is pixel level detail still clearly observable in the image, indeed it is a very sharp image, the handrails on the top cables and the poles on the top of the further tower are all pixel level details.

Hopefully the forum algorithms won't mess with the quality too much.


----------



## privatebydesign (Jun 13, 2015)

Here are some 100% crops from my LR + PS version showing great detail, sharpness and dark tonality.


----------

