# I tested out the theory behind DxOMark's F-Stop blues article



## Kit Lens Jockey (Dec 29, 2016)

I was kind of put off when I read the DxOMark article referencing how very fast lenses on digital cameras are a waste due to the sensor's limited ability to gather light from a lens with a very wide aperture. You can read the article here, but the basic gist is that digital sensors are somewhat incapable of registering light from very wide aperture lenses, so cameras secretly bump up the ISO at wide apertures in order to compensate for this.

https://www.dxomark.com/Reviews/F-stop-blues

I really didn't want to believe that dropping serious money on a 50mm f1.0 was completely pointless, so I decided to do a sort of "blind" test with my 5D Mark III.

I set the camera up on a tripod, attached a Canon 50mm 1.4 lens to the camera, obtained focus, and then removed the lens, stuck a piece of tape over the electrical contacts on the lens so that the camera could not discern what kind of lens was attached to it, and I took a photo in manual mode at 1/50 of a second and ISO 1000. I then did the same thing with a Canon 50mm 1.0 lens.

This gave me two photos, one taken with a 1.4 lens wide open, and another with a 1.0 lens wide open, both at the same camera settings, and both taken so that the camera had no way of knowing what lens it was taking the photo with.

The result? the 50mm 1.0 was noticeably brighter than the 50mm 1.4. I guess if I really wanted to I could get into a big technical analysis of whether or not the 1.0 lens gave _exactly_ a 1 stop increase in brightness or not. But for me, right now, just knowing that having a 1.0 aperture does actually make a difference on a digital camera is enough.

Here are the photos. Photo 5540 is with the 50mm 1.0 lens, 5541 is with the 50mm 1.4. The only modifications I made to them were to convert them both to sRGB and resize them down to 3000px. I admit the focus on the 1.4 image is pretty bad, but for the purposes of this, I wasn't too concerned with perfect focus.


----------



## raptor3x (Dec 29, 2016)

Playing around with the exposure of the two images in LR and looking at the histograms I'd say there's only ~1/3-1/2 stop difference in exposure between the two, which is what you would expect if you extrapolated DxO's data.


----------



## Kit Lens Jockey (Dec 29, 2016)

Hmm, interesting. Maybe it is pointless to have this lens after all. Is going to f1.2 from f1.4 more significant of a difference than going from f1.2 to f1.0? The wide aperture 50mm that I most commonly use is the Sigma 50mm art, but I was considering changing to the Canon 50 1.2 to gain that extra little bit. The f1.0 is fun to play around with, but I'm too worried about something happening to such an expensive and non-repairable lens to it to use it frequently.


----------



## Kit Lens Jockey (Mar 28, 2017)

So, I was curious to see how the new sensor in the 5D4 would handle super wide apertures in comparison to the 5D3. So, I got out the 50mm f1.0, focused it, taped off the electrical contacts, and mounted it on a 5D3 and then a 5D4 back to back at the same shutter speed and ISO settings.

Both of these are shot at a 10 second exposure and an ISO of 100. The cameras they were taken with are in the file names. These are straight out of the cameras, just re-sized to the same size and converted to sRGB. Unfortunately I moved the tripod a little while switching cameras. It might just be the difference in white balance, but the 5D3 actually looks _brighter_ than the 5D4. :-\

Could this mean that the 5D4's higher megapixel count hinders its ability to pull in light from very wide aperture lenses? Hard to say, but seems possible.


----------



## AlanF (Mar 28, 2017)

The real iso sensitivity often differs from the nominal, as measured by DxO. The MkIII is reported to be slightly higher iso than the MkIV. https://www.dxomark.com/Cameras/Compare/Side-by-side/Canon-EOS-5D-Mark-IV-versus-Canon-EOS-5D-Mark-III___1106_795


----------



## C-A430 (Mar 28, 2017)

What is the point of comparing between different lenses? You were supposed to test the same lens at f1.0 vs f1.4

24-105 f4 never gets more than t5 - it is partly because aperture is not open at f4 and partly because its low quality glass reflects part of the light.


----------



## Kit Lens Jockey (Mar 28, 2017)

C-A430 said:


> What is the point of comparing between different lenses? You were supposed to test the same lens at f1.0 vs f1.4
> 
> 24-105 f4 never gets more than t5 - it is partly because aperture is not open at f4 and partly because its low quality glass reflects part of the light.


And how exactly was I supposed to stop down the f1.0 lens to f1.4 while also preventing it from communicating with the camera so that the camera could not make its own ISO adjustments behind the scenes as the article states?


----------



## neuroanatomist (Mar 28, 2017)

Kit Lens Jockey said:


> C-A430 said:
> 
> 
> > What is the point of comparing between different lenses? You were supposed to test the same lens at f1.0 vs f1.4
> ...



Set the aperture to f/1.4 (or whatever), press and hold the DoF preview button, then unlock and very slightly rotate the lens barrel so the electronic connection is broken.


----------



## Kit Lens Jockey (Mar 28, 2017)

Aren't the apertures spring loaded to return to full open when power is removed from the lens? Or does it require power to move each way? Either way, I don't want to fool around with such a rare lens too much.


----------



## neuroanatomist (Mar 28, 2017)

Kit Lens Jockey said:


> Aren't the apertures spring loaded to return to full open when power is removed from the lens? Or does it require power to move each way? Either way, I don't want to fool around with such a rare lens too much.



Well, I don't make a habit of suggesting things that don't work. Incidentally, that 'trick' is how people who use macro reversing rings are able to use a stopped-down lens. It won't harm the lens. 

As was pointed out, using two different lenses, which have different T-stops relative to their F-stops, is a significant confounding factor in your test.


----------



## midluk (Mar 28, 2017)

You could shoot the same f/1.0 lens at f/1.0 with and without the contacts taped to see how much correction the camera applies.

By having a look at the aperture scaling values in the camconst.json file of RawTherapee, you can learn that it indeed seems like the 5D4 has a higher scaling than the 5D3. There are no measured values below f/2 yet for the 5D4, though.
So to get real results, shoot a completely saturated image with the f/1.0 lens with different apertures and see how the camera scales the saturation values in the raw file. Assuming Canon has implemented correct scaling factors in the camera, that should be the numbers you are searching for.


----------



## gruhl28 (Mar 28, 2017)

Interesting subject, Kit Lens Jockey; thanks for taking a first stab at testing this. I agree with some of the previous comments, though - you can't draw conclusions comparing different lenses as they may have different transmission. And it would have been better to adjust the shutter speed also when comparing f/1.0 with f/1.4. 

I think midluk had the best suggestion - shoot at f/1.0 with the contacts taped and untaped and compare - that will show how much the camera compensates. You could also shoot at f/1.4 using Neuro's procedure, but halve the shutter speed (twice as long). The exposure at f/1.0 at one shutter speed and the exposure at f/1.4 at half the shutter speed (twice as long a shutter speed) should look the same if the sensor is able to capture all the light. If the f/1.0 exposure is darker, then you have proven what DxOMark has written. You can then use Lightroom or PS to see how much darker the f/1.0 exposure is. 

Please let us know the results.


----------



## IglooEater (Mar 28, 2017)

midluk said:


> You could shoot the same f/1.0 lens at f/1.0 with and without the contacts taped to see how much correction the camera applies.



+1. 

Op, thanks for doing the tests, we don't all have a 1.0 to experiment on!


----------



## hendrik-sg (Mar 28, 2017)

For me, the comparision is reasonable between 2 lenses. Maybe it depends on what for one buys a fast lens.

The testshows, that the 1.0 lens transmits not much more light than the 1.4 lens. To gatter most light possible, the 1.0 is of very few value, it seems to be only about bokeh then.

For me this is a big point, because i use the fastlenses for low light applications. In the 35m range it would be specially interesting, just because the 2.0 IS lens is really good and much less expensive then the 1.4 version and has IS on top


----------



## Mikehit (Mar 28, 2017)

hendrik-sg said:


> For me, the comparision is reasonable between 2 lenses. Maybe it depends on what for one buys a fast lens.
> 
> The testshows, that the 1.0 lens transmits not much more light than the 1.4 lens. To gatter most light possible, the 1.0 is of very few value, it seems to be only about bokeh then.
> 
> For me this is a big point, because i use the fastlenses for low light applications. In the 35m range it would be specially interesting, just because the 2.0 IS lens is really good and much less expensive then the 1.4 version and has IS on top



It depends on what the question is.
As I understood the OP, at very wide apertures the camrea bumps up the ISO artificially but at narrower apertures _with the same lens _it does not (need to?) do that.
So if you use two different lenses, you have to consider not only the amount of light coming into the camera through the aperture but also the amount of light the glass itself transmits. If the two lenses transmit different amounts of light then the comparison becomes flawed especially if the false-ISO is accounting for less than one stop of illumination.

Then again my assumption behind the OP may be incorrect.


----------



## neuroanatomist (Mar 28, 2017)

Kit Lens Jockey said:


> I really didn't want to believe that dropping serious money on a 50mm f1.0 was completely pointless, so I decided to do a sort of "blind" test with my 5D Mark III.
> 
> But for me, right now, just knowing that having a 1.0 aperture does actually make a difference on a digital camera is enough.





hendrik-sg said:


> The testshows, that the 1.0 lens transmits not much more light than the 1.4 lens. To gatter most light possible, the 1.0 is of very few value, it seems to be only about bokeh then.



Interesting that two people look at the same images and come to opposite conclusions... The one who spent $$$$ on a 50/1.0L concludes that it was money well spent, the one who didn't spend $$$$ concludes it would not be worth spending it. 

'Round these parts, we call that *bias*.


----------



## scottburgess (Mar 29, 2017)

The conclusion made in the article is that this is a problem with the sensors. However, the data presented do not fully support this conclusion. To more carefully tease out the relative contributions of lens and sensor one could do this experiment: hook up a Canon 35mm film camera, load some Fuji Velvia, and compare exposures of a flat target at f/1.2-1.4 with the same exposure at other apertures.

If the exposures are noticeably different, then the lens could be contributing to the problem. If the film frames are indistinguishable (ignoring DOF and vignetting), then the sensor and gain are probably the primary culprits. I am wondering if vignetting or metering matters here, or if the manufacturers are cheating the measurement of the maximum aperture by 10-20% or so like they do for the focal length of lenses.

Anyone want to try that out? Might be fun if the article turns out to be wrong. I'm not funding the research, mind you, but will kibitz from the sidelines.


----------



## Mt Spokane Photography (Mar 29, 2017)

I've always thought that the reason for buying a ultra wide aperture lens was more related to shallow depth of field than getting more light. There are known ways to design cameras and lenses to do a better job at ultra wide apertures, but the cost is very high. Still, we have seen recent patents trying to improve on the situation by using modified pixels or microlenses near the edges so they pick up more light.


----------



## sulla (Mar 29, 2017)

IF the sensor really didn't at all register light incident at high angles (coming form the outermost part of the lens, i.e. from the parts of the lens between 1.4 and 1.0) then the bokeh should not be different either. So, if this was true, very high aperture lenses should not provide better bokeh than more narrow lenses.
So an alternative test for the DxO-claimed effect could be to compare the diameter of bokeh circles of the EF50 1.0 at 1.0 and at 1.4. In this test it would not be necessary to block the contacts, as the camera could forge ISO, but cannot forge bokeh.

On the other hand, IF the lens transmission in the outermost regons was only gradually reduced, this would make all and every very high aperture lens an apodization lens, making its bokeh only much smoother. So, I guess, the money would be well spent... 
A test could be done to compare the shape of bokeh circles of the EF50 1.0 at 1.0 and 1.4. If circles at 1.4 show a distinctive edge and those at 1.0 don't, then this should measure the "sensor apodization".


----------



## applecider (Mar 29, 2017)

Doesn't vignetting difference at progressively wider apertures account for the lower amounts of light that DXO is measuring? And shouldn't the actual iso be noted in some exif if it is being changed under the hood? And finally the newest body that I saw listed in the referenced article was the 5d, just seemed odd. Granted it was 2010 would current bodies handle iso in the same way one would want to know. That is change to a higher effective iso as apertures get larger, and were in body lens corrections on and available then?

And to the original poster, where did you get those models? Voodoo? They kinda creeped me out.


----------



## midluk (Mar 30, 2017)

applecider said:


> And shouldn't the actual iso be noted in some exif if it is being changed under the hood? And finally the newest body that I saw listed in the referenced article was the 5d, just seemed odd. Granted it was 2010 would current bodies handle iso in the same way one would want to know. That is change to a higher effective iso as apertures get larger, and were in body lens corrections on and available then?


I have already pointed to the camconst.json file of RawTherapee.
That unfortunately does not include the original 5D, but for newer Canon cameras (also 5D4), it can be concluded that the values in the raw file are silently multiplied by a global constant correction factor (<1.2).


----------



## IglooEater (Mar 30, 2017)

neuroanatomist said:


> Kit Lens Jockey said:
> 
> 
> > I really didn't want to believe that dropping serious money on a 50mm f1.0 was completely pointless, so I decided to do a sort of "blind" test with my 5D Mark III.
> ...



We also call that different needs and priorities..


----------



## C-A430 (Mar 30, 2017)

sulla said:


> IF the sensor really didn't at all register light incident at high angles (coming form the outermost part of the lens, i.e. from the parts of the lens between 1.4 and 1.0) then the bokeh should not be different either.



Well said. That also opens another question - does the same LENS on same CAMERA camera at same APERTURE give different bokeh at center and near the edges?




applecider said:


> Doesn't vignetting difference at progressively wider apertures account for the lower amounts of light that DXO is measuring? And shouldn't the actual iso be noted in some exif if it is being changed under the hood? And finally the newest body that I saw listed in the referenced article was the 5d, just seemed odd. Granted it was 2010 would current bodies handle iso in the same way one would want to know. That is change to a higher effective iso as apertures get larger, and were in body lens corrections on and available then?
> 
> And to the original poster, where did you get those models? Voodoo? They kinda creeped me out.



Actually it is widely known that (RGB) micro-lenses on digital sensors increase vignetting (more vignetting than on film). According to rumors everyone is working on technology to bypass that, but it seems none has been implemented. Also it is believed that mirrorless will be the first to get that technology since they are most affected (flange distance?).

In meantime camera manufacturers (supposedly) add a cheat to make sure it is difficult to make a test showing that from f1.8 you are getting diminishing returns (less sharpness and more vignette but barely any difference in light gathering or bokeh) and with it they protect sales of f1.4 and f1.2 lenses.


----------



## Sporgon (Mar 30, 2017)

I seem to remember reading somewhere that the best digital camera to gain most benefit from an f1 or 1.2 lens was the original 12.7 mp 5D :-X

No wonder the prices are going up


----------



## IglooEater (Mar 30, 2017)

Sporgon said:


> I seem to remember reading somewhere that the best digital camera to gain most benefit from an f1 or 1.2 lens was the original 12.7 mp 5D :-X
> 
> No wonder the prices are going up



I would ass-ume a larger pixel works better with a higher angle of incidence. I wonder how the A7s does.


----------



## Sporgon (Mar 30, 2017)

IglooEater said:


> Sporgon said:
> 
> 
> > I seem to remember reading somewhere that the best digital camera to gain most benefit from an f1 or 1.2 lens was the original 12.7 mp 5D :-X
> ...



I think the article I read was before the time of the A7s. With an adapter and the longer distances from rear element to sensor needed for the EF f/1 or 1.2 I imagine it would be better than the old 5D. However with native f/1.2 lenses, should they exist, on the much shorter flange distance I am guessing that it would not be as good.


----------



## funkboy (Apr 6, 2017)

Mt Spokane Photography said:


> I've always thought that the reason for buying a ultra wide aperture lens was more related to shallow depth of field than getting more light. There are known ways to design cameras and lenses to do a better job at ultra wide apertures, but the cost is very high. Still, we have seen recent patents trying to improve on the situation by using modified pixels or microlenses near the edges so they pick up more light.



Yep, IIRC Leica claims they do that on their rangefinder cameras as the wide-aperture angle of incidence is even higher than a DSLR due to the short flange-back distance.


----------



## AJ (Apr 7, 2017)

All lenses exhibit vignetting (peripheral light fall-off) to some extent.
For a given lens, vignetting is typically worse at lower fstops.
Is this a function of optics, or the angle with which the light hits the sensor, or both?
Suppose it's optics, and the vignetting extends all the way to the center of the image. Then fstop is indeed a meaningless ratio and we should talk about tstop. Furthermore we should then recognize that tstop varies across the image (gets worse toward the corners).


----------



## gwooding (Apr 7, 2017)

After reading this thread I was intrigued so I decided to do some tests myself.
The gear I used is
1DX MK2
7D MK2
50 f1.2.

I shot 6 different scenarios with each camera:
Shutter Speed: 1/40s f1.2 contacts not taped
Shutter Speed: 1/40s f1.2 contacts taped
Shutter Speed: 1/40s f1.8 contacts not taped (effectively 1EV underexposed)
Shutter Speed: 1/40s f1.8 contacts taped (effectively 1EV underexposed)
Shutter Speed: 1/20s f1.8 contacts not taped
Shutter Speed: 1/20s f1.8 contacts taped

I shot in RAW and loaded the images into Capture One Pro 10. I used the "Linear Response" curve. I white balanced the images using the gray card. All lens corrections were turned off. I then added a colour readout in the middle of the gray card to assist with quantifying the difference. This readout is also roughly in the center of the frame to eliminate the impact of vignetting (which can be clearly seen in how bright the corners in the f1.8 1/20s images are).

Looking at the 1DX results:
The first thing to notice is that there is a definite reduction in exposure when shooting at f1.2 and taping the lens contacts. In order to rectify this difference I had to add 0.2EV to the exposure in Capture One.
In order to match the exposure of the f1.8 1/20s I had to add 0.35EV to the exposure of the f1.2 taped image.

As a sanity check I tested out how much I needed to add to the f1.8 1/40s image to match the exposure of the 1/20s image and I found adding the expected value of 1.0EV to be correct.

I believe that we can therefore conclude that on the 1DX MK2 the 50 f1.2 loses 0.35EV wide open due to this phenomenon.

Interestingly the 7D results differ
I found that in this instance to match the correct exposure with the taped f1.2 I had to add 0.6EV (adding to 0.4EV to the 1EV underexposed f1.8 1/40s image matches the f1.2 taped exposure).

From this test it seems that the 7D reduces the advantage of a wide aperture lens even further. The question remains as to whether this is due to the higher pixel density or micro lens structure. I think if we could get someone to perform a similar test on a 5DS and something APSC but with a low pixel density we could get a better understanding.


----------



## gruhl28 (May 8, 2017)

gwooding said:


> After reading this thread I was intrigued so I decided to do some tests myself.
> The gear I used is
> 1DX MK2
> 7D MK2
> ...


I'd lost track of this thread, surprised that no one has responded to these test results. Thanks for posting this, gwooding. These look like the most valid results so far. They seem to confirm that there is a loss of light with fast lenses, although at least with your lens/camera combinations it isn't a complete waste using a faster lens; there is still some exposure benefit.


----------



## Arahn (May 8, 2017)

So this means that when I am shooting in very low light situations, let's say f1,2 ISO 25600, it would actually make sense to step down to f1,6 ISO 25600, underexpose -0,66 EV and later ramp up the image by 2/3 of a stop in post? The result then would be the same level of noise as the f1,2 image but feature even have more DOF!? 

To make my trail of thought clearer*:
My Settings: f1,2 - ISO 25600 - 1/100s
Camera really shoots with: f1,2 - ISO 40000 - 1/100s (+0,66 ISO to compensate for "lost light")

New settings: f1,6 - ISO 25600 - 1/100s (-0,66EV - camera does not compensate)
+0,66EV in post f1,6 - ISO 40000 - 1/100s (Same Noise as in "real camera settings" but 0,66 f-stop more DOF)

*assuming the cameras do not compensate for the phenomenon when reaching f1,6 or smaller. 


Or am I totally of track here?


----------



## ajfotofilmagem (May 8, 2017)

Arahn said:


> So this means that when I am shooting in very low light situations, let's say f1,2 ISO 25600, it would actually make sense to step down to f1,6 ISO 25600, underexpose -0,66 EV and later ramp up the image by 2/3 of a stop in post? The result then would be the same level of noise as the f1,2 image but feature even have more DOF!?
> 
> To make my trail of thought clearer*:
> My Settings: f1,2 - ISO 25600 - 1/100s
> ...


Not exactly. Tests show that the cameras distort the ISO value to compensate for the greater waste of extremely steep light rays on F1.2 lenses. It would be something like the correction of vignette in camera. But as with vignettes, the correction is most needed near the edges of the sensor, and in lesser quantity in the center of the image.


----------

