# DxO & MTF Charts ... a little help please!



## Rienzphotoz (Feb 13, 2014)

*BACKGROUND:*
I am not into lens/sensor scores or MTF chart readings etc ... I have nothing against it, in fact I envy those who can understand all the math/science (that goes into the lens/senor scores/charts) and make their purchase decisions based on it ... but I generally do not have patience/skill to go into scores and charts ... I buy lenses based on the reviews from a few sources that I trust (including several members in CR) ... sometimes, I buy camera gear based on how comfortable I am with it ... but generally, I feel that most modern camera gear is more capable then my skills as a photographer. 

*WHAT I WANT TO LEARN & UNDERSTAND:*
I would like to learn & understand how someone gives a score to a particular lens or sensor. In this case I chose DxO because, over the past few years, I've read several discussions about DxO (prior to that I was not even aware such a company exists) and it got me interested in wanting to learn a bit more. Generally discussions related to DxO seem to invite war, blood bath and then end up with personal insults (I am also equally to blame for some of those discussions). So, I'd like to request your assistance to help me (and anyone else who is interested) understand DxO scores in "layman's" terms ... but please, I beg you not to turn this into DxO worshiping / hating contest. Also, please refrain from using this topic to poke fun or insult or score a point over fellow member ... we come here because of our passion for photography, so we are basically like minded citizens ... so please help me (and others) learn. 

*NOW THE QUESTION:*
I was looking at some lens comparison scores over at DxO today (a screen shot of which I've posted below). According to this a $3000 dollar lens seems to have better sharpness than a $12000 lens. I do understand that a lens is not just about sharpness but also about various factors such as how well it auto focuses ... but for now I just want to understand only the sharpness part of the scores. From my understanding, the scores suggest that the Sigma 120-300 f/2.8 is sharper than the EF 200-400 L IS (without using 1.4 TC), when both have them are tested at f/4 ... is that correct?
... also, is there an explanation to DxO scores? as in how they have arrived at that score? 
... and if there is no explanation (for how they allotted those scores), how does one know if they are accurate?
Appreciate if you can assist in helping me understand in simple terms without jargon ... and once again, no blood bath please.

Thanks in advance


----------



## neuroanatomist (Feb 13, 2014)

*Re: DxO ... a little help please! *



Rienzphotoz said:


> (a screen shot of which I've posted below)



?

But if you intended to post a shot of this, then yes they are showing that the Sigma 120-300/2.8 is slightly sharper than the 200-400L. It's probably a subtle difference - 1 P-Mpix on the 5DIII, but no difference on the 6D for example. 



Rienzphotoz said:


> ... also, is there an explanation to DxO scores? as in how they have arrived at that score?
> ... and if there is no explanation (for how they allotted those scores), how does one know if they are accurate?



For the Scores, not really. That's one of the big complaints about DxO - their Scores are a 'black box'. There are some details available. When you look at the table that I presume you meant to post as a screenshot, you would be tempted to think that the Score at the top is derived primarily from the Measurements below…but that's not the case, with the exception of transmission. The Score is based on 'performance in 150 lux illumination at 1/60 s and ISO 100'. That means that 1) a faster lens will score higher (thus, the 50/1.8 scores higher than the 600/4) and 2) the same lens will score higher when tested on a body with more DR/color depth at ISO 100 (which means the Lens Score is as much about the sensor as the lens). The reason the 120-300/2.8 scores higher than the 200-400/4 is really down to f/2.8 vs. f/4.

There are several examples of their measurements being incorrect, some of which they've corrected (silently).


----------



## Rienzphotoz (Feb 13, 2014)

*Re: DxO ... a little help please! *



neuroanatomist said:


> Rienzphotoz said:
> 
> 
> > (a screen shot of which I've posted below)
> ...


----------



## Rienzphotoz (Feb 13, 2014)

*Re: DxO ... a little help please! *



neuroanatomist said:


> yes they are showing that the Sigma 120-300/2.8 is slightly sharper than the 200-400L. It's probably a subtle difference - 1 P-Mpix on the 5DIII, but no difference on the 6D for example.
> The Score is based on 'performance in 150 lux illumination at 1/60 s and ISO 100'. That means that 1) a faster lens will score higher (thus, the 50/1.8 scores higher than the 600/4) and 2) the same lens will score higher when tested on a body with more DR/color depth at ISO 100 (which means the Lens Score is as much about the sensor as the lens). The reason the 120-300/2.8 scores higher than the 200-400/4 is really down to f/2.8 vs. f/4.


Thanks for the reply ... but I may have not understood correctly, let me explain what I understand from that in my own stupid way:
So, they took a 50 f/1.8 lens and shot an image at f/4 ... in order to compare sharpness they would have to do that with a totally different image made with a 600 f/4 L IS lens correct? (because the FOV would be totally different) so they then compared those 2 different images and found that the image made with 50mm was sharper (from my limited understanding, generally subjects closer are lot more sharper then those at a distance). 

Now, if they had compared the same image for the both lenses, then the 50mm lens would obviously lose out as they would need to crop the image to get the same FOV).

But if they compared two different images at 2 different focal lengths (and obviously in two different conditions), how can sharpness be defined in a generalized score? wouldn't that be like scoring a 1st grade student's score (who gets 95 out of 100 in math) with a University student's score (who gets 85 out of 100 in math) and then come up with the result that the 1st grade student is sharper?


----------



## neuroanatomist (Feb 13, 2014)

*Re: DxO ... a little help please! *

I'm distinguishing between the DxOMark Score (where the Sigma 120-300/2.8 gets a 29 and the Canon 200-400/4L gets a 24) vs. sharpness (which DxO reports in P-Mpix). On the 5DIII, the 50/1.8 gets a Score of 28, higher than the 200-400/4, although the 50/1.8 only delivers 14 P-Mpix vs. 19.


----------



## Rienzphotoz (Feb 13, 2014)

*Re: DxO ... a little help please! *



neuroanatomist said:


> I'm distinguishing between the DxOMark Score (where the Sigma 120-300/2.8 gets a 29 and the Canon 200-400/4L gets a 24) vs. sharpness (which DxO reports in P-Mpix). On the 5DIII, the 50/1.8 gets a Score of 28, higher than the 200-400/4, although the 50/1.8 only delivers 14 P-Mpix vs. 19.


So, do they provide any explanation for that?


----------



## neuroanatomist (Feb 13, 2014)

*Re: DxO ... a little help please! *



Rienzphotoz said:


> neuroanatomist said:
> 
> 
> > I'm distinguishing between the DxOMark Score (where the Sigma 120-300/2.8 gets a 29 and the Canon 200-400/4L gets a 24) vs. sharpness (which DxO reports in P-Mpix). On the 5DIII, the 50/1.8 gets a Score of 28, higher than the 200-400/4, although the 50/1.8 only delivers 14 P-Mpix vs. 19.
> ...



It's the one I gave above - in a dimly lit warehouse (150 lux) with a camera that has no setting higher than ISO 100, the 50/1.8 would be a better choice than an f/4 lens. 

The point is, the DxOMark Score isn't derived from the optical measurements (except transmission).


----------



## Rienzphotoz (Feb 13, 2014)

*Re: DxO ... a little help please! *



neuroanatomist said:


> Rienzphotoz said:
> 
> 
> > neuroanatomist said:
> ...


Sorry, I still didn't get it ... when you say they shoot in a dimly lit warehouse, they are using f/1.8 for EF 50mm f/1.8 lens and f/4 for the 600mm f/4 lens? or are they shooting both at f/4?
Thanks for your patience


----------



## neuroanatomist (Feb 13, 2014)

*Re: DxO ... a little help please! *



Rienzphotoz said:


> Sorry, I still didn't get it ... when you say they shoot in a dimly lit warehouse, they are using f/1.8 for EF 50mm f/1.8 lens and f/4 for the 600mm f/4 lens? or are they shooting both at f/4?
> Thanks for your patience



Well, I'm sure they don't shoot in a dimly lit warehouse, but they are illuminating the target with 150 lux, and that's aout the level of light in a dim warehouse. 

For the Overall Score, they are comparing the lenses at the aperture they determine is best, with is stated for each lens on the comparison pages (although not in the table you posted). That 'best at' value is pretty much always wide open for every lens (which makes sense in 150 lux illumination).


----------



## Rienzphotoz (Feb 13, 2014)

*Re: DxO ... a little help please! *



neuroanatomist said:


> Rienzphotoz said:
> 
> 
> > Sorry, I still didn't get it ... when you say they shoot in a dimly lit warehouse, they are using f/1.8 for EF 50mm f/1.8 lens and f/4 for the 600mm f/4 lens? or are they shooting both at f/4?
> ...


Hmmm :-\ ... I don't mean to take sides, but such a test does not seem fair or reasonable for a tool (lens) that is made to be used in diverse conditions.


----------



## neuroanatomist (Feb 13, 2014)

*Re: DxO ... a little help please! *



Rienzphotoz said:


> ...but such a test does not seem fair or reasonable for a tool (lens) that is made to be used in diverse conditions.



Exactly.


----------



## mackguyver (Feb 13, 2014)

*Re: DxO ... a little help please! *

The best way to use DxO's measurements are for comparison in scenarios you would actually use. I particularly like to use the the "Field Map" view. For instance, I compared the following aspects of lenses (see screen shots below):

1. How does the new Tamron 150-600 compare to my 300 f/.28 II IS at 300mm & f/5.6 in terms of sharpness? 
DxOMark shows the 300mm is sharper across the entire frame.

2. For architectural work, how much worse is the distortion on my 24-70 f/2.8 II than my 24mm 1.4 II?
DxOMark shows the 1.4 II has better control of distortion.

3. How does much better is chromatic aberration on the Zeiss Otus 55mm vs. my 50 f/1.2 when shot wide open? This is something I'm not fond of on , but I wondered, is it that much better to justify the cost?
DxOMark show almost no CA on the Zeiss, while the Canon, well...

Note: I just noticed that I forgot to change the camera body to 5DIII on all of them, which is important for sharpness measurements, but not distortion and others as long as you match FF to FF or crop to crop.


----------



## Rienzphotoz (Feb 13, 2014)

*Re: DxO ... a little help please! *



neuroanatomist said:


> Rienzphotoz said:
> 
> 
> > ...but such a test does not seem fair or reasonable for a tool (lens) that is made to be used in diverse conditions.
> ...


Thanks for your explanations Neuro ... all this got me thinking that for a company like DxO that is capable/equipped to conduct sophisticated lens/sensor tests, what stops them from comparing lenses/sensors at 10 different ISO measurements and in 10 different conditions with 10 different subject movements and then come up with scores in a chart, so people can come to a more informed conclusion :-\ 

For example they could, in controlled environment, throw a ball and see how well the camera/lens can auto focus at various apertures and then come up with several scores for sharpness ... I think such scores would help customers to figure out what would be a good lens for them for a particular shooting style e.g. if I like to shoot birds in flight in a certain lighting condition, I would have a ready reference of tests/scores to see what fits my needs. Coz they seem to do that very well for images shot at ISO 100 and pretty much everyone seems to agree with DxO that the Sony/Nikon 36MP sensor delivers great results at that ISO ... surely a company that boasts of sophisticated lab tests, is capable of conducting varied tests and producing different scores for different situations. Would that be an unreasonable expectation?


----------



## neuroanatomist (Feb 13, 2014)

*Re: DxO ... a little help please! *



mackguyver said:


> The best way to use DxO's measurements are for comparison in scenarios you would actually use. I particularly like to use the the "Field Map" view.



Just watch out for their mistakes. For example, compare the 17-40L wide open to the 16-35L II stopped down two stops to f/5.6, and check out the field maps. The 17-40 is infamous for mushy corners wide open, but you don't see that on DxO. Is it believable that the corners of the 17-40 wide open are sharper than the corners of the 16-35 at f/5.6? Not to me…


----------



## Rienzphotoz (Feb 13, 2014)

*Re: DxO ... a little help please! *



mackguyver said:


> 1. How does the new Tamron 150-600 compare to my 300 f/.28 II IS at 300mm & f/5.6 in terms of sharpness?
> DxOMark shows the 300mm is sharper across the entire frame.


In the first set of images (150-600 vs 300), the 300mm is evenly colored, while the 150-600 isn't ... is that representing sharpness or CA?


----------



## mackguyver (Feb 13, 2014)

*Re: DxO ... a little help please! *



neuroanatomist said:


> mackguyver said:
> 
> 
> > The best way to use DxO's measurements are for comparison in scenarios you would actually use. I particularly like to use the the "Field Map" view.
> ...


I'm still scratching my head on those measurements, too, at least after getting over the 70-200 f/2.8 IS I/II review 

I always like to use multiple resources - Lenstip, SLRgear, & Photozone are usually the three that seem to have the best reviews/measurements. As they're all free, it makes it pretty easy to be an informed consumer now that we've reached the death of almost all photo shops. I like to borrow lenses from CPS, too, before making big purchases.



Rienzphotoz said:


> mackguyver said:
> 
> 
> > 1. How does the new Tamron 150-600 compare to my 300 f/.28 II IS at 300mm & f/5.6 in terms of sharpness?
> ...


That shot represents sharpness.


----------



## meli (Feb 13, 2014)

*Re: DxO ... a little help please! *



Rienzphotoz said:


> ... all this got me thinking that for a company like DxO that is capable/equipped to conduct sophisticated lens/sensor tests, what stops them from comparing lenses/sensors at 10 different ISO measurements and in 10 different conditions with 10 different subject movements and then come up with scores in a chart, so people can come to a more informed conclusion :-\


...and with 10 different samples of the same camera/lens picked with appropriate criteria.
They probably could, but they wont. What you wish takes time, personnel, and money. And all this to run a free service. That doesnt fly really well.


> For example they could, in controlled environment, throw a ball and see how well the camera/lens can auto focus at various apertures and then come up with several scores for sharpness ...


Dxo doesnt do AF. Its a whole other beast and frankly its difficult to come up with an objective/exhaustive suite of lab/field AF tests.


----------



## lescrane (Feb 13, 2014)

*Re: DxO ... a little help please! *

So since some of you are willing to explain wtf these charts mean to those of us who have never seen them, can you give a very basic explain. of this chart?
Obviously the darker green the better, the more uniform the better, but what does the square area mean? Is it the frame of the image area? eg, darker in the middle is sharper, lighter around the edges, less sharp?


----------



## mackguyver (Feb 13, 2014)

*Re: DxO ... a little help please! *



lescrane said:


> So since some of you are willing to explain wtf these charts mean to those of us who have never seen them, can you give a very basic explain. of this chart?
> Obviously the darker green the better, the more uniform the better, but what does the square area mean? Is it the frame of the image area? eg, darker in the middle is sharper, lighter around the edges, less sharp?


Yes, that it exactly right. The rectangle is the sensor area and darker (green, vs. yellow or red) and more uniform is better 

In this case, the Tamron puts in a pretty good showing considering the price difference.


----------



## AlanF (Feb 13, 2014)

*Re: DxO ... a little help please! *

Mac
Thanks for bringing up this method of comparison. I have just used it to compare the 150-600mm with the 100-400 in a thread about the Tammy to disprove an assertion that it is a confirmed soft lens. 

DxO measurements are very useful for comparing different lenses on the same body or even better the same lens on different bodies.


----------



## mackguyver (Feb 13, 2014)

*Re: DxO ... a little help please! *



AlanF said:


> Mac
> Thanks for bringing up this method of comparison. I have just used it to compare the 150-600mm with the 100-400 in a thread about the Tammy to disprove an assertion that it is a confirmed soft lens.
> 
> DxO measurements are very useful for comparing different lenses on the same body or even better the same lens on different bodies.


No problem and I saw you _feeding the troll_ on that post, LOL. It is a nice tool and you can compare up to three lenses at a time. As Neuro says, it's not perfect, but it's free and relatively easy to use. Also, I forgot to mention the-digital-picture.com in my last post. Their tools are nice to use for comparisons as well.


----------



## Rienzphotoz (Feb 13, 2014)

*Re: DxO ... a little help please! *



mackguyver said:


> I forgot to mention the-digital-picture.com in my last post. Their tools are nice to use for comparisons as well.


When I first started out with DSLRs, the only source I used to trust for reviews was the-digital-picture.com ... also, they make it very simple to understand.


----------



## jrista (Feb 13, 2014)

*Re: DxO ... a little help please! *



Rienzphotoz said:


> mackguyver said:
> 
> 
> > I forgot to mention the-digital-picture.com in my last post. Their tools are nice to use for comparisons as well.
> ...



Bryan's reviews at TDP are indeed excellent. I probably have more respect for his reviews than anyones. They are just strait forward, cover the key technical aspects but also demonstrate real-world usage. Really doesn't get better than that. Just wish he would test more brands. 

BTW, Rienz...I'd be happy to explain how to use an MTF. It seems complicated, all those lines, but once you get the general idea, they are actually EXTREMELY informative, and it isn't all that difficult to understand.

As for using DxO, @mackguyver pretty much nailed it. Use their measures, which are comparable, and ignore the scores. The measures are pretty decent (except transmission, that one is pretty useless because they don't account for differing apertures, so it really is an aperture measure, not a transmission measure.) DxO's lens measures are handy because they can be directly compared. If you want similar detailed information about lenses that can be manually compared, DPR lens reviews have similar information. Two windows side-by-side would let you compare lenses with DPR information.


----------



## Rienzphotoz (Feb 13, 2014)

*Re: DxO ... a little help please! *



jrista said:


> BTW, Rienz...I'd be happy to explain how to use an MTF. It seems complicated, all those lines, but once you get the general idea, they are actually EXTREMELY informative, and it isn't all that difficult to understand.


Hi Jon,

Please do ... I'd appreciate that very much ... and am sure others like me will also appreciate it very much.

Thanks in advance


----------



## lescrane (Feb 13, 2014)

*Re: DxO ... a little help please! *



mackguyver said:


> lescrane said:
> 
> 
> > So since some of you are willing to explain wtf these charts mean to those of us who have never seen them, can you give a very basic explain. of this chart?
> ...



thanks. I compared it to the 100-400L in 70D crop body and the tammy looks pretty good at 400.


----------



## jrista (Feb 13, 2014)

*Re: DxO ... a little help please! *



Rienzphotoz said:


> jrista said:
> 
> 
> > BTW, Rienz...I'd be happy to explain how to use an MTF. It seems complicated, all those lines, but once you get the general idea, they are actually EXTREMELY informative, and it isn't all that difficult to understand.
> ...



Alright. Here goes. First, I'm just going to cover MTF charts. You don't actually need to know all the fundamental science that goes into resolving power to actually understand an MTF chart. All you really need to know is how to read the MTF chart, and that will tell you pretty much everything you need to know about the theoretical characteristics of a lens. Since most manufacturers publish MTF charts, and most use the same general standard (30lp/mm resolving power\sharpness & 10lp/mm contrast), they are pretty easy to compare brand to brand as well.

So, first off, an example MTF chart. This is the EF 24-70mm f/2.8 L II:











There are two charts here, because they represent the wide angle and narrow angle extremes of the zoom ratio. This one single chart tells you everything you need to know about a lens, and you can compare two MTF charts to each other to determine differences in each lenses performance (or, determine the differences between one extreme and the other of a zoom.) 



So, first a breakdown of the chart itself. The MTF chart (absent any plot) represents the resolving power (sharpness and contrast) of a lens, from the center of the frame to the corner of the frame. The center of the frame is represented by the leftmost edge. The corner of the frame is represented by the rightmost edge. The vertical (y-axis) scale is an indication of how close to "ideal" resolving power gets. The vertical scale ranges from 0.0 to 1.0. In "historical" terms (and this really stems from the film days, so take it with the understanding that it definitely does not apply quite the same way today), MTF above 0.6 is "good/satisfactory" and MTF above 0.8 is "excellent/superior". Personally, I make the assumption, given how crisply modern digital sensors resolve detail, and the fact that people are increasingly publishing their photos online in full 1080p resolution, that MTF above 0.7 is "good/satisfactory" and above 0.9 is "excellent/superior".



Now for the plot. There are usually eight lines plotted on an MTF chart. These lines are blue and black (for Canon, other manufacturers may use other colors), thick and thin, solid and dashed. These lines represent how the lens reproduces sets of diagonal lines, the first set angled 45° and the second set 90° perpendicular to the first. These lines are called Sagittal and Meridional lines. They are angled at 45° and 135° because that is the ideal orientation to measure the behavior of optics from center to corner (vs. 0° and 90°, which would be more ideally suited to testing a lens center to edge...which is really insufficient.) The reason perpendicularly angled lines are used is because lenses do not behave the same when resolving detail at all angles...astigmatism in the lens design will often affect how lenses perform with fine detail angled differently across the lens. From the center to top right/bottom left corners, sagittal lines are angled parallel to the vector from the center to the corners of the lens. Meridional lines are angled perpendicular to the vector from center to top right/bottom left corners.

There are four sets of lines in total used in a standard MTF test. The first set are the 45° & perpendicular 10lp/mm lines. These lines are alternating white and black (technically speaking, the chart base is white, and there are thicker lines drawn at even spacing with thick white gaps between them, and thinner lines drawn at even spacing with thin white gaps between them.) The second set are 45° & perpendicular 30lp/mm lines. The 10lp/mm lines, which represent lower resolution detail, are used to measure lens contrast. The softer the transition between thick dark and thick white, the lower the contrast of the lens. The 30lp/mm lines, which represent higher resolution detail, are used to measure resolving power. 






Can the black lines be resolved as fully separated with a white line in-between (resolved, high contrast)? If parallel black lines are separated, how quickly does the white line in-between become fully white (sharpness)?



So, we have an MTF chart that represents resolving power (y-axis) from center to corner of a lens (x-axis). This chart has eight curves plotted on it, that represent four sets of sagittal and meridional lines on a test chart. Why eight lines, rather than four? The MTF chart contains plots for both wide-open (max aperture) performance as well as f/8 performance. So, four sets of lines for max aperture: 

[list type=decimal]
[*]*f/2.8 sagittal thick solid (10lp/mm)*
[*]*f/2.8 meridional thick dashed (10lp/mm)*
[*]f/2.8 sagittal thin solid (30lp/mm)
[*]f/2.8 meridional thin dashed (30lp/mm)
[*]*f/8 sagittal thick solid (10lp/mm)*
[*]*f/8 meridional thick dashed (10lp/mm)*
[*]f/8 sagittal thin solid (30lp/mm)
[*]f/8 meridional thin dashed (30lp/mm)
[/list]



Now, onto interpreting an MTF chart.

Using these eight curves plotted on an MTF chart, you can derive everything you need to know about the lens. The higher up the chart they are plotted, the better the contrast and resolution. You will normally see that as the curves move from the left edge to the right edge of the chart, they tend to "fall off", they drop lower. This is an indication of how the lens' performance chances from center to corner. You will also notice that one set of curves, either the solid curves or the dashed curves , tend to perform better than the other. This is an indication of astigmatism in the lens...sagittal curves (solid) may maintain higher performance than meridional curves (dashed). 

This so happens to be exactly the case with the 24-70mm lens. At both wide (24mm) and narrow (70mm), the 24-70 resolves sagittal lines (45° angle) better than meridional lines (90° angle), and the meridional curves tend to fall off quicker in the midframe to the corners than sagittal curves do. Sometimes you may notice that some of these curves don't have a consistent falloff, they may turn "bump", resulting in slightly better resolving power just past midframe, then fall off again all the way into the corner. The nature of each curves falloff is an indication of how the various optical aberrations affect a lens' performance from center to corner. Depending on exactly what aberrations a lens may suffer from and to what degree, the nature of the curves' falloff will differ. Astigmatism and other imperfections in lens manufacture will result in falloff differing even from sagittal to meridional lines.

You will also find, once you start reading MTF charts, that wide angle lenses tend to have more chaotic late midframe and corner performance than normal and longer lenses. Telephoto/supertelephoto lenses will often have nearly flat MTF curves from center to corner (especially if they are higher end). It is not unusual to see crazy meridional performance in lenses as you approach the corners in wider angle lenses, especially ultrawide to wide angle zooms. There are simply certain compromises that must be made in order to produce a wide angle zoom lens that performs acceptably at as many focal lengths as possible.



That's pretty much it. A crash course in reading and understanding MTF charts. There is a lot more theory that builds up to WHY these kinds of tests are used to accurately measure lens resolving power, but you don't necessarily need to understand all that underlying grit in order to effectively use MTF charts themselves. All they really are is a plot of four types of detail from the center to the corner of the lens. Those plots measure contrast and resolution (sharpness & acutance) across the surface of the lens, and offer a fairly precise indication of how optical aberrations will affect your IQ in different regions of your frame.


----------



## candc (Feb 13, 2014)

*Re: DxO ... a little help please! *

are you interested in the sigma 120-300? dxo analysis is useful for comparing similar lenses on the same body. the dxo accutance map measures sharpness across the field as does an mtf chart but dxo measures actual performance of the lens, i think mtf charts are based on a model?


----------



## mackguyver (Feb 13, 2014)

*Re: DxO ... a little help please! *



candc said:


> are you interested in the sigma 120-300? dxo analysis is useful for comparing similar lenses on the same body. the dxo accutance map measures sharpness across the field as does an mtf chart but dxo measures actual performance of the lens, i think mtf charts are based on a model?


Most manufacturers (other than Zeiss & Leica, I think, and maybe a few others) use theoretical MTFs generated by computer instead of actual measurements. That doesn't make them invalid, but they aren't going to be as precise.


----------



## zim (Feb 13, 2014)

*Re: DxO ... a little help please! *

Jrista, thank you for that excellent concise explanation.


----------



## candc (Feb 13, 2014)

*Re: DxO ... a little help please! *

that is a great explanation but i will never remember it all. is there a simple way to look at it to get a quick summary? such as the black line for a general indicator and how tight the lines are together? do these mtf charts generally translate into what is found in testing of real lenses?

thanks,


----------



## jrista (Feb 13, 2014)

*Re: DxO ... a little help please! *



candc said:


> that is a great explanation but i will never remember it all. is there a simple way to look at it to get a quick summary? such as the black line for a general indicator and how tight the lines are together? do these mtf charts generally translate into what is found in testing of real lenses?
> 
> thanks,



You can't really narrow it down. That's the fundamental fallacy everyone falls into, and why a lot of the stuff DXO, DPR, and the whole host of other "lens testers" produce is largely useless.

If you want to eliminate anything, eliminate the set of curves that don't fit with the apertures you will most use. If you will mostly shoot wide open, then you can ignore the blue curves. If you will mostly shoot stopped down, then you can ignore the black curves. You could ignore dashed curves, and just use the solid curves, but then your not really getting the whole picture. 

There is no "one number tells all". That's just a fallacy. The attempt to utterly simplify everything is really what gets you into trouble. 

When it comes to the difference between an MTF and a lens test, lens tests all ultimately run into the "sensor bound" problem. Sensors have a fixed resolving power...it's the same across the entire area of the sensor. Lenses, on the other hand, have a non-linear resolving power that falls off as you stop down. At apertures wider than f/8, the potential for a lens to resolve much finer detail than the sensor becomes very real. The problem is, final "output" resolution has an asymptotic relationship with the lowest common denominator. Since the sensor usually IS that lowest common denominator, at faster apertures, where lenses have the potential to resolve a LOT of detail, may all end up looking the same in the end. Why? Well, let's say your sensor can resolve 50lp/mm, and you have four lenses capable of resolving, at f/4, 100lp/mm, 130lp/mm, 150lp/mm, and 173lp/mm (the latter is the maximum diffraction-limited resolving power of an f/4 lens.) The problem is that all of these lenses will all appear to resolve somewhere between 45-49lp/mm with a "real world" lens test, like the kind that DXO does. They are all SENSOR BOUND! The SENSOR cannot resolve more than 50lp/mm, so that is your absolute limit on final output resolution (the resolution measured in the RAW images by computer algorithms.)

So, first off, your standard lens test that tests lenses attached to cameras are largely useless for any apertures above f/8, however from f/8 and narrower, the vast majority of lenses are diffraction limited, so they will all perform the same anyway. 

Second, MTFs really don't have any relationship with artificial lens test results, because they are either performed algorithmically based on fairly accurate computer models that account for overall lens construction and design, as well as material traits; or they are performed with optical lens bench testing, which uses a special apparatus to test JUST the lens. Synthetic MTFs will usually indicate just a little bit better performance than Real MTFs generated with a optical test bench, however both will be largely similar, and neither will bear any resemblance to your "standard lens+camera" tests. 

If you want to keep it simple: Pick one set of solid lines, for max aperture or f/8 (depending on whichever you use most), and go with that.


----------



## golubiewac1 (Feb 13, 2014)

*Re: DxO ... a little help please! *

Thanks to jrista for the tutorial on MTF charts. It will be very helpful to me.


----------



## candc (Feb 13, 2014)

*Re: DxO ... a little help please! *

ok, i am getting closer i think, bold lines on chart = bold lines in testing for contrast, fine lines = resolution. black is wide open, blue is f/8 solid lines are radians, dashed are perpendicular, i am a bit fuzzy on what the radian and perpendicular lines tell you but i am getting the idea, 

thanks


----------



## jrista (Feb 13, 2014)

*Re: DxO ... a little help please! *



candc said:


> ok, i am getting closer i think, bold lines on chart = bold lines in testing for contrast, fine lines = resolution. black is wide open, blue is f/8 solid lines are radians, dashed are perpendicular, i am a bit fuzzy on what the radian and perpendicular lines tell you but i am getting the idea,
> 
> thanks



Sagittal (radian) and meridional (perpendicular) tell you about lens astigmatism (they will usually diverge when plotted across the chart).


----------



## privatebydesign (Feb 13, 2014)

*Re: DxO ... a little help please! *

http://www.luminous-landscape.com/tutorials/understanding-series/lens-contrast.shtml

http://www.luminous-landscape.com/tutorials/understanding-series/understanding-mtf.shtml


----------



## candc (Feb 14, 2014)

*Re: DxO ... a little help please! *

i am going to have to read up on that, i hear about astigmatism in a lens all the time but don't fully understand what it is and how it translates to a lens performance. i know it is an unwanted characteristic in a lens and now i think i can see from the mtf chart that the closer together the dashed and solid lines of a particular pair are then the better the astigmatism is controlled?


----------



## candc (Feb 14, 2014)

*Re: DxO ... a little help please! *



privatebydesign said:


> http://www.luminous-landscape.com/tutorials/understanding-series/lens-contrast.shtml
> 
> http://www.luminous-landscape.com/tutorials/understanding-series/understanding-mtf.shtml



those are good articles thanks, i am going to have to read through them again a few times but what grabbed me is that you should look at the uppermost set of lines, the bold ones because they translate best to what you visually correlate with lens performance? that is to say that the contrast seems to be more important than absolute resolving power. i noticed that on the mtf chart for the 24-70ii, the bold lines were flat at the top and the fine lines dropped off significantly more.


----------



## jrista (Feb 14, 2014)

*Re: DxO ... a little help please! *



candc said:


> privatebydesign said:
> 
> 
> > http://www.luminous-landscape.com/tutorials/understanding-series/lens-contrast.shtml
> ...



It is easy to get absolute in an online article. I wouldn't say things are as simple as "contrast is more important than absolute resolving power". It depends. As a bird photographer, absolute resolving power is very important to me, so I pay close attention to the thin lines myself. Some lenses have great contrast with lower resolution detail, but they simply can't separate finer details. That would ultimately mean that I might be able to resolve the major details of a birds feathers, but not actually be able to cleanly separate the structure of the feathers themselves. I'd be ok with a slight loss of contrast for larger details, if it means I have the ability to separate finer detail.

You can reduce the information in an MTF according to your needs. Don't let someone else tell you what *your *needs are, though.  If contrast is more important to you, well then sure, pay attention to the thicker solid lines.


----------



## candc (Feb 14, 2014)

*Re: DxO ... a little help please! *



jrista said:


> candc said:
> 
> 
> > privatebydesign said:
> ...



That's the thing I am trying to get straight because its a bit confusing. What the mtf measures is micro contrast which we see as resolution at some point, not overall image contrast. The mtf chart for the 24-70ii shows really good contrast center to edge but the resolution lines don't look that impressive. Its known to be a super sharp lens by all accounts so that seems to reinforce the point that what we see as resolution is really closely related to contrast?


----------



## jrista (Feb 14, 2014)

*Re: DxO ... a little help please! *



candc said:


> jrista said:
> 
> 
> > candc said:
> ...



An MTF measures both microcontrast as well as larger-scale contrast. The 30lp/mm is more of your "microcontrast" (sharpness and acutance), where as 10lp/mm is general contrast. That's why both sets of thin and thick line pairs are used.

The 24-70 is an excellent lens...but it IS both wider angle and it is a zoom. Both of those facts require that certain compromises be made. Compare the 24-70 II with the previous, or with third-party lenses in the same general range, and you'll see why it is considered a very high quality lens with high resolving power. It's got very good resolution for the focal length bracket it falls into. 

You want to see some REALLY high resolution lenses? Here:

EF 300mm f/2.8 L II:





EF 600mm f/4 L II:





As you get to longer focal lengths, the falloff towards the corner drops. More light is collimated, you don't have to worry about bending highly oblique light. The more you bend light, the tougher it is to maintain resolution from center to corner. 

Even the older EF 300 f/2.8 L Mark I was an unbelievably stellar lens:






But again, these are all primes, and they are telephotos. Mostly collimated light, no highly oblique incident light angles to deal with, and they use the highest quality optical glass around.

Also, keep in mind that you have blue and black lines, meaning f/8 and max aperture. Depending on what the maximum aperture is, the black lines may drop considerably due to optical aberrations. If you want to look at diffraction limited performance, use the blue lines. That is f/8, and it is easier to compare lenses at f/8, as more likely than not most lenses you compare will be diffraction limited, or close to it, at that aperture. You'll notice that f/8 resolution lines tend to edge higher up the chart. 

Another example would be the 70-200 f/2.8 L II. At 70mm and 200mm:











Notice the differences here. At 70mm, a shorter focal length that has to deal with more oblique off-axis light, has greater falloff in the corners, and struggles more with meridional test lines (especially the high resolution ones). However at 200mm, you can see the whole set of lines has moved up the chart, and that faloff to corner is slower and less dramatic.

This is the power of an MTF chart. They tell you a LOT about optics, and generally at a glance (once you know how to read them.) Learning how to read them just takes some time and practice. Eventually, comparing lenses or focal lengths really boils down to some momentary glances at the charts, and you absorb a whole lot of information all at once. (At least...that's how it works with me...)


----------



## mackguyver (Feb 14, 2014)

*Re: DxO ... a little help please! *

Another point on MTF charts - you don't want to compare a super telephoto to a wide angle lens - as you can see the IQ looks insanely higher on the former, but it's not really that way. You should compare similar focal lengths to each other if using MTF charts, i.e 24mm prime to a 24-70 lens at the wide end.


----------



## candc (Feb 14, 2014)

*Re: DxO ... a little help please! *

Thanks to you both for explaining all this, I undestand it a lot better than I did before which was:

Flat at the top=good
Ski slope to the right=bad

That's still true but I see why its relative and why


----------



## jrista (Feb 14, 2014)

*Re: DxO ... a little help please! *



mackguyver said:


> Another point on MTF charts - you don't want to compare a super telephoto to a wide angle lens - as you can see the IQ looks insanely higher on the former, but it's not really that way. You should compare similar focal lengths to each other if using MTF charts, i.e 24mm prime to a 24-70 lens at the wide end.



Oh, the telephoto focal lengths are indeed that good. Thats WHY I invested $12,000 in the 600mm f/4 L II! You have no idea how much better it is. And that conforms with the theory. At 600mm, the vast majority of incident light is collimated. There isn't much bending going on. Not nearly as much as at 70mm or 24mm. An MTF is an MTF. There are no special considerations for focal length. You can compare any MTF chart to any other. They are about as honest a review of lens quality at any given focal length as you can hope for.


----------



## Rienzphotoz (Feb 14, 2014)

Hi Jon,

Thank you very much for the AWESOME explanations .. I think I understood most of it and would need to re-read a few times to burn into my brain ... please continue the discusiion with as simple/stupid as you can make for people like me ... much appreciated and many thanks.

*PS.* As you may have noticed, I changed the title from "DxO ... a little help plesase!" to "*DxO* *& MTF Charts* *... a little help please!*" ... with all the very useful info shared here, I think, that is a more appropriate title.

*PPS.* Also, thank you everyone for being very kind in obliging my request to not turn this into a war zone for/against DxO ... much appreciated, as this thread has provided some good education for me.


----------



## mackguyver (Feb 14, 2014)

*Re: DxO ... a little help please! *



jrista said:


> mackguyver said:
> 
> 
> > Another point on MTF charts - you don't want to compare a super telephoto to a wide angle lens - as you can see the IQ looks insanely higher on the former, but it's not really that way. You should compare similar focal lengths to each other if using MTF charts, i.e 24mm prime to a 24-70 lens at the wide end.
> ...


jrista, this is something I had read from Canon (will have to look for it), and I guess I didn't phrase it correctly. It's not to say that the MTF charts lie, just that wide angle lenses may look really soft in comparison to telephoto charts, but that doesn't mean that they are as soft as they appear, and the value of MTFs is better used to compare similar focal lengths to each other.

That makes me think of another great resource (even if it's not as current as the edition CPS members receive when they join) for learning about all of this stuff - Canon Lens Work. You can download the 11 PDFs free from Canon's European CPS website:
http://www.canon-europe.com/Support/Documents/digital_slr_educational_tools/en/ef_lens_work_iii_en.asp

The 10th PDF, OPTICAL TERMINOLOGY & MTF CHARACTERISTICS, has tons of great information on optics and MTF charts. It's not light reading, but it will tell you everything you wanted to know and probably a whole lot more!

One of the best explanations I've seen on how to read MTF charts is on page 14 of Sony's Alpha lens brochure which also contains great diagrams and explanations of all core lens concepts:
www.docs.sony.com/release/Alpha_Lens_Brochure_Fall_2012.pdf


----------



## jrista (Feb 14, 2014)

*Re: DxO ... a little help please! *



mackguyver said:


> jrista said:
> 
> 
> > mackguyver said:
> ...



I see what you are saying. Indeed, a wider angle lens is compressing a very wide field of view into the same physical region of the sensor. That compression certainly mitigates the impact of higher angle light being bent into the lens. The MTF charts really don't lie, though, and they are a normalized plot, so they tell you the same thing about lenses regardless of focal length. The illusion would be when you think your ultra wide angle lens performs as well as a telephoto "for what it is"...simple fact of the matter is, quite often, they really don't! I have the 16-35 and the 600mm, two of Canon's most extreme lenses. The corner performance of the 16-35 is well and truly ATROCIOUS. Even its center performance isn't all that great on a camera like the 7D (with small pixels/high resolving power.) This is clearly indicated by the MTF:











At the wide end, sagittal performance drops to 0.3!! Meridional performance drops to almost zero!!! Even at the long end, resolution suffers considerably in the corners.

My 600mm lens, on the other hand, produces what I consider perfection from center to corner. Both lenses live up to their MTFs. Even though you don't necessarily "feel" as though the 16-35mm lens is all that bad, all it takes is a little bit of investigation of the corners to see the MTF doesn't lie. Corner performance on the 16-35 is about as bad as it gets. (Canon REALLY needs to update that lens.) 



mackguyver said:


> That makes me think of another great resource (even if it's not as current as the edition CPS members receive when they join) for learning about all of this stuff - Canon Lens Work. You can download the 11 PDFs free from Canon's European CPS website:
> http://www.canon-europe.com/Support/Documents/digital_slr_educational_tools/en/ef_lens_work_iii_en.asp
> 
> The 10th PDF, OPTICAL TERMINOLOGY & MTF CHARACTERISTICS, has tons of great information on optics and MTF charts. It's not light reading, but it will tell you everything you wanted to know and probably a whole lot more!



Aye! I read all the EF Lens Work documents years ago. Excellent, if complicated, stuff.



mackguyver said:


> One of the best explanations I've seen on how to read MTF charts is on page 14 of Sony's Alpha lens brochure which also contains great diagrams and explanations of all core lens concepts:
> www.docs.sony.com/release/Alpha_Lens_Brochure_Fall_2012.pdf



I've never looked into Sony's documents. I'll have to check it out.


----------



## candc (Feb 14, 2014)

I read the Sony document. It is not a highly detailed technical paper but a summary of sorts. I found some useful info in the article. What I learned is that when interpreting an mtf chart the relative closeness of a dashed and solid line pair will generally indicate the quality of the defocused areas. That is to say if the dashed and dotted lines of the line pairs are close together or superimposed then that indicates the lens will have good bokeh. I didn't know that was something that could be measured but I guess it can. Maybe its not all telling but apparently a good indicator?


----------



## jrista (Feb 15, 2014)

candc said:


> I read the Sony document. It is not a highly detailed technical paper but a summary of sorts. I found some useful info in the article. What I learned is that when interpreting an mtf chart the relative closeness of a dashed and solid line pair will generally indicate the quality of the defocused areas. That is to say if the dashed and dotted lines of the line pairs are close together or superimposed then that indicates the lens will have good bokeh. I didn't know that was something that could be measured but I guess it can. Maybe its not all telling but apparently a good indicator?



Aye, what Sony says is true. When the lines are on top of each other, you generally have very flat out of focus blur circles. They tend to be uniform from the center of the circle to the edge of the circle. When there is divergence, any number of alternate forms of blur circles can arise. Some even divergence and a relatively even falloff actually means the lens still has some spherical aberration, which usually results in good boke as well. When the meridional lines are all squirrely, then your boke will usually be pretty nasty.


----------

