# What about DxO Mark?



## Vamp898 (Jan 31, 2012)

I think the most of you know DxO Mark and there Sensor Benchmarks.

They also test lense and i recently seen that most of Tamrons SP Lenses are mostly at the same quality as the L-Senses of Canon.

But they cost only 1/4 of the L-Lense of Canon.

They have a proper test which is not to misunderstand and tell very exactly how they test, also the results seems to be very realistic.

My Question is why? Why are the Tamron SP Lenses at the same quality as the L-Lenses of Canon, but cost only 1/4.

I doubted it and bought some L-Lenses and SP Lenses with the same focal length for example 70-300mm.

really, the SP lense from Tamron seems to own the Canon lense in nearly every way but cost only ~330 €.

The Canon L-Lense 70-300 is about ~1250 €

I tested a lot of things, outside an inside. Checked the lp/mm with a lot of test photos and it seems to be true. 

At DxO even the before version of the current 70-300 without IS and without SP seems to be a hard competitor for the L-Lense.

With the lenses, it seems that those L-Lenses from Canon are something like a religion. You just believe that its the best on the marked and so it worths the money.

But Tamron is no replace for Canons EF-Lenses, Tamron have only a hand-full ob objectives and only 30-50% of them (depends on APS-C or Full-Frame) are SP-Lenses.

Or do you think DxO is wrong at all?


----------



## Mt Spokane Photography (Jan 31, 2012)

A lot of people think that DXO has the capability to test lenses, but they do not!. Just like other amateur testers, they test a lens / body combination. This means that the results only apply to a lens when used on that particular body, and you can't compare numbers with other lenses tested on a different model camera body. 

http://www.dxomark.com/index.php/About/In-depth-measurements/DxOMark-testing-protocols/MTF

The information is useful for those who realize what the test actually is.

Zeiss is one of the few companies that actually has the equipment and tests / publishes the actual MTF of their lenses. The equipment is expensive, and intrepreting the test results is difficult.

Here is one company that makes lens testing equipment. you will not see it in use at the popular lens testing sites, its just out of their reach. http://www.trioptics.com/


----------



## Vamp898 (Jan 31, 2012)

Sure i know, that is way i always compare lenses mounted on the same body.

I would never compare a Lense mounted on a 7D Body with a lense on a Nikon D300 Body.

But if a Tamron SP on a 5D Mark II have more lp/mm than a Canon L on a 5D Mark II or at least the same amount, than i think this result is not wrong.

So if you choose the same camera, the test "should" show correct results. And if the test shows that 2 objectives mounted on a 7D have the same vignetting, chromatic aberration and the same lp/mm but one cost only 1/4 from the other, it lets me start thinking.


----------



## Mt Spokane Photography (Jan 31, 2012)

Vamp898 said:


> Sure i know, that is way i always compare lenses mounted on the same body.
> 
> I would never compare a Lense mounted on a 7D Body with a lense on a Nikon D300 Body.
> 
> ...



I have nothing against Tamron, I had a Tamron 200-500mm zoom with no IS. It was OK, but IQ not even close to my Canon 100-400mm L, but then, it was half the price. At least, all the Tamron AF lenses I've had actually focus on a Canon body, unlike the Sigma lenses I had for my EOS film cameras that would not Auto focus on a Canon DSLR, and could not be upgraded. All the other Tamron lenses I've owned were cheap consumer grade, and just fine for cheap lenses.

Tokina tends to be my favorite 3rd party lens maker. They have a very limited selection, but are very well made, and their wide angle lenses tend to be excellent.

I'm overloaded with lenses right now, and, having both a FF and crop body, my only ef-s lens is the Canon 15-85, and its pretty good for the price.

I tend to prefer wide aperture primes, and most 3rd party manufacturers have mostly zooms.

I took a look at the Tamron 17-50 f/2.8 versus the Canon 17-55, they seemed to test very closely, MSRP was $650 versus $1046 for the Canon. Seems well worth looking into for a crop body

You might also want to take a look at the Photozone reviews, Klaus is a tough grader, and points out things that may impact you depending on how you work. 

I would be using manual focus, for example, and its a difficult lens to manual focus due to the very short throw.

I tend to read all the reviews, each reviewer adds some insite into the use and operation of a lens. They usually come up with the same general result, but the devil is in the details.


----------



## BlueMixWhite (Feb 1, 2012)

Dont we have to take into consideration of other factors such as AF speed, colour reproduction...etc. I know a lot of ppl complaint about Tamron slow focusing and it's not slient at all. This could makes the huge premium of L lenses.


----------



## ejenner (Feb 1, 2012)

Obviously for many people Tamaron, Sigma and Tokina make better value for money lenses than Canon or Nikon, otherwise they would not be in business.

I don't know about the 70-300, but for many other lenses there are definite trade-offs and you don't get exactly the same for cheaper and it certainly isn't all about lp/mm in the center. I personally have several L lenses and a sigma 1.4. I would certainly buy all 3rd party is I thought they really gave me what I wanted.

I would argue that while DxO Mark is useful, even on the same body, results are not always consistent with 'real world' applications. Many experienced photographers have questioned some of their scores, perhaps it is simply manufacturing variation. But I honestly find that while shooting test charts or walls can be useful to see if a lens is way off and maybe get a broad idea of resolution/sharpness, but is not very representative of how much better or worse a lens is for a particular use.


----------



## bvukich (Feb 1, 2012)

*Commonly cited reasons:*
AF Speed/Accuracy/Consistency
Weather sealing
Color rendition
Resale value
Build quality

Resale value alone is a huge factor for me. An L or one of the "good" non-L (15-85, 17-55/2.8, 10-22, 60 Macro) loses very little value vs. a 3rd party lens that sheds 20% of it's value as soon as you open the box and only goes down hill from there.


----------



## neuroanatomist (Feb 1, 2012)

Vamp898 said:


> ...for example 70-300mm.
> really, the SP lense from Tamron seems to own the Canon lense in nearly every way but cost only ~330 €.
> The Canon L-Lense 70-300 is about ~1250 €



Can you link to the DxOMark comparison between the Canon 70-300mm f/4-5.6L IS and the Tamron SP 70-300mm f/4-5.6 Di VC, tested on the same Canon body? Because I can't find such a comparison (the Canon mount of the Tamron SP 70-300mm is Preview, not Tested). FWIW, even though it's not really valid, the comparison of the Canon 70-300L on a 40D vs. the Tamron on a Nikon D300 (both 12 MP APS-C sensors) shows that the Tamron doesn't come close to 'owning' the Canon L.


----------



## Mt Spokane Photography (Feb 1, 2012)

neuroanatomist said:


> Vamp898 said:
> 
> 
> > ...for example 70-300mm.
> ...



It is tested on Photozone, and Klaus gives us a lot of insite into the little things that are important to photographers. On a CROP BODY (NOT FF) It compares well to the $600 canon 70-300mm USM IS zoom, but does not match the "L"

It is a good buy for those who want good value for their dollar, and well worth looking into. but not for those who want the best at any cost, or have a FF body.

From his conclusion. http://www.photozone.de/canon-eos/592-tamron70300f456vceosapsc?start=1

"The most interesting question is, of course, how it compares to the genuine Canon lenses in this range. The Tamron manages to stay a little ahead of the consumer-grade Canon EF 70-300mm f/4-5.6 USM IS but it doesn't totally reach the professional-grade Canon L variant (especially in terms of bokeh quality). Even so it represents an excellent value offer in the APS-C market. "

Unfortunately, DXO assigns a numeric value to their rating which does not tell you all the things that might be important to you, and is of little or limited value to me.


----------



## kubelik (Feb 1, 2012)

as others have pointed out here, there are tons of things that even a successfully performed, accurate Lines per MM resolution test will not tell you about:

build quality
weathersealing
bokeh (blur quality! not amount of blur)
AF speed
AF accuracy (yeah Sigma, I'm talking about you)
QC consistency (giving Sigma the evil eye again here)
color rendition (why some people pay for Zeiss glass over Canon L glass)
flare resistance (you don't think you need it until you have track wildlife close to sun angle)
IS quality (really 4 stops? maybe so...)

so ... try out a lens before you buy. I'd love for a $300 lens to match a Canon L series lens in every aspect (or even 1/2 of these aspects) but unfortunately I don't see that happening anytime soon.


----------



## bonedaddy.p7 (Feb 1, 2012)

Mt Spokane Photography said:


> Unfortunately, DXO assigns a numeric value to their rating which does not tell you all the things that might be important to you, and is of little or limited value to me.



As with synthetic tests of this nature this is exact problem. They test with many possibly useful, pre-defined parameters and try to take it down to a numbers game. However what you are shooting is not their pre-defined shots. I also find that many great lenses just are not the right tool for a particular shot. A test like this is kinda like any other review, they can often get you into a ballpark idea of how things can compare but they are not the end-all be-all when evaluating a lens.

FWIW, I have the Tamron SP17-50mm VC and it is a very nice lens for the money and I feel that in many of the environments I shoot in it is nearly equal to some L glass, but there are other factors to consider such as weather sealing, ergonomics, etc. I have been using Tamron CCTV lenses for a very long time and they are quite good and I feel that the 17-50 is nearly the same quality as those, but the price difference is also a factor of many things beyond just quality of glass and other components. there is the quality and frequency of QA testing, brand recognition, marketing, in-field testing, and more that factors into pricing and Canon tends to be a bit better tested with higher QA and the L glass offers other benefits like the weather sealing that can be invaluable to some.


----------



## neuroanatomist (Feb 1, 2012)

Mt Spokane Photography said:


> Unfortunately, DXO assigns a numeric value to their rating which does not tell you all the things that might be important to you, and is of little or limited value to me.



Completely agreed. I was just curious about the OP's example in relation to the topic, since it doesn't appear to me that DxOMark had test data relevant to the example.

Looking back at that first post, it reads that the OP actually purchased both the Tamron 70-300 and the Canon 70-300L and tested them personally. 

But I do question some other statements, such as: "_At DxO even the before version of the current 70-300 without IS and without SP seems to be a hard competitor for the L-Lense._" First off, there is no Canon 70-300mm without IS, only 75-300mm lenses lacking IS. But assuming the 70-300 IS non-L was the lens being compared to the L, I think I see the cause of some of the confusion. If you look at the scores, the L is better (overall 18 points vs. 13 points). If you look just at the resolution, they both achieve 56 lp/mm - and I would bet that is the real source of the contention that the lenses perform similarly. But those are peak values for the center of the lens, and if you start digging into the actual results, you see substantial differences between the lenses, especially at the long end.

The details matter, and there is a wealth of data in the DxOMark tests if you dig beyind the top-line scores.

Here's a great but totally meaningless comparison - EF 50mm f/1.8 II vs. EF 300mm f/2.8L IS II, both tested on a 5DII, a $100 lens vs. a $7000 lens. The 50/1.8 II has a higher overall score (27 vs. 15) and better resolution (59 lp/mm vs. 49 lp/mm) - it must be better, right? :


----------



## unruled (Feb 1, 2012)

I have no issues believing tam ron glass gets close to L. I think though that with 3rd party mfct. mileage between different lens models varies more , ie. the 17-50 2.8 non VC is great and sharper than the VC version, but there may be duds in other mm lenses.

all other factors such as build quality etc. are justified, but in my book if I save 60‰ on a tamron even if it failed completely I could afford to buy 2 replacements before reaching the price of L glass. if it never fails,I saved money. a pro may not be able to afford the risk down timewise but for me, an amateur its no issue.

I wish people would stop looking down on non L glass. it definitely has valid use cases. even the crappiest lens now is far better than photogs had in the 19 century, let's not forget what photography is about.


----------



## Kernuak (Feb 1, 2012)

When I first got back into photography and first dragged myself into the digital age, I bought a Tamron 55-200 as a second lens. It cost me £99 as part of a package, although the real price at the time was £199. It was very sharo and punched well above it's weight on that aspect. However, that didn't tell the whole story, granted it wasn't part of Tamron's pro range, but the contrast and colour rendition didn't match any of my L lenses now. Beyond actual image quality though, the build quality was pretty poor, it was put together well enough, but felt cheap and flimsy, worse though, was the manual focus ring, which was about 1mm thick, a bit like the Canon 18-55 kit lens. While the manual focus ring is probably better on Tamron's pro range lenses, that is an area where L lenses excell and really only Zeiss can match or exceed them in that area.


----------



## unruled (Feb 1, 2012)

how can you compare a 99-200 buck lens with a £££ lens? you are right to compare to the canon kit though  

think about it, the canon 50 1.8 is built like a plastic cup..but we forgive it for price and IQ.


----------



## Mt Spokane Photography (Feb 1, 2012)

neuroanatomist said:


> Here's a great but totally meaningless comparison - EF 50mm f/1.8 II vs. EF 300mm f/2.8L IS II, both tested on a 5DII, a $100 lens vs. a $7000 lens. The 50/1.8 II has a higher overall score (27 vs. 15) and better resolution (59 lp/mm vs. 49 lp/mm) - it must be better, right? :



Ha, I did not try that comparison. It just confirms that their value rating point system is useless. But then, most of us know that. I much prefer those lens testers who add personal notes based on their actual use of the equipment along with their measurements. Both are valuable, and one without the other is misleading.

I'd also like to see a database of lens focusing speed and accuracy. I think its coming in the next year more or less. Certainly, software like FoCal is providing a base for adding more and more lens parameters. What is needed, of course, is a average of hundreds of tests, so we can begin to see if there is a particularly good or bad one with regard to focus accuracy. Right now, its just individuals opinions or observations with one lens.


----------



## ejenner (Feb 3, 2012)

After having some issues with my 17-40 and having it to be repaired and then sent back again, I started testing my lenses in real world situations with some scientific reasoning/experiments. I was never much of a brick wall fan, so if it seemed to do what I wanted, I was OK - which is what got me into bother with the 17-40 after moving to FF.

But anyway, that's an aside, the one thing I really learned from all this is that each lens has a very definite set of characteristics that cannot be dumbed down into 1, 2 or even 3 numbers. Or even the center, mid center and a corner on a (flat) test chart.

For instance, no test chart or DxO score tells me how quickly the resolution of the 17-40 drops off compared to say a 24-105 at 24mm f11. (No, even a test chart does not). In fact the more I actually look at real images, the less the lenses seem to be properly characterized buy the test scores even when I can see how those were obtained.

Point is modern lenses are complicated, they have a lot of glass and a lot of corrective elements that make things that should be simple (like DOF) more complicated. Unless you are shooting wide open in the center or at f22 (no wait, not all lenses are the same at f22 despite the fact that every lens should be equally mushy from diffraction)........no, no not going there.

Blah. Rant over. DxO, photozone, blah. Maybe good for a quick general 'is this a lemon' glance, but even then I'm suspicious. All those numbers for people to read so much into, reminds me of the difference between precision and accuracy.


----------



## briansquibb (Feb 3, 2012)

I think DxO can measure the bodies reasonably well. It gives a broad brush view on the bodies capability - for example the comparison of the iso between the 7D and the 5DII gives the 5DII having better iso performance, however the actual numbers they give are pretty meaningless to the average user. The 'quality' of the noise is not mentioned and also the 'quality' is very much a perception that is different for each user.

Another example would be that the 1D4 is noiser than the 5DII at iso100 to iso200 and iit upsets some people. Personally I cant see the noise on an A3 print - so to me it isn't relevant like it is to others.

Another example was that on paper the 1ds3 should be trounced by the 5DII. In reality the 1Ds3 gives much better quality (to me) IQ within its limited iso capability (native iso 1600 max) - again my opinion.

I dont look at DxO lens comparisons - I go to more real world review sites - such as Photozone ( http://photozone.de/reviews where they tell it as they see it rather than being reliant on pseudo scientific measurement assesment.


----------



## JR (Feb 3, 2012)

briansquibb said:


> The 'quality' of the noise is not mentioned and also the 'quality' is very much a perception that is different for each user.



Does anyone know if any body figured out a way to measure this "quality" of noise? Like dpreview or other site/magazine? I am curious to know because I have seen this comment a few time lately on this site and would like to better understand it!

Jacques


----------



## nightbreath (Feb 3, 2012)

It's also interesting to know whether or not DxO sensor ratings is something we should be guided by. If you compare new APS-C sensor ranks even with current full-frame sensors, new ones will have better performance: DxO Mark Sensors Comparison: Sony NEX-7, Nikon D7000, Canon 5D Mark II

Can someone explain me what's happening? Is this an improved technology used by all manufacturers (Nikon, Sony, Pentax) or some kind of hack (algorithms applied to RAW files in-camera) manufacturers use to make their images look better than they initially are?


----------



## ejenner (Feb 4, 2012)

JR said:


> briansquibb said:
> 
> 
> > The 'quality' of the noise is not mentioned and also the 'quality' is very much a perception that is different for each user.
> ...



Not sure if I totally understand it, but say I have two images with the same signal/noise level, but in one the noise is high spatial frequency (speckly sp?) and the other is lower amplitude but also lower frequency. The speckly noise will likely be less objectionable even if it is higher magnitude (for the same S/N it is likely to be). It is less likely to be seen in print and more easily attenuated in NR algorithms.

This is once reason when comparing sensors I like to download raw files and process them. Sometimes the slightly noisier raw file is actually easier to clean up in PP. Having those raw files available is one of the things I really appreciate, although even then it is not perfect becasue a particular ISO on one camera is not necessarily the same exposure as that ISO on another camera. 

In general the most objectionable noise is low-frequency chroma noise. Small amount of luminance noise can actually make an image appear sharper. Of course some of this is personal taste, possibly depending on what you compare it to. I think people who used fast color film generally don't mind the speckle-look as much as the broad smeary-color look of a lot of chroma noise.


----------



## ejenner (Feb 4, 2012)

nightbreath said:


> Can someone explain me what's happening? Is this an improved technology used by all manufacturers (Nikon, Sony, Pentax) or some kind of hack (algorithms applied to RAW files in-camera) manufacturers use to make their images look better than they initially are?



I haven't looked at those specifically, but much of the improvements to date (the last few years) are indeed improved technology. Actually much of it is reduced sensor read noise which basically comes down to being able to manufacture known electronics technology more cheaply. This really helps with high ISO noise levels.

However, I would say that even the latest generations have some tricks, although 'cooking' raw files seems to be a new phenomenon. I mentioned different ISO behavior, there may also be things about the smaller sensors that I am unaware of.

Honestly though, how does a crop sensor have a higher DR than a FF? It would have to have a very much lower noise floor. I'm not sure, but looking at DxO's definition it might well be to do with the way they compute their scores. For instance it is possible that the 5DII will show more shadow detail, but with noise beyond the DxO cutoff, were as the smaller sensors suddenly become very noisy in the shadows. Or maybe it is normalized so comparing different formats is not really applicable. However, combined with the ISO scores, they don't really make sense to me. Also you would improve the ISO score by processing the raw file, but even processing the raw file I find it hard to believe you would increase the DR that much.

EDIT: I just looked at the S100 - DR 11.6 EV, almost as good as the 5DII? Yea, that's normalized or something.


----------



## unruled (Feb 4, 2012)

nightbreath said:


> It's also interesting to know whether or not DxO sensor ratings is something we should be guided by. If you compare new APS-C sensor ranks even with current full-frame sensors, new ones will have better performance: DxO Mark Sensors Comparison: Sony NEX-7, Nikon D7000, Canon 5D Mark II
> 
> Can someone explain me what's happening? Is this an improved technology used by all manufacturers (Nikon, Sony, Pentax) or some kind of hack (algorithms applied to RAW files in-camera) manufacturers use to make their images look better than they initially are?



call in moore's law -- advancements in miniturization and the design of IC's is boosting improvement each year. If you take the 5D classic and compare its output at 1600ISO, and put it next to a 60D at 1600... full frame does not always win.


----------



## well_dunno (Feb 4, 2012)

70-200 f/2.8 IS II has a lower score than all the 70-200 lenses with the exception of f/4 non-IS version in DxO Mark lens rating list, which was disputed by some users there...


----------



## elflord (Feb 4, 2012)

unruled said:


> call in moore's law -- advancements in miniturization and the design of IC's is boosting improvement each year. If you take the 5D classic and compare its output at 1600ISO, and put it next to a 60D at 1600... full frame does not always win.



If you compare the DXOMark scores, you'll see that on the ISO component, it is more or less determined by sensor size. In the example above, the Nikon 7000 vs Sony NEX7 vs 5D Mark II, if you look at the details, 

(1) the 5D Mark II wins on the ISO component, and is consistently better in the SNR18% chart. 
(2) the other cameras do better on the dynamic range part of the DXOMark test due to better dynamic range at _low ISOs_. The 5D Mark II has better dynamic range at ISOs 800-3200. The NEX7 has a curious inflection point in its dynamic range chart. It looks as though they change some in-body parameters at very high ISOs. 

Regarding the 60D and the 5D classic, the 5D classic does beat the 60D on the ISO portion of the DXOMark, and at ISO 1600, it does better than the 60D on all measures in the DXOmark. In fact with the exception of dynamic range at ISOs under 400, the 5D classic beats the 60D on every measure in across its ISO range.


----------



## unruled (Feb 4, 2012)

really? I hadn't checked dxo,but I've shot them side by side and felt the 60d did better. I don't know, maybe I have insufficient faith in artificial benchmarks.


----------



## elflord (Feb 4, 2012)

nightbreath said:


> It's also interesting to know whether or not DxO sensor ratings is something we should be guided by. If you compare new APS-C sensor ranks even with current full-frame sensors, new ones will have better performance: DxO Mark Sensors Comparison: Sony NEX-7, Nikon D7000, Canon 5D Mark II
> 
> Can someone explain me what's happening? Is this an improved technology used by all manufacturers (Nikon, Sony, Pentax) or some kind of hack (algorithms applied to RAW files in-camera) manufacturers use to make their images look better than they initially are?



I think if you take a look at the graphs (click the measurements tab and look at dynamic range and SNR18%), you can see where it comes from -- improved dynamic range at low ISOs. Maybe this suggests that sensor dynamic range is improving, but signal to noise is largely constrained by the sensor.The numbers are consistent with this reading of the graph -- the dynamic range score is based on the low ISO part of the dynamic range curve. The ISO component of the benchmark reflects performance in the higher ISO range on SNR, dynamic range and bit depth curves.


----------



## LetTheRightLensIn (Feb 5, 2012)

Vamp898 said:


> I think the most of you know DxO Mark and there Sensor Benchmarks.
> 
> They also test lense and i recently seen that most of Tamrons SP Lenses are mostly at the same quality as the L-Senses of Canon.
> 
> ...



I think they have good sensor data but terrible lens data.

I did a super careful controlled comparison between the Tamon 70-300 VC and Canon 70-300L and my results did not remotely match their results. Nor have the results from most other lens test sites.

They have weird stuff, they say that at 200mm f/2.8 that the 70-200 f/2.8 IS is sharper than the 70-200 2/8 non-IS is sharper than the 70-200 2.8 IS II. Pretty much everyone who has used them has found the 100% exact opposite results.

They claim that the wide angle Canon FF zooms are sharper at the edges at f/2.8 than at f/5.6. Very bizarre.

I believe they ranked the Canon 70-300 IS non-L better than the Canon 70-300L at 300mm. All I can say is that sure didn't match my results. In fact I think they even had the 70-300L coming in second to last for 300mm and the 300mm IS coming in last! Something crazy like that.

Lens testing is VERY tricky. I bet they just do one quick AF or MF try and are done with it when and that would lead to very random results.

Anyway I do not think the Tamron matches the 70-300L, at all in center frame sharpness anywhere near wide open anywhere along the entire focal range (it gets closer at the edges, oddly enough, or when stopped to f/8, not so surprising). The Tamron AF is also only half the speed. But the price is amazing so it's certainly the best value in tele-zooms.

And I actually do think their 17-50 2.8 is better overall, on APS_C only of course, than the Canon 17-40L, even though it costs less. I added the Tamron, compared them and sold off the L. They don't pay as much for marketing and can charge less than someone like Canon.


----------

