# DXO: Fair and Balanced



## ahsanford (Jan 15, 2016)

As part of our recurring 'DXO is Fair and Balanced' investigative journalism series, we offer you the finest insights.

Today's wonderful test of DXO's impartiality is the Sigma 24-35 f/2. It's always great to put effectively the same lens on both a Nikon and Canon body and see what DXO thinks about it.

See how things turned out below. Classic DXO. 

- A


----------



## Mt Spokane Photography (Jan 15, 2016)

Lens tests cannot be compared across models for obvious reasons. I realize that you were doing this tongue in cheek, but its a good example. The measurements vary from camera to camera because they are not measuring a lens. Some people seem to think they are measuring a lens, but that's not the case.


----------



## ahsanford (Jan 15, 2016)

Mt Spokane Photography said:


> Lens tests cannot be compared across models for obvious reasons. I realize that you were doing this tongue in cheek, but its a good example. The measurements vary from camera to camera because they are not measuring a lens. Some people seem to think they are measuring a lens, but that's not the case.



Understood and agree. I just want to see the aggregate lens DxOMark Score die a fiery death. 

Let DXO report it's nutty numbers with test shots taken in absurd conditions, state that two very similar lenses have a sharpness difference because one stops down to f/22 while the other only stops down to f/16, etc. but don't dangle such a piñata at us like that overall score. It's misleading at best.

- A


----------



## JMZawodny (Jan 15, 2016)

ahsanford said:


> Mt Spokane Photography said:
> 
> 
> > Lens tests cannot be compared across models for obvious reasons. I realize that you were doing this tongue in cheek, but its a good example. The measurements vary from camera to camera because they are not measuring a lens. Some people seem to think they are measuring a lens, but that's not the case.
> ...



The quickest way to bring on a fiery death would be for folks to stop using their website or spreading their opaque results elsewhere around the web.

This is my personal "Don't feed the monsters" philosophy.


----------



## Sporgon (Jan 15, 2016)

Mt Spokane Photography said:


> Lens tests cannot be compared across models for obvious reasons. I realize that you were doing this tongue in cheek, but its a good example. The measurements vary from camera to camera because they are not measuring a lens. Some people seem to think they are measuring a lens, but that's not the case.



Great fun ! I think ahsandford's point is that percentage-wise (judging by the 'scale') most metrics are either identical or within a gnats whisker of each other, with the expection of sharpness, where the 50 mp Canon has a clearly higher score - yet overall the lens scores lower on the Canon.

So this must suggest that DXO's weighting of vignetting and transmission is *huge* compared with sharpness. 

Personally I think it's hilarious, but the problem is a growing number of other sites on the 'net quote them.


----------



## Neutral (Jan 15, 2016)

Sporgon said:


> Mt Spokane Photography said:
> 
> 
> > Lens tests cannot be compared across models for obvious reasons. I realize that you were doing this tongue in cheek, but its a good example. The measurements vary from camera to camera because they are not measuring a lens. Some people seem to think they are measuring a lens, but that's not the case.
> ...


Look at this from the other angle.
DXO gives perceptial resolution of the system containing two components - lens and camera sensor. 
This is very useful to see overal system performance and how well lens is suited for particular camera/sensor.
So on Nikon D800 this combination allows to get 75% of 36mpx sensor resolution (27/36=0.75) but on Canon 5DSR only 64% of 50mpx resolution (32/50=0.64).
So on 5DSR lens degrades overall resolution by 36% and on Nikon d800 only by 25%.
So clearly on D800 score for this lens should by higher than on 5DS.
In other words this lens is less optimal for Canon 5DS than for Nikon D800.

There were number of discussions earlier regarding DXO lenses perceptial resolutions on different bodies and what I see that a lot of people are totally confused and do not undestand well what it is all about.


----------



## bdunbar79 (Jan 15, 2016)

Statements like that confuse me. Who cares if it doesn't resolve as high of a % of MP's off the 5DsR sensor? It still has MORE resolution and MORE detail in the final photo than on the D800. The % or how it gets there are completely irrelevant. So I suppose if I can resolve 100% of the pixels off an 18 MP sensor that is better than 60% of the pixels off a 50 MP sensor? What an asinine comment. 

The score is not useful at all. People/consumers see a score. They don't go, "Oh yeah, but wait, that means the camera/lens combo so we should be more careful in evaluating the scores."

This is the problem with DxOmark. And I just don't know why people cannot understand this. I'm convinced that some photographers just simply choose to be ignorant.


----------



## ahsanford (Jan 15, 2016)

I still see value in DXO's lens testing for one reason: I think having 100 lenses tested on both a 5D3 and a 5DS R is useful.

It lets someone with a 5D3 consider a 50 MP rig (or other higher MP rig) to see how well my current glass might perform on it. I shun the notion that a lens is black/white 'rated for 50 MP' or not -- Roger's early 5D3 vs. 5DS data showed that 4 very different lenses all improved -- but the _degree_ to which they did varied. 

I just want to know if a given lens of mine will shine on a higher-resolving sensor, or if I need to budget to replace a few lenses to go along with that shiny new SLR. That is _*one*_ area DXO can help the world -- crappy metrics and scoring systems be damned. They retest on higher resolving rigs, while other sites like LensTip and PZ are absolutely crawling to upgrade their test rigs and re-test lens.

- A


----------



## neuroanatomist (Jan 15, 2016)

Neutral said:


> Look at this from the other angle.
> DXO gives perceptial resolution of the system containing two components - lens and camera sensor.
> This is very useful to see overal system performance and how well lens is suited for particular camera/sensor.
> So on Nikon D800 this combination allows to get 75% of 36mpx sensor resolution (27/36=0.75) but on Canon 5DSR only 64% of 50mpx resolution (32/50=0.64).
> ...



This logic of yours is completely wrong. Let's apply it...

Sigma 50/1.4A: 
• Nikon D800E, 23 P-Mpix / 36 MP = 64%
• Canon 5DIII, 21 P-Mpix / 22 MP = 95% 
• Clearly on the 5DIII the score should be much higher than the D800E, the lens is far less optimal for the D800E
• DxO Lens Score on D800E = 36, Score on 5DIII = 35

Zeiss 50/1.4 Otus 
• Nikon D800E, 33 P-Mpix / 36 MP = 92%
• Canon 5DIII, 21 P-Mpix / 22 MP = 95% 
• Clearly on the 5DIII the score should be slightly higher than the D800E, the lens is a bit less optimal for the D800E
• DxO Lens Score on D800E = 50, Score on 5DIII = 38

The same would be true of pretty much every 3rd party lens tested on the D800E and the 5DIII, by your flawed logic they should all score higher on the 5DIII - but they don't. 




Neutral said:


> There were number of discussions earlier regarding DXO lenses perceptial resolutions on different bodies and what I see that a lot of people are totally confused and do not undestand well what it is all about.



Unfortunately, it seems you should include yourself in that population of those people who are totally confused and do not understand well what it is all about.

What it is all about is that the DxO Score for lenses is based on performance when shooting at base ISO at 1/60 s in 150 lux – the light level of a dimly-lit warehouse. That means the three factors that primarily drive the Lens Score are lens transmission, sensor DR at base ISO, and sensor color depth at base ISO. That's why the nifty-50/1.8 scores higher than the 600/4 II on the same camera, and that's why the score for the Sigma 24-35/2 is higher on the D800E than on the 5DSR. 

Good job swallowing DxO's BS hook, line and sinker. Alternatively, maybe you understand perfectly well how DxO's Lens Scores work, and your 'other angle' was an expression of bias rather than flawed logic.


----------



## jthomson (Jan 15, 2016)

I still like their software. Their testing aggregate score is a joke and is recognized as such.


----------



## Mt Spokane Photography (Jan 16, 2016)

Neutral said:


> Look at this from the other angle.
> DXO gives perceptial resolution of the system containing two components - lens and camera sensor.
> This is very useful to see overal system performance and how well lens is suited for particular camera/sensor.
> So on Nikon D800 this combination allows to get 75% of 36mpx sensor resolution (27/36=0.75) but on Canon 5DSR only 64% of 50mpx resolution (32/50=0.64).
> ...



Good example of a person confused and not understanding what its all about.

Many of us do not understand, and that's because DXO keeps their grading of a lens score a secret. The only thing we know for sure is that companies that buy their products get a higher score, and DXO made cameras get a higher score. They grading their own camera equal to a DSLR, but keep the method a secret as well.
I keep expecting them to get called out on that one by someone like the FTC that requires claims to be substantiated, and you can't keep the method a secret and still claim yours is better.


----------



## Monchoon (Jan 16, 2016)

dilbert said:


> Until a better website comes along that provides the same or similar metrics for lenses and sensors, it is the best we've got to use.



The best to use for what? And why would you want another that does the same or similar? :


----------



## Aglet (Jan 16, 2016)

dilbert said:


> Until a better website comes along that provides the same or similar metrics for lenses and sensors, it is the best we've got to use.



try 

www.lenstip.com


----------



## Aglet (Jan 16, 2016)

neuroanatomist said:


> Unfortunately, it seems you should include yourself in that population of those people who are totally confused and do not understand well what it is all about.



Perhaps you can dial the vituperation down a stop or 2?

There are plenty of ways of pointing that out without making yourself look like the south end of a north-facing donkey.


----------



## Neutral (Jan 16, 2016)

neuroanatomist said:


> Neutral said:
> 
> 
> > Look at this from the other angle.
> ...



Hahaha, this is very funny when someone is trying like a clown to jump out of his pants to show how bad is someone compared to him ( in logic, knowlege, skills etc.) and doing so intentionally distorts what other person was trying to tell or maybe fails to to grasp main point ). 
I frequently read this forum just for fun )

May be here was also my fault that I did not formulate precisely main point and left room to allow some people to play with the words. This is the internet forum and not scientific media.
Next time will be more careful to formulate things which I want to share.

For those who did not understand what I was trying to tell repeate just one more time but a bit differently :
Forget about DXO scores - consider this as some sort index in their database.
Other indexes could be created and used for sorting data in better way.
What is important about their lens database that this is just tool which allows to estimate how optimal this or that lens on particular body to get maximum performance of the combo.
In other words which combo makes better overall SYSTEM - in particular for system resolution.
This especially important for the latest high resolution high performance sensors and see how well lenses match that sensors performance.
This helps a lot (measurement, not scores) to evalute possible SYSTEM components upgrade path.
No more than that, simple as this.
And as for any tool one need to know what it is for and how to use it in a best way.
All heated discussing about scores bashing DXO is just useless waste of time.


----------



## StudentOfLight (Jan 16, 2016)

I posted this anomaly on the Third Party forum:

http://www.canonrumors.com/forum/index.php?topic=27611.msg568914#msg568914


----------



## Neutral (Jan 16, 2016)

Aglet said:


> neuroanatomist said:
> 
> 
> > Unfortunately, it seems you should include yourself in that population of those people who are totally confused and do not understand well what it is all about.
> ...


Alas, there is some category of living beings who think that if they would try to thrust their sh*t onto someone else this could add respect to them or make them more important within community 
This particular one seems to belong to this category.
BUT - one need to remember that trying to thrust his sh*t onto others could have result just opposite to the intended - all sh*t could be repelled back and initiator could end up covered with his own sh*t )
Luckily this category is in extreme minority here, most are normal people who do not need to behave the same way to stress that they are better than others.


----------



## Neutral (Jan 16, 2016)

StudentOfLight said:


> I posted this anomaly on the Third Party forum:
> 
> http://www.canonrumors.com/forum/index.php?topic=27611.msg568914#msg568914


Tehnical discussion of variances in test measurement and trying to interprete differences and trying to understand what is the cause of them is perfectly good one.
Normally with better sensors results should be better but sometimes there could be something hidden under surface that affects results. 
Sometimes yes, there could be errors in measurements or records due to number of reasons.
Most outstanding case was with Canon EF 70-200 f/2.8L IS USM II when DXO reported lower perceptial resolution compared to the previous generation, did not publicully admitted that this was wrong but eventially corrected measurements results.
This could be really dissapointing for some but this is not the end of the world ).


----------



## neuroanatomist (Jan 16, 2016)

Neutral said:


> Hahaha, this is very funny when someone is trying like a clown to jump out of his pants to show how bad is someone compared to him ( in logic, knowlege, skills etc.) and doing so intentionally distorts what other person was trying to tell or maybe fails to to grasp main point ).
> 
> May be here was also my fault that I did not formulate precisely main point and left room to allow some people to play with the words.



Distort? Not understand? Play with words? Let's review your words:



Neutral said:


> This is very useful to see overal system performance and how well lens is suited for particular camera/sensor.
> So on Nikon D800 this combination allows to get 75% of 36mpx sensor resolution (27/36=0.75) but on Canon 5DSR only 64% of 50mpx resolution (32/50=0.64).
> So on 5DSR lens degrades overall resolution by 36% and on Nikon d800 only by 25%.
> So clearly on D800 score for this lens should by higher than on 5DS.
> In other words this lens is less optimal for Canon 5DS than for Nikon D800.



You stated quite clearly that the lens is less optimal on the D800 than on the 5Ds, and thus it makes sense that the score on the D800 is higher. To what 'score' were you referring? Surely not the P-Mpix measurement, because assuming you have basic counting skills you're aware that the system resolution is higher on the 5Ds. No other measurement has relevance. Therefore, you were justifying the higher DxO Score given to the lens on the D800 based on the differential loss of system resolution on the two cameras. 




Neutral said:


> For those who did not understand what I was trying to tell repeate just one more time but a bit differently :
> Forget about DXO scores...



When called on your obvious fallacy, you responded: 'Oh, I am just misunderstood, forget about the scores,' instead of simply acknowledging your error. 




Neutral said:


> What is important about their lens database that this is just tool which allows to estimate how optimal this or that lens on particular body to get maximum performance of the combo.
> In other words which combo makes better overall SYSTEM - in particular for system resolution.



For system resolution, the Sigma 24-35 delivers 18% better performance on the 5Ds than on the D800 (32 vs. 27 P-Mpix). So by your own statements, you are confirming that the lens is better on the 5Ds, yet earlier you concluded the opposite – that the lens should get a higher score on the D800. Perhaps you should retract your earlier conclusion. Or just spew more sh!t, it's up to you. 




Neutral said:


> This helps a lot (measurement, not scores) to evalute possible SYSTEM components upgrade path.
> No more than that, simple as this.



Yes, that makes perfect sense. DxO's Scores are useless, their measurements are generally useful (except when they make errors). I've probably stated that more times here than your total number of posts on this forum.


----------



## privatebydesign (Jan 16, 2016)

Neutral said:


> Forget about DXO scores - consider this as some sort index in their database.
> Other indexes could be created and used for sorting data in better way.
> What is important about their lens database that this is just tool which allows to estimate how optimal this or that lens on particular body to get maximum performance of the combo.
> In other words which combo makes better overall SYSTEM - in particular for system resolution.
> ...


But without knowing and understanding how the tests are made and the results formulated there is nothing of value to that database.

Do we know if there is a 10% gain applied to results from manufacturers that start with S or N? No we don't! Do we know how relevant the testing protocol is to our own actual shooting situations? In general it is nothing like most peoples. Do we know the weighting that is given to lens characteristics that have little impact in real life with the software we have, things like vignetteing? Again we don't. 

DXO is entirely flawed and without transparency the 'results', ratings, and numbers are entirely irrelevant.


----------



## Orangutan (Jan 16, 2016)

privatebydesign said:


> DXO is entirely flawed and *without transparency the 'results', ratings, and numbers are entirely irrelevant*.



This is the key: DxO has insufficient transparency with regard to testing protocols and calculations. Without transparency, all inferences are meaningless. Furthermore, promoted inferences that lack supporting data should be considered marketing rather than science or engineering.


----------



## Neutral (Jan 17, 2016)

Orangutan said:


> privatebydesign said:
> 
> 
> > DXO is entirely flawed and *without transparency the 'results', ratings, and numbers are entirely irrelevant*.
> ...



I do agree that better transparency could be beneficial for people who want to be confident that DXO measurements results are 100% correct.
On the other hand lack of such trasparency does not mean at all that their measurements are incorrect , wrong or fake.
Morever stating that are just assumptions not supported by any facts/data, just personal emotions.
I personally do not see any issues with the results of their measurements (not scores) looking at their data and doing my own tests. One interesting thing though that using best canon glass on my Sony a7rII I get perceptially better results than DXO publish for Canon 5DSR. They resolve to the every single pixel on a7rII. But this is after sharpness corrections in LR and a7rII sensor allows very agressive sharpening without any visual side effects.
Also DXO could be caught by other independant test labs if they have done something wrong.
Also it is not very difficult to do reversed calculations to prove that they are wrong in perceptual resolution measurements. Just use MTF50 of one known lens (e.g. sigma art 50 published at lensrental) and use DXO measurements data for this lens on Canon 5dsr as starting point to calculate lens resolution lp/mm and to compare with lens MTF50 results on lensrental. Also this would allow to estimate DXO measurements initial condition. This (reversed calculations) is common practice for proving that something correct or incorrect.
So I trust their data and have no bias or prejustice to them and this based on their history and my own evaluation of their measurements data.
DXO started long time back as company doing optical test/measurements with their DXO optical analyser and then later started with DXO Optic RAW converter which is now well recognized. They do all their lens/sensor measurements for DXO lens correction modules - for themselves - to provide best possible lens correction results in their SW. It would be very strange to think that they would be using wrong/fake measurements to get best results for their SW.
Having accumulated huge amount of measurement data they started to publish that for other people benefit and of course for marketing themselves.
More useful info for their DXO Analyzer and what they doing could be found here:
http://www.dxo.com/us/image-quality-evaluation/dxo-analyzer


----------



## Orangutan (Jan 17, 2016)

Neutral said:


> Orangutan said:
> 
> 
> > privatebydesign said:
> ...



You've made my point for me, it's a question of the burden of proof. If DxO is to be considered science or engineering, then the burden of proof rests fully on DxO to demonstrate, through transparency and supporting data, that all the work is correct. Otherwise, it is merely marketing, where the burden of proof shifts more toward the "you haven't proved it's wrong" end of the spectrum.

So long as DxO continues to obfuscate their methods, their work is marketing crap, and should not be given our respect.



> Also DXO could be caught by other independant test labs if they have done something wrong.


On the other hand, we could just stop paying any attention at all to DxO's marketing crap, and read the independent tests instead.


----------



## Neutral (Jan 17, 2016)

Orangutan said:


> Neutral said:
> 
> 
> > Orangutan said:
> ...


OK, then why not to ask officially ( kind of public petition ) their customers (including NASA, Samsung, Pentax, Panasonic, Sigma, FujiFilm etc.) to impose sanctions on DXO and stop using their measurement facilities/services crap. May be this would force DXO to publish all their measurements methods, conditions and calculations to general public ) Also to all users of DXO Optic to stop using this crappy product and return it back DXO )


----------



## Orangutan (Jan 17, 2016)

Neutral said:


> Orangutan said:
> 
> 
> > Neutral said:
> ...



Non sequitur, these are separate questions. What they do on contract is a matter between them and their customer. I'm referring to their publicly promoted material. There is no reason to believe they use equally rigorous methods for both, and some reason t believe they don't because they've been caught in errors several times. In short, their public database should be considered marketing material, not engineering data, except in those cases where they provide enough information to replicate their results, and where their testing is internally consistent.


----------



## Larsskv (Jan 17, 2016)

Neutral said:


> privatebydesign said:
> 
> 
> > DXO is entirely flawed and *without transparency the 'results', ratings, and numbers are entirely irrelevant*.
> ...



Consider what you are writing here. You do not see any issues, but you get better "perceptually better results" with the A7RII than the 5DSR. Considering that extra resolution, that seems a little weird, don´t you think?

Did you compare apples to apples? 

Did you take your test picture in a room with 150 lux, at 1/60 sec, at base ISO and pushed the file to correct lighting when you took your comparison picture? No? Because that is what DXO seems to do. I get "perceptually better results" with all my f/4 lenses, than what is reported by DXO. Why? Because I make a correct exposure in a sufficient amount of light, when I am shooting, and therefore DXO is irrelevant for my type of shooting, which I guess is quite common - at least with Canon shooters.. DXO is very misleading, and it´s hard to know when they don´t let you know how they rate their scores.

(Edit: I don´t see why my text is quoted here...) <fixed by mod>


----------



## bdunbar79 (Jan 17, 2016)

I guess as a scientist, I find them calling their measurements and scores "scientific" quite offensive. It's literally just wrong and it is not scientific. Whenever I submit my results, I HAVE to disclose ALL of my methodology and measurements, so why don't they have to? Well, because they're not doing real science because that is every part of real science. Disclose the scoring system and parameters and I'm fine, but until then, they are absolutely meaningless to the scientific community.


----------



## Orangutan (Jan 17, 2016)

Larsskv said:


> (Edit: I don´t see why my text is quoted here...)



You may be missing some quote tags somewhere. It's got someone else's words attributed to me.


----------



## Larsskv (Jan 17, 2016)

Orangutan said:


> Larsskv said:
> 
> 
> > (Edit: I don´t see why my text is quoted here...)
> ...



Your absolutely right! Sorry about that. I can´t recall exactly what I did. Maybe I deleted some quote tags, and the numbers of quote tags dont add up.


----------



## neuroanatomist (Jan 17, 2016)

Neutral said:


> I personally do not see any issues with the results of their measurements (not scores) looking at their data





Neutral said:


> Sometimes yes, there could be errors in measurements or records due to number of reasons.
> Most outstanding case was with Canon EF 70-200 f/2.8L IS USM II when DXO reported lower perceptial resolution compared to the previous generation, did not publicully admitted that this was wrong but eventially corrected measurements results.



Perhaps you don't have any problem with unethical behavior, but when an organization purports to be scientifically-based but fails to fully disclose their methods, they are not acting in accordance with the established practice of science. Moreover, when they are made aware of an error and defend their data rather than publicly acknowledging their error, then subsequently alter their underlying data but fail to correct their conclusions based on those faulty data, that behavior is counter to the ethical principles to which scientists are expected to adhere. 







DxO Image Science...Bad Science...Biased Scores...BS.


----------



## Neutral (Jan 17, 2016)

Larsskv said:


> Neutral said:
> 
> 
> > Orangutan said:
> ...


[

I agree that using this combination of ISO, shuter speed and luminance creates a bit difficult overall setup and gives latest better tech sensors some advantage.
With higher luminance results could be better.
And this applies in particular to the a7rII sensor. As I mentioned I can push sharpness very agressively using 0.5 radius without seeing any side effects. As results on final processed image I can see that my EF24-70 f/2.8L IS USM II resolve to every single pixel on a7rII. The same for the latest 70-200 and 100-400 . 
So better visual results is contribution of both the lens quality and sensor quality and indicator of this combo performace.

On the other hand if all measurements are done using the same conditions then they could be compared on the same body without any problems/issues . For comparison it important that all done exactly the same with the same conditions
If I want to compare lenses on the same body this is all OK.
As for luminance conditions - this could be considered as some negative shift/bias that could be taken into account if needed.
For me I use DXO data to see what could be the best performance lens for particular body so I do not have any problems with their measurements.


----------



## Larsskv (Jan 17, 2016)

Neutral said:


> Larsskv said:
> 
> 
> > Neutral said:
> ...



DXO does not tell you how the lens performs at it´s sharpest aperture, given you have sufficient light. Further, one must be aware that wide aperture lenses have a major advantage over smaller aperture lenses, such as f/4 lenses. That is because of the peculiar lighting set up, 150 lux, and 1/60 sec shutter speed. Since I never shoot lets say f/4 lenses under such conditions, I find their measurements irrelevant.


----------



## Aglet (Jan 17, 2016)

neuroanatomist said:


> Perhaps you don't have any problem with unethical behavior..



oh, the irony



> DxO Image Science...Bad Science...Biased Scores...BS.



doesn't bad ol' Dr. Brain use DXO Pro software on his little mac computer at home?


----------



## 3kramd5 (Jan 18, 2016)

Aglet said:


> neuroanatomist said:
> 
> 
> > Perhaps you don't have any problem with unethical behavior..
> ...



Pardon my skepticism that those people who turn lens data into useful image processing code are the same people turning lens data into a web-accessible database and scoring system. 

The composite scores are bathwater.


----------



## Mt Spokane Photography (Jan 18, 2016)

neuroanatomist said:


> Neutral said:
> 
> 
> > I personally do not see any issues with the results of their measurements (not scores) looking at their data
> ...



Yes, I ran a Aerospace Lab, and would have been fired if I produced reports for NASA or other organizations that were so sloppy.

First, I had to use recognized and documented test methods and they had to make sense in the context that they were used for. I had to document the test setup and record all the results. The original data was always available for review. I was not allowed to make up my own tests. 

Unfortunately, the lack of standard test methods applies to all camera testers, there is no standard test method for lenses attached to digital cameras, so the results we see can not be taken as reliable.

There are some popular tests that use software called Imatest, but it is not a accepted standard, and results definitely have flaws because they can not account for so many variables in a lens/camera combination. Even so, Imatest results can be duplicated by anyone willing to take the time and purchase the software.

Numbers invented by DXO like mpix are undefined and their use in final test scores is very confusing. A product can do very well in all the published results and get a lower score than a product that received lower results. Its because their methods don't make sense as far as pushing out a overall score.


----------



## Don Haines (Jan 18, 2016)

Aglet said:


> neuroanatomist said:
> 
> 
> > Perhaps you don't have any problem with unethical behavior..
> ...


I think there is a huge difference between the two......

The software has a well defined goal and allows the user to control it to their desired preferences.

The camera and lens rating is pre-determined mush where all the variables are pre-defined to values that are realistic.

If the software was similar, it would permananently set the white balance slider way over to one side, tint to the other side, boost gain by 5 stops, and only allow highly compressed jpg output at 640x400 resolution.


----------



## Don Haines (Jan 18, 2016)

The problem with DXO's single number rating system is a conceptual problem. We can argue over how that number is determined, but in the end, it does not matter. The problem is the concept.

Any attempt to reduce a complex system used by diverse people for diverse goals down to a single rating metric is ******* to failure.


----------



## candc (Jan 18, 2016)

I will be captain obvious. They compile a lot of specific data and then diminish its usefulness by combining it all into one number.

For instance. The sharpness value is an average across the field at all apertures. Maybe that's important to know for some reason? I don't care how sharp it is stopped all the way down, I don't use lenses that way.


----------



## Kwwund (Jan 18, 2016)

Well...consider the service provided by a restaurant critic. He tells you that the food is good, the service was slow, the ambience was poor, and the price was reasonable. Two out of four stars. You don't know the technical specs of the ratings on any dimension, nor the weights applied to each dimension. Is the 2-star rating useless because you don't know how it was mathematically determined?

There are lots of composite rating systems in all fields. They do tend to obscure detail, and they force the reader to trust the values/judgment/biases of the reviewer. But I would not go so far as to say that such composite ratings are *******.


----------



## neuroanatomist (Jan 18, 2016)

Kwwund said:


> Well...consider the service provided by a restaurant critic. He tells you that the food is good, the service was slow, the ambience was poor, and the price was reasonable. Two out of four stars. You don't know the technical specs of the ratings on any dimension, nor the weights applied to each dimension. Is the 2-star rating useless because you don't know how it was mathematically determined?
> 
> There are lots of composite rating systems in all fields. They do tend to obscure detail, and they force the reader to trust the values/judgment/biases of the reviewer. But I would not go so far as to say that such composite ratings are *******.



It the restaurant critic's motto, _Food Science_? :


----------



## Don Haines (Jan 18, 2016)

Kwwund said:


> Well...consider the service provided by a restaurant critic. He tells you that the food is good, the service was slow, the ambience was poor, and the price was reasonable. Two out of four stars. You don't know the technical specs of the ratings on any dimension, nor the weights applied to each dimension. Is the 2-star rating useless because you don't know how it was mathematically determined?
> 
> There are lots of composite rating systems in all fields. They do tend to obscure detail, and they force the reader to trust the values/judgment/biases of the reviewer. But I would not go so far as to say that such composite ratings are *******.


Yes, but imagine if the food is all rated from 1 to 100.... Does this mean that you would be better off ordering the 87 rated pancakes than the 79 rated pie? Even though you really wanted the 73 rated cup of coffee?


----------



## Orangutan (Jan 18, 2016)

neuroanatomist said:


> Kwwund said:
> 
> 
> > Well...consider the service provided by a restaurant critic. He tells you that the food is good, the service was slow, the ambience was poor, and the price was reasonable. Two out of four stars. You don't know the technical specs of the ratings on any dimension, nor the weights applied to each dimension. Is the 2-star rating useless because you don't know how it was mathematically determined?
> ...



+1

This is the point I made above: if this is done "for fun" (as marketing) then who cares? The problem is that DxO is promoting this as objective.

If you have a single restaurant critic go into a restaurant (presumably in disguise), then it's entertainment.

If you have a blind test-test using 100 graduates of a culinary academy, you can begin to call it science.


----------



## ahsanford (Jan 18, 2016)

candc said:


> I will be captain obvious. They compile a lot of specific data and then diminish its usefulness by combining it all into one number.
> 
> For instance. The sharpness value is an average across the field at all apertures. Maybe that's important to know for some reason? I don't care how sharp it is stopped all the way down, I don't use lenses that way.



This is why the 35 Art was considered sharper than the 35L II on the 5D3. The 35L II goes to f/22 while the 35 Art only stops down to f/16. The poor sharpness value of f/22 pulled down the average of the 35L II, which is nonsense.

- A


----------



## PhotographyFirst (Jan 18, 2016)

I think DXO is quite genius and deliberate with their scoring system. 

The male gender sure has a strange willingness to argue over anything and everything. Just like with sports, audio gear, computers, video game systems, cars, and so on. Having numbers to argue over regardless of how meaningless and wrong those numbers might be, is like fuel for the flames. Even better when those numbers are dumbed down as much as possible. "7 is better than 6.9! Your choice sucks! You loser of purchasing crappy low scoring gear!" 

I would be very curious to see if DXO has calculated the number of sales conversions they get when people visit to see DXO scores. Are most of those people just visiting to get numbers for arguing or are most of them actually using DXO to make a purchase choice? I would think that someone who is already in a camera system is purchasing their gear based on a need rather than just trying to find the highest scoring lens to purchase regardless of being useful for their work. I don't think many people are trying to decide between a 50mm prime and a 600mm prime based on which one scores higher. 

All of the photo rumors and gear websites know very well that DXO is prime clickbait for readers. All they have to do is post what DXO does and they get their clicks regardless of how controversial the DXO content really is. Much better than having to post a link to a complex review with test shots, charts, graphs, and other more useful data. 

Next time I am at a sporting event, I am going to sneak down to the sidelines with a Sony A7s II, the Canon 50mm STM, and a printout of the DXO scores. Those guys using the 600 II lens and 1dx will be full of regret when they see my lens is several points better than theirs and my camera body totally destroys them for sports shooting scores.


----------



## 3kramd5 (Jan 18, 2016)

PhotographyFirst said:


> I would be very curious to see if DXO has calculated the number of sales conversions they get when people visit to see DXO scores.



I don't see how they possibly could unless they are exceedingly heavy-handed with tracking cookies.


----------



## bdunbar79 (Jan 18, 2016)

PhotographyFirst said:


> I think DXO is quite genius and deliberate with their scoring system.
> 
> The male gender sure has a strange willingness to argue over anything and everything. Just like with sports, audio gear, computers, video game systems, cars, and so on. Having numbers to argue over regardless of how meaningless and wrong those numbers might be, is like fuel for the flames. Even better when those numbers are dumbed down as much as possible. "7 is better than 6.9! Your choice sucks! You loser of purchasing crappy low scoring gear!"
> 
> ...



Great post! LOL!

Sales? No need. DxOmark consistently scores all of Nikon's gear higher than Canon's, year after year after year, yet Nikon gains no market share on Canon in the camera market. Love it!


----------



## neuroanatomist (Jan 19, 2016)

Factual errors and a refusal to admit mistakes, combined with an obvious bias in conclusions. Dilbert and DxOMark have much in common.


----------



## Don Haines (Jan 19, 2016)

dilbert said:


> PhotographyFirst said:
> 
> 
> > ...
> ...


The DXO scores are extremely heavily biased towards T-stop values....so much so, that realistically, that is the only metric that counts. 

The problem that most people have with DXO is that their rating system is based on unrealistic values and is kept secret. This is not an open and unbiased system.

As a researcher, when I run an experiment I have to show the exact setup of the equipment, show all the settings, include the calibration data for the equipment ( even the serial numbers), and all the data. Good runs and bad runs are included. multiple runs for repeatability..... And in the end, when I write up the report, different conclusions for different conditions.... If I were to say that "this radio is the best for everything because it handled this one unrealistic scenario the best", I would be encouraged to either take a long vacation or to retire.....


----------



## 3kramd5 (Jan 19, 2016)

dilbert said:


> PhotographyFirst said:
> 
> 
> > I think DXO is quite genius and deliberate with their scoring system.
> ...



Am I blind? I'm failing to see an reference to size in the quoted post.


----------



## dolina (Jan 19, 2016)

I like DXO's Mobile review. Got me to consider the Nexus 6P.


----------



## Larsskv (Jan 19, 2016)

dilbert said:


> Don Haines said:
> 
> 
> > ...
> ...



How would anybody outside DXO know?

I suspect it is lower t stop=better, because small aperture lenses suffer so much in DXO tests. If tests are done at 150 lux, 1/60 sek and base ISO, lower t stop will be better.


----------



## 3kramd5 (Jan 19, 2016)

dilbert said:


> 3kramd5 said:
> 
> 
> > dilbert said:
> ...



You are implying that the poster thinks the 600 is better than the 50 because it's bigger than the 50.

Rather, he was CLEARLY (see red) talking about applicability to the given subject matter, which is the major failing of a composite scoring system applied to items with a wide range of intended uses.

Size is irrelevant.


----------



## Maiaibing (Jan 19, 2016)

bdunbar79 said:


> Whenever I submit my results, I HAVE to disclose ALL of my methodology and measurements, so why don't they have to?


Because no one demands you buy their products or spend your time finding their site and reading the info they put on the web for you to enjoy/hate for free.

Lots of companies claim science backs their products and statements without any such disclosure. Cosmetics companies are a case in point. Methodology and measurements is often - correctly - considered a valuable intellectual property that should be protected. No surprise here.


----------



## bdunbar79 (Jan 21, 2016)

Maiaibing said:


> bdunbar79 said:
> 
> 
> > Whenever I submit my results, I HAVE to disclose ALL of my methodology and measurements, so why don't they have to?
> ...



No. Cosmetic companies have to do regulated toxicological testing, where everything, every little single thing, MUST be recorded and disclosed, and audited by a regulating body.

Anytime you publish a journal article, or file a patent, etc. 

The other point, is that everything you just typed doesn't make DxOmark scientific. My only point was that they are not scientific...because they are not. Not by a long shot. So why claim to be?


----------



## neuroanatomist (Jan 21, 2016)

dilbert said:


> The scoring system is fine. There is nothing wrong with the scoring system.



There is nothing wrong with measuring weight in stone or length in furlongs, but it's not too useful for the vast majority of people. 

There _is_ something wrong with a scoring system that is biased, undisclosed, and may be determined differently for different products.


----------



## Don Haines (Jan 21, 2016)

dilbert said:


> The scoring system is fine. There is nothing wrong with the scoring system.


The scoring system rates the T-stop value drastically higher than any other optical properties of the lens....

If this is fine with you, then so be it.... but many others seem to disagree.

My problem with DXO is the entire concept of reducing things to a single number is deeply flawed. No matter how they arrive at that number, no matter what their magic formula is, the concept of a single number is deeply flawed. They would have been better off to have stopped at the sub-scores.


----------



## Maiaibing (Jan 21, 2016)

Don Haines said:


> dilbert said:
> 
> 
> > The scoring system is fine. There is nothing wrong with the scoring system.
> ...



While I understand all reviews at DpReview etc. etc. etc. are imminently painful to you, single number scores - sometimes much less developed than DXO's - enjoy extremely wide spread popularity on the net, in newspapers, magazines etc. 

Why single DXO out for this?


----------



## Maiaibing (Jan 21, 2016)

bdunbar79 said:


> No. Cosmetic companies have to do regulated toxicological testing, where everything, every little single thing, MUST be recorded and disclosed, and audited by a regulating body.



I hope this is not a desperate attempt to turn around from a bad argument by drawing a strawman. Safety testing has nothing at all to do with their claims of scientific testing supporting the value of their products.

This is what you were claiming was wrong of DXO to do:

"Experience the first creme with IntuiGen Technology™ that knows your skin's needs.
This high-performance, multi-action moisturizer dramatically reduces the look of multiple signs of aging: lines, wrinkles, loss of firmness, dullness and dehydration.
Breakthrough IntuiGen Technology™ helps activate skin's own revitalization. It specifically addresses your unique anti-aging needs all at once, revealing a younger and more beautiful look.
The lightweight, silky soft creme feels like no other, cushioning your skin with sensuously rich, refreshing moisture.
_Tested and proven: Instantly, skin looks smoother, clearer, more radiant. In just 4 weeks, 92% of women showed significantly firmer skin.
_

I suggest you contact Estee Lauder and ask them for disclosure.


----------



## ahsanford (Jan 21, 2016)

Maiaibing said:


> Why single DXO out for this?



Numerical metrics/ratings make sense for individual aspects of a product's performance, for instance:


I am curious to see if lens X does well making the jump from a 5D3 to a 5DS.
I am curious to know how much light a lens really lets in.
Sensor DR, high ISO performance and other metrics can be measured and reported

Skipping Neuro's (absolutely legitimate) beef with DXO's individual test methods, I'm not opposed to measuring things like sharpness, low light sensitivity, etc. and giving it a number.

Where I go Defcon 1 is how DXO rolls things up:

[list type=decimal]
[*]Roll those individual metrics together into a nonsensical, inconsistent overall score that seems to prioritize minutiae like Transmission (or even the outright max aperture) over the more bread and butter considerations of sharpness, aberrations, vignetting, etc.
[*]Overwhelmingly weight the use of that overall score to peg if something is 'ground-breaking' or 'disappointing'
[*]Apply a noogie to other products without considering usage, photographic need, features, etc. like some sterile United Nations of optical quality.
[/list]

Again, if DXO simply did away with the Overall Score, people could go there and mine the data they needed when considering new gear. That's all they need to do.

- A


----------



## bdunbar79 (Jan 21, 2016)

Maiaibing said:


> bdunbar79 said:
> 
> 
> > No. Cosmetic companies have to do regulated toxicological testing, where everything, every little single thing, MUST be recorded and disclosed, and audited by a regulating body.
> ...



I'm sorry, but where in their claim does it ever say "science" or "scientific"? Nowhere, because they are not allowed to. 

Things starting to click now?

If you have a therapeutic cream, then they must do efficacy testing to make scientific claims, but that's more pharmaceutical.

But anyways, I guess since cosmetic companies mislead with garbage science, it's okay for DxOmark to do so. So in a weird way, by using your example, you're admitting DxOmark is garbage science, and also okay with that...because afterall, the cosmetic industry does it.

_Unfortunately, cosmetic companies have to make misleading claims because this is what consumers respond to._

Just like DxOmark...


----------



## 3kramd5 (Jan 21, 2016)

dilbert said:


> 3kramd5 said:
> 
> 
> > ...
> ...




?

_"Next time I am at a sporting event, I am going to sneak down to the sidelines with a Sony A7s II, the Canon 50mm STM, and a printout of the DXO *scores*. Those guys using the 600 II lens and 1dx will be full of regret when they see my lens is several *points *better than theirs and my camera body totally destroys them for sports shooting *scores*."_

If he were drawing a conclusion that bigger is better, the analogy is backwards. Rather, he was demonstrating is that *scores *don't relate to *applicability*. 

You, not the quoted poster, are introducing an irrelevant variable (size) which no one else is considering. You're making a gross assumption that he thinks the 1Dx + 600 II is a better combo for the sidelines of a sporting event than the A7S2 + 50 STM because the former combo is bigger, when most people would consider it better because of higher framerate, longer focal length, better autofocus, easier handling, etc.


----------



## PhotographyFirst (Jan 21, 2016)

3kramd5 said:


> dilbert said:
> 
> 
> > 3kramd5 said:
> ...



It's best to just put the trolls on your ignore list. Every time you reply, he gets his e-jollies and thus wins every time no matter how wrong. The guy is arguing over a part of a post that was written to be humorous, let alone arguing something completely imaginary.


----------



## Don Haines (Jan 21, 2016)

Maiaibing said:


> Don Haines said:
> 
> 
> > dilbert said:
> ...


It is because it is too wide.

Just like rating the best vehicle based on the number of cupholders is ludicrous, rating the best lens on one parameter is silly. If one wanted to rate the best economy car based on cost/reliability/fuel economy then that makes sense.... but rating the best dumptruck on those same parameters means that a mine should get a Toyota Prius rather than a Caterpillar 795D to haul that 240 ton load up out of the mine......


----------



## ahsanford (Jan 21, 2016)

Don Haines said:


> Maiaibing said:
> 
> 
> > Don Haines said:
> ...



+1. Just kill the overall score metric and my respect for DXO goes from a 1 out 10 to perhaps a 4 out of 10. I'm not kidding. 

Yes, their methods are nuts, but no one else on the planet is attempting a 'human genome project'-level of comprehensive testing *that updates when sensor resolutions increase* like they do. Though I always prefer dedicated reviews from folks I respect more than DXO, sometimes you want to know how a 28mm f/2.8 IS USM does on 50 MP compared to 22 MP, and DXO -- despite all it's flaws -- has that information.

- A


----------



## Maiaibing (Jan 21, 2016)

bdunbar79 said:


> I'm sorry, but where in their claim does it ever say "science" or "scientific"? Nowhere, because they are not allowed to.
> 
> Things starting to click now?
> 
> If you have a therapeutic cream, then they must do efficacy testing to make scientific claims, but that's more pharmaceutical.



Glad you spotted the difference yourself this time. 

Do your scientific papers include the word "scientific" or "science"? I actually read such papers and cannot - off hand - remember ever having seen the words "scientific" or "science" mentioned (except in philosophical papers discussing what science is or is not). Not excluded - but seems very unusual to me.



bdunbar79 said:


> But anyways, I guess since cosmetic companies mislead with garbage science, it's okay for DxOmark to do so.



This is the second time you try to draw me into a discussion of whether DXO is scientific. I never commented on this question - because I don't care. Yes, there's lots of garbage science around - also in the scientific world - be happy you can spot the difference. Too many cannot.

DXO has one lens measurement that I find very valuable - and I'm very happy to be able to get that info from them. YMMV.


----------



## neuroanatomist (Jan 21, 2016)

Maiaibing said:


> Do your scientific papers include the word "scientific" or "science"? I actually read such papers and cannot - off hand - remember ever having seen the words "scientific" or "science" mentioned (except in philosophical papers discussing what science is or is not).



Some are published in this high impact journal.


----------



## ahsanford (Jan 21, 2016)

neuroanatomist said:


> Maiaibing said:
> 
> 
> > Do your scientific papers include the word "scientific" or "science"? I actually read such papers and cannot - off hand - remember ever having seen the words "scientific" or "science" mentioned (except in philosophical papers discussing what science is or is not).
> ...



+1. Scientists and engineers are probably the largest part of the enthusiast-class of CR forum dwellers. 

- A


----------



## Maiaibing (Jan 21, 2016)

ahsanford said:


> Scientists and engineers are probably the largest part of the enthusiast-class of CR forum dwellers.



Do you have stats to back that claim? ;D


----------



## bdunbar79 (Jan 21, 2016)

Maiaibing said:


> bdunbar79 said:
> 
> 
> > I'm sorry, but where in their claim does it ever say "science" or "scientific"? Nowhere, because they are not allowed to.
> ...



Then what are you arguing? Cosmetic companies (your example) aren't claiming to be scientific organizations providing scientific data. DxOmark is. That's why they are getting singled out. Because, they're not.


----------



## AlanF (Jan 21, 2016)

Maiaibing said:


> Do your scientific papers include the word "scientific" or "science"? I actually read such papers and cannot - off hand - remember ever having seen the words "scientific" or "science" mentioned (except in philosophical papers discussing what science is or is not). Not excluded - but seems very unusual to me.



A search of Web of Knowledge gives 597,267 papers published with the word "science" in their title.


----------



## ahsanford (Jan 21, 2016)

Maiaibing said:


> ahsanford said:
> 
> 
> > Scientists and engineers are probably the largest part of the enthusiast-class of CR forum dwellers.
> ...



 

I honestly don't, but call it a hunch -- too many people on this forum speak in the language I'm accustomed to hearing at work. (Though I traded writing peer-reviewed publications for creating patents some time ago.)

- A


----------



## Maiaibing (Jan 21, 2016)

ahsanford said:


> I honestly don't, but call it a hunch -- too many people on this forum speak in the language I'm accustomed to hearing at work. (Though I traded writing peer-reviewed publications for creating patents some time ago.)
> 
> - A



Good enough for me!


----------



## neuroanatomist (Jan 21, 2016)

AlanF said:


> Maiaibing said:
> 
> 
> > Do your scientific papers include the word "scientific" or "science"? I actually read such papers and cannot - off hand - remember ever having seen the words "scientific" or "science" mentioned (except in philosophical papers discussing what science is or is not). Not excluded - but seems very unusual to me.
> ...



Surely you're not suggesting that ISI has better data than some random person on the Internet?!


----------



## Larsskv (Jan 22, 2016)

dilbert said:


> Don Haines said:
> 
> 
> > Maiaibing said:
> ...



Dilbert, please tell me what the DXO scores mean, what are they actually telling us about a lens? Because when buying that f/4 zoom lens, I'm obviously not going to shot it at 150 lux at base ISO and 1/60 shutter speed, but in sufficient light. Is the DXO scores relevant for a purchasing decision in such a situation? Why/why not?


----------



## Maiaibing (Jan 22, 2016)

AlanF said:


> A search of Web of Knowledge gives 597,267 papers published with the word "science" in their title.



Yeah - you're right. So not only unusual but _extremely _rare. Thanks for the check-up.


----------



## Don Haines (Jan 22, 2016)

dilbert said:


> Don Haines said:
> 
> 
> > Maiaibing said:
> ...


Ok.... How about this......

It is because it is too wide.

Just like rating the best lens based on the T-stop is ludicrous, rating the best lens on one parameter is silly. If one wanted to rate the best economy lens based on lowest cost then that makes sense.... but rating the best birding lens on those same parameters means that a birder should get a Nifty-fifty rather than a Big White to haul that 240 meter away target up onto the sensor....


----------



## Hector1970 (Jan 22, 2016)

I've no problem with DXO.
There is probably no perfect measurement.
It tells you something about the camera / lens.
I've no idea if any individual sensor from Nikon is actually better than Canon.
I just think Canon produce on balance of everything a more usable and reliable camera.
I've used Nikon's and maybe they are great on sensors but I just prefer the layout of Canon, the glass is better etc.... 
I prefer myself more descriptive reviews of camera gear.
I like the Digital Picture reviews by Bryan Carnathan .
I haven't gone wrong with any of his recommendations so far.


----------



## neuroanatomist (Jan 22, 2016)

dilbert said:


> Well the problem is that it isn't easy to draw any conclusions about which parameter is most important.



So DxO used dilbertland logic to decide the most important parameters for a Lens Score are lens transmission, sensor DR at base ISO, and sensor color depth at base ISO.


----------



## AlanF (Jan 22, 2016)

Talking about Lens score, there is a site called lenscore.org (+senscore), which claims to be scientific. But, I don't understand their overall ratings either. For instance, with zoom lenses they give an overall score for the lens without, unless I have missed it, stating the focal length etc. Seems nonsenscore to me.


----------



## 3kramd5 (Jan 22, 2016)

dilbert said:


> 3kramd5 said:
> 
> 
> > ...
> ...



The analogy here is sports, and in these discussions that's the recurring topic. The 600mm is the longest autofocusing (600 X 2) lens they make, which is why most people would choose it rather than the larger, heavier, more expensive lens.

If people were discussing studio, or landscape, or street photography, it's unlikely the 600 would be invoked.


----------



## GMCPhotographics (Jan 25, 2016)

Mt Spokane Photography said:


> Lens tests cannot be compared across models for obvious reasons. I realize that you were doing this tongue in cheek, but its a good example. The measurements vary from camera to camera because they are not measuring a lens. Some people seem to think they are measuring a lens, but that's not the case.



While I generally agree, I do wonder why the same lens on the Canon seems to have a dimmer T stop value than they the same lens it attached to a Nikon. The vignetting is worse too on Canon...hmmm.


----------

