# IQ comparison; or how meaningful is DXO



## aj1575 (Sep 2, 2013)

Hello, I made a table with different samples from different cameras at different settings. I took the pictures from dpReview, and arranged them in 6 rows with 4 pictures each. The picture in each row are taken with the same setting (ISO, Raw or JPG). The pictures are mainly from Canon or Nikon cameras, but others are also included.

Feel free to rank the pictures in each row, or to just point a interesting things. I will post the picture with the settings and the cameras later.

_ You should download the picture to see it at 100% _

The goal is not to spot the 70D. The idea is to rank the picture according to your impression, and then see later from what cameras they were.


----------



## ajfotofilmagem (Sep 2, 2013)

Ok, I'll kick: 
3A is the Canon 7D
3B is the Nikon D7100
3C is the Sony A77
3D is the Canon 70D
Did I kicked too far? ???


----------



## ajfotofilmagem (Sep 2, 2013)

Someone else will guess?


----------



## JohnUSA (Sep 3, 2013)

Here's my picks:

1. D
2. D
3. D
4. A
5. A
6. A


----------



## aj1575 (Sep 3, 2013)

ajfotofilmagem said:


> Ok, I'll kick:
> 3A is the Canon 7D
> 3B is the Nikon D7100
> 3C is the Sony A77
> ...



Mostly Wrong. Try to sort the rows from best to worst, I will post the pic the names on it soon.


----------



## aj1575 (Sep 3, 2013)

I will do a ranking.

1. DCBA
2. DACB
3. DBAC
4. BDCA
5. DCAB
6. DABC


----------



## Kristofgss (Sep 3, 2013)

Do you mean sort per row or sort once to pick the most pleasing one of the four cameras?


----------



## luigig75 (Sep 3, 2013)

1) DCBA
2) DCAB
3) DBAC
4) DBCA
5) ABCD
6) DABC


----------



## aj1575 (Sep 3, 2013)

Kristofgss said:


> Do you mean sort per row or sort once to pick the most pleasing one of the four cameras?



You can also only pick the best and the worst. What I like to do is a blind test; just look at the pictures, without knowing what camera it is, and what the settings were.

It is very interesting. I made the table, but I can remember the position of only a few cameras, so there is nothing else to do than just look carefully at the pitures, and rate them by what I see. 

I think I will do a table with only the DXO mark attached to it, this would also be interesting.


----------



## Cannon Man (Sep 3, 2013)

I spot that they are all bad.. None of them a 1DX


----------



## Kristofgss (Sep 3, 2013)

1. acbd
2. cabd
3. dbac
4. bcda
5. dcab
6. adbc


----------



## Woody (Sep 3, 2013)

My ranking:

1 DCBA
2 DACB
3 DBAC
4 BDCA
5 BACD
6 DABC


----------



## aj1575 (Sep 3, 2013)

I like to see some more ratings, so that it makes sense to put the results into some nice graphs.


----------



## rpt (Sep 3, 2013)

1. DCAB
2. ADCB
3. DABC
4. BDAC
5. ACBD
6. ADBC

There you go...


----------



## ajfotofilmagem (Sep 3, 2013)

I'll try again. From best to worst ...
1- D- C- A- B-
2- D- A- B- C-
3- D- B- A- C-
4- B- D- A- C-
5- C- A- B- D-
6- D- A- B- C-


----------



## Mt Spokane Photography (Sep 3, 2013)

DXO measures the sensor characteristics. Those are hard facts. DXO does not test cameras.

For those that understand this, the information can be useful in selecting a camera that has a sensor which meets your special needs.

Unless its a extreme application, the photographer is 90%, the lens is 9.99 % and the body makes little difference to IQ. 

Since DXO does not test bodies for other things that may matter to photographers like metering accuracy, their attempting to put a number to a camera body based on sensor testing is of little value to me, but the raw data is valuable. 

However, if you are one who needs a particular sensor characteristic, understanding the strengths and weaknesses of your sensor might be important. So far, I have found lenses to be so much more critical, that the sensor makes little or no difference, unless its defective.


----------



## Joynt Inspirations (Sep 3, 2013)

I'm game.

1. DCBA
2. ADCB
3. DBAC
4. BDCA
5. ACBD
6. DBAC

Fool me once …


----------



## hutjeflut (Sep 3, 2013)

its just not possible to siply rank them as each row has different pixtures that have different qualitys better like colour sharpness and noise are not always equally good on each shot.

1. D simply beats all the others both in detail and sharpness but on it seems somewhat lower on the exposure side as its a little darker resulting in lower saturation.

2. D and A seem the have about the same iso preformance but A seems to be slightly out of focus and a little flat colour wise while C seems to be best colour wise but had poor details and noise preformance.

3. D is the best on everything here

4. B seems to have the best noise preformance but seeing all the details of the blocks has fallen away this immage on full size will look flat and lack detail so your mostlikely better off with D which had the second best noise preformance but maintains detail.

5. D has the best noise preformance but lacks detail and saturation and contrast making me think C would give the best end result as its luminous noise mostly and has plenty detail and saturation. A has more detail but far worse noise preformance making you lose detail on noise reduction.

6. B has the best colour reproduction and a nice even move from dark to light blue while A as the most detail/sharpness/contrast and D has the best noise preformance.

so depending on what you look for different photos might fit your need best.


----------



## aj1575 (Sep 4, 2013)

hutjeflut said:


> its just not possible to siply rank them as each row has different pixtures that have different qualitys better like colour sharpness and noise are not always equally good on each shot.
> 
> ....
> 
> so depending on what you look for different photos might fit your need best.



Thanks, a very nice analysis.


----------



## aj1575 (Sep 4, 2013)

Here is the comparison with the camera models and the settings of the pictures taken.

I leave it to you to judge your results; I will make some kind of statistics later.


----------



## ajfotofilmagem (Sep 4, 2013)

aj1575 said:


> Here is the comparison with the camera models and the settings of the pictures taken.
> 
> I leave it to you to judge your results; I will make some kind of statistics later.


Well, I guessed right the line 3-D image was even canon 70D.  Overall, my preferences this blind test, was also Canon 6D and 70D. By coincidence, are these cameras I would buy right now.


----------



## ajfotofilmagem (Sep 4, 2013)

It did not seem very fair to compare images of APS-C vs full frame. But in the end, the results were what one might expect. No big surprise. Just a little disappointed with the sharpness of the D7100, which having no AA filter, I expected to be sharper than other APS-C. :


----------



## privatebydesign (Sep 5, 2013)

I came out with this, I didn't cheat for what that is worth.

D,C,B,A
A,D,C,B
B,C,D,A
B,C,D,A
C,D,A,B
D,A,B,C

So the 70D didn't fair too well for me, but the 6D did. However small crops of jpegs that haven't been normalised really don't illustrate much.


----------



## Pi (Sep 5, 2013)

What does this have to do with DXO?


----------



## ajfotofilmagem (Sep 5, 2013)

Pi said:


> What does this have to do with DXO?


In this blind test, each person chose the images that seemed best. Now we know what we were looking cameras, can compare with the DXO ranking. In my case, the images that I found most enjoyable were 70D and 6D. Two models with DXO scores worse than the competitors. For me, no surprise. The criteria DXO does not show which camera is the ideal image to my eyes.


----------



## privatebydesign (Sep 5, 2013)

It depends on what you think the images illustrate.

I don't consider them relevant to my buying decisions for two key reasons:
[list type=decimal]
[*]They are not normalised, that is, they are different sizes at 100%, this is not how I use images, I want a comparison of a set reproduction size, if I am making a 12" x 18" print I don't care how many pixels I have, I just need that size.
[*]I don't have any use for jpegs, indeed I practically don't use them at all, I even send 16 bit data to my printers now. Jpeg output is irrelevant to me, I need to know what I can get out of the RAW file and how much time it takes to get it.
[/list]


----------



## Pi (Sep 5, 2013)

ajfotofilmagem said:


> Pi said:
> 
> 
> > What does this have to do with DXO?
> ...



Some of the crops are OOF, you have different NR applied to them, etc. Your title is too provocative for what you are doing. You do not need to take cheap shots at DXO.


----------



## CR00 (Sep 5, 2013)

I think he mentioned at the beginning of the post that he took the pictures from dpReview, and arranged them in 6 rows with 4 pictures each. I think you can try it at http://www.dpreview.com/reviews/nikon-d7100/20 and see it for yourself.


----------



## dtaylor (Sep 5, 2013)

Mt Spokane Photography said:


> DXO measures the sensor characteristics. Those are hard facts.



No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.


----------



## qwerty (Sep 5, 2013)

dtaylor said:


> Mt Spokane Photography said:
> 
> 
> > DXO measures the sensor characteristics. Those are hard facts.
> ...



I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses). The disagreements are to 1) whether the reported scores (scores, as opposed to test results) are fair, useful, meaningful or what-have-you 2) whether the differences matter for a given user and 3) the fact the DxO only measures sensor performance, not camera performance (and does not claim to do anything different).


----------



## ajfotofilmagem (Sep 5, 2013)

qwerty said:


> dtaylor said:
> 
> 
> > Mt Spokane Photography said:
> ...


That is why it is said: "Statistics is the prostitute of mathematics". ??? And also: "Statistics is a form of lying, using numbers". :-X What is the use of a collection of correct data, if the end result will be totally subjective score, and mysterious criteria? :-\


----------



## Pi (Sep 5, 2013)

ajfotofilmagem said:


> qwerty said:
> 
> 
> > dtaylor said:
> ...



This is left to the intelligence of the reader. The "end result" has nothing to do with statistics, it is some kind of cumulative score for readers who are too busy to try to understand the data. The data is there for everybody who cares; the score is not data and every intelligent user would ignore it.


----------



## dtaylor (Sep 6, 2013)

qwerty said:


> I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).



Yes there is. The biggest debate I've seen in other forums is over their DR scores which do not match the results from other testers (dpreview; IR) and do not seem to match real world experience. IMHO a simple Stouffer transmission step wedge test is far more accurate and reliable then DxO's methodology.

DxO also "interprets" DR based on output/viewing size, which is absurd to me.


----------



## Robert Welch (Sep 6, 2013)

When it comes to applying the DXO ISO ratings to actual usage, I once heard someone suggest to round their ISO rating up to the next standard ISO setting (i.e. 1000 to 1600, 2000 to 3200), then double that. The result will generally be a reasonable high ISO to use the camera on with decent results for many applications.

In my experience, this translation has applied pretty well for the cameras I've owned, to some degree. My 40D & 7D are DXO rated about 700, and I generally don't like to use them much above 1600 if I can help it (the 7D does a little better if pushed higher, but some of that may be because of the higher resolution, meaning enlargement isn't as high). My 1DmkIII is rated about 1000, and while I feel at 3200 it's being pushed a bit, it is usable there, but I try to keep it no more than 2000-2500 if I can. The 6D & 5Dmk3 are both rated just above 2000, and both those cameras work very well at 6400. So, in broad strokes, this method seems to have some validity.


----------



## qwerty (Sep 6, 2013)

dtaylor said:


> qwerty said:
> 
> 
> > I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).
> ...



Sorry, too long of a day for a fully coherent post, but:

If I am not mistaken, DPreview reports the dynamic range of processed jpegs, not of the raw files themselves. Basing the measurement on processed files means that the DR reported depends on the processing applied during conversion (which is why they have several different dynamic ranges reported for each camera at each ISO). (Also, the files they post for "raw" comparisons are obviously not really the raw files; I believe they are ACR conversions to jpeg from the raw files). 

DxO's real business is making raw converters; they analyze the sensor outputs to optimizer their converters (which is why they look at the actual raw data, not processed images). They would tell you that the DxOmarks are just a side effect of their core business...

With regard to DxO's normalization, I think it would be absurd to compare non-normalized results. Normalization tells you how things will perform for a fixed print size (which is what most of us care about, as opposed to per-pixel values).

I checked their math a few years ago and they do it right (see http://www.dxomark.com/index.php/Reviews/Detailed-computation-of-DxOMark-Sensor-normalization  and check for yourself if you do not believe me).

Comparing non-resolution-normalized results for cameras with different sizes is akin of comparing prints of different sizes. It would be like comparing a 4x5 print from a 4 MP 1D with 8x10 from a 18 MP 1DX (from the same viewing distance)...


... ... ...
But, all that aside, if you just click the "screen" button when viewing the DxO results, they will happily give you the non-normalized values you want.


----------



## jebrady03 (Sep 6, 2013)

I shared the first image on facebook and tagged a bunch of my friends and asked them to pick their favorites. I haven't posted the results yet but so far, the preference for Canon is OVERWHELMING. The greatest part about it is that the majority of the people I tagged are DIEHARD Nikon shooters who trash Canon every chance they get, and they're picking Canon left and right. Only one guy split Canon and Nikon down the middle at 3 and 3 (and he's one of the more vocal Canon-haters) and everyone else is at least 2-4 with some being 1-4 (Nikon:Canon).
Crow... it's what's for dinner!


----------



## Pi (Sep 6, 2013)

jebrady03 said:


> I shared the first image on facebook and tagged a bunch of my friends and asked them to pick their favorites. I haven't posted the results yet but so far, the preference for Canon is OVERWHELMING. The greatest part about it is that the majority of the people I tagged are DIEHARD Nikon shooters who trash Canon every chance they get, and they're picking Canon left and right. Only one guy split Canon and Nikon down the middle at 3 and 3 (and he's one of the more vocal Canon-haters) and everyone else is at least 2-4 with some being 1-4 (Nikon:Canon).
> Crow... it's what's for dinner!



This resolves a long standing question. Canon is better!


----------



## jebrady03 (Sep 6, 2013)

Pi said:


> This resolves a long standing question. Canon is better!



AGREE WHOLEHEARTEDLY! lol 

But seriously, I've always said, and will continue to say, that the photographer > lenses > camera and if a person is iffy about which camera to buy, hold them and play with them. Very often, ergonomics will decide. After all, nobody wants to lug around a camera they don't enjoy using.

...and that's how I'm going to let them down easy  lol

Any of the cameras in that test are GREAT choices and anyone who owns them should be VERY satisfied with their choice.


----------



## ajfotofilmagem (Sep 6, 2013)

Pi said:


> jebrady03 said:
> 
> 
> > I shared the first image on facebook and tagged a bunch of my friends and asked them to pick their favorites. I haven't posted the results yet but so far, the preference for Canon is OVERWHELMING. The greatest part about it is that the majority of the people I tagged are DIEHARD Nikon shooters who trash Canon every chance they get, and they're picking Canon left and right. Only one guy split Canon and Nikon down the middle at 3 and 3 (and he's one of the more vocal Canon-haters) and everyone else is at least 2-4 with some being 1-4 (Nikon:Canon).
> ...


The question is not as simple as: "Canon is the best!" The real question is: Pictures of Canon cameras are more pleasing to the eye when made and processed on equal terms, for most uses in the real world.  However, there are exceptions. ??? If I need to take pictures in RAW only, exclusively at ISO 100, with only prime lenses stopped down 3 stops from maximum aperture diaphragm, only objects with great brightness variation, certainly has the best Nikon camera for that. : For all other types of photo, the Canon system seems more advantageous, with images closer than I think it should look like. Obviously some people disagree, but the sun rises for all, and would not benefit a monopoly Canon, or any other company. 8)


----------



## Woody (Sep 6, 2013)

After comparing my image choices vs actual cameras used, this is what I notice:
i) FF is much better than cropped cameras
ii) Canon jpeg engine rules


----------



## Apop (Sep 6, 2013)

Pi said:


> ajfotofilmagem said:
> 
> 
> > qwerty said:
> ...



yes ,but
The scores/numbers should at least represent the data ?
Regardless of ones intelligence , some people are not interested or don't have the time to interpret the data themselves. I think DXO should deserve credit for conducting all these tests, if you agree or disagree with their scores/data shouldn't matter. They are putting in a lot of time and effort and if you do disagree , set up an experiment and try to disprove them rather than bashing them because your camera doesn't score the highest( this is just a general opinion of me, not aimed at you at all pi)

Still I think the score is derived from data and should not necessarily be ignored, since the score should represent (some of) the data.

For example; does anyone know where the image quality of a sensor is based on @ dxo?
95 vs 82 image quality, sensor 1 has around 20% better image quality.
To be honest I never value that score, because I have no clue where it is based on
(Resolution? DR? Noise? , no clue).

In that aspect I agree that scores should be ignored , they might have made up some formula with several variables and come up with such a score. again I have no clue how they get the number.

On the other hand , thins like
2.853 ISO or 2.340 ISO don't have to be ignored, If it's shot RAW and was a good test ( so repeated by others with the same result), that is enough information for me.


----------



## aj1575 (Sep 6, 2013)

What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.

I also found the test interessting, because it even worked for myself, since I forgot most positions of the cameras, and also did a blind test (and I judged the pictures differently then when I knew from what camera they were).

My conclusion.
-The DXOmark score difference between the 70D and the D7100 is definitly not justified.
-The Fujifilm x-pro1 makes some nice pictures.
-The Sony a99 is a bit dissapionting, the D7100 and the 70D produce pictures that are about on the same level.
-FF is better, but not but the difference is not as big as I thought (the 70D was often rated higher than the D600 at JPEG).

I really tried to make a fair test; I took samples from colorcards to show noise performance at low ISO, I took parts with high contrast and some with details. So I think the comparison is quite fair. If it is meaningful to you, I don't know, this is up to you.


----------



## Apop (Sep 6, 2013)

aj1575 said:


> What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.
> 
> I also found the test interessting, because it even worked for myself, since I forgot most positions of the cameras, and also did a blind test (and I judged the pictures differently then when I knew from what camera they were).
> 
> ...





It's a nice thing you showed, but does DP review even use the same lenses for the cameras?( for example all sigma 35 1.4?, same apertures?, don't think so).
(One thing i noticed they have trouble framing every camera in exactly the same way , and there are also quite some focus differences , making it a bit hard to compare.)

But in general I guess it's safe to say they are close.(70d/d7100).

I think to say that DXO is complete nonsense isn't fair.


----------



## AmbientLight (Sep 6, 2013)

Apop said:


> I think to say that DXO is complete nonsense isn't fair.



I disagree on this point. How can criticism be unfair, if DXO's ratings can at best be said to provide a distorted view?

We have to assume that DXO has some form of interest of misinterpretations. Otherwise we must decry them as unprofessional, which I assume they are not, so there must be some sort of intend behind those misleading ratings. To claim that all is good based on underlying correct measurements is not sufficient nor is it good enough. For example half-truths may not be outright lies, but they are not to be trusted either. If we must strain ourselves to properly interpret DXOs findings then there are at least some obvious shortcomings. In light of DXO rating's discrepancies with what others call reality or if this is not a good enough description in light of their rather mysterious way of creating a rating, calling their ratings complete nonsense is a sufficient first approximation of what they do.

If this were not so, you would have to make a claim that their ratings are useful as they are.


----------



## Pi (Sep 6, 2013)

aj1575 said:


> What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.
> 
> I also found the test interessting, because it even worked for myself, since I forgot most positions of the cameras, and also did a blind test (and I judged the pictures differently then when I knew from what camera they were).
> 
> ...



You do not really think that posting selected OOF crops of JPEGs with different noise reduction proves anything, do you?

Tell me which brand you want to see a winner, and I will post similar crops from IR proving that that brand is the best.


----------



## Apop (Sep 6, 2013)

Pi said:


> aj1575 said:
> 
> 
> > What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.
> ...



Nokia ?

If people want to proof that DXO is meanigless they need to do a bit better than posting samples like this.
See if you can replicate their results , and hope that they were wrong , then people try replicate yours and hope you were right !


----------



## aj1575 (Sep 6, 2013)

Apop said:


> I think to say that DXO is complete nonsense isn't fair.



I did not say that DXO is complete nonsense, I said that the *DXOmark score* is nonesense.


----------



## aj1575 (Sep 6, 2013)

Pi said:


> You do not really think that posting selected OOF crops of JPEGs with different noise reduction proves anything, do you?
> 
> Tell me which brand you want to see a winner, and I will post similar crops from IR proving that that brand is the best.



Two things here, dp uses different lenses. For Canon APS-C they use a EF 50mm f1.4 for Nikon the 50mm f1.4 AF-S and for the FF they use the 85mm f1.8 from each brand. They shot at f8. So the differencr in quality should be rather small (check the tests of these lenses at various sites, also DXOmark)

The OOF claim is understandable. I was thinking the same when I looked at the samples. The D600 looks worse in the poker card than the 70D. But I do not think that this is an OOF problem, for several reasons.
1. The 7100D and other Nikons show this problem, so either they focus many Nikon cameras wrong, but not the Canons; or the Nikon AF is not accurate enough...
2. This softnes does not appear on the whole picture, you just need to move down on dp-site tool a little bit to the black and white circle. This seems to be on the same plane, but it looks sharp enough to me.

So to me, this is not an OOF problem, the Nikon sensors just have a problem to resolve that properly.


----------



## Pi (Sep 6, 2013)

aj1575 said:


> Pi said:
> 
> 
> > You do not really think that posting selected OOF crops of JPEGs with different noise reduction proves anything, do you?
> ...



How about DOF and focus plane differences?



> So to me, this is not an OOF problem, the Nikon sensors just have a problem to resolve that properly.



Tell me that you are kidding.

http://www.the-digital-picture.com/Reviews/ISO-12233-Sample-Crops.aspx?Lens=640&Camera=614&Sample=0&FLI=0&API=2&LensComp=287&CameraComp=453&SampleComp=0&FLIComp=0&APIComp=2

D7100:


----------



## Mt Spokane Photography (Sep 6, 2013)

dtaylor said:


> Mt Spokane Photography said:
> 
> 
> > DXO measures the sensor characteristics. Those are hard facts.
> ...


 
As someone who ran a lab for NASA, I think the test data represents hard facts. I've been thru hell repeating and defending my tests by experts who are skeptics, so I can recognize when someone is incompetent. I also think they are accurate results for the samples they had. someone can always find a unit that performs differently. You should be able to create the same setup and compare results. 

Now, interpretation of the data is where people tend to disagree, and assigning a value number to a sensor based on a secret formula is silly, particularly when you link it to a camera model and give the camera a score. There is a lot more to a camera than the sensor, and DXO does not test that.


----------



## qwerty (Sep 6, 2013)

Just to try to drag the thread back on-topic, here are my favorites in each row:
1d
2a
3d
4b
5c (d had less noise, but seemed oversmoothed)
6d (d had less noise and seemed "sharp enough", but I get the feeling that if there were more detail it would have looked oversmoothed)

I want to thank the OP for making this comparison, and would be curious to see more of the same. I know a few issues have been pointed out in the thread, and a second pass might be even better.

If I had the time, what I would do is download the raw files of a standard scene from somewhere (I thought dpreview posted them somewhere, but either I am misremembering, bad at searching, or they took them down when they added their scene comparison widget) and convert them yourself using lightroom with the same settings, except scaled so that the image sizes are the same for the different cameras. You can even keep them as tiffs until final conversion to keep everyone (less un-)happy.

Oh, and how about a less inflammatory title for the next thread? : o )


----------

