# DxOMark Splits from DXO Labs



## Canon Rumors Guy (Jan 8, 2018)

```
DxOMark has split from DXO Labs and the latter will now be focused on software development, most notably the recently acquired Nik Software suite, which they say <a href="http://www.canonrumors.com/great-news-dxo-to-release-new-nik-collection-software-in-2018/">will be re-released in 2018</a>.</p>
<p>DxOMark will continue its camera and lens testing and will reportedly become “<a href="http://image-sensors-world.blogspot.com/2018/01/dxomark-separates-from-dxo-labs.html">more objective and less affected by DXO Lab products</a>” according to <a href="http://image-sensors-world.blogspot.com/2018/01/dxomark-separates-from-dxo-labs.html">Image Senors World</a>.</p>


<p>DxOMark does have some credibility issues in certain circles when it comes to testing, so hopefully this will be a good thing and will start changing public perception. We all love objective reviews and those are harder and harder to come by.</p>
<p><strong>The email from DXOMark</strong></p>
<blockquote><p>We’ve had an important internal change as well: In September, DxOMark Image Labs was spun off from DxO Labs. DxOMark Image Labs is now a privately-owned, independent company. As such, we continue to pursue the development and commercialization of image quality solutions and services that support our customers in designing the best-quality camera systems for a range of markets, including smartphones, DSC/DSLRs, drones, action cams, surveillance, and automotive.</p></blockquote>
<p>My head’s spinning a bit, too many “DxOs”, so hopefully I understand this correctly.</p>
<span id="pty_trigger"></span>
```


----------



## ahsanford (Jan 8, 2018)

If they...

1) Stop scoring their own product's multi-shot mode and comparing it against competitors with similar multi-shot feature but only report their single shot performance,

2) Abolish their overall score or publish the overall score algorithm in full, so the arbitrary and inconsistent rankings cease,

3) Stop taking the exact same lens from Sigma or Zeiss and declaring it disappointing on Canon and mind-blowing on Nikon, 

4) Have f/22 performance affect a lens's sharpness rating (see 35L II vs. 35 Art for giggles)

...they have a fighting chance of my taking them seriously.

- A


----------



## neuroanatomist (Jan 8, 2018)

5) Stop heavily weighting base ISO in the sensor scoring. Based on one assessment comprising analysis of EXIF data for ~20 million images, ISO 100 is used for ~30% of pictures.


----------



## 3kramd5 (Jan 8, 2018)

neuroanatomist said:


> 5) Stop heavily weighting base ISO in the sensor scoring. Based on one assessment comprising analysis of EXIF data for ~20 million images, ISO 100 is used for ~30% of pictures.



6. Stop considering camera dynamic range for lens scoring.

7. Start using appropriate (realistic) subject distance in lens testing; putting an ultra wide angle lens 5 feet from a chart is hardly how it will be used, and certainly isn’t what it was designed for.


----------



## neuroanatomist (Jan 8, 2018)

ahsanford said:


> 3) Stop taking the exact same lens from Sigma or Zeiss and declaring it disappointing on Canon and mind-blowing on Nikon





3kramd5 said:


> 6. Stop considering camera dynamic range for lens scoring.



More generally, base lens scores on the optical measurements of the lens itself and not on the combination of camera+lens.


----------



## neuroanatomist (Jan 8, 2018)

Canon Rumors said:


> My head’s spinning a bit, too many “DxOs”, so hopefully I understand this correctly.



DxO Labs = DxO PhotoLab (RAW converter), DxO FilmPack, DxO ViewPoint, the Nik suite, and the DxO One camera.

DxOMark Image Labs = DxO Analyzer (B2B industry sensor testing software) and the camera score database (which is basically DxO Analyzer data from production cameras repackaged for public release)


----------



## ahsanford (Jan 8, 2018)

neuroanatomist said:


> 5) Stop heavily weighting base ISO in the sensor scoring. Based on one assessment comprising analysis of EXIF data for ~20 million images, ISO 100 is used for ~30% of pictures.



Killing off the overall score or breaking Dynamic Range into two (or three) scores...

Landscape, Studio / "Controlled"

General use / "Field"

Low-Light / "Challenging"

...would address this. Seeing as the overall score is the metacritic/RT reveal for the internet masses to frenzy over -- and it must drive page hits -- I just don't think the former will happen.

Also, they could do this ISO dependent output with the noise and color depth metrics as well.

- A


----------



## 3kramd5 (Jan 8, 2018)

neuroanatomist said:


> ahsanford said:
> 
> 
> > 3) Stop taking the exact same lens from Sigma or Zeiss and declaring it disappointing on Canon and mind-blowing on Nikon
> ...



There is validity to the pairing. Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars). 

A transfer function doesn't tell the whole story. For example when I was still using an A7Rii I often adapted Canon lenses. However, Canon lenses aren't designed for the unique filter stack in front of the Sony camera sensor, and perform better on the native platform than adapted, a fact which would not turn up on an optical bench. 

As a tool to help determine optical performance of lenses you're interested in on bodies you're interested in, their numbers are useful. It would be better if they stopped scoring lenses with different mounts against each other. Best would be to abandon the composite scoring scheme, but that is probably their biggest web traffic generation mechanism.


----------



## ahsanford (Jan 8, 2018)

3kramd5 said:


> Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).



_Obi Rog Kenobi at LR pets his beloved OLAF... and feels a powerful disturbance in the force._

- A


----------



## neuroanatomist (Jan 8, 2018)

3kramd5 said:


> There is validity to the pairing. Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).



Except that DxOMark _does_ report sensor scores independent of lenses (or rather, without regard to lens). But their DxOMark Lens Score is camera-dependent. Moreover, they draw conclusions such as 'this brand's 24-70/2.8 zoom is better than that brand's equivalent' based on the sensor contribution, when the lens' optics alone would lead to the opposite conclusion. 

Then there's:

8. Stop assuming that we all use our cameras and lenses in a dimly lit warehouse. 

Lens scores are based on 'performance in lighting of 150 lux' which is why, according to the DXOMark lens score, the 50/1.8 is a better optic than the 300/2.8 or 600/4.


----------



## 3kramd5 (Jan 8, 2018)

ahsanford said:


> 3kramd5 said:
> 
> 
> > Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).
> ...



Right. There he is testing it, but not using it. I betcha when Roger uses his camera lenses, they’re mounted to cameras 

Granted you have to have to use it to test it, but the distinction is clear enough.


----------



## ahsanford (Jan 8, 2018)

neuroanatomist said:


> 8. Stop assuming that we all use our cameras and lenses in a dimly lit warehouse.
> 
> Lens scores are based on 'performance in lighting of 150 lux' which is why, according to the DXOMark lens score, the 50/1.8 is a better optic than the 300/2.8 or 600/4.



Also:

9. When the company you constantly batter for too having few pixels behind the lens finally puts out a sensor that outresolves your preferred gear maker, stop rewriting your unpublished overall lens score to still find your preferred gear maker makes better stuff. You can't scream 'but the resolution on this D800E is amazing' for a couple years and then start scream 'no no, hear me out: _ transmission is the new resolution_, yo' the minute the 5DS R is on the scene. :

- A


----------



## 3kramd5 (Jan 8, 2018)

neuroanatomist said:


> 3kramd5 said:
> 
> 
> > There is validity to the pairing. Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).
> ...



And changing both of those things would improve their credibility.


----------



## candc (Jan 8, 2018)

Their measurements can be useful but the conglomerate score they assign can be misleading for sure. 

I use dxo software and I really like the new local adjustment tools in "photolab"


----------



## neuroanatomist (Jan 8, 2018)

3kramd5 said:


> And changing both of those things would improve their credibility.



I agree, although to be honest their credibility with photographers is probably rather irrelevant given that their primary source of income is B2B.


----------



## Valvebounce (Jan 9, 2018)

Hi Folks.
I’m not certain, but I don’t think I have ever used DxO lens or camera scores, but I do use DxO Optics Pro and Photolab, my concern is whether the software side will still have access to the excellent lens correction data? 
This question is based on the assumption that the data for the lens corrections comes from the lens score tests, if this is wrong I have just reaffirmed the “to ass-u-me” saying! ;D

Cheers, Graham.


----------



## RogerCicala (Jan 9, 2018)

3kramd5 said:


> ahsanford said:
> 
> 
> > 3kramd5 said:
> ...



Science, as I was trained to do it, is about eliminating as many variables as possible to test one variable in a reasonably large sample size. Hence, we test just the lenses and a reasonably large sample size of the lenses. 

Carefully measuring one third grader and then stating that all third graders are 67.325 secret measurements tall is bad science.


----------



## 3kramd5 (Jan 9, 2018)

RogerCicala said:


> 3kramd5 said:
> 
> 
> > ahsanford said:
> ...



Agreed, however

1) some of those variables are pertinent and may be unwise to eliminate (for example if an 85mm lens from brand A has fewer losses than an 85mm lens from brand B, it matters little to someone who can not use brand B lenses. Similarly, if you want an 85mm lens, focal length is a variable which should not be eliminated, whereas if you’re after the most perfect optics and will work around focal length, it can be), and
2) not all scientific data is readily usable. Your work is great and very interesting; I read your blogs including bench level lens testing. However I’ve yet to come up with a way to use an MTF chart.

In this particular instance, the variable I question eliminating is the camera sensor's optical stack.

I think we can agree that

A) the MTF of an optical system is a function of the MTF of the constituent members. That is true even when the optical system is a camera lens: each element has a transfer function, which contribute to the lens transfer function as a whole. Some elements are included to correct the affects of others. 

B) since the advent of digital ILCs, lenses have been designed for the cameras to which they'll be mounted, ergo the sensor and its optical stack (as opposed to lenses designed for film cameras). 

Some third party lenses may be sensor agnostic, but they probably should not be. Interviews with both Sigma and Laowa/Venus refer to the importance of designing for the optical stack. It's part of the path, indeed it's the final element before the sensor. Putting a lens on an optical bench without that stack to eliminate the variable eliminates the proper environment for the lens, similar to (though less impactful than) pulling the rear element off a camera lens.

Occam's razor and all, maybe I'm just dense. However, it seems to me that by isolating the variables to [camera lens], one introduces the risk that the remaining variable is being used inappropriately.


----------



## ahsanford (Jan 9, 2018)

RogerCicala said:


> 3kramd5 said:
> 
> 
> > ahsanford said:
> ...



I'm sorry, let's just take a moment and reflect that I just summoned Uncle Rog with Bat-symbol like post. Kind of proud of myself at the moment. 

Thanks, Uncle Rog. We love what you do, especially the bit about transparency of methods, not coming up with some BS unpublished aggregate score and (most of all) being a wonderful resource with a name that we can ask questions to. Please travel to France (or wherever Le DXO Scoremaster Generale is located in his/her secret bunker) and straighten them out.

- A


----------



## RogerCicala (Jan 9, 2018)

3kramd5 said:


> RogerCicala said:
> 
> 
> > 3kramd5 said:
> ...



You do realize that it was our work on sensor stack thickness a few years ago that started all the public discussion on stack thickness, and that we place the appropriate sensor stack in the optical path for each lens tested? We do this for the tests you read, and for the ones you don't read that we do for various manufacturers (who, BTW, always approve the stack thickness we're testing their lenses at).


----------



## 3kramd5 (Jan 9, 2018)

RogerCicala said:


> 3kramd5 said:
> 
> 
> > RogerCicala said:
> ...



Clearly I did not. I did read in a blog (I believe) about optical glass being added for a given test but did not know it is done as a matter of course.

Kudos. My above post is redundant.


----------



## jeffa4444 (Jan 9, 2018)

3kramd5 said:


> neuroanatomist said:
> 
> 
> > ahsanford said:
> ...


What DXO doesn't do and If I'm reading your post correctly you don't seem to understand is all reputable companies would a. test the camera sensor excluding the lens b. test the lens (can be two part of three part i.e. test on MTF, test on projector, test on f stop / t stop machine) c. test the lens on the camera its to be used with. 
The lens tells you nothing about whether the camera dynamic range is accurate or not or a host of other electronic measurements about its performance. Likewise all the top lens manufactures Ive had the pleasure to visit or indeed the company I work for all primarily want to see how a lens projects as much as any MTF reading its far more simple than shooting on the camera to see vingetting at different stops or in the case of zooms at different focal lengths. Its also easier to see colour fringing & aberrations. The MTF will also tell you with multiple off-centre readings if an element is out of alignment.


----------



## 3kramd5 (Jan 9, 2018)

Hi Jeff, I don’t think you read my post correctly. Of course OEM’s independently test components. It’s at the end used level where I’m questioning the value of isolated data.


----------



## Bdube (Jan 10, 2018)

3kramd5 said:


> RogerCicala said:
> 
> 
> > 3kramd5 said:
> ...



Ah, this is a great misunderstanding. Roger has already handled the whole sensor stack thing.

However, it is certainly untrue that the individual elements have transfer functions, that, when multiplied, produce the system transfer function. This is an area where optics departs from ECE, or general LSI theory.

Imagine you have a 1 inch diameter, 2 inch focal length lens which is diffraction limited. This gives you an MTF described by an arc cosine formula you are free to to find online. Consider that you get another, and place it in contact with the previous one. You now have a 1 inch diameter, 1 inch focal length doublet. Since each piece was diffraction limited, the result is also diffraction limited and will have diffraction limited f/1 performance. Not the square of diffraction limited f/2 performance, which would be quite substantially worse and incorrect.

FWIW, the UVIR cut filter and OLPF are under the coverglass. It is not the last thing before the sensor.

I recommend reading Intro to Fourier Optics by Goodman, or Principles of Optics by Born & Wolf, before writing about transfer functions and LSI theory in optics.


----------

