# Inconsistencies in Lens Review Sites



## AlanF (Feb 24, 2017)

The old hands know very well how much individual copies of lenses vary and that lens review sites can be unreliable. The unreliability comes from not doing measurements well or because they usually look at only one copy of a lens or both. The gold standard is Lensrentals who do very careful analyses of a large number of lenses using the best equipment manned by competent users. Nevertheless, people choose lenses based on those unreliable measurements. It's worth bringing this up for discussion, and I'll give two examples, based on lenses that many of us use and know well. 

First collage, the Canon 100-400mm II. The profiles from two sites of good reputation are quite different. Photozone has the lens sharpest wide open from 200 to 400mm. On the other hand, ePhotozine at 400mm has a great dip in sharpness at 400mm at f/5.6 that leaps up at f/8. 

Second collage, Canon 100-400mm II vs Sigma 150-600mm C at 400mm. The profiles from Lenstip has the Canon slightly sharper at 400mm at f/8 than at f/5.6, between the Photozone and ePhotozine results. It has the Sigma performing very poorly on going from 300mm to 450mm. Roger of Lensrentals has compared the average of many copies of each lens at 400mm, and there is very little to choose between them, _on average_, and points out that individual copies of the lens vary to the extent that either one may be better than the other. 

My two copies of the 100-400mm II fit the Photozone pattern, sharpest wide open. And the my Sigma is very close to if not better than the Canons at 400mm.

The conclusion is that the most important comparison is that of your own individual lens and that the sites that just look at one copy of a lens can be taken only as a very rough guide.


----------



## dak723 (Feb 24, 2017)

Very good points! Maybe it's just a generational thing, but it seems to be that people put way too much emphasis on things they read on the internet. Not just lens testing sites, which as you mention often just test one copy of the lens, but any review site. People seem to forget that ANYONE can start a website. You need not have any knowledge or experience or be an expert on your subject. The best way, in my opinion, to find out about a lens or a camera is to buy it and try it out yourself. Make sure you buy from a place that accepts returns. That is the only way to really know.


----------



## ahsanford (Feb 24, 2017)

You have a lot going on here. Good points, fair questions, but consider:

1) A _site_ is not reliable / unreliable categorically. (Other than DXO with lenses. ) A site presents a finding and we need to review it, question it, consider it, and file it alongside others. With one copy comes great responsibility. NEVER render an opinion on one lens tested on one body. TDP gets 1-3 copies (usually 1 or 2) and tests it on a few bodies with some clearly defined methods. LR famously gets 10 lenses because they can -- but they only test IQ -- handling and AF are not reported. PZ is now testing one lens on both 21 MP and 50 MP bodies, which is nice way to consider the future-proofing of the lens for the high MP world. Everyone else (to my knowledge) is writing _War and Peace_ about *one lens on one body*, which is dangerous to bet the farm on. So always always always read multiple reviews from trusted sources and mentally aggregate their findings:


When they agree, draw confidence. When TDP + PZ + LT + Dustin Abbott unanimously scald the 16mm f/2.8 vignetting on the 16-35 f/2.8L III, I consider it 'effectively so'. When everyone and their mother unanimously loses their mind over Sigma 85mm Art resolution, it's a pretty sure bet that it's sharp.


When reviews are mixed, I rely on this forum to ask those that actually bought the product to provide insight.


And when I'm in doubt on a product I am interested in (either conflicting feedback or lack of info that I need to make a buying decision), I always rent before I buy.
2) Getting a bit particular, ePhotozine appears to scale resolution results depending on frame position (and they are not consistent about this). I say this because no lens on the planet is as sharp in the corners as it is in the center yet they often seem to show lenses resolving 90-95% of the value of the center in their 'edge' reporting. I think _scaling_ is slightly misleading (but their call). But in instances where they don't scale and they state the center and edges have nearly the same resolution, I find the results somewhat unbelievable. As a result, I tend to personally 'weight' their findings less than folks with fully transparent methodology.

3) Not all reviewers are testing resolution on a 50 MP canvas. In your very post above, you presented at least 3 different test bodies as far as I could tell. 


DXO: Tests a lens on everything. ;D (I kid, some downsampling/interpolation surely must be going on.)
PZ: Tests on a 5D2 and 5DS R
TDP: Numerous bodies including the 5DS R, but you tend to care less as they aren't putting a number to it. It's more of a comparative IQ tool than a 'woohoo! new high score!' sort of tool.
LT: Still stuck on a 1Ds3 or 5D3 at 21-22 MP. They have not made the painful climb up to 50 MP
LR: OLAF is famously/gloriously sensor-independent in that it just checks the lens. 
ePhotozine: I believe they shoot the 5D4 (and 5D3 with older reviews) 

It goes without saying that a 50 MP test is a more difficult bar to clear. So a 22 MP test on a devastatingly sharp lens like the 85 Art may _trend-wise_ describe what you'll see on a 50 MP rig (i.e. it would be great on both), but a strong copy of an EF 50 f/1.4 USM might look good on a 5D3 and leave you wanting a lot more on 50 MP. Caveat emptor.

- A


----------



## AlanF (Feb 24, 2017)

TDP rarely tests more than one copy of a lens. I checked the first 35 Canon lenses in ascending order on his site, before giving up, and only two had tests on more than one copy. I find the tests on the 5DS R less useful than the previous ones: the lines are so far apart on the charts that you can't see where they merge and can no longer be resolved, you can just see the sharpness of the edge transition and any CA.


----------



## Mt Spokane Photography (Feb 24, 2017)

It is always a good idea to read all the reviews. The better lens testers know what to expect by looking at published MTF charts, and, if a lens is not reasonably similar, they send it in for a calibration, or exchange it for another. Lenses never match the theological MTF charts, but the general shape of the curves, and level of sharpness are a good guide as to what to expect.

Some factors are not published by the manufacturers, and we rely on a testers expertise to discover them. That's where we rely on experience and intuition for them to spot a lens that is not performing properly due to a lens issue and not the design. However, everyone makes mistakes or overlooks something. Some of the testers will take a second look when readers tell them of a inconsistency. 

So, I'm agreeing with you, you can never take a test of a single lens, or even lenses from the same manufacturing batch as being typical. I think that testers have found issues peculiar to a entire batch of lenses.


----------



## neuroanatomist (Feb 24, 2017)

Mt Spokane Photography said:


> Lenses never match the *theological* MTF charts



God tests lenses?

;D


----------



## AlanF (Feb 24, 2017)

neuroanatomist said:


> Mt Spokane Photography said:
> 
> 
> > Lenses never match the *theological* MTF charts
> ...



God = Roger


----------



## neuroanatomist (Feb 24, 2017)

Well, I don't know if He tests them, but He does occasionally loan them out. For example, He loaned His EF 17-40mm f/4L to DxO, which is the only explanation I can think of for why it's better in the wide open corners than the 16-35/2.8 II and as good as the 16-35/2.8 III.


----------



## KeithBreazeal (Feb 24, 2017)

neuroanatomist said:


> Mt Spokane Photography said:
> 
> 
> > Lenses never match the *theological* MTF charts
> ...


LOL We just pray that we get a good copy.


----------



## Mt Spokane Photography (Feb 24, 2017)

neuroanatomist said:


> Mt Spokane Photography said:
> 
> 
> > Lenses never match the *theological* MTF charts
> ...



I need him to check my typing, a spell checker doesn't catch these errors.


----------



## Orangutan (Feb 24, 2017)

Mt Spokane Photography said:


> neuroanatomist said:
> 
> 
> > Mt Spokane Photography said:
> ...



Dew knot trussed yore spelt chequer two fined awl yore mist steaks.


----------



## rfdesigner (Feb 24, 2017)

I think I agree with most of what's said, especially about theologial testing 

I do have a query about Rogers methodology as his results reflect pretty badly on the 100f2.0 which I own and in my experience is very sharp wide open, and the TDP and other results suggest it's sharp too.

The difference?.. I believe his tests are at infinity and my usage is at portrait distance, TDP and others use test charts which are rarely if ever at infinity.

For me the jury's still out, I should take my copy and find a good test at infinity and compare to portrait focus distance, probably a shot of a full moon vs a white test disk held closer to the camera.


----------

