My Setup for Comparing Medium Format Lenses (80mm, 150mm...) 
Note: Random number #121  in center for on-film Blind Identification

Blind Lens Testing Project
by Robert Monaghan

Related Local Links:
Blind Lens Testing Proposal
Blind Lens Testing Program Results
How to Test Cameras and Lenses
Lens Testing Chart Download
Lens Testing Pages
Lens Testing Resolution Charts

Introduction

Why should you participate in this blind lens testing project?  The benefits and fun to you include:

What? You are afraid of looking bad?  Don't worry.  We will use an anonymous code identifier in reporting results. So you won't be embarrassed if your results don't turn out quite as you hoped they would. Besides, you have got to do better than me. I have already admitted I can not reliably tell which camera or lens took which slide. So what have you got to lose, and look at all you have to gain by participating! Go for it!!!

Procedure to Participate

Start by sending me an email with your mailing address and stating you want to participate in our blind medium format normal lens testing project. [Coming up next (if this project is successful) are wide angle lenses, battle of the ultrawides, and 150mm-ish portrait lenses and 200 to 250mm-ish telephotos]. 

You will be mailed a set of slide shots for direct comparison. On your answer sheet, you will be asked to rank each slide, best to worst, using the visible random number that identifies each slide. Look at the photo at the top of this page to see the test scene showing a sample random number and camera test setup. The test shot includes a lot of fine detail (brickwork in building etc.) and a fountain spraying water in the foreground (for bokeh fans).   

We will also ask some information on your experience level in medium format and general photography, so we can get some idea of how people's experience impacts results. I would expect that folks who are more experienced might be better able to tell different lenses and cameras.  So this data will help us test that idea too. We will also ask what kind of loupe(s) or viewing setup you were using (e.g., maybe the differences are not clear at 4X, but obvious at 15X or 22X?). 

You can email your ratings and answers to us (at [email protected])  if you wish.  We will shortly reply with an "answer key" corresponding to your numbered slide set which will tell you which slide was taken by which camera and lens, and under what conditions (e.g., handheld, tripod etc.).  

PLEASE do NOT share this answer key with anyone, as doing so will bias their results and ruin the value and benefit to other participants!! Thanks!!!

Naturally, with the answer key in hand, you can now go back to the test slides and know which camera and lens took which picture.  You will be able to see which shots were with the same lens, on different camera bodies (e.g., motorized vs. non-motorized). You will know which shots were from cameras and lenses costing kilobucks, and which lenses and cameras cost under $100 US.  

Hopefully, you will be able to get a number of the suggested benefits at the top of this page from your participation in this blind lens testing project. But I wanted to add that if you haven't had much experience with lens testing, you will probably benefit the most from participating and seeing these results yourself.

We ask that you please mail your slide set back as soon as possible.   It already takes a week in the mails to get these slides to you and back to us. We would like to be able to quickly respond to requests to participate.  If you hold onto and keep a slide set, replacing that slide set will take a half-day of setup and shooting and cost over $100 US for the dozen+ rolls of film and processing needed. 

Thank you for your help in participating in this project!!! 

Homebrew Loupes and Light Tables Ideas:

Besides our test slides, you will need some way to light them for viewing (e.g., lightbox) and a loupe or similar magnifier to view them.  We have some suggestions for homebrew loupes if you don't have one.  If you have a removable waist level viewfinder, chimney finder (often ideal), or even prism finder, you may already have a good loupe.  If not, you can often use a 35mm or medium format lens as a loupe, outer lens end near your eye, the film end of the lens pointing at the film.  Enlarger lenses may work even better.  For high power loupes, an old movie camera (8 or 16mm) glass lens can provide surprising amounts of magnification too. 

Light tables can be easily jury-rigged too.  Do you have a daylight fluorescent lighting fixture (as in many dorm rooms?).  You can often turn them so the tubes are facing upwards. A piece of frosted glass or plastic can be used as a temporary light table (or sheet(s) of white paper under a regular clear piece of glass or plastic). See our studio resources pages for more ideas. 

Sample Blind Test Participation Form:

Your name: _____________________________________________ Anonymous code (pick 5 letters) _ _ _ _ _

Your email address: ______________________________________

Your mailing address: ____________________________________________________________________

Which slide set do you have (circle Roman numeral:)  Set I    II    III     IV     V     VI    VII   VIII   IX   X  XI  XII

Split the slides into even number and odd numbered piles.  The even numbers were taken at 1/125th at f/16, the odd numbered slides at 1/500th at f/8, both focused at infinity using the lens markings. Lighting varys slightly, as you can see it is a day with some clouds. The tripod was moved and setup several times in each session both before and after lunch, so distances and subject sizes will vary slightly even with the same lens(es). The edges of the film have been trimmed to remove tell-tale film back markings in corners and side of film backs.  

Are you a (Circle one):    amateur photographer   semiprofessional photographer    fulltime pro photographer
Years of experience in photography?  _____    using 35mm? ____    medium format? _____  large format? ___
How many rolls of film do you shoot in the average year?  35mm: _____  rollfilm _____   sheet film (#) _____

What equipment do you use for 35mm? ______________________________________________
What equipment do you use for medium format? _______________________________________

What type and power of loupe or viewing setup did you use? _______________________________X

List the odd numbered slides, from BEST  to WORST, based on your assessment of their image quality (i.e., both resolution and contrast).  You may also wish to add an optional rating and your comments on each too:

BEST: #_______  (rating: ____)  Comments:__________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
WORST:  #________ ( ____)  _________________________________________________________________

List the even numbered slides, from BEST  to POOREST, based on your assessment of their image quality (i.e., both resolution and contrast).  You may also wish to add an optional rating and your comments on each too:

BEST: #_______  (rating: ____)  Comments:__________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
Next:  #________ ( ____)  _________________________________________________________________
WORST:  #________ ( ____)  ______________________________________________________________

Notes: Comments can be helpful notes on why you assigned this rank or rating, quality of bokeh etc.
Ratings are optional, and should be on a 0 to 5 scale (e.g., my scale 0=bad (bottle glass), 1= poor (no-name import label lenses), 2=good (consumer lenses), 3=very good (serious amateur) 4= excellent (pro quality glass) 5=perfect).  For example, if the top five lenses were very similar, maybe from the same camera and lens, you might assign them 4.50, 4.52, 4.54, 4.56, and 4.58 ratings. The next group might be a bit less refined, and so get scored 4.20, 4.22, 4.24.  Using the rating scale will help reviewers to see which lenses you thought performed similarly, and which you thought clustered out as lesser performing, and get a sense of by how much they differed.

Can you tell if any of the shots were handheld?  Which one(s)? #______   #______  #______ #________
Can you tell which shot(s) were taken without a lens hood? #_____ #____ #____ #_____ #_____ #_____
Can you tell which shot(s) were taken with a cheapy Chinese 3 el. lens? #______ #________ #________
Any shots you can identify as being from the same camera(s)? 

Can you positively identify the lens brand which made any particular shots? Which ones? How? 

 

What features or clues did you use to make your determinations? (e.g., mortar between brickwork, contrast of white blocks in triangle atop columns, fine lines in windows, bokeh of water droplets etc.?). 

 

Email us your listing of best to worse (with optional ratings). We will mail back the "answer key" for your review: 

Were you surprised by the answer key results versus your own scores and ratings? What surprised you most?  

 

Which lenses did better or worse than you expected?



Any suggestions for improving this blind test project?

 

Final comments - was it a useful project etc.? Why or why not?

 

 

Related Postings

From: Stephe [email protected]
Newsgroups: rec.photo.equipment.medium-format
Subject: Re: buying Mamiya offshore -- Hong Kong or UK?
Date: Mon, 15 Apr 2002


Jon N. wrote:

> If I can't compare
> slides side by side, I can't tell the difference at all, and I seriously
> doubt you could.

Well looking at slides done with a box camera with nothing to compare them
too might look OK as well..  (I actually have a box camera that works
pretty good!)

If you can't tell which camera they came from looking at the slides, that
IS a different story, but if you can see the difference in "side by side"
obviously there is something there.

Example I can't tell which slides were shot with my rolleicord and which
came from my zeiss tessar folder. There isn't any visible difference even
though people claim a front cell focusing camera isn't as sharp.
--
stephe
http://www.geocities.com/kievgurl/





From: Morton Linder [email protected]
Newsgroups: rec.photo.equipment.35mm
Subject: Re: Are Leica Lenses Really Better Than Zuiko?
Date: Mon, 15 Apr 2002


About 15(?) years ago, when I switched over from  Nikon F2 to Olympus OM-4, I
tested the comparable sets of Nikon and Zuiko lenses, with tripod and
electronic flash and U.S. Govt. engraved lens targets, on Kodachrome ASA 25
film. I viewed them with a very fine binocular Leitz medical microscope.

Basically, it was a wash. Some of each brand were a bit sharper than the other,
both ways.

The point is that, with top brand top line lenses, they are basically all
excellent, and the differences are minor.(I am not referring to recent
all-plastic entry level lenses, of which I have no experience.)

Morton




From: "Meryl Arbing" [email protected]
Newsgroups: rec.photo.equipment.35mm
Subject: Re: Leica M lens tests URL followup was Re: Leica...Is It Worth It?
Date: Mon, 22 Apr 2002


One frequently hears the "and NOBODY can tell the difference" argument cited
on these postings. Usually, it is the digital people claiming that "NOBODY
can tell the difference" between their dithered inkjet prints and
traditional lab prints.

The unfortunate part is that it is a faulty argument...it is an attempt to
prove the "null hypothesis". Just because a particular observer can't see
the differences does not mean that the differences do not exist. Sometimes,
all it takes is for someone to point out a specific and, from then on, the
differences stand out like the proverbial "sore thumb". Sometimes it is just
a question of not knowing what to look for or where to look for it.

"Typical real world shots" are notorious for disguising defects. For
example...is the softness at the edges of a shot due to DOF or poor lens
design? I suppose this is one reason why traditional test shots are made of
a single plane to eliminate DOF effects.

But do we HAVE to know exactly where to look for the differences between a
Leica or Zeiss lens and a Sakar, for example? No, I don't think so. And that
is because we have the cumulative experience of thousands and thousands of
photographers making millions of shots to prove that there is indeed a
difference! Even if we don't see it in an individual photograph, the
differences are apparent over years of photography so as to provide us with
a lengthy body of evidence that certain lenses have qualities that are
exceptional. This has come down to us simply as "reputation" and, as we all
know, a good reputation is difficult gain and easy to lose. If Leica lenses
were no better than ordinary glass, I doubt that people would be interested
in them.

I have an old Zeiss Ikon Contaflex Super (@1965) with a Zeiss Tessar 50/2.8
and I can see where the reputation of Zeiss and Contax came from and that it
is completely accurate and much deserved. I still take it out to shoot and I
have been having wonderful results with Portra BW400 even though that film
wasn't even conceived when the Contaflex was made.

I don't think we can dismiss "reputation" as real evidence for certain
lenses being significantly better than others. Making an excellent lens
isn't magic. All it takes is careful design; using the best materials;
manufacturing to exacting tolerances...Canon has done it with their 'L'
lenses and...Canon 'L' lenses have a deserved reputation for being among the
best in the world right up there with Leica and Zeiss.

If other...lesser names...were as good...they would have as good a
reputation. What is a reputation anyway if it isn't the honest reports of
our peers born out over the course of time? A false or hollow reputation
built by marketing hype is quickly dispelled by the real-world experiences
of the photographic community.





From: [email protected]
Newsgroups: rec.photo.equipment.35mm
Subject: Re: zeiss vs bronica tests, who wins? Re: Lens testing realities
Date: 22 Mar 2002


Robert Monaghan [email protected] wrote:
> well, the problem with viewing scans or prints is that they add multiple
> generations between the original slide and what is evaluated, and you
> can't tell if what you are seeing is limited by the enlarger lens or
> scanner optics and software, or other issues.

I suppose. I see your point, but I think it's very difficult
to compare slides, especially that number of slides, using a loupe.
It's especially difficult, too, as one can only see one slide at
a time, so you have to "remember" the differences.

Based on your Z-scores, your point is that (at f8 and f16, anyway)
all MF lenses are about the same. Maybe slides don't allow enough
of...a unique signature?

For example, imagine trying to pick the violin with the nicest
tone, only they're played at such faint volume it's impossible
to tell the Strad. from a cheap Yamaha.


> You can't really compare a f/2.8 lens against an f/4 lens, both
> wide open, and be a fair comparison.

That's a good point. Is there no aperture wider than f8 that
all the lenses shared?

> I could have easily had shots at every aperture in the test, but as it is
> for f/8 and f/16, there are dozens of slides to compare, and it gets
> tiresome as you noted. Perhaps I will include a wide open series with
> the telephoto blind lens test, if there is enough interest, since such
> lenses are more often used wide open for narrow DOF uses than wide angles
> or normal lenses?...

Ideally, what I'd like to see is a test of only two images --
one from the cheapest and one from the most expensive lens.
Both shot at the widest common aperture. Then, the images
would be scanned, a portion cropped, and displayed side/side.
*Then* if no one could tell the difference, I'd start to
believe there's really no difference. :)

Steve





From: Gary Frost [email protected]
Newsgroups: rec.photo.equipment.35mm
Subject: Re: zeiss vs bronica tests, who wins? Re: Lens testing realities
Date: Fri, 22 Mar 2002

...(quote above posting)


We have gone from "Price is not a reliable indicator of image quality
of lenses." to "All lenses perform the same regardless of price."
(at least for practical photographic uses...)
While the first statement I have found to be true, the second is
definitely not.

Robert is refering to a specific, limited comparison of like designed
lenses from different manufacturers. (at widely differing prices)
Don't infer that it means all lenses are the same, looking on a
light table at 10X. ...At least that is not my experience!

If you compare a bunch of different 35mm 4-element tessars, I suspect
you will find similiar limitations wide open (f/2.8-3.5) towards the
edges. (Minox is not able to magically overcome this...)
Compared to a Sonnar at 2.8, yes you will see the differences.
By f/8 it may be hard to tell though.


Keep in mind too, with reference to medium format lenses, the fastest
tend to be around 2 stops slower than the fastest 35mm lenses.

Testing medium format lenses at f/8 makes more sense than testing 35mm
at this aperture.

Gary Frost





From: fotocord 
Subject: Re: lens variations (enlarger..)
Newsgroups: rec.photo.equipment.medium-format
Date: Wed, 19 Jun 2002 

Robert Monaghan wrote:

> Considering these are kilobuck+ lenses, letting 2 out of 10 get out with
> "sever decentering problems" makes me wonder who has got those lenses,
> and if they wonder why everybody else raves about that zoom lens? ;-0) ;-)

I'm willing to bet most users after reading how great a certain lens is
either thinks their technique is the problem or having nothing to compare
it to things this is as good as it gets.  The first flektogon I bought I
thought "What are these people talking about, this lens sure isn't that
great..." but given my past experience with this, bought another sample and
saw why people rave so much.

I suppose it would be a drag to buy a $2800 lens that isn't that great,
can't get it replaced as it isn't actually broken have to sell it for $1500
to buy another $2800 lens to see if it's any better. When the lenses are
>$300 this becomes realistic to try it and this is why a buyer of high
dollar lenses doesn't know what we are talking about  :-)

--
Stacey 




From: John Stafford [email protected]
Newsgroups: rec.photo.equipment.medium-format
Subject: Re: Are you contemplating going digital?
Date: Mon, 09 Dec 2002

Bryan Olson at [email protected] wrote 

> [...]
> To get a little bit back to topic: overall, photographers are not
> as bad as audiophiles, but we see way too many subjective
> judgements made without appropriate controls.

No, photographers are worse by far.

If you test a random sample of artists and nonartists to determine which can
see differences in color values, the outcome is within 5% of the same:
usually 60% fail regardless of status.  (Color professional-printing lab
people do lots better than any other single group.)




From: [email protected] (Collin Brendemuehl)
Newsgroups: rec.photo.equipment.35mm
Subject: Re: Leica Lenses vs. Zeiss
Date: 14 Nov 2003 

You who suffer from the "brand loyalty" disease make me laugh. :)
All this bantering is meaningless.
Please read this from the Pentax list:

---------------------------------------------------------

Date: Fri, 14 Nov 2003 
From: -- name withheld for his privacy --
To: [email protected]
Subject: Re: Pentax enthusiast (long)

LOL

A few years ago I showed some prints to a couple of Leica aficionados. The
photos were made with a Super Takumar 50mm 1.4. The Leica people knew I was a
Leica user. Every one of them commented on how great the Leica glass was.

-- name withheld --


> Like Bob I've owned
> Contax and still own Leica and great Pentax glass and it's pretty difficult
> differentiate between film shot using them without resorting to a microscope.





From: Mxsmanic [email protected]
Newsgroups: rec.photo.equipment.35mm
Subject: Re: Leica Lenses vs. Zeiss
Date: Fri, 14 Nov 2003 

Collin Brendemuehl writes:

> But today Pentax, Mamiya, Fujinon, Nikon have the best
> stuff around.

I disagree as far as Nikon goes. My best lenses are the Zeiss and Leica
glass. The high-end Nikon glass is close to or equal to the Zeiss/Leica
stuff, but it's not _better_. It is true that my best Nikon lenses are
also zooms, which actually means that Nikon is doing extremely well, if
it can rival fixed-focal-length Zeiss or Leica with a zoom. The zooms
still have their drawbacks, though. My 80-200 is perfect in almost
every respect, except that it vignettes pretty badly when wide open and
racked out.

Also, Nikon makes a lot of cheap consumer stuff, so you have to look
carefully at a Nikon lens before buying it in order to be sure that
you're getting the professional goods.

Alas, in most cases, your choice is conditioned mainly by the camera on
which you want to mount the lens, not by a choice of lens manufacturers.
You generally mount Nikon glass on a Nikon body, Leica glass on a Leica
body, and so on.





From: "Jeremy" [email protected]
Newsgroups: rec.photo.equipment.35mm
Subject: Re: Pentax Lens Bokeh
Date: Mon, 24 Nov 2003

> >In a comparison performed by PDML members a comparison of the A*85/1.4
> >to the Zeiss equivalent was performed. And to eliminate all variables
> >they used only one roll of film!
> Are these tests on the web? Could you supply an URL, please?

I thought I saw that very article quoted in a post on this NG within the
past few days.

The poster, who was known to shoot with Leica, uploaded an image made by a
Pentax, and he intentionally lied about it as having come from a Leica. The
image received a number of comments from Leica users praising the high
quality of Leica glass.

It was nothing scientific, of course.

Bob Monaghan has conducted tests where people were asked to identify which
brand of lens took which photo--and they were unable to consistently
associate the correct images with their respective lenses.




End of Page