The Great Club Snap SLR Autofocus Speed Test

[Ian]

Following on from discussions on other threads here how about everyone test their camera and lens AF speed using a common testing proceedure.


The Testing method - Outline.

In order to test the AF speed a common testing proceedure is required, one that is applicable to all AF SLR's regardless of age or focusing system employed.

The proceedure is very straight forward and can be used with any AF lens and all that is required is a notepad, pen/pencil and a stopwatch.

Multiple testing for each lens and focal length will need to be employed to produce a more even result and to provide more consistent results. For this reason 7 tests per focal length will give a good level of data and population accuracy.

Two focal test points should be used, near focus and infinity. Selecting an object that is around 700-1000m distant for the infinity target would be acceptable for all but the longest supertele, in which case an object around 1500m should be used.

The objects selected as test points should be checked to make sure the lens doesn't focus hunt when aquiring focus.

A single sensor point (center) should be used to ensure maximum compatibility with both new and older SLR autofocusing systems. Multipoint focusing tests should be performed seperately.

All predictive or continious focusing methods should be disabled, so that the camera lens combination under test reflects the real capabilities of the AF system to respond to a maximum focus 'thow' situation.

The use of a tripod is mandatory.

Use fresh batteries to ensure maximum AF performance.

100 ISO shall be used as the standardised ISO rating for the test.

Each tester shall test their 'lag' time with the stopwatch by a simple test method to determine their average response time.

Tests shall be carried out in full sun and also near sunset to guage the preformance differential under good and adverse lighting conditions.


Testing method - Detail

Bring the lens to it's closest AF focal distance and disable any predictive or other focusing systems that will cause the lens to be AF active while lining up the infinity target.

Measure the time taken to focus on the infinity target.

Repeat until you have 7 readings.

Note down the aperture used (wide open value) and shutter speed so that an evaluation of the lighting conditions can be cross-compared with other cameras/lenses.

Repeat the test by focusing on the infinity target and measuring the time to focus on the minimum focus target Follow the procedure described above.

Measuring your stopwatch reaction time.

This is straight forward, click on the stopwatch and then as quickly as possible click on the stop button. Do this 7 times, noting the time taken each time. This will be used to determine your 'average' lag which will be used to bias the AF results you obtain.

--- end of test method ---

So what do you guys think? Any flaws anyone can spot in the proceedure?

If no one can find any major flaws I'll code up a web page for inputting the results and set up a statistical analysis of the data provided by the good members of Club Snap.

 

[Zoomer]

For casual testing, it should be comprehensive enough.

However, I think that it should be made an option for testers to include with their report an image taken of the test object (to spot potential differences in timings due to complexity of subject?)

The method for obtaining lag time is hardly the best, time taken to rapidly pressing a stopwatch 2 times would be different from pressing 2 buttons on 2 hands, and then, using your eyes to watch for the focous lock (make it compulsary to enable beeps and base on that? ) press the left (no offense of left handers) hand, and then stopping the stopwatch.

 

[erwinx]

since there're upcoming zoo and JBP shoots, the people attending can organise a SLR AF speed shootout... everyone point at the same subject and who focuses first....

 

[tsdh]

wahh.. the Great ClubSnap SLR AF test.

The procedure is comprehensive enough, but as Zoomer said, the target may affect AF performance. It is better to require the target as vertical object with high contrast against the background, and make sure the vertical edge is in the middle of viewfinder.

Manual starting & stopping the timer induce time lag which may vary. The best is actually using electronic timer connected to cable release and flash hotshoe (or PC-flash connector). This will ensure accurate measurement. Pressing the cable release, will trigger the timer and camera's shutter, but in AF-mode the shutter will not be fired at that moment until AF is locked onto target (this is with Nikon body, I don't know with others), once locked the shutter and the flash-hotshoe will be fired, which will stop the timer.

Unfortunately, electronic timer with external input is not used by many. So Ian's manual timer will still be the choice, but to further minimise error, I suggest to use flash instead of watching the focus-lock display. Just stop the timer when the flash is fired.

For DSLR, a programmer can easily write a small software running in PC using its SDK to trigger the shutter and detect focus-lock. The problem is: Canon & Nikon do not release their SDK to individuals.

 

[Zoomer]

Can't we listen for the soft beep sound that indicates that AF is done?

 

[Ian]

Ok, time for a bit of an explaination.

1) Not all cameras have beeps to indicate focus lock.

2) The most accurate timing method is to connect a datalogger across the control inputs of the Af motor, however major surgery is required to use this method and I can't see people volunteering their gear to the knife, saw and soldering iron!

The next most accurate method is to place a highly sensitive microphone on the lens or body and use that to trigger a timer by using a voice operated relay. However this technology is beyond the scope of most Club Snap members and has several false triggering issues that could be problematic.

3) Using a stopwatch one handed is the best/easiest method for the majority of people, after all the other hand is going to be busy triggering the AF on the camera. In this case the lag calculation method described above is acceptable (just)

4) Using a flash to measure the end of focus is an idea, however it may not be foolproof across all camera brands, and also assumes the person testing has a flash unit.

5) Using a vertical or horizontal target is a good point, however regardless of orientation it's essential to eliminate AF hunting. Though it's orientation should be noted as an after thought.

 

[erwinx]

i agree that getting absolute measurements is difficult, thats why you should line up several cameras side by side and shoot the same target.

You could even place a running stopwatch next to the target and look at the time 'captured'

of course, that would factor in shutter lag as well, but then, we should look at overall performance - AF + shutter lag

 

[YSLee]

Ian, wouldn't different lenses give completely different results?

 

[Ian]

Originally posted by YSLee
Ian, wouldn't different lenses give completely different results?

Yes they will give different results, however that's a minor problem, what will emerge in time is that brand X body with lens Y is faster/slower than brand Z body with it's equivalent to lens Y.

Thus finally answering the big questions or rather at least providing some imperical evidence.

 

[YSLee]

3rd party then? But even between mounts, I know of at least one 3rd party lens that has quite different AF speeds.. Or are you talking about comparing AF speeds within a brand?

 

[Ian]

Originally posted by erwinx
i agree that getting absolute measurements is difficult, thats why you should line up several cameras side by side and shoot the same target.

You could even place a running stopwatch next to the target and look at the time 'captured'

of course, that would factor in shutter lag as well, but then, we should look at overall performance - AF + shutter lag

Not at all, lag performance (ie wake up times, shutter lag) are already documented online for many AF bodies, but there's no real results for purely AF testing, which is why I proposed the test be conducted in the manner described.

A line up of several cameras with different users and using a common target will only work if the people are honest in their reporting, and a large enough stopwatch display can be found (also there's the problem of people reading a fast moving stopwatch display rather than stopping a stop watch etc).

 

[Jed]

Ian, shame on you.

Multiple party tests never ever work out. To many variables.

And I've said it once, twice, goodness knows how many times and I'll say it again. Modern AF systems are good enough for shooting the vast, vast majority of things that casual users, of which the majority of Clubsnap are, would want to shoot. Which renders the entire debate moot.

 

[tsdh]

Originally posted by Ian
A line up of several cameras with different users and using a common target will only work if the people are honest in their reporting, and a large enough stopwatch display can be found (also there's the problem of people reading a fast moving stopwatch display rather than stopping a stop watch etc).

Shoot a clock in film/digital, no need to watch the stopwatch.
e.g.: a clock is included in the frame beside the target, then once the second arm hit 12, everybody press his shutter. The camera will do AF before actually firing its shutter. Then we can see the result from the photo.

 

[Zoomer]

Originally posted by tsdh

Shoot a clock in film/digital, no need to watch the stopwatch.
e.g.: a clock is included in the frame beside the target, then once the second arm hit 12, everybody press his shutter. The camera will do AF before actually firing its shutter. Then we can see the result from the photo.

Maybe a digital clock that counts in milliseconds is better?

 

[Jed]

Chaps. Shooting a clock. Brilliant idea. That happens to measure in seconds as the smallest unit and I honestly struggle to think of any camera that I've ever used that would not acquire an acquirable target within a tiny fraction of that time. And it's not for lack of using cameras.

A digital thing that measures in milliseconds. Great. Slightly better, but what happens to our shutter lag? Different cameras have different shutter lags.

Not to mention YS has already been talking about the differences in lenses, the amount of glass that needs moving, contrast of subject, reaction time of the person tripping the shutter, reaction time of the person timing the AF.

Frankly, the best way to judge AF performance is to get out there and use the cameras. Use it enough and you'll know how good the AF is, and whether it's up to the task of what you need it to do.

I've no idea how applicable this analogy is, and my brain's too addled to reason it out, but as the saying goes, if you have to ask the price, you can't afford it. Well, with AF, if you need to compare, you don't need it (to push it to its limits).
PS/= I've not read Ian's well thought out proposal thoroughly at all. As mentioned, my brain's addled. But I do know that some of the replies it's generating show a vague lack of consideration.

 

[tsdh]

Originally posted by Jed
A digital thing that measures in milliseconds. Great. Slightly better, but what happens to our shutter lag? Different cameras have different shutter lags.

Yes Jed, we can't avoid shutter lag as part of the measurement error. But it is more predictable and constant as compared to the error generated by manually press the stopwatch. And for some cameras, the shutter lag is documented.

Frankly, the best way to judge AF performance is to get out there and use the cameras. Use it enough and you'll know how good the AF is, and whether it's up to the task of what you need it to do.

Agree with you, every camera (and lenses) has different AF algorithm which will react differently at different condition.
So, back to Ian, what is the goal of this 'Great SLR AF test'.
If a brief test can achieve the goal, why must use a very precise and complex one?

 

[erwinx]

1. 2 cameras in AF mode, pointing at same target with a stopwatch in miliseconds next to it (and within the frame).
Each hand hold 1 cable release.

2. Press cable release at same time, look at time difference in stopwatch readings.

3. Repeat same exercise in manual/prefocused mode, look at time difference in stopwatch readings to ascertain shutter lag.

4. Ignore above and go and shoot birds in flight at JBP. When's the last time anyone posted birds in flight from JBP (or anywhere else)?

 

[denizenx]

I think what they want is a big gathering at some beer diner and then all AF on some guy and see if they can catch his beer before it disappears...

 

[denizenx]

anyway as long as humans are initiaters in timing must add + - 2 secs... adjusting the thumb takes almost 100ms liao...

 

[denizenx]

someone write a program that uses a serial port with a custom soldered remote to AF on a sharp pic on the screen? should eliminate most of the msecs...
hahaha leave it to the labs... go for beer