Thanks for your interest in my article. Apologies if it was confusing. I can see why it was. All I was trying to do was to explain what effect size is by giving an example. It’s always difficult to avoid sounding too geeky and losing people’s attention when you talk about statistics and you use terms like “effect size”! So I decided to explain it by coming up with the New York / London example. What I was trying to say is that when you can predict big effect sizes (before you test) you can reduce the number of people you will test (the “handful” term was used colloquially, rather than formally). Usually, sample sizes need to be decided before the test, survey, etc. is run, and one of the things that should be taken into account is the size of the effect you expect to measure. If you predict that the behaviour you will be measuring will only have tenuous variations between people you will need to boost those numbers to be confident that your results will have solid statistical basis. You can, of course, run a power analysis post hoc to determine whether your sample was big enough to give you high confidence levels in your results. But by then it is too late.
I hope this clarifies things (although Steve made my job very easy by explaining this very clearly).

Great blog and comments in general. :-)
Looking forward to reading the UXmatters article.

If the question you’re trying to answer is “Is design A better (in real terms) than design B?” then no, it doesn’t really depend on the certainty. If the observed difference between the two designs is relatively large (the relative measurement is important), then the underlying theory says you can be fairly sure.

Do we ever need to write a test report that reads “we can be certain with a 95% degree of confidence that…”? Rarely, if ever. However, the key point I’m trying to make is that – with large observed differences – the point at which you reach that 95% level of confidence is much fewer observations than with small relative differences.

BTW: you’re not alone in the lack of intuition with this. One of the smartest mathematicians I ever met completely failed to understand this topic on anything other than a “I don’t get it, but I’ll take your word for it” basis.

PS: I’m in the process of writing this topic up into a column for uxmatters, which I hope will provide extra clarity.

When I had to do this stuff as part of signal processing my intuition was (and apparently still is) wrong on pretty much all of it. That’s why I said it was dangerous to talk about it without actually working out the numbers…

This explains why I felt “like a deer caught in the headlights” during my stats course at uni :)

You run that comparison as an A/B test for 15 minutes and check the result. If it’s a big difference, you could stop right there. If it’s a small difference, keep running it and check again at the hour mark. At that point, you probably have a decent sized sample and can crunch some numbers; run a chi-squared test and see what you see. What you’re trying to do at this stage is determined the likelihood that your small observed difference is *real* or random.

Steve

For the sort of research we do, our methods and sample sizes are appropriate, taking into account Stephen’s comments.

I, for one, am much clearer on the matter now.

Sorry, my reading of the original example takes a handful to be something like four or five and height differences to be much smaller even for a large difference.

I think it’s dangerous to talk about these things in vague terms numerically whilst giving a salient example of New York and London.

So yeah, my “gripe” is with the example.