I agree that the article is way too complicated 😉 also, please change the backgound on this page https://vwo.com/ab-split-test-significance-calculator/ It’s killing!

Thanks for your feedback. I agree with you that a t-test for the difference between two independent means should be appropriate in this case and I share your observation that due to large standard deviations, required sample sizes can be quite large.

It is therefore that I advise to seriously think through the desired alpha. Not for going easy on significance, but for optimized decision making.

Apart from that, I’d afterwards also want to analyse differences in response percentages and differences in the average value amongst the respondents, for further exploration of the effects of the test variant(s).

Regards, Kees

great article. Thanks a lot!
I was wondering though, how to employ this method of precaclulating required sample sizes for tests where the key metric isn’t conversion rate, but revenue per visitor (RPV).

To my understanding, it would be fine to use a t-test on the difference between two independent means (two group). Do you agree? If not, could you provide a sample of how you would approach it?

From my observations, the usually rather large standard deviation (compared to the mean) of RPV tests generates the requirement for huge sample sizes in order to get power up to an acceptable level.

Many thanks!

I’ve seen a range of suggested methods and it’s something that’s often mentioned, but I believe perhaps oversimplified and/or misunderstood.

Thanks!

Also, don’t hesitate if you want to propose new topics you would find interesting.