Experiments on the Web: How to

This is a great 2007 guide by Ron Kohavi et al. on how to perform controlled experiments on the Web and trust the data instead of the wise man: "Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO". Protocol, statistics, measures, pitfalls, anecdotes. It has it all.