With respect to multiple comparisons I’ve always thought about it on a few levels. First, what is a ‘family’ of tests? Is it multiple comparisons from one experiment? What about comparisons from multiple related experiments? I think this framework might be helpful.

Let me pose an observational example – a researcher goes and tests a correlation between, say, species richness and plant biomass in 50 different fields. Let’s say they perform 50 correlations, and a researcher has a critical alpha of 0.05. Now, if each correlation has a p value of around 0.06, except for 1 which is 0.04, what would you conclude under the old versus new framework? Would it be the same or different if each correlation had a p value of 0.5 except for one which had a correlation of 0.04? Would you always conclude that something was different about that one field, or not? What inferences from each set of p values could you draw about that general relationship? How would those conclusions change if you corrected your alpha for multiple comparisons to 0.001, and would this be more likely to give you a true impression of the correlation between these variables in nature?

(and, yes, I know a multilevel approach is far more appropriate to this particular example, but, for the sake or argument, consider the two examples)