Samples in many cell-based experiments are matched/paired but taking this into account does not always increase power of statistical tests for differences in means

Abstract

Power of statistical tests for differences in means is the probability of obtaining a statistically significant p value when means differ. When samples in experimental replicates come from a single cell culture, they are matched or paired because they share between-trials biological variability. This can cause positive correlation between values from conditions in a replicate. Correlation can also be caused in otherwise independent samples by shared technical variability. However, correlation is reduced by noise that affects samples individually. I investigated how to maximize power in experiments with two conditions over a range of correlations. Normalizing data to control increases the rate of false positives, if Student’s t test is used. Paired t tests, theoretically the correct test for matched samples, have higher power than Student’s t test when correlation is high, but lower power when correlation is low. Testing correlation to select a test for differences in mean can affect the subsequent rate of false positives. Ultimately, components of experimental variability must be considered to choose the most powerful two sample test for differences in mean. This contrasts with experiments with more than two conditions, where random-block ANOVA, a matched samples test, can be used as a default.