The Effects of Dominance, Regular Inbreeding and Sampling Design on QST, an Estimator of Population Differentiation for Quantitative Traits

Abstract

Abstract To test whether quantitative traits are under directional or homogenizing selection, it is common practice to compare population differentiation estimates at molecular markers (FST) and quantitative traits (QST). If the trait is neutral and its determinism is additive, then theory predicts that QST = FST, while QST > FST is predicted under directional selection for different local optima, and QST < FST is predicted under homogenizing selection. However, nonadditive effects can alter these predictions. Here, we investigate the influence of dominance on the relation between QST and FST for neutral traits. Using analytical results and computer simulations, we show that dominance generally deflates QST relative to FST. Under inbreeding, the effect of dominance vanishes, and we show that for selfing species, a better estimate of QST is obtained from selfed families than from half-sib families. We also compare several sampling designs and find that it is always best to sample many populations (>20) with few families (five) rather than few populations with many families. Provided that estimates of QST are derived from individuals originating from many populations, we conclude that the pattern QST > FST, and hence the inference of directional selection for different local optima, is robust to the effect of nonadditive gene actions.