Correlation of coalescence times in a diploid Wright-Fisher model with recombination and selfing

Abstract

AbstractThe correlation among the gene genealogies at different loci is crucial in biology, yet challenging to understand because such correlation depends on many factors including genetic linkage, recombination, natural selection and population structure. Based on a diploid Wright-Fisher model with a single mating type and partial selfing for a constant large population with sizeN, we quantify the combined effect of genetic drift and two competing factors, recombination and selfing, on the correlation of coalescence times at two linked loci for samples of size two. Recombination decouples the genealogies at different loci and decreases the correlation while selfing increases the correlation. We obtain explicit asymptotic formulas for the correlation for four scaling scenarios that depend on whether the selfing probability and the recombination probability are of orderO(1/N) orO(1) asNtends to infinity. Our analytical results confirm that the asymptotic lower bound in [King, Wakeley, Carmi (TPB 2018)] is sharp when the loci are unlinked and when there is no selfing, and provide a number of new formulas for other scaling scenarios that have not been considered before. We present asymptotic results for the variance of Tajima’s estimator of the population mutation rate for infinitely many loci asNtends to infinity. When the selfing probability is of orderO(1) and is equal to a positive constantsfor allNand if the samples at both loci are in the same individual, then the variance of the Tajima’s estimator tends tos/2 (hence remains positive) even when the recombination rate, the number of loci and the population size all tend to infinity.