On the number of segregating sites for populations with large family sizes

Abstract

We present recursions for the total number,Sn, of mutations in a sample ofnindividuals, when the underlying genealogical tree of the sample is modelled by a coalescent process with mutation rater>0. The coalescent is allowed to have simultaneous multiple collisions of ancestral lineages, which corresponds to the existence of large families in the underlying population model. For the subclass of Λ-coalescent processes allowing for multiple collisions, such that the measure Λ(dx)/xis finite, we prove thatSn/(nr) converges in distribution to a limiting variable,S, characterized via an exponential integral of a certain subordinator. When the measure Λ(dx)/x2is finite, the distribution ofScoincides with the stationary distribution of an autoregressive process of order 1 and is uniquely determined via a stochastic fixed-point equation of the formwith specific independent random coefficientsAandB. Examples are presented in which explicit representations for (the density of)Sare available. We conjecture thatSn/E(Sn)→1 in probability if the measure Λ(dx)/xis infinite.