The coalescent in two partially isolated diffusion populations

Abstract

SummaryThen0coalescent of Kingman (1982a,b) describes the family relationships among a sample ofn0individuals drawn from a panmictic species. It is a stochastic process resulting fromn0− 1 independent random events (coalescences) at each of whichn(2 ≤n≤n0) ancestral lineages of a sample are descended fromn− 1 distinct ancestors for the first time. Here a similar genealogical process is studied for a species consisting of two populations with migration between them. The main interest is with the probability density of the time length between two successive coalescences and the spatial distribution ofn− 1 ancestral lineages over two populations whennton− 1 coalescence takes place. These are formulated based on a non-linear birth and death process with killing, and are used to derive several explicit formulae in selectively neutral population genetics models. To confirm and supplement the analytical results, a simulation method is proposed based on the underlying bivariate Markov chain. This method provides a general way for solving the present problem even when an analytical approach appears very difficult. It becomes clear that the effects of the present population structure are most conspicuous on 2 to 1 coalescence, with lesser extents onnton− 1 (3 ≤n) coalescence.This implies that in a more general model of population structure, the number of populations and the way in which a sample is drawn are important factors which determine then0coalescent.