A discrete approach to the external branches of a Kingman coalescent tree. Theoretical results and practical applications

Abstract

AbstractThe Kingman coalescent process is a classical model of gene genealogies in population genetics. It generates Yule-distributed, binary ranked tree topologies—also calledhistories—with a finite number ofnleaves, together withn−1 exponentially distributed time lengths: one for each each layer of the history. Using a discrete approach, we study the lengths of the external branches of Yule distributed histories, where the length of an external branch is defined as the rank of its parent node. We study the multiplicity of external branches of given length in a random history ofnleaves. A correspondence between the external branches of the ordered histories of sizenand the non-peak entries of the permutations of sizen−1 provides easy access to the length distributions of the first and second longest external branch in a random Yule history and coalescent tree of sizen. The length of the longest external branch is also studied in dependence of root balance of a random tree. As a practical application, we compare the observed and expected number of mutations on the longest external branches in samples from natural populations.