Five-leaf generalizations of theD-statistic reveal the directionality of admixture

Abstract

AbstractOver the past 15 years, theD-statistic, a four-taxon test for organismal admixture (hybridization, or introgression) which incorporates single nucleotide polymorphism data with allelic patterns ABBA and BABA, has seen considerable use. This statistic seeks to discern significant deviation from either a given species tree assumption, or from the balanced incomplete lineage sorting that could otherwise defy this species tree. However, while theD-statistic can successfully discriminate admixture from incomplete lineage sorting, it is not a simple matter to determine the directionality of admixture using only four-leaf tree models. As such, methods have been developed that use 5 leaves to evaluate admixture. Among these, theDFOILmethod, which tests allelic patterns on the “symmetric” treeS= (((1, 2), (3, 4)), 5), succeeds in finding admixture direction for many five-taxon examples. However,DFOILdoes not make full use of all symmetry, nor canDFOILfunction properly when ancient samples are included because of the reliance on singleton patterns (such as BAAAA and ABAAA). Here, we take inspiration fromDFOILto develop a new and completely general family of five-leaf admixture tests, dubbed Δ-statistics, that can either incorporate or exclude the singleton allelic patterns depending on individual taxon and age sampling choices. We describe two new shapes that are also fully testable, namely the “asymmetric” treeA= ((((1, 2), 3), 4), 5) and the “quasisymmetric” treeQ= (((1, 2), 3), (4, 5)), which can considerably supplement the “symmetric”S= (((1, 2), (3, 4)), 5) model used byDFOIL. We demonstrate the consistency of Δ-statistics under various simulated scenarios, and provide empirical examples using data from black, brown and polar bears, the latter also including two ancient polar bear samples from previous studies. RecentlyDFOILand one of these ancient samples was used to argue for a dominant polar bear → brown bear introgression direction. However, we find, using both this ancient polar bear and our own, that by far the strongest signal using bothDFOILand Δ-statistics on treeSis actually bidirectional gene flow of indistinguishable direction. Further experiments on treesAandQinstead highlight what were likely two phases of admixture: one with stronger brown bear → polar bear introgression in ancient times, and a more recent phase with predominant polar bear → brown bear directionality.Code and documentation available athttps://github.com/KalleLeppala/Delta-statistics.