Mathematical constraints onFST: multiallelic markers in arbitrarily many populations

Abstract

Interpretations of values of theFSTmeasure of genetic differentiation rely on an understanding of its mathematical constraints. Previously, it has been shown thatFSTvalues computed from a biallelic locus in a set of multiple populations andFSTvalues computed from a multiallelic locus in a pair of populations are mathematically constrained as a function of the frequency of the allele that is most frequent across populations. We generalize from these cases to report here the mathematical constraint onFSTgiven the frequencyMof the most frequent allele at amultialleliclocus in a set ofmultiplepopulations. Using coalescent simulations of an island model of migration with an infinitely-many-alleles mutation model, we argue that the joint distribution ofFSTandMhelps in disentangling the separate influences of mutation and migration onFST. Finally, we show that our results explain a puzzling pattern of microsatellite differentiation: the lowerFSTin an interspecific comparison between humans and chimpanzees than in the comparison of chimpanzee populations. We discuss the implications of our results for the use ofFST.This article is part of the theme issue ‘Celebrating 50 years since Lewontin's apportionment of human diversity’.