A unified characterization of population structure and relatedness

Abstract

AbstractMany population genetic activities, ranging from evolutionary studies to association mapping to forensic identification, rely on appropriate estimates of population structure or relatedness. All applications require recognition that quantities with an underlying meaning of allelic identity by descent are not defined in an absolute sense, but instead are made “relative to” some set of alleles other than the target set. The early Weir and Cockerham FST estimate made explicit that the reference set of alleles was across independent populations. Standard kinship estimates have an implicit assumption that pairs of individuals in a study sample, other than the target pair, are unrelated, whereas other estimates assume alleles within individuals are not identical by descent. However, populations lose independence when there is migration between them, and when individuals in a study are related it is difficult to see how they can also be non-inbred. We have therefore re-cast our treatments of population structure, relatedness and inbreeding to make explicit that the parameters of interest involve differences of probabilities of identity by descent in the target and the reference sets of alleles and so can be negative. We take the reference set to be for the population from which study individuals have been sampled. We provide simple moment estimates of these parameters, phrased in terms of allele matching within and between individuals for relatedness and inbreeding, or within and between populations for population structure. A multi-level hierarchy of alleles within individuals, alleles between individuals within populations, and alleles between populations allows a unified treatment of relatedness and population structure. Our new estimates appear to be sensitive to rare or private variants, to give indications of the effects of natural selection, and to be appropriate for use in association studies.