Local fitness and epistatic effects lead to distinct patterns of linkage disequilibrium in protein-coding genes

Abstract

Abstract Selected mutations interfere and interact with evolutionary processes at nearby loci, distorting allele frequency trajectories and creating correlations between pairs of mutations. Recent studies have used patterns of linkage disequilibrium between selected variants to test for selective interference and epistatic interactions, with some disagreement over interpreting observations from data. Interpretation is hindered by a lack of analytic or even numerical expectations for patterns of variation between pairs of loci under the combined effects of selection, dominance, epistasis, and demography. Here, I develop a numerical approach to compute the expected two-locus sampling distribution under diploid selection with arbitrary epistasis and dominance, recombination, and variable population size. I use this to explore how epistasis and dominance affect expected signed linkage disequilibrium, including for nonsteady-state demography relevant to human populations. Using whole-genome sequencing data from humans, I explore genome-wide patterns of linkage disequilibrium within protein-coding genes. I show that positive linkage disequilibrium between missense mutations within genes is driven by strong positive allele-frequency correlations between mutations that fall within the same annotated conserved domain, pointing to compensatory mutations or antagonistic epistasis as the prevailing mode of interaction within conserved genic elements. Linkage disequilibrium between missense mutations is reduced outside of conserved domains, as expected under Hill–Robertson interference. This variation in both mutational fitness effects and selective interactions within protein-coding genes calls for more refined inferences of the joint distribution of fitness and interactive effects, and the methods presented here should prove useful in that pursuit.