Recombination rate inference via deep learning is limited by sequence diversity

Abstract

AbstractA common inference task in population genetics is to estimate recombination rate from multiple sequence alignments. Traditionally, recombination rate estimators have been developed from biologically-informed, statistical models, but more recently deep learning models have been employed for this task. While deep learning approaches offer unique advantages, their performance is inconsistent across the range of potential recombination rates. Here, we generate and characterize data sets (genotype alignments with known recombination rates) for use by deep learning estimators and assess how their features limit estimator performance. We find that certain input parameter regimes produce genotype alignments with low sequence diversity, which are inherently information-limited. We next test how estimator performance is impacted by training and evaluating neural networks on data sets with varying degrees of diversity. The inclusion of genotype alignments with low diversity at high frequency results in considerable performance declines across two different network architectures. In aggregate, our results suggest that genotype alignments have inherent information limits when sequence diversity is low, and these limitations need to be considered both when training deep learning recombination rate estimators and when using them in inference applications.