Quantitative Evaluation of Nonlinear Methods for Population Structure Visualization & Inference

Abstract

Population structure (also called genetic structure and population stratification) is the presence of a systematic difference in allele frequencies between sub-populations in a population as a result of non-random mating between individuals. It can be informative of genetic ancestry, and in the context of medical genetics it is an important confounding variable in genome wide association studies. Recently, many nonlinear dimensionality reduction techniques have been proposed for the population structure visualization task. However, an objective comparison of these techniques has so far been missing from the literature. In this paper, we discuss the previously proposed nonlinear techniques and some of their potential weaknesses. We then propose a novel quantitative evaluation methodology for comparing these nonlinear techniques, based on populations for which pedigree is either known a-priori through artificial selection or simulation. Based on this evaluation metric, we find graph-based algorithms such as t-SNE and UMAP to be superior to PCA, while neural network based methods fall behind.