Evidence from multiple studies conducted in the past few decades converges on the conclusion that numerical properties can be associated with specific directions in space. Such Spatial-Numerical Associations (SNAs), as a signature of elementary number processing, seem to be a likely correlate of math skills. Nevertheless, almost three decades of research on the Spatial Numerical Association of Response Codes (SNARC) effect, the hallmark of SNAs, has not provided conclusive results on whether there is a relation with math skills. Here, going beyond reviewing the existing literature on the topic, we try to answer a more fundamental question about WHY the SNARC effect should (and should not) be related to math skills. We propose a multi-route model framework for a SNARC-math skills relationship. We conclude that the relationship is not straightforward and that several other factors should be considered, which under certain circumstances or in certain groups, can cause effects of opposite directions. The model can account for conflicting results, and thus may be helpful for deriving predictions in future studies.