In the past few years, the role of both domain-specific and domain-general factors on numerical development and mathematics achievement has been debated. In this paper, we focus on the role of visuo-spatial processes. We will more particularly review the numerical abilities of populations presenting atypical visuo-spatial processes: individuals with blindness, hemineglect, children presenting low visuo-spatial abilities, non-verbal learning disorder or Williams syndrome. We will show that math abilities of each population are relatively unique and are not necessarily associated with generalized math impairment. We will show that a better understanding of the strengths and weaknesses of each population gives further insights into our conceptual understanding of the development of numerical cognition. We will finally demonstrate how the comparison across disorders can impact on practical rehabilitation and educational strategies.