While some researchers place negative numbers on a so-called extended mental number line to the left of positive numbers, others claim that negative numbers do not have mental representations but are processed through positive numbers combined with transformation rules. We measured spatial associations of negative numbers with a modified implicit association task that avoids spatial confounds present in most previous studies. In two lab-based magnitude classification experiments (each including 24 participants) and two online replications (with 74 and 77 participants, respectively), positive and negative numbers were combined with two spatial contexts: either directional symbols (left- or right-pointing arrows) or rectangles of varying sizes. In all experiments, we found a robust distance effect for negative numbers. However, there were no consistent associations of negative numbers with directional or size contexts. In the context of directional symbols, holistic processing was prevalent only in the small negative number range (-9, -8, -7, -6) when ensured by the stimulus set, supporting an extended mental number line. In the context of rectangles, however, large negative numbers from -4 to -1 were perceived as small, thus supporting rule-based processing. For negative number processing in the context of size, we further suggest the Semantic-Perceptual Size Congruity Cuing model (SPeSiCC model). We show that associations of size with negative numbers underly more complex processing mechanisms than mere recruitment of a transformation rule. In general, we conclude that associations of negative numbers with space and size are situated in the context, as they depend on the presented number range and differ for spatial direction and size.