Our perception of the passage of time can suffer from significant distortions as time flies when we are busy and drags when we are bored. A prominent mechanistic model proposes that this perceptual volatility reflects changes in the activity dynamics of distributed neuronal ensembles referred to as population clocks because they encode time. In this framework, time is understood similarly to space (both can be segmented in seconds or centimeters) and duration estimation is primarily internal (the brain tells time). Here, I challenge this framework from the angle of Bergson’s proposal that the inner experience of time is unlike space because it is ever-changing and indivisible (2 successive seconds are not experienced equivalently). Quantifying and communicating this inner experience requires its externalization and immobilization through distance measurements derived from stereotyped movements and spatial metaphors (“short/long” durations; time “flies/drags”), which explains the habit of thinking time like space. In support of Bergson’s proposal, humans and animals heavily rely on movements in a variety of duration estimation tasks and the neural underpinnings of duration estimates overlap those of motor control and spatial navigation. Thus, philosophical and empirical arguments question whether duration estimation is fundamentally internal. Rather than being explained by ad hoc changes in the speed of population clocks, the puzzle of the volatility of time perception might resolve itself by considering that living beings lack the ability to internally measure time, which they compensate through interactions with regularities afforded by the world and symbolic representation drawn from space.