According to phenomenological and enactive approaches, human sociality does not start from isolated individuals and their hidden inner states, but from intercorporeality and interaffectivity. This paper introduces first a general concept of embodied affectivity: it conceives emotions as a circular interaction of the embodied subject and the respective situation with its affective affordances. This leads to a concept of embodied interaffectivity (with others) as a process of coordinated interaction, bodily resonance, and “mutual incorporation,” providing the basis for a primary empathic understanding. Finally these empathic capacities are also based developmentally on an intercorporeal memory acquired in early childhood, which conveys a basic sense of social attunement or a “social musicality” and also manifests itself in an individual’s habitus. Basic empathy mediated by embodied interaction may subsequently be extended by higher-level cognitive capacities such as perspective-taking and imaginary transposition. Nevertheless, intercorporeality and interaffectivity remain the basis of social understanding.