Distinguishing between straight and curved sounds: Auditory shape in pitch, loudness, and tempo gestures

Abstract

AbstractSound-based trajectories or sound gestures draw links to spatiokinetic processes. For instance, a gliding, decreasing pitch conveys an analogous downward motion or fall. Whereas the gesture’s pitch orientation and range convey its meaning and magnitude, respectively, the way in which pitch changes over time can be conceived of as gesture shape, which to date has rarely been studied in isolation. This article reports on an experiment that studied the perception of shape in uni-directional pitch, loudness, and tempo gestures, each assessed for four physical scalings. Gestures could increase or decrease over time and comprised different frequency and sound level ranges, durations, and different scaling contexts. Using a crossmodal-matching task, participants could reliably distinguish between pitch and loudness gestures and relate them to analogous visual line segments. Scalings based on equivalent-rectangular bandwidth (ERB) rate for pitch and raw signal amplitude for loudness were matched closest to a straight line, whereas other scalings led to perceptions of exponential or logarithmic curvatures. The investigated tempo gestures, by contrast, did not yield reliable differences. The reliable, robust perception of gesture shape for pitch and loudness has implications on various sound-design applications, especially those cases that rely on crossmodal mappings, e.g., visual analysis or control interfaces like audio waveforms or spectrograms. Given its perceptual relevance, auditory shape appears to be an integral part of sound gestures, while illustrating how crossmodal correspondences can underpin auditory perception.