Predicting Hand Movements With Distributional Semantics: Evidence From Mouse‐Tracking

Abstract

AbstractAlthough mouse‐tracking has been taken as a real‐time window on different aspects of human decision‐making processes, whether purely semantic information affects response conflict at the level of motor output as measured through mouse movements is still unknown. Here, across two experiments, we investigated the effects of semantic knowledge by predicting participants’ performance in a standard keyboard task and in a mouse‐tracking task through distributional semantics, a usage‐based modeling approach to meaning. In Experiment 1, participants were shown word pairs and were required to perform a two‐alternative forced choice task selecting either the more abstract or the more concrete word, using standard keyboard presses. In Experiment 2, participants performed the same task, yet this time response selection was achieved by moving the computer mouse. Results showed that the involvement of semantic components in the task at hand is observable using both standard reaction times (Experiment 1) as well as using indexes extracted from mouse trajectories (Experiment 2). In particular, mouse trajectories reflected the response conflict and its temporal evolution, with a larger deviation for increasing word semantic relatedness. These findings support the validity of mouse‐tracking as a method to detect deep and implicit decision‐making features. Additionally, by demonstrating that a usage‐based model of meaning can account for the different degrees of cognitive conflict associated with task achievement, these findings testify the impact of the human semantic memory on decision‐making processes.