Multisensory Plucked Instrument Modeling in Unity3D: From Keytar to Accurate String Prototyping

Abstract

Keytar is a plucked guitar simulation mockup developed with Unity3D that provides auditory, visual, and haptic feedback to the player through a Phantom Omni robotic arm. Starting from a description of the implementation of the virtual instrument, we discuss our ongoing work. The ultimate goal is the creation of a set of software tools available for developing plucked instruments in Unity3D. Using such tools, sonic interaction designers can efficiently simulate plucked string prototypes and realize multisensory interactions with virtual instruments for unprecedented purposes, such as testing innovative plucked string interfaces or training machine learning algorithms with data about the dynamics of the performance, which are immediately accessible from the machine.