Speechreading Supplemented by Single-Channel and Multichannel Tactile Displays of Voice Fundamental Frequency

Abstract

The benefits of two tactile codes of voice fundamental frequency (F o ) were evaluated as supplements to the speechreading of sentences in two short-term training studies, each using 12 adults with normal hearing. In Experiment 1, a multichannel spatiotemporal display of F o , known as Portapitch, was used to stimulate the index finger. In an attempt to improve on past performance with this display, the coding scheme was modified to better cover the F o range of the talker in the training materials. For Experiment 2, to engage kinesthetic/proprioceptive pathways, a novel single-channel positional display was built, in which F o was coded as the vertical displacement of a small finger-rest. Input to both displays consisted of synthesized replicas of the F o contours of the sentences, prepared and perfected off-line. Training with the two tactile F o displays included auditory presentation of the synthesized F o contours in conjunction with the tactile patterns on alternate trials. Speechreading enhancement by the two tactile F o displays was compared to the enhancement provided when auditory F o information was available in conjunction with the tactile patterns, by auditory presentation of a sinusoidal indication of the presence or absence of voicing, and by a single-channel tactile display of the speech waveform presented to the index finger. Despite the modified coding strategy, the multichannel Portapitch provided a mean tactile speechreading enhancement of 7 percentage points, which was no greater than that found in previous studies. The novel positional F o display provided only a 4 percentage point enhancement. Neither F o display was better than the simple single-channel tactile transform of the full speech waveform, which gave a 7 percentage point enhancement effect. Auditory speechreading enhancement effects were 17 percentage points with the voicing indicator and approximately 35 percentage points when the auditory F o contour was provided in conjunction with the tactile displays. The findings are consistent with the hypothesis that subjects were not taking full advantage of the F o variation information available in the outputs of the two experimental tactile displays.