Vibrotactile Speech Communication: Perceptual Studies with a Phonemic-Based Display

Tactile communication systems provide an alternative channel of communication for people with all levels of sensory capabilities and can help those with sensory impairments to receive information through another sensory modality. Recently, a TActile Phonemic Sleeve (TAPS) has been developed with the objective of enabling people to “hear through the skin." This thesis presents three studies that evaluate the feasibility of the TAPS system for phoneme and word acquisition and for two-way tactual communication. The TAPS system is based on a phonemic-based coding scheme that uses an array of 24 (6-by-4) tactors to convey haptic stimuli on the forearm. In Study 1, an effective mechanism for learning phonemes and words with TAPS based on the theory of memory consolidation was explored. Four naive participants learned to recognize 51 words made up of 10 phonemes within 60 min of experimental time. A fifth naive participant demonstrated the ability to learn all 39 phonemes of the English language as haptic codes after a total of 80 min with a phoneme identification score of 93.8%. We found that with the distinctive set of haptic symbols that had been developed prior to this thesis, participants were able to learn phonemes and words in a short amount of time. We also validated the memory consolidation theory by showing an improvement of phoneme recognition score when the fifth naive participant was re-assessed the day after he had learned the phonemes. In Study 2, we evaluated the learning performance of longer (four-phoneme) words. A total of three experienced participants spent 20 min per day for 3 days to review 39 phonemes and 500 words (with most of the words containing two or three phonemes) that they had learned in an earlier study. They then spent 10 more days to practice and test with a word list consisting of 100 four-phoneme words (List #1). A generalization study was conducted by testing the same participants with a different set of 100 four-phoneme words (List #2) during the last 2 days of the experiment. After the 15-day experimental period, the average word percent-correct (PC) scores of the three participants for List #1 and List #2 were 80.2% and 72.3%, respectively. Both results were well above the corresponding chance levels (1% for the closed set of words in List #1 and near 0 for the open set of words in List #2) which demonstrated that the participants were able to learn longer words with the TAPS system within a reasonable amount of time. In Study 3, the feasibility of TAPS for tactile communication of spontaneous speech was evaluated. Two of the three experienced participants from Study 2 sent text messages to each other through two identical TAPS systems with an open set of words. The average percent-correct (PC) scores for the two participants for messages (PC_msg) and words (PC_word) were 73.4% and 81.7%, respectively. These results are impressive considering that the participants had to recognize words and phrases using an open vocabulary. Overall, the three studies demonstrate that the users of the TAPS system can successively receive phonemes, isolated words up to 4 phonemes in length, phrases, and sentences in a two-way exchange that simulates daily communication scenarios. Future work will explore the design of additional haptic symbols for conveying punctuation marks and investigate the efficacy of the TAPS system in helping people with sensory impairments to communicate via the sense of touch.