Tactile Speech Communication: Design and Evaluation of Haptic Codes for Phonemes with Game-based Learning

This thesis research was motivated by the need for increasing speech transmission rates through a phonemic-based tactile speech communication device named TAPS (TActile Phonemic Sleeve). The device consists of a 4-by-6 tactor array worn on

the forearm that delivers vibrotactile patterns corresponding to English phonemes. Three studies that proceeded this thesis evaluated a coding strategy that mapped 39 English phonemes into vibrotactile patterns. This thesis corresponds to a continuation of the project with improvements summarized in two parts. First, a design and implementation of a training framework based on theories of second language acquisition and game-based learning is developed. A role playing game named Haptos was designed to implement this framework. A pilot study using the first version of the game showed that two participants were able to master a list of 52 words within 45 minutes of game play. Second, an improved set of haptic codes was designed. The design was based on the statistics of spoken English and included an additional set of codes that abbreviate the most frequently co-occurring phonemes in duration. The new set included 39 English phonemes and 10 additional abbreviated symbols. The new codes represent a 24 to 46% increase in word presentation rates. A second version of the Haptos game was implemented to test the new 49 codes in a learning curriculum distributed over multiple days. Eight participants learned the new codes within 6 hours of training and obtained an average score of 84.44% in symbol identification tests with error rates per haptic symbol below 18%. The results demonstrate the feasibility of employing the new codes for future work where the ability to receive longer sequences of phonemes corresponding to phrases and sentences will be trained and tested.