Tactile Communication System for the Interaction between Deafblind and Robots

This paper presents a touch-based system for communication by deafblind people with humans and robots. With presented system the deafblind people can send control signal to robots by writing block letters on an array of 8 × 8 diamond-shaped capacitive touch sensors. For better conformability with curved surfaces, the touch sensors arrays have been fabricated on flexible polyvinyl chloride (PVC) substrate. The characters written by deafblind on the tactile sensing array are recognized by Convolution Neural Network (CNN), which is trained with data from Extended Modified National Institute of Standards and Technology (EMNIST) database. A mobile app was also developed to efficiently recognize and display the written letters. Finally, a Baxter robot was simulated using Robot Operating System (ROS) and was remotely controlled by deafblind through English block letters written on the tactile sensing array. The result shows that the robot could successfully execute the `pick or drop' task. The developed system demonstrates the efficacy of tactile based communication by deafblind people to independently interact with a robot and their environment.