Interactive sonification of images in serious games as an education aid for visually impaired children
Loading...
Date
2019
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
The paper presents an application for interactive sonification of images intended for use on mobile devices in education of blind children at elementary school level. The paper proposes novel sonification algorithms for converting colour and grayscale images into sound. The blind user can interactively explore image content through multi-touch gestures and select image sub-regions for sonification with real-time control of the synthesized sound parameters such as amplitude, frequency and timbre. Additionally, images may contain text fields read by text-to-speech synthesizer. The usability of one of the proposed sonification schemes is tested by collecting data on the tracking accuracy and recognition speed of basic shapes, such as lines, curves, as well as figures and simple functions. In order to facilitate the learning process for blind children, a number of games were proposed that use the devised sonification schemes. The first game-"Hear the Invisible"-is intended for two players-one child draws a shape and the task of the other one is to guess the displayed shape by means of the available sonification methods. The second proposed game is "Follow the Rabbit" and is intended for a single player who tracks a colourful "Rabbit" that runs along a path representing a given geometric shape. The obtained results show that the proposed sonification methods allow to reach previously unattainable levels of flexibility in exploration of shapes and colours. The main application of the described interactive tool is to teach geometry and mathematics in schools for blind children. The developed games are meant to enhance the learning process and motivate the children.
Description
Subject(s)
system, trials