Huawei Launches StorySign App that Uses AI and Augmented Reality to Help Deaf Children Read

Currently, the app is only available in select sign languages and can only translate one book, but the company hopes to add support for more books in the future.


While Huawei continues to make strides in the smartphone world, the Chinese smartphone maker is also aware of its social responsibilities, and its latest app for the deaf children is a testimony to that. Called StorySign, the app leverages artificial intelligence and augmented reality to help deaf children interpret the words on a storybook and enjoy an immersive reading experience without the help of an adult.

A Gift From Huawei To The 32 Million Deaf Children In The World

Huawei partnered with the nonprofit European Union for the Deaf, animation experts Aardman, and the publisher Penguin to develop the app. It employs AI tools like optical character recognition (OCR) and image recognition to translate storybooks into sign language. This would break the language barrier that currently exists, as nearly 90 percent of the deaf children are born to hearing parents. As sign language usually varies according to the country, the app first asks you to select one. Currently, sign language in British English, German, French, Italian, Dutch, Spanish, Irish, Portuguese, Swiss-German, and Belgian Flemish are supported. Once that is done, the app can be hovered over a book for the translation process to begin.

To make the storytelling session more immersive, a virtual avatar called Star would translate the words in sign language in real time. Since sign language doesn’t translate directly into any given language and the order changes, each word is highlighted as it is being gestured. This would make it easier for kids to associate words with signs. The word-by-word translation process would greatly enhance the learning experience because as compared to hearing children, deaf kids do not have the advantage of using the sound-based phonics system which supplements the learning process. The app can recognize words even when a phone is positioned at an awkward angle and since the AI engine controls the loading speed, there would be no lag time in the app.

There Is Still A Long Way To Go

The app can now be downloaded for free from Google Play and Huawei’s AppGallery. It is currently available in 10 European countries but unfortunately, it is only compatible with the book ‘Where’s Spot’ right now. However, Huawei is working with Penguin to increase the library of titles in the future. The company isn’t sure whether the app would make it to iPhones. Although StorySign works on all phones running Android 6.0 and over, it is optimized for Huawei’s AI-enhanced smartphones such as the Huawei Mate 20 Pro.

With the app, Huawei wants to make an impact in the deaf community by helping the deaf children read at the same level as kids who can hear. The company also hopes that the app will spark a wider conversation about equal opportunities for deaf individuals across Europe. Moreover, Huawei also believes that the app will increase awareness about deaf literacy issues and encourage people to support charities that work for the deaf community. Alongside the app, Huawei has also started the StorySign campaign which will collect donations for the BDA projects for deaf children. The collected money would also be invested in the future development of StorySign.

Find out more about StorySign in the video below: