HearMe
Visual Based Learning Tool for Hearing-Impaired Children
Hearing Loss in Children
According to the WHO, approximately 466 million people worldwide
have disabling hearing loss, which represents about 6.1% of the
world population. [1] Among those people, most of the people
with hearing impairments can hear after getting an implant or
after starting to use a hearing aid. But those people still tend
to use sign language for communication because they do not have
an improved vocabulary. Vocabulary development is mostly
affected in between the age of 3 to 6 years. [2] Which is why a
customized tool for hearing impaired people who are recovering
is needed.
The aim of this research is to develop a learning tool to
provide the ability to use visual based materials from the
surrounding environment to improve vocabulary in early ages.
CEHIC worked in collaboration with us the entire period of the
research and the tool is developed under their supervision.
Furthermore, this tool has four main areas of development. It
will initially discover the level of the student each day when
they try to use the application. Then the student can explore
what’s around the child through the camera, which enables
seamless and effective learning experience for the child to
interact with his surrounding environment and learn based on
them, after that it generate meaningful sentences and phrases
from the objects that observed previously and the words they
have learnt. Finally, lip reading training which basically
involves with training lip movements for the alphabet and that
gives visual feedback while monitoring the mouth movements of
the child.
Detect the level of the child when they start to use the application using a pre trained model developed using Random Forest Algorithm and a dataset gathered from hearing impaired children and then measuring the progress of the child each time they visit the application
Exploring objects in the surrounding environment and teach the student in order to develop their vocabulary
Introduce new words to the child based on other words they learn in the application in order to enhance their vocabulary
Detect the accuracy of the pronunciation of the child using lip movement detection and voice detection
The proposed application solution has 4 main components. They are,
Here illustrates the overall system diagram of the proposed solution which was intended to provide a smart approach for stakeholders, researchers, and the contributors from CEHIC who were working with us in collaboration. Using a mobile app with gamified tasks and explorative learning this application proposes a method which can improve language and cognitive abilities of a hearing-impaired children who are recovering. It was observed that all the models perform with performance with better than 84% accuracy. The application should be used in the field for many years to gather its real impact on the students. The research work is done based on English language and further extension can be taken to do the enhanced the visual learning to other languages. Currently the models are compatible with mobile versions and further enhancement related to increase the compatibility of mobile phones can be done. The learning process is currently done with the supervision of a teacher on premises. But further improvements can be made to accommodate remote learning.
,
