BEYOND HEARING

Overview
“Unleashing Deaf people’s hearing superpower”

Beyond Hearing is an augmented reality interface, localising sound with a multi-sense feedback system to improve Deaf people’s safety, communication and daily sound engagement.

Deaf people lacks the ability of sound localisation which might lead to dangerous accidents. Besides, car flashing by without having any sound cue increases mental pressure when Deaf people go out. Beyond Hearing assists them with visual and haptic signals to solve this problem.
Taiwan and the US patent pending.

Role
Solo project

Duration
2022 Jan. - 2025. Aug. (3 year 7 month)

Category
Interaction Design
Inclusive Design
AR

PROBLEM

My sister was traumatised and suffered from hearing loss since the year 2021. She lost her right hearing ability which obstructs her from identifying the source of the sound correctly.

448M people in the world have defects with hearing ability. Sound localisation and engagement for Deaf people are the major problems in their daily life. 80% of my interviewees from the international Deaf community mentioned these issues limit their activities and sound experiences.

USER RESEARCH

To understand more about hard of hearing people needs, the quantitative and qualitative survey have been done. In the interview with Deaf community, 80% of them point out sound localisation is the major problem in their daily life. The safety and communication problems are the two category they cared most.

PRIOR ARTS

To differenciate this project to other exist projects, the prior arts table was made.
It shows that currently there is no great solution for multi-senses and multi-dimension sound localisation feedback in the martket.

OPPORTUNITY: Sound Localisation

There’s no a mature product that tackles this problem accurately. Existing similar products aim to provide amplified sound or transfer dialogue into captions. Both of them didn’t provide a solution to identify the localisation of sound.

EXPERIMENT:

What is the better way to present sound Localisation? 

To Answer this quesion, the following experiments were conducted. Including shape of sound, sound tatto, meta-balls and AR glasses interface.

CONCEPT: AR Glasses for Deaf people localising sound

‘How might I use Argument Reality for them feel where the sound come from and create a safe environment for deaf/HoH people (even blind space)?’

Inspired by a video game I played. When the enemy approaches or an emergency situation happens, the game interface will give corresponding visual cues.

PROTOTYPE TEST I

To test if the AR glasses interface is suitable for sound localisaion, I made a prototype with sound sensors on the sides, and with LED lights to distinguish the direction of it. The prototype can successfully distinguish the sound from left or right. After testing with cohorts, the feedback was great. So, I continued.

TECHNOLOGY  |  Sound localisation

To detect the sound for all 360 degrees, the next prototype I built was with the microphone arrays. After collecting and analyzing the sound signal, I send the information to the display. Therefore, the visual cues will appear on the OLED screen if the sound is from the sides or in the front of the user. The motor will also provide a haptic vibration if the sound comes from the back.

DESIGN  |  Sound-Visual Language Design

The visual cues can not only show the localisation of sound, but also can display different kind of sounds. By analysing various frequency bands to distinguish differnet kinds of sound. For example, indication of which person is talking to you in crowd conversation, and car honking on the side. This allow Deaf people incrasing the environmental sound awareness.

USER VALIDATION

This prototype tested by 5 hearing loss people and 100+ hearing people. The feedback is generally positive. One Deaf person said that’s what she needs. Another one suggested that I could design the on/off mode because if it’s always on, the visual information might overwhelm people.

USER SCENARIO

1. Walking on the street
Users will get notification while there’s a car/bike honking or passing by.

2. Communication
The OLED screen indicates the user the speaker in a conversation, which assists the user to switch their focus on them.

3. Noisy Environment
If someone is using dangerous tools nearby, the interface will keep the user aware.

OUTCOME

Beyond Hearing provides a safer environment, easier communication for Deaf People in their daily sound engagement. With this interface, the inclusiveness for approaching sound has been created. Deaf people can intuitively know the location of sound, experience it with visual and tactile cues. At the same time, this multi-senses feedback system will potentially unleash people's hearing superpower. Beyond Hearing increases the awareness and the capability of noticing subtle sounds which are normally ignored.

This project is still ongoing! Looking forward to sharing with you about more detail in the future!

Special Thanks:

Dr. Lorenzo Picinali
Audio Experience Design Lab Lead
Imperial College London

Audrey Gaulard
Innovation Design Engineering 2nd year Lead
Imperial College London

Dr. Elena Dieckmann
Innovation Design Engineering 2nd year Lead
Imperial College London

Iulia Ionescu
Visiting tutor
Royal College of Art

Matt Johnson
Visiting tutor
Royal College of Art

Exhibition:

2022 RCA show (Jun. 26th-30th 2022)
2022 Imperial College London Summer show (July 1st-7th 2022)
2024 Reddot Award Museum Exhibition

Press:
RCA2022 featured project
Nike waffle shoes featured project

Award:
D&AD side hustle pencil 2023
Core77 design award student winner 2023
Red dot design concept shortlist 2023
IDA Official Selection 2022
IDA Honorable Mention 2022
IDA Awards (2022)

Global Student Project Competition
Winner of the year
RS grass root  (2022)

Global Design Graduate Show in collaboration with GUCCI Shortlisted
ARTS THREAD (2022)