Reading Lips in a Noisy Room: Why a Robot's Mouth Is Not Just Decoration

Seventy years of research show that we hear with our eyes as well as our ears, and that a well-animated robot face can lend a listener some of the same help a human face does.

The cocktail party problem, solved by looking

Anyone who has tried to follow a conversation in a crowded bar knows the trick without being taught it: you look at the speaker's mouth. When the acoustic signal is degraded, the visible movements of the lips, jaw and tongue tip fill in what the ears miss.

The classic demonstration came from William Sumby and Irwin Pollack in 1954. They asked listeners to identify words played against masking noise at a range of signal-to-noise ratios, sometimes with the talker's face visible and sometimes without. Seeing the face made a large difference, and the difference mattered most where it was needed most. In the noisiest conditions, words that were close to unrecognisable by ear alone became substantially easier to identify once the face was in view. Sumby and Pollack also noticed something subtle: when they expressed the visual benefit relative to how much room listeners had left to improve, it stayed roughly constant across noise levels. Vision, in other words, was not a last resort that switched on in emergencies. It was contributing all along.

When eyes overrule ears

If Sumby and Pollack showed that vision helps hearing, Harry McGurk and John MacDonald showed in 1976 that vision can actively rewrite it. In their now famous experiment, a recording of someone saying one syllable was dubbed onto video of a face articulating a different one. The textbook case pairs an auditory 'ba' with lips producing 'ga'. Many viewers report hearing neither: they perceive a blend, typically 'da'. Close your eyes and the sound snaps back to 'ba', even when you know the trick.

The McGurk effect matters because it rules out a tidy story in which the brain processes sound first and consults the face only when in doubt. Speech perception is multisensory from the start. For anyone designing a machine that talks, that is both an opportunity and a warning. A face that moves in step with the voice can boost understanding. A face that moves out of step, or in the wrong shapes, can actively confuse.

Can a synthetic face do the same job?

For decades the speech group at KTH Royal Institute of Technology in Stockholm has asked whether animated faces can deliver this benefit. One strand of that work, SynFace, used a speech recogniser to drive a talking head that showed lip movements for telephone audio, aimed at people with hearing loss. The results are instructive precisely because they are not uniformly glowing. A 2009 study by Samer Al Moubayed, Jonas Beskow and colleagues tested SynFace with hard-of-hearing participants at sites in three countries. Averaged across people, the face did not produce a significant improvement in the speech reception threshold. Looking at individuals, however, the variation was wide, and many participants did benefit, especially when the speech was buried in babble noise. Synthetic lip-reading support works for some listeners in some conditions, which is the honest version of the claim.

Other KTH work, summarised in Al Moubayed's 2012 doctoral thesis, showed that the face can help beyond the lips. In an intelligibility test with degraded sentences, adding head nods or eyebrow raises timed to prominent syllables made the talking head significantly easier to understand than one that moved only its mouth, with nods helping more than eyebrows. Visual prosody, the way faces mark emphasis, seems to carry information about the words themselves.

Lip-reading a robot

The obvious next question was whether this survives the jump from screen to robot. Furhat, developed at KTH, projects an animated face from behind onto a translucent mask shaped like that face. The mask does not move. The jaw is an illusion of light. Would people still be able to read its lips?

Al Moubayed, Gabriel Skantze and Beskow tested exactly this in a study published in 2012. Ten participants listened to short sentences with degraded audio while watching either the face on a flat screen or the same animation on Furhat, viewed from several angles. The back-projected face delivered an intelligibility gain at least as large as the flat screen, and in some conditions slightly larger. A static plastic shell, lit from within, was readable as a moving mouth.

That finding has a practical sting. Many social robots ship with stylised faces, glowing eyes or no mouth at all. Charming, perhaps, but such designs throw away one of the most reliable perceptual aids in human communication.

What designers should take away

Three cautions follow from the research. First, synchrony matters: the McGurk effect is a reminder that the brain fuses what it sees and hears, so lip movements that are late or wrong are worse than a neutral face. Second, benefits vary between listeners, as the SynFace studies make clear, so lip animation should be seen as support rather than a guarantee. Third, the whole face counts. Nods and eyebrow movements placed on the right syllables add information that lips alone do not carry.

The deeper lesson is that a robot face is not a skin stretched over the real machinery. For the listener it is part of the signal. Get it right and speech becomes easier to understand. Get it wrong and you are not just failing to help, you may be adding noise of your own.

References

1. Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26(2), 212-215. Link
2. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746-748. Link
3. Al Moubayed, S., Beskow, J., Öster, A.-M., Salvi, G., Granström, B., van Son, N., & Ormel, E. (2009). Virtual speech reading support for hard of hearing in a domestic multi-media setting. Proceedings of Interspeech 2009, 1443-1446. Link
4. Al Moubayed, S., Skantze, G., & Beskow, J. (2012). Lip-reading: Furhat audio visual intelligibility of a back projected animated face. Intelligent Virtual Agents (IVA 2012), Lecture Notes in Computer Science 7502, Springer. Link
5. Al Moubayed, S. (2012). Bringing the avatar to life: Studies and developments in facial communication for virtual agents and robots. Doctoral thesis, KTH Royal Institute of Technology. Link

Feature drafted with AI from the published research listed above, for social-robotics.com.