Mm-hm, Go On: The Tiny Noises That Make Listening Visible

Backchannels look like filler, but they shape what speakers say, and teaching robots to produce them well turns out to be a surprisingly deep problem.

The listener is not silent

We tend to picture conversation as one person talking while the other waits. In reality, the listener is busy. They nod, murmur 'mm-hm', say 'yeah' or 'right', raise their eyebrows, smile at the funny bit and wince at the painful one. These signals, known as backchannels, do not claim the floor. They travel on a side channel, telling the speaker 'I am with you, carry on'.

Because they are small and frequent, backchannels are easy to dismiss as noise. They are anything but. Remove them and conversation starts to feel like talking into a void. Anyone who has spoken on a video call to a frozen face knows the creeping unease of not knowing whether anyone is there.

Listeners as co-authors

A memorable demonstration came from Janet Bavelas, Linda Coates and Trudy Johnson in 2000. They asked people to tell stories of personal close calls to listeners, and in some conditions they quietly gave the listeners a distracting task. Distracted listeners produced fewer responses, especially the specific ones that react to the content of the story, such as wincing at the right moment, as opposed to generic nods and 'mhm's.

The striking finding was what happened to the narrators. Speakers telling their story to a distracted audience told it noticeably less well. The listener's responses were not decoration; they were part of how the story got told. Bavelas and colleagues described listeners as co-narrators, and the label captures something important. Conversation is built by both parties, even when only one of them seems to be talking.

When to say mm-hm

If backchannels matter, the next question is how listeners know when to produce them. A badly timed 'mm-hm' can feel like an interruption, while one that comes too late can seem distracted. Research by Nigel Ward and Wataru Tsukahara, published in 2000, looked for prosodic cues that invite backchannel responses in both English and Japanese. They argued that one important cue in both languages is a stretch of low pitch late in the speaker's utterance, which seems to work as an invitation for feedback.

Such findings gave dialogue system builders their first practical recipes: listen for certain pitch contours or pauses, then respond. Rule-based approaches like this can work surprisingly well, but they are brittle. Real speakers combine many cues at once, and the right moment depends on content as well as sound.

Robots that listen out loud

Social robots have been used to test whether backchannels change how people feel about a machine. In 2017 Hae Won Park and colleagues at the MIT Media Lab studied children aged four to six telling stories to two robots called Tega. One robot used a model that detected backchannel opportunities from prosodic features in the child's speech and responded at those moments; its twin did not respond contingently. The responsive robot was seen as a more attentive listener, and children preferred telling stories to it.

In Japan, Tatsuya Kawahara's group at Kyoto University built an attentive listening system for the android ERICA, which produces backchannels alongside repeats, follow-up questions and short assessments. In a 2020 study with 20 older adults, the autonomous system was compared with a version operated by a hidden human. It performed comparably on basic listening skills, such as being easy to talk to and appearing to listen seriously. It fell short on higher-level qualities: understanding the conversation, showing interest and expressing empathy. That gap is telling. Producing well-timed noises is now within reach; making them mean something is harder.

From rules to prediction

The newest work treats backchannels as part of the same predictive problem as turn-taking. In a paper accepted at NAACL 2025, Koji Inoue, Divesh Lala, Gabriel Skantze and Kawahara fine-tuned a Voice Activity Projection model, first trained on general conversation, to predict both the timing and the type of backchannels continuously and in real time. Working on Japanese dialogue data, they reported that it outperformed baseline methods on both tasks.

Type matters as much as timing. A neutral 'mm' and an enthusiastic 'oh!' do different work, and a robot that only ever produces one flavour quickly sounds like a metronome.

The ethics of the nod

There is a slightly uncomfortable side to all this. Backchannels signal attention and understanding, and a robot can produce them without either in any deep sense. Used well, they make interactions smoother and help people express themselves, which is valuable in settings such as reminiscence for older adults. Used carelessly, they risk creating an illusion of empathy that the system cannot back up.

The honest goal is not to fake listening but to make a machine's actual processing visible in a way people can read. A good backchannel should mean 'I have registered that', and ideally it should be true.

References

1. Bavelas, J. B., Coates, L. and Johnson, T. (2000). Listeners as co-narrators. Journal of Personality and Social Psychology, 79(6), 941-952. Link
2. Ward, N. and Tsukahara, W. (2000). Prosodic features which cue back-channel responses in English and Japanese. Journal of Pragmatics, 32, 1177-1207. Link
3. Park, H. W., Gelsomini, M., Lee, J. J. and Breazeal, C. (2017). Telling stories to robots: the effect of backchanneling on a child's storytelling. Proceedings of HRI 2017, 100-108. Link
4. Inoue, K., Lala, D., Yamamoto, K., Nakamura, S., Takanashi, K. and Kawahara, T. (2020). An attentive listening system with android ERICA: comparison of autonomous and WOZ interactions. Proceedings of SIGDIAL 2020, 118-127. Link
5. Inoue, K., Lala, D., Skantze, G. and Kawahara, T. (2025). Yeah, Un, Oh: continuous and real-time backchannel prediction with fine-tuning of Voice Activity Projection. NAACL 2025 (arXiv:2410.15929). Link

Feature drafted with AI from the published research listed above, for social-robotics.com.