Three's a Crowd: What Happens When a Robot Joins the Group

Most dialogue technology assumes one user and one machine, but real social robots meet families, visitors and colleagues, and a group changes everything.

The dyad assumption

Almost every voice interface you have used is built for a pair: you and the machine. You speak, it answers. That assumption is baked deep into the design, from the wake word to the single stream of transcribed text.

Social robots rarely live in that world. A robot in a museum lobby, a hospital reception or a classroom meets people in twos and threes. They arrive and leave mid-conversation, talk to each other as well as to the robot, and sometimes talk about the robot in its presence. Each of these situations raises questions that a one-to-one system never has to answer. Who is speaking? Who are they speaking to? Is this remark for me, or for their friend? And when the robot speaks, who should it be addressing?

Engagement before conversation

Before a robot can take part in a group conversation, it has to work out who is in the group at all. Dan Bohus and Eric Horvitz at Microsoft Research tackled this in a series of systems built for what they called open-world dialogue. Their 2009 models broke the problem into three parts: sensing the engagement state and intentions of each person nearby, deciding whom to engage with and when, and carrying out those decisions through coordinated behaviours in an embodied agent.

Their systems were deployed for real rather than only in the lab. One long-running example, known as the Assistant, sat outside Horvitz's office and helped visitors coordinate meetings with him, dealing with whoever turned up, alone or in groups. Working in the wild made one thing obvious: people approach, hover, chat with each other and drift away, and a robot that cannot track these shifts will either ignore people who want its attention or pester those who do not.

Footing and addressees

The sociologist Erving Goffman gave us a useful vocabulary for this: footing, the stance and role each person takes in an interaction. In a group, some people are addressed directly, some are ratified participants who are not currently addressed, and others are bystanders or overhearers. These roles shift constantly, and they are signalled mostly through gaze, body orientation and the content of what is said.

Robots can both read and shape footing. As described in the previous article on gaze, Bilge Mutlu and colleagues showed in 2009 that the Robovie robot could cast people as addressees, bystanders or overhearers using gaze alone, and that people's participation followed the roles they had been given. Bohus and Horvitz found a similar steering effect with their agent: when it released the floor to one of two participants during confirmations, that person spoke first in about 86 per cent of cases. In groups, attention is a lever, and robots can learn to pull it.

Furhat in the museum

Some of the richest multiparty data comes from KTH in Stockholm, where the Furhat robot head was developed. Furhat's back-projected face can show fine shifts of gaze, which makes it well suited to group settings where it matters exactly whom the robot is looking at.

In one project, Furhat was exhibited at the Swedish National Museum of Science and Technology, playing a collaborative card-sorting game with two visitors at a time. The players discussed the cards with each other and with the robot, which meant their gaze was split between partners, robot and objects. Martin Johansson and Gabriel Skantze used the recordings to build a data-driven model that detects when the robot has an opportunity, or an obligation, to take the turn. The key was combining cues that signal whom a remark is aimed at with cues that signal a turn is ending, drawing on voice activity, syntax, prosody, head pose, card movements and dialogue context. The best results came from combining several modalities, reaching a weighted F1 score of about 0.88 on an interaction the model had not seen before, using only features that could be extracted automatically.

Why groups are hard

The museum study captures the central difficulty of multiparty interaction. In a pair, a turn ending usually means it is your turn. In a group, a turn ending is only half the information; you also need to know whether it was yielded to you. Gaze, the most obvious addressee signal, is also needed for looking at whatever is being discussed. Speech recognition becomes harder with overlapping voices, and simply knowing which person said what requires sound localisation and visual tracking working together.

There are social subtleties too. A robot that always answers the loudest person, or that keeps addressing the same individual, can quietly exclude others. Group interactions involve relationships between the humans that the robot cannot see: a parent and child, a manager and a new employee. Gabriel Skantze's 2021 review of turn-taking in human-robot interaction treats multiparty settings as a distinct challenge in their own right, not simply a harder version of the one-to-one case.

The prize

Despite the difficulty, multiparty interaction may be where social robots earn their keep. A screen-based assistant can serve one person; an embodied robot with a head and eyes can moderate a discussion, draw a quiet participant in, or manage a queue of visitors with a glance. Those are capabilities that only a physical, socially aware presence can offer. Getting there means treating the group, not the individual, as the basic unit of conversation.

References

1. Bohus, D. and Horvitz, E. (2009). Models for multiparty engagement in open-world dialog. Proceedings of SIGDIAL 2009, 225-234. Link
2. Bohus, D. and Horvitz, E. (2010). Facilitating multiparty dialog with gaze, gesture, and speech. Proceedings of ICMI-MLMI 2010. Link
3. Mutlu, B., Shiwa, T., Kanda, T., Ishiguro, H. and Hagita, N. (2009). Footing in human-robot conversations: how robots might shape participant roles using gaze cues. Proceedings of HRI 2009, 61-68. Link
4. Johansson, M. and Skantze, G. (2015). Opportunities and obligations to take turns in collaborative multi-party human-robot interaction. Proceedings of SIGDIAL 2015, 305-314. Link
5. Skantze, G. (2021). Turn-taking in conversational systems and human-robot interaction: a review. Computer Speech and Language, 67, 101178. Link

Feature drafted with AI from the published research listed above, for social-robotics.com.