How social robots work

The basic loop

Almost every social robot runs some version of the same loop, many times a second:

1. Sense: cameras, microphones and sometimes touch or depth sensors capture what is going on.
2. Perceive: software turns raw data into facts: there are two people, the one on the left is speaking, she is looking at me, she sounds puzzled.
3. Understand: speech is turned into text and meaning, and combined with what the robot knows about the situation and the person.
4. Decide: the robot chooses what to say and do next, including whether to speak at all.
5. Express: speech is synthesised and coordinated with facial expression, gaze, head and body movement.

Each step is hard on its own. The real difficulty is doing them all together, fast enough. In human conversation the typical gap between one person stopping and the next starting is around a fifth of a second. A robot that takes two seconds to respond feels slow and awkward, however clever its answer.

Seeing people

Computer vision gives the robot most of its social awareness. Modern systems, almost all based on deep learning, can:

• detect faces and bodies, and track them as people move around;
• recognise individuals the robot has met before (where this is allowed and wanted);
• estimate head pose and gaze direction, to tell who is looking at the robot and who is looking away;
• estimate facial expressions and body posture;
• recognise gestures such as pointing, waving or a thumbs up.

Emotion recognition deserves caution. Software can detect a smile reliably, but a smile is not the same as happiness. Facial expressions vary by person and culture, and inferring inner feelings from the face is scientifically contested. Good systems treat these signals as hints, not facts.

Hearing people

Social robots usually have an array of several microphones. Comparing tiny differences in when a sound reaches each microphone lets the robot work out where a voice comes from, so it can turn towards the speaker and filter out other noise.

Speech recognition turns audio into text. It has improved enormously in the last decade, but robots face harder conditions than phones: the speaker may be a metre or more away, there may be several people talking, the robot's own motors make noise, and children's and older adults' voices are underrepresented in training data.

Voice activity detection and end-of-turn prediction decide when someone has stopped talking. This sounds trivial but is one of the biggest sources of awkwardness. Pause too short and the robot interrupts; wait too long and it seems slow. Humans predict the end of a turn from intonation, grammar and gaze before the speaker actually stops, and current research tries to give robots the same ability.

Managing conversation

For most of the field's history, robot conversation was carefully scripted. Designers wrote dialogue trees or state machines: if the user says something about the weather, respond with one of these lines. This was reliable and safe but brittle. Anything off-script led to "Sorry, I didn't understand".

Large language models (LLMs) changed this almost overnight. Since around 2023, robots can hold open-ended conversations on nearly any topic, in many languages, with a consistent personality. That has revived interest in social robots and made many old limitations disappear.

LLMs bring their own problems for robots:

• Latency. Generating a reply takes time. Systems stream the answer and start speaking before it is finished, and use filler behaviour such as a thinking gaze or "hmm" to cover gaps.
• Grounding. A language model does not know what the robot can see or do. It has to be fed a description of the scene and of the robot's abilities, and its output has to be connected to real actions safely.
• Reliability. LLMs can state false things confidently. In a care home or a classroom that matters more than in a chat window.
• Persona and boundaries. Designers must decide what the robot will and will not talk about, and keep its character consistent over long relationships.

Most serious systems combine the two: an LLM for open conversation, wrapped in rules and scripts for anything safety-critical, such as medication reminders or emergency handling.

Talking to groups

Talking to one person in a quiet lab is very different from talking to three people in a busy reception. In a group the robot has to track who is speaking to whom, decide who it is addressing, use gaze to signal that, and avoid talking over people who are speaking to each other. Multi-party interaction is one of the most active research areas, and it is where gaze and turn-taking become essential.

Expressing itself

Voice

Text-to-speech has become very natural, with control over speed, pitch and emotion. Designers face a choice: a fully human voice raises expectations the robot may not meet, while a slightly synthetic voice reminds people they are talking to a machine.

Faces

There are three main approaches. Mechanical faces use motors under a rubber skin, which can be very lifelike but are expensive and fragile. Screen faces show an animated face on a display, which is cheap and flexible but flat. Back-projected faces, as used by Furhat, project an animated face from inside onto a translucent mask, combining a three-dimensional shape with the flexibility of animation.

Gaze

Where a robot looks may be its most powerful social signal. Gaze shows who it is talking to, what it is paying attention to and whether it is listening. Looking away briefly before answering, as people do, makes a robot seem to be thinking. Looking at an object while talking about it helps people follow. Research consistently finds that good gaze behaviour makes robots seem more intelligent and likeable.

Gesture and movement

Head nods, tilts and body orientation add meaning and rhythm to speech. Even robots without arms use their head and torso. Movement must be legible: people should be able to tell what the robot is about to do before it does it.

Backchannels

When people listen, they nod, say "mm-hm" and react with small expressions. These backchannels tell the speaker they are being followed. Robots that produce them at the right moments are rated as much better listeners.

Memory and personalisation

A robot that forgets you every time you leave the room cannot form a relationship. Long-term social robots keep a model of each user: name, preferences, history of past conversations, goals, and how they like to be addressed. This is what makes a robot useful in therapy or eldercare, and it is also where privacy questions are sharpest.

Wizards behind the curtain

Many impressive robot demonstrations, and a large share of research studies, use a technique called Wizard of Oz: a hidden human operator controls some of the robot's behaviour while the participant believes it is autonomous. This is a legitimate research method, used to study how people respond to abilities robots do not have yet. But it means you should be careful when judging what a robot in a video can really do on its own.

The hardware

Most social robots are far simpler mechanically than industrial or humanoid robots. They rarely need to lift anything or walk. What matters is expressive movement in the head, eyes and face, quiet motors, good microphones and cameras, and enough onboard computing to run perception quickly. Heavier processing, such as language models, often runs in the cloud, which raises latency, reliability and privacy trade-offs.