The Smile Is Not the Feeling: What Robot Faces Can and Cannot Tell Us About Emotion

Robots have been pulling expressive faces since the 1990s, but the science of what a face actually reveals about emotion has turned out to be far messier than the cartoon version.

Kismet and the readable machine

In the late 1990s, at the MIT Artificial Intelligence Lab, Cynthia Breazeal built a robot head called Kismet. It had large eyes, mobile eyebrows, ears that could perk up or droop, and rubbery lips. It could not walk or grasp anything. Its purpose was purely social: face-to-face exchange of the kind a caregiver has with an infant.

Breazeal's 2003 paper in the International Journal of Human-Computer Studies laid out the logic. Kismet's expressions were not a fixed menu of six faces. Instead, its internal state was placed in a three-dimensional affect space defined by arousal (how energised), valence (how positive or negative) and stance (whether it was inclined to approach or withdraw). A set of prototype poses anchored the corners of this space, and Kismet's face blended between them as its state shifted. The result was a face that could slide from interest to boredom to mild alarm rather than flicking between stock emotions.

Kismet also went the other way. Following research on how adults talk to babies, it was built to classify the melody of a person's voice into broad categories such as praise, prohibition, soothing and bids for attention, and to respond with a fitting expression. People who scolded it saw it look crestfallen. Praise brought a brighter face.

The tidy theory behind many robot faces

Kismet's design drew on a long tradition in emotion research, and so did a great deal of the robotics and affective computing that followed. The popular version of that tradition holds that there is a small set of basic emotions, each with a characteristic facial configuration that people everywhere produce and recognise: the scowl of anger, the wide eyes of fear, the smile of happiness. If that is true, building an emotional robot face is a solved design problem. Implement the canonical expressions and people will read them.

The 2019 reckoning

In 2019, Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix Martinez and Seth Pollak published a long review in Psychological Science in the Public Interest that put this assumption under serious strain. They examined evidence on how people actually move their faces when experiencing emotions, and how observers infer emotion from faces, across adults, children and people from different cultures.

Their conclusion was not that faces are meaningless. It was that the link between a specific facial configuration and a specific emotion is far less reliable and specific than commonly believed. People do scowl when angry more often than chance, but often they do not, and they also scowl for reasons unrelated to anger, such as concentrating hard. Smiles can mean happiness, embarrassment, appeasement or politeness. Context, culture and the individual matter a great deal. Recognition studies that seem to show universal reading of expressions often rely on posed faces and multiple-choice labels, and agreement drops when people must label faces freely. The authors suggested that researchers talk about facial configurations rather than emotional expressions, to avoid smuggling in the conclusion.

Other work adds texture. Rachael Jack, Philippe Schyns and colleagues used a computer graphics platform that generated large numbers of animated facial movements, and recorded which ones observers judged as fitting each emotion, to reconstruct the mental templates that observers hold for six emotions. Western participants' templates clustered into six distinct groups. East Asian participants' templates did not separate as neatly, and they signalled emotional intensity partly through dynamic movements around the eyes. Not everyone accepts the strongest readings of such findings, and the review drew pointed commentary from researchers who see more structure in facial behaviour than Barrett and colleagues allow. But the debate has clearly moved on from the cartoon.

What this means for robots

Interestingly, the critique cuts differently for robots than for emotion recognition software. A robot is not experiencing anything, so there is no hidden emotion to misread. Its face is a communicative display, designed to be interpreted, more like an actor's performance than a symptom. Posed, exaggerated, stereotyped expressions are exactly what the recognition studies show people can match to labels, so a robot that uses them will often be read as intended, at least by people who share the designers' cultural reference points.

The risks lie elsewhere. First, stereotyped faces may travel poorly across cultures, which matters for products sold worldwide. Second, a face divorced from context can feel hollow: humans read expressions together with voice, posture and situation, so a smile that ignores what was just said may land as creepy rather than warm. Third, designers working on the input side, building robots that try to detect a user's mood from their face, inherit all the reliability problems the 2019 review describes.

Design choices themselves are wide open. A 2018 survey by Alisa Kalegina, Maya Cakmak and colleagues at the University of Washington catalogued 157 robots with faces rendered on screens and found enormous variety in realism and detail, with people's preferences shifting depending on the job the robot was meant to do. There is no single correct emotional face.

The sober lesson from Kismet onwards is that robot expressions work best as conversation, not diagnosis. A face that responds sensibly to what is happening gives people something to read. A face that claims to know what they feel is making a promise the science cannot yet keep.

References

1. Breazeal, C. (2003). Emotion and sociable humanoid robots. International Journal of Human-Computer Studies, 59(1-2), 119-155. Link
2. Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019). Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest, 20(1), 1-68. Link
3. Jack, R. E., Garrod, O. G. B., Yu, H., Caldara, R., & Schyns, P. G. (2012). Facial expressions of emotion are not culturally universal. Proceedings of the National Academy of Sciences, 109(19), 7241-7244. Link
4. Kalegina, A., Schroeder, G., Allchin, A., Berlin, K., & Cakmak, M. (2018). Characterizing the design space of rendered robot faces. Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, 96-104. Link

Feature drafted with AI from the published research listed above, for social-robotics.com.