The Mona Lisa Problem: Why Flat-Screen Faces Cannot Look at Anyone in Particular

A face on a screen seems to look at everyone in the room at once, which is a party trick for a portrait and a serious bug for a robot that needs to choose who speaks next.

A very old illusion

Walk past a good portrait and the eyes follow you. The Mona Lisa is the famous example, but the effect is far older than Leonardo and far more general than eyes. A recruiting poster with a pointing finger aims at you wherever you stand. So does a face on a television, a video call or a tablet strapped to a robot.

Psychologists and artists have offered explanations for centuries. The clearest framing for engineers came from Samer Al Moubayed, Jens Edlund and Jonas Beskow at KTH in a 2012 paper in ACM Transactions on Interactive Intelligent Systems, playfully titled Taming Mona Lisa. Their argument is that we treat a flat picture as a window onto its own virtual space, with its own forward direction pointing straight out of the surface. Because we perceive the shapes in a picture as stable even when we view it from an angle, a face drawn looking straight ahead keeps looking straight out at us as we move. Every observer gets the same view, so every observer feels addressed.

Why this breaks conversation

For a single user standing in front of a kiosk, this hardly matters. If anything it is flattering. The trouble begins when two or more people share the space. Human conversation leans heavily on gaze to manage who talks. Speakers look at the person they are addressing, glance away while planning, and look back to hand over the turn. If a screen face looks straight ahead, everyone feels looked at. If it turns its eyes 20 degrees to its left, everyone sees it looking to the same relative side, regardless of where they actually are. Exclusive eye contact with one person becomes impossible.

The KTH team spelled out the implication: on a flat display the face either meets everybody's eyes or nobody's. For a robot receptionist, a tutor working with a small group or a game host, that is not a cosmetic issue. It removes one of the main channels people use to coordinate.

Testing it with five chairs

Their experiment was elegant. Five people sat in an arc facing an animated face. The face looked in various directions, and each person wrote down whom it seemed to be looking at. Seats were rotated so everyone sat in every position. The same face was shown in two ways: on a flat surface, and projected onto a physical mask shaped like a human head.

The predictions were sharp. On the flat surface, all five observers should see the gaze relative to themselves, so a frontal gaze would be claimed by everyone at once. On the 3D mask, everyone should agree on a single target, as they would with a real person. That is close to what happened. The projected 3D face almost completely removed the Mona Lisa effect and gave viewers a consistent, absolute sense of where it was looking. The data also let the team fit functions mapping the face's eye rotation to perceived direction in the room, which made it possible to calibrate gaze for real interaction.

From perception to turn-taking

Perception is one thing, but people are clever. Perhaps in a real conversation they could work out whom a flat face meant, even if it looked wrong. Al Moubayed and Gabriel Skantze tested this in a follow-up presented in 2011. Again five people sat in front of the face, but this time it asked them questions, and the only cue to the intended addressee was its gaze.

According to Al Moubayed's thesis summary, with the flat display the right person took the turn only about half of the time, against around 84 per cent with the 3D projection. Half is better than the one-in-five rate you would get by guessing, so people do compensate somewhat. But the compensation had a cost: responses were roughly half a second slower with the flat face. Participants also rated the 3D face as more human-like and its eyes as easier to read.

Furhat and the case for curved faces

These results led directly to Furhat, a back-projected robot head described by Al Moubayed, Beskow, Skantze and Björn Granström in 2012. An animated face is projected from inside onto a translucent mask moulded to the same shape as the face model, and the head sits on a motorised neck. The aim was a hybrid: the expressive flexibility of computer animation with the physical directionality of a real head. Early public tests included a multiparty dialogue system shown at the London Science Museum, where Furhat talked with two visitors at once and used head turns and gaze to regulate who spoke.

There are other ways out of the Mona Lisa trap. Mechanical eyes in a physical head work, as does a flat screen mounted on a neck that turns, at least for coarse direction. The broader point is that direction is a property of objects in space, not of pictures. A screen can depict a glance. Only something with a real shape can aim one.

For designers, the takeaway is blunt. If your robot is meant to deal with more than one person, look hard at how it signals attention. A flat face with beautiful eyes may be quietly lying to everyone in the room.

References

1. Al Moubayed, S., Edlund, J., & Beskow, J. (2012). Taming Mona Lisa: Communicating gaze faithfully in 2D and 3D facial projections. ACM Transactions on Interactive Intelligent Systems, 1(2), Article 11. Link
2. Al Moubayed, S., & Skantze, G. (2011). Turn-taking control using gaze in multiparty human-computer dialogue: Effects of 2D and 3D displays. Proceedings of AVSP 2011, 99-102. Link
3. Al Moubayed, S., Beskow, J., Skantze, G., & Granström, B. (2012). Furhat: A back-projected human-like robot head for multiparty human-machine interaction. In Cognitive Behavioural Systems, Lecture Notes in Computer Science 7403, 114-130. Springer. Link
4. Al Moubayed, S. (2012). Bringing the avatar to life: Studies and developments in facial communication for virtual agents and robots. Doctoral thesis, KTH Royal Institute of Technology. Link

Feature drafted with AI from the published research listed above, for social-robotics.com.