Pay No Attention to the Human Behind the Curtain: Wizard of Oz in HRI

Much of what we know about how people respond to social robots comes from robots secretly operated by researchers, a method that is indispensable, often underreported, and part of a wider reckoning about how reliable HRI findings really are.

A trick with a long history

Anyone who has worked in social robotics, or in magic, knows the value of a well-hidden assistant. In a Wizard of Oz study, participants believe they are interacting with an autonomous system, while in fact a human operator, the wizard, is listening and choosing some or all of the machine's responses from another room. The name comes from the scene in which the great and powerful Oz turns out to be an ordinary man pulling levers behind a curtain.

The technique predates social robots. In 1984 Kelley described an iterative design method for natural language office software in which a hidden human simulated the parts of the system that did not yet exist, then gradually handed over to real code as it was developed. The logic is sound: if you want to know how people will talk to a future system, you need to let them talk to something convincing now.

Why HRI leans on it

Robots are hard. Speech recognition in noisy rooms, understanding intent, judging when to speak and tracking several people at once are all difficult problems. If a researcher wants to study, say, how children respond to a robot that expresses empathy, waiting for a fully autonomous empathic robot could take years. A wizard lets the social question be studied today, with the engineering catching up later.

The trouble is that this convenience comes with methodological costs, and for a long time the field rarely talked about them.

What Riek found

In 2012 Laurel Riek published a systematic review of Wizard of Oz experiments in the first issue of the Journal of Human-Robot Interaction. She examined 54 studies from the main HRI venues between 2001 and 2011, assessing them against criteria drawn from earlier methodological work, including Kelley's.

The picture was sobering. Wizards were mostly used to handle language: about 72 per cent of studies used them for verbal processing and around 48 per cent for nonverbal behaviour. Most studies constrained what the wizard could make the robot do, typically through a fixed menu of responses, but only about one in ten constrained what the wizard was allowed to perceive. Very few reported measuring wizard errors, fewer than one in twenty reported training their wizards before the experiment, and only about a quarter used the technique iteratively, as Kelley had originally intended.

Each of these gaps matters. A wizard who can hear tone of voice and see facial expressions has perceptual abilities far beyond any robot of the time, so participants were in effect responding to a human-level perceiver dressed as a machine. Untrained or inconsistent wizards introduce variation that nobody measures. And if the paper does not explain any of this, readers cannot judge what the results actually mean for real autonomous robots. Riek proposed reporting guidelines to address exactly these problems.

A broader replication worry

Wizard of Oz is one thread in a larger knot. In 2015 the Open Science Collaboration attempted to replicate one hundred published psychology studies and found that many effects came out weaker or failed to reappear. Because HRI borrows heavily from social psychology, the shock travelled.

Irfan and colleagues described in 2018 their own unsuccessful attempts to reproduce established social facilitation effects with robots. They argued that HRI had been too ready to accept results from psychology, and too slow to develop reference tasks of its own. Two years earlier, Baxter and colleagues had characterised more than a hundred full papers from three years of the HRI conference, and proposed methodological and reporting recommendations, drawn from more rigorous practice in other fields, that had not yet become common in HRI.

Honest puppetry

None of this means Wizard of Oz should be abandoned. It remains one of the most useful tools for exploring interactions that are not yet technically possible, and for prototyping before investing in expensive engineering. The point is to be explicit about it: what the wizard controlled, what they could perceive, how they were trained, how often they erred and how consistent they were across participants.

There is also a fresh twist. Large language models now make it possible to automate many functions that once required a wizard, but they bring their own invisible variability, from changing model versions to unpredictable outputs. The curtain has not disappeared; it has simply moved. Reporting what sits behind it, whether a person or a model, is what turns a convincing demonstration into a finding other people can build on.

References

1. Riek, L. D. (2012). Wizard of Oz Studies in HRI: A Systematic Review and New Reporting Guidelines. Journal of Human-Robot Interaction, 1(1). Link
2. Kelley, J. F. (1984). An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems, 2(1). Link
3. Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science. Link
4. Irfan, B., Kennedy, J., Lemaignan, S., Papadopoulos, F., Senft, E. and Belpaeme, T. (2018). Social Psychology and Human-Robot Interaction. Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction. Link
5. Baxter, P., Kennedy, J., Senft, E., Lemaignan, S. and Belpaeme, T. (2016). From characterising three years of HRI to methodology and reporting recommendations. 11th ACM/IEEE International Conference on Human-Robot Interaction. Link

Feature drafted with AI from the published research listed above, for social-robotics.com.