Following the Robot into the Dark: Overtrust and the Problem of Calibration

When a machine sounds confident and the corridor fills with smoke, people will follow it past a clearly marked exit, which tells us that the real design problem is not building trust but keeping it honest.

The fire that was not a fire

In 2016 a team at Georgia Tech led by Paul Robinette, with Alan Wagner and Ayanna Howard, ran one of the most quoted experiments in human-robot interaction. Volunteers, mostly students, were told they were taking part in a study about following directions. A brightly coloured machine labelled as an emergency guide robot led each of them to a meeting room. For some participants the trip went smoothly. For others the robot took a wrong turn, circled about, or simply stopped, at which point an experimenter announced that it had broken down.

Then the real test began. While the participant sat in the closed room, artificial smoke filled the corridor and a smoke alarm went off. Stepping outside, they found the robot, now lit up in red, pointing with its arms towards an exit at the back of the building rather than the clearly marked door they had come in through. In the main experiment reported in the conference paper, all 26 participants followed it. Half of them had watched the same robot bungle a simple navigation task minutes earlier. In follow-up conditions, the robot pointed towards a darkened room with no visible way out, partly blocked by furniture, and most people still did not turn back to the entrance they knew.

Why would anyone do that?

The researchers offered a cautious reading. Under time pressure and alarm, a machine that presents itself as an emergency guide may be treated less as a tool to be evaluated and more as an authority to be obeyed. The label, the lights and the confident gesture all signal that someone has thought about this situation, and in an emergency thinking is precisely what people would like to outsource. The team also noted that in earlier versions of their work, run in simulation without a convincing emergency, participants did withdraw trust from a robot that had made mistakes. Stakes and stress seem to change the calculation.

It is worth being careful about what the study shows. The sample was small and drawn from a university population, the robot was operated by a hidden researcher, and a staged fire is not a real one. Participants may also have suspected that nothing truly dangerous could happen inside a psychology experiment, which is a classic difficulty with any lab emergency. Still, the result is a useful provocation precisely because it inverts the usual worry. Much of the field has fretted that people will not trust robots enough. Here the problem was the opposite.

Trust as a dial, not a switch

Engineers who study automation have long framed this as a question of calibration. In an influential review, John Lee and Katrina See argued that trust matters most exactly when a system is too complex for its user to understand fully, so that people lean on an attitude rather than a model of how the thing works. The goal, in their account, is not maximum trust but appropriate reliance: trust that rises and falls with what the system can actually do in the situation at hand. Overtrust leads to misuse, where people lean on a system beyond its competence. Undertrust leads to disuse, where a capable system is ignored.

What drives trust in robots specifically? A meta-analysis by Peter Hancock and colleagues pooled the available experimental and correlational studies and found that characteristics of the robot, above all its performance, were the strongest predictors, with environmental factors playing a moderate role and surprisingly little evidence for the influence of human factors. That is reassuring in one sense, since performance is something designers can work on. The Georgia Tech study complicates the picture, though. Performance was manipulated, and it barely mattered once the alarm sounded. Context can swamp track record.

Designing for doubt

For anyone building social robots, the lesson is uncomfortable. The very qualities that make a robot feel competent and companionable, such as confident speech, purposeful movement, a clear role and a friendly face, are also the qualities that can encourage people to stop checking. A robot that sounds equally sure about everything gives its users no way to tell when it is guessing.

Calibrated trust therefore has to be designed for, not just hoped for. One approach is to make uncertainty visible, letting a system signal when it is outside familiar territory. Another is to make failure legible, so that a mistake does not quietly disappear into a smooth performance. A third is to avoid dressing up a machine in borrowed authority it has not earned, whether that is a uniform, an official label or a persona borrowed from a trusted profession. None of these is free, since honest hesitancy can make a product feel less magical, and commercial pressure usually pushes the other way.

There is a broader point here about the social psychology of machines. People do not evaluate robots from scratch each time; they bring scripts about guides, helpers and experts, and a robot that slots into one of those scripts inherits the deference that comes with it. That makes trust partly a matter of theatre. Magicians and con artists have understood this for centuries, and it is both fascinating and slightly alarming to see it carry over to machines. The best robot, in a crisis, may be the one that knows when to say it is not sure.

References

1. Robinette, P., Li, W., Allen, R., Howard, A. M. and Wagner, A. R. (2016). Overtrust of robots in emergency evacuation scenarios. Proceedings of the 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 101-108. Link
2. Georgia Institute of Technology (2016). In emergencies, should you trust a robot? News release via EurekAlert. Link
3. Lee, J. D. and See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50-80. Link
4. Hancock, P. A., Billings, D. R., Schaefer, K. E., Chen, J. Y. C., de Visser, E. J. and Parasuraman, R. (2011). A meta-analysis of factors affecting trust in human-robot interaction. Human Factors, 53(5), 517-527. Link

Feature drafted with AI from the published research listed above, for social-robotics.com.