Reading the Future of a Sentence: How Machines Learn to Know When You Are Done Talking

From silence timers to self-supervised neural networks, the quest to predict the end of a turn has quietly become one of the most important problems in conversational robotics.

The silence trap

Ask a voice assistant a question, hesitate halfway through, and there is a good chance it will cut you off. Speak smoothly and it may still leave you hanging for a second before answering. Both problems have the same root: the system is deciding that you have finished based mostly on how long you have been quiet.

Silence is a poor guide. People pause inside turns constantly, while searching for a word or gathering a thought, and they often finish turns without any pause at all, because the listener jumps in on time. Human listeners know the difference because they are tracking much more than sound level. They follow the grammar ('I was thinking we could...' is plainly unfinished), the intonation, the gaze of the speaker, and the shape of the exchange. The research challenge is to give machines a similar sense of where a turn is heading.

From decisions to forecasts

Early end-of-turn models in dialogue systems were usually classifiers attached to particular moments. When a pause is detected, look at the last stretch of speech and decide: is this a turn shift or a hold? Such models improved on fixed timers, but they still treated turn-taking as a sequence of separate yes-or-no decisions, often triggered by silence in the first place.

In 2017 Gabriel Skantze at KTH Royal Institute of Technology in Stockholm proposed a different framing. Instead of training a model to make a specific decision, he trained a recurrent neural network on recordings of human-human dialogue to do something more general: continuously forecast who would be speaking over the next few moments. Once a model can do that, many practical questions become readouts of the same prediction. In pauses, it predicted whether a turn shift would follow better than both human judges and more traditional models. At the onset of speech, it could estimate whether an utterance would be a brief 'mm-hm' or a longer contribution.

Voice Activity Projection

This idea matured into what Erik Ekstedt and Skantze called Voice Activity Projection, or VAP, presented in 2022. The model listens to the audio of both speakers and, frame by frame, predicts the pattern of voice activity for each of them over a short window into the future. Crucially, the training is self-supervised: the 'labels' are simply what happened next in the recording, so the model can learn from large amounts of unannotated conversation.

The authors argued that earlier approaches had missed how upcoming voice activity events depend on each other within that projection window, and they designed the objective to capture those dependencies. They then evaluated the model on several tasks it was never explicitly trained for, including predicting turn shifts and backchannels, and reported better performance than prior work. The appeal for robotics is obvious. A single predictive model, running continuously, can tell a system when it is safe to speak, when it is being invited to speak, and when the user is merely pausing.

Words matter too

Sound is only part of the picture. The same group also built TurnGPT, a language model based on GPT-2 and fine-tuned on dialogue, which predicts likely turn-completion points from the words alone. Their analysis suggested it was using dialogue context and a sense of pragmatic completeness, not just surface grammar. A sentence can be grammatically complete and still obviously unfinished as a turn, and text-based models can pick up on that.

Text and audio models have complementary strengths. Words tell you whether something could be complete; prosody and timing tell you whether the speaker is actually letting go. Combining them is the natural next step.

Putting it on a robot

In 2025 Skantze and Bahar Irfan reported what they described as the first application of these general turn-taking models to human-robot conversation. Their system on the Furhat robot used TurnGPT and VAP together to decide when to start preparing a response, when to take the turn and how to handle interruptions, with a large language model generating the content. Neither turn-taking model was fine-tuned for the task. In a within-subject study with 39 adults, the system was compared with a conventional silence-based baseline. Participants significantly preferred the predictive version, and it produced shorter response delays and fewer interruptions.

That is a modest-sounding result with large implications. It suggests that turn-taking competence learned from ordinary human conversation can transfer to a robot without hand-crafted rules for each application.

What remains hard

Prediction is not mind-reading. Models trained on two-person phone or lab conversations meet very different conditions in a noisy museum, a care home or a group. Cultural and individual differences in timing, discussed in the previous article, mean that a single threshold of confidence will not suit everyone. And predicting that a turn is ending is only half the job: the robot also needs something ready to say, which is where response latency and language models enter the story later in this series.

Still, the direction of travel is clear. The field is moving away from asking 'has the user stopped?' towards asking 'what is about to happen?', which is, after all, the question human listeners are answering all the time.

References

1. Skantze, G. (2017). Towards a general, continuous model of turn-taking in spoken dialogue using LSTM recurrent neural networks. Proceedings of SIGDIAL 2017, 220-230. Link
2. Ekstedt, E. and Skantze, G. (2020). TurnGPT: a Transformer-based language model for predicting turn-taking in spoken dialog. Findings of EMNLP 2020, 2981-2990. Link
3. Skantze, G. (2021). Turn-taking in conversational systems and human-robot interaction: a review. Computer Speech and Language, 67, 101178. Link
4. Ekstedt, E. and Skantze, G. (2022). Voice Activity Projection: self-supervised learning of turn-taking events. Interspeech 2022 (arXiv:2205.09812). Link
5. Skantze, G. and Irfan, B. (2025). Applying general turn-taking models to conversational human-robot interaction. ACM/IEEE International Conference on Human-Robot Interaction (HRI 2025). Link

Feature drafted with AI from the published research listed above, for social-robotics.com.