Google DeepMind's RT-2 turns web knowledge into robot actions

Google DeepMind introduced RT-2, a transformer trained on web text and images that outputs robot actions directly rather than relying on separate planning and control systems. Across more than 6,000 trials, it matched the earlier RT-1 on familiar tasks and raised success on unseen scenarios from 32% to 62%. It could, for example, recognise rubbish it had never been trained on and work out that it should be thrown away. Vision-language-action models like RT-2 pointed towards robots that understand everyday instructions in context, which matters greatly for robots that share homes and workplaces with people.

Source: Google