Scientists Teach Robots To “Watch” YouTube Videos And Learn From Them

People have talked about getting personal robots as assistants for years, but Cornell is doing more than just talking. The university’s “Robo Brain,” which launched online in July, teaches robots how to function in daily life from an extensive database of YouTube videos.

The Robo Brain database was created by Aditya Jami, a visiting researcher at Cornell. It essentially stores information in a robot-friendly format so the machines can pull from the cloud-like database whenever they need to perform a task like pouring a drink or finding keys.

“Our laptops and cell phones have access to all the information we want,” explained Ashutosh Saxena, an assistant professor of computer science, in the Cornell Chronicle. “If a robot encounters a situation it hasn’t seen before it can query Robo Brain in the cloud,”

Saxena led the information-gathering project, along with colleagues from Cornell, Stanford, Brown, and University of California, Berkeley. In July, according to The Washington Post, the team started downloading about 120,000 YouTube videos, over 100 million instruction manuals, billions of images, and more data for robots to reference.

Their work is paying off. So far, robots have learned when to hold a coffee cup upside-down  (i.e. how liquid interacts with the mugs), and how to scoop ice cream and drizzle syrup over the top. Additionally, robots have studied human interactions and learned when it’s appropriate not to intervene, like when two humans are talking. One robot even observed how a human became frustrated when the robot passed in front of the TV, so the next time it went behind the couch instead of in front of the human’s line of sight.

Check out the video below of the robot learning not to interrupt TV-watching. Down the line, you’ll also be able to visit the Robo Brain website if you’d like to be a “teacher,” leaving addictions or making corrections to the database information.