Learning to Drive in the Dreams of a Diffusion Model

Abstract:
End-to-end machine learning is gaining traction in vehicle autonomy. A dataset of human driving is easy to collect, and contains demonstrations of high-quality driving. Learning to mimic that behavior is an effective way to teach a machine how to drive that scales well with data and compute. This talk describes the latest training strategy developed at comma.ai to teach machines how to drive. It involves training a large world-model, which can act as a simulator that takes in driving actions and generates video. This simulator can produce realistic driving scenarios much like a video game. Policy models are allowed to act in this simulator where they can be trained on-policy, to drive like a human. The latest models trained like this are already deployed in openpilot, an open-source ADAS project, and used by thousands of people every day in the real world.

Bio:
Harald is the CTO of comma.ai. He has worked at comma for 7+ years, where he has been developing an end-to-end self-driving solution for openpilot, the most popular open-source self-driving car project in the world.