How to build a self-driving car

By Peter

Transcript

1. Look at speaker notes
3. Autopilot
4. How to build a self-driving car Side-project and lessons learned With demos! 🍿
5. CARLA: Open-source simulator for autonomous driving
6. Familiar stack
7. import carla client = carla.Client('localhost', 2000) world = client.get_world() my_model_3 = world.spawn_actor(...)
8. bp = blueprint_library.find('sensor.camera.rgb') center_rgb = world.spawn_actor(bp, attach_to=my_model_3)
9. for frame in range(args.frames): sensor_data = ... control_at_frame = my_tesla_3.get_control() # e.g. {'steer': 0.7, 'throttle': 0.3, 'brake': 0.01} sensor_data.save_to_disk(f'{frame}.png') # e.g. 000003.png
10. 000003.png {'steer': 0.01, 'throttle': 0.4, 'brake': 0.0}
12. Problem Definition Given X {'steer': 0.01, 'throttle': 0.4, 'brake': 0.0} Predict Y
13. Simpler Problem Definition Given X {'steer': 0.01, 'throttle': 0.4, 'brake': 0.0} Predict Y -1.0 1.0 0.0
14. ‘s Pilotnet
15. ‘s Pilotnet
16. ‘s Pilotnet X X Y
17. Training
18. Self-driving solved?
19. “The devil is in the detail”
20. Training data contains single images sampled from the video, paired with the corresponding steering command (1/r). Training with data from only the human driver is not sufficient. The network must learn how to recover from mistakes. Otherwise the car will slowly drift off the road. The training data is therefore augmented with additional images that show the car in different shifts from the center of the lane and rotations from the direction of the road. Bojarski, Mariusz, et al. “End to End Learning for Self-Driving Cars.” ArXiv.org, 25 Apr. 2016, arxiv.org/abs/1604.07316.
21. Training data contains single images sampled from the video, paired with the corresponding steering command (1/r). Training with data from only the human driver is not sufficient. The network must learn how to recover from mistakes. Otherwise the car will slowly drift off the road. The training data is therefore augmented with additional images that show the car in different shifts from the center of the lane and rotations from the direction of the road. Bojarski, Mariusz, et al. “End to End Learning for Self-Driving Cars.” ArXiv.org, 25 Apr. 2016, arxiv.org/abs/1604.07316.
22. Problem No data that shows how to recover from mistakes Solution Data augmentation In summary...
23. Add left and right cameras
24. Center Camera {'steer': 0.0}
25. Left Camera {'steer': 0.2}
26. Right Camera {'steer': -0.2} Label?
27. Random Translation Augmentation
28. Center Camera {'steer': 0.0}
29. Center Camera w/ Random Translation e.g. + 37px {'steer': -0.148} 1px : 0.004 steer 37 * 0.004 = 0.148
30. Center Camera w/ Random Translation e.g. - 40px {'steer': 0.16} 1px : 0.004 steer 40 * 0.004 = 0.16
32. Problem Solved! Before Data Augmentation After Data Augmentation
33. Let’s see it in action!
34. Cool! But what is the neural network actually “seeing”?
35. Saliency Map
36. ‘s VisualBackProp
37. VisualBackProp Explained Deconvolution Layer
40. Inside the neural network
41. We gave only this... {'steer': 0.01}
42. ...and we got this! Lane Detection
43. But how do we get to here?
44. Our End-to-End (E2E) Solution Neural Network Steering / Speed Image
45. What about for other cars?
46. Industry Standard? Neural Network Steering / Speed Image
47. Do not predict controls Neural Network Image
48. Predict Paths, instead! Neural Network Image Paths
49. Path Prediction (X, Y) Coordinates
50. One last thing...
51. “Don’t reinvent the wheel” Model Predictive Control
52. Vehicle Control is a Physics Problem
53. That’s why airplanes already have autopilot
54. Solved by Mechanical Engineering (기계 공학)
55. The Self-driving Stack in the Industry Image Paths Steering
56. Lessons Learned (or re-learned) • Open-source spreads knowledge and creates opportunities for all - here’s mine • EC2 is great for on-demand GPUs i.e. ML side projects • Pytorch >> TensorFlow • Machine Learning is 99% infrastructure and 1% Machine Learning • Best way to learn is to build it on your own
57. For a side project, do something that inspires you