Share
Embed
Slide 1
Slide 1 text
Look at speaker notes
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
Autopilot
Slide 4
Slide 4 text
How to build a
self-driving car
Side-project and lessons learned
With demos! 🍿
Slide 5
Slide 5 text
CARLA: Open-source simulator for autonomous driving
Slide 6
Slide 6 text
Familiar stack
Slide 7
Slide 7 text
import carla
client = carla.Client('localhost', 2000)
world = client.get_world()
my_model_3 = world.spawn_actor(...)
Slide 8
Slide 8 text
bp = blueprint_library.find('sensor.camera.rgb')
center_rgb = world.spawn_actor(bp, attach_to=my_model_3)
Slide 9
Slide 9 text
for frame in range(args.frames):
sensor_data = ...
control_at_frame = my_tesla_3.get_control()
# e.g. {'steer': 0.7, 'throttle': 0.3, 'brake': 0.01}
sensor_data.save_to_disk(f'{frame}.png')
# e.g. 000003.png
Slide 10
Slide 10 text
000003.png
{'steer': 0.01,
'throttle': 0.4,
'brake': 0.0}
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
Problem Definition
Given X
{'steer': 0.01,
'throttle': 0.4,
'brake': 0.0}
Predict Y
Slide 13
Slide 13 text
Simpler Problem Definition
Given X
{'steer': 0.01,
'throttle': 0.4,
'brake': 0.0}
Predict Y
-1.0 1.0
0.0
Slide 14
Slide 14 text
‘s Pilotnet
Slide 15
Slide 15 text
‘s Pilotnet
Slide 16
Slide 16 text
‘s Pilotnet
X
X
Y
Slide 17
Slide 17 text
Training
Slide 18
Slide 18 text
Self-driving solved?
Slide 19
Slide 19 text
“The devil is in the detail”
Slide 20
Slide 20 text
Training data contains single images sampled from the video,
paired with the corresponding steering command (1/r).
Training with data from only the human driver is not
sufficient. The network must learn how to recover from
mistakes. Otherwise the car will slowly drift off the road.
The training data is therefore augmented with additional
images that show the car in different shifts from the center of
the lane and rotations from the direction of the road.
Bojarski, Mariusz, et al. “End to End Learning for Self-Driving Cars.” ArXiv.org, 25 Apr. 2016, arxiv.org/abs/1604.07316.
Slide 21
Slide 21 text
Training data contains single images sampled from the video,
paired with the corresponding steering command (1/r).
Training with data from only the human driver is not
sufficient. The network must learn how to recover from
mistakes. Otherwise the car will slowly drift off the road.
The training data is therefore augmented with additional
images that show the car in different shifts from the center of
the lane and rotations from the direction of the road.
Bojarski, Mariusz, et al. “End to End Learning for Self-Driving Cars.” ArXiv.org, 25 Apr. 2016, arxiv.org/abs/1604.07316.
Slide 22
Slide 22 text
Problem
No data that shows
how to recover from mistakes
Solution
Data augmentation
In summary...
Slide 23
Slide 23 text
Add left and right cameras
Slide 24
Slide 24 text
Center
Camera
{'steer': 0.0}
Slide 25
Slide 25 text
Left
Camera
{'steer': 0.2}
Slide 26
Slide 26 text
Right
Camera
{'steer': -0.2}
Label?
Slide 27
Slide 27 text
Random
Translation
Augmentation
Slide 28
Slide 28 text
Center
Camera
{'steer': 0.0}
Slide 29
Slide 29 text
Center
Camera
w/ Random
Translation
e.g. + 37px
{'steer':
-0.148}
1px : 0.004 steer
37 * 0.004 = 0.148
Slide 30
Slide 30 text
Center
Camera
w/ Random
Translation
e.g. - 40px
{'steer':
0.16}
1px : 0.004 steer
40 * 0.004 = 0.16
Slide 31
Slide 31 text
No content
Slide 32
Slide 32 text
Problem Solved!
Before Data Augmentation After Data Augmentation
Slide 33
Slide 33 text
Let’s see it in action!
Slide 34
Slide 34 text
Cool!
But what is the neural
network actually “seeing”?
Slide 35
Slide 35 text
Saliency Map
Slide 36
Slide 36 text
‘s VisualBackProp
Slide 37
Slide 37 text
VisualBackProp Explained
Deconvolution
Layer
Slide 38
Slide 38 text
No content
Slide 39
Slide 39 text
No content
Slide 40
Slide 40 text
Inside the
neural network
Slide 41
Slide 41 text
We gave only this...
{'steer': 0.01}
Slide 42
Slide 42 text
...and we got this!
Lane Detection
Slide 43
Slide 43 text
But how do we get to here?
Slide 44
Slide 44 text
Our End-to-End (E2E) Solution
Neural Network Steering / Speed
Image
Slide 45
Slide 45 text
What about for other cars?
Slide 46
Slide 46 text
Industry Standard?
Neural Network Steering / Speed
Image
Slide 47
Slide 47 text
Do not predict controls
Neural Network
Image
Slide 48
Slide 48 text
Predict Paths, instead!
Neural Network
Image Paths
Slide 49
Slide 49 text
Path Prediction
(X, Y) Coordinates
Slide 50
Slide 50 text
One last thing...
Slide 51
Slide 51 text
“Don’t reinvent the wheel”
Model
Predictive
Control
Slide 52
Slide 52 text
Vehicle Control is a Physics Problem
Slide 53
Slide 53 text
That’s why airplanes already have autopilot
Slide 54
Slide 54 text
Solved by Mechanical Engineering (기계 공학)
Slide 55
Slide 55 text
The Self-driving Stack in the Industry
Image
Paths
Steering
Slide 56
Slide 56 text
Lessons Learned (or re-learned)
● Open-source spreads knowledge and creates
opportunities for all - here’s mine
● EC2 is great for on-demand GPUs i.e. ML side projects
● Pytorch >> TensorFlow
● Machine Learning is 99% infrastructure and 1%
Machine Learning
● Best way to learn is to build it on your own
Slide 57
Slide 57 text
For a side project, do something that inspires you