Magic Tricks for Self-driving Cars

A highlighted talk at the Defcon 2018 CAAD Village. I conducted this work at Baidu X-Lab in summer 2018 while I was an intern researcher.

Transcript

1. Magic Tricks for Self-driving Cars Weilin Xu, Zhenyu Zhong, Yunhan Jia
2. • This is a proof-of-concept. • We are NOT targeting at any autonomous vehicle vendors. • Don’t try to fool your neighbor’s car. AUTHORS’ WARNING
3. Autonomous Vehicle Framework Radar LiDAR Camera Image: https://medium.com/toyota-ai-ventures/https-medium-com-toyota-ai-ventures-announcingblackmore-7947eacc9e9e Camera
4. Camera-based Obstacle Detection
6. Our Target: YOLOv3 YOLO v3 Object Detection Model [147 Layers, 62M Parameters] Input [416x416x3] Output [3549 Bounding Boxes] Image: http://media.nj.com/traffic_impact/photo/all-way-stop-sign-that-flashes-in-montclairjpg-30576ab330660eff.jpg
7. Trained with the COCO Dataset • Common Objects in Context • 80 Classes: person, [car, truck, bus], [bicycle, motorcycle], [stop sign, traffic light], etc. Source: http://cocodataset.org/
8. car 0.01% YOLOv3 Inference 116x9 0 156x 198 373x326 Anchor Boxes !" = $ %" + '" !( = $ %( + '( !) = *) +,- Center Point Object Size !. = *. +,/ *. *) 13 x 13 Grid ('" , '( ) = (11,2) Prediction Vector Bounding Box 80 Class Confidence Objectness %" %( %) %. *567 '8 '9 … … ':; '<= ▪ stop sign 99% × car 0.01% car 0.01%
9. Threat Model: Image Patch Attack Company Logo
10. Threat Model: Image Patch Attack Company Logo
11. Threat Model: Image Patch Attack Company Logo
12. Threat Model: Image Patch Attack Company Logo
13. Attack Algorithms • Input Construction • Objectives • Optimization
14. Resize Perspective Transform Differentiable Input Construction Company Logo Company Logo
15. Objectives • Object Production • Object Vanish • Object Transformation
16. Object Production - Coarse We want more certain objects on the whole image. • Easy to implement. • May be difficult to optimize. Company Logo
17. Object Production - Precise We want a certain object of a specific size in a specific location. Company Logo
18. Object Vanish - Coarse We want a certain object class to vanish on the whole image.
19. Object Transformation - Coarse We want certain object class to transform to other class.
20. Optimization • Change of variable Convert to tanh() space to encode the [0, 1] interval constraint. Friendly to many off-the-shelf optimizers, e.g. Adam. • To optimize logits Skip sigmoid() to avoid vanishing gradient. Carlini, Nicholas, and David Wagner. “Towards Evaluating the Robustness of Neural Networks.” IEEE S&P (Oakland) 2016.
21. But, Image Sensing is not an Identity Function Model Input [416x416x3] § Limited Resolution § Distortions § Random Noise § … Digital Image Scene
22. Towards Robust Physical Adversarial Examples • [Limited Resolution] Smoother patch with the total variation regularization. • [Distortions] Color management with the non-printability loss. • [Inaccurate Patch] Random transformation during optimization iterations. • …… Sharif, Mahmood, et al. "Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition." ACM CCS 2016.
23. Conclusion • Magicians can fool object detection models, so can attackers. • We should be cautious with self-driving cars that rely on computer vision.