信頼されるLiDARに向けて 吉岡 健太郎 2022/9/26 情報処理学会 連続セミナー2022第6回 Assistant Prof., Keio University Japan Rotating-mirror LiDARs (Yoshioka, ISSCC 2018) Solid-state LiDARs (Kondo,Tan, ISSCC/VLSI 2020)

・2014 Graduate@Keio Univ. ・2014-2021 Toshiba ・2017-2018 Stanford Visiting Scholar @Mark Horwitz Group ・2021- Keio Univ. Assistant Prof. Keio Computing and Sensing Group https://sites.google.com/keio.jp/keio-csg/ About me Kaggle/Signate Master

・コアコンピタンス: Mixed-signal circuit design(ADC/TDC), LiDAR signal processing, MLアクセラレータ、コンピュータアーキテクチャ 回路・LiDARの研究遂行。 2022-よりLiDAR自動運転セキュリティの研究開始。 About me WiFi6 product chip VLSI 2020 ISSCC2018 JSSC 2018 ISSCC 2020 JSSC 2020 LiDAR Product SoCs ISSCC2017 ISSCC2018 JSSC2018 TVLSI2019

Slide 3 ・LiDARとは？ ・高精度、高解像度な次世代型LiDAR LiDARの集積化により低コスト＋高性能LiDARへ ・LiDARセンサセキュリティ LiDARの抱える脆弱性、セキュリティ危機 アジェンダ

Slide 4 LiDARデータってどんな感じ？

Slide 5 自動運転と距離センサ 機械はどう見て運転している？ https://eng.uber.com/atg-dataviz/

Slide 6 距離センサのベンチマーク • 自動運転の盛り上がり – トヨタも2020年代前半に市街地自動運転の実現を明言 • 自動運転には周辺環境のセンシングが必須 – しかしミリ波レーダやソナーでは性能不足 センサ 特徴 超音波 近距離の物体検知を安価に実現。長距離は苦手。 カメラ 自動車以外に歩行者・自転車の検出、標識の認識が可能。距離情報の取得はやや不得意 ミリ波レーダー 耐天候性があり長距離測距が可能。検知範囲が狭く、距離精度・空間分解能が低い。 LiDAR ミリ波レーダーに比べ環境の影響を受けやすいが、距離精度・空間分解能が高い。 自動運転システムではLiDARが重要なセンサに

Slide 7 iPad ProにLiDARが搭載され話題に 最新iPad, iPhoneにも搭載 一気に身近な存在に

Automobile LiDAR Requirements Slide 8 ● Long range (>200m) 200m Braking distance@120km/h >150m 17cm debris = 0.1 deg. @ 100m ● High image resolution (<0.1deg)

LiDAR Fundamentals Slide 9 Detects the distance to object by measuring ToF LiDAR System Laser SoC SiPM Laser pulse time time SiPM output ToF Light Reflected ToF Laser emitted Distance to object Light speed x ToF 2 = ToF

Noise problem of automotive LiDAR • Sunlight affects the measurement result Direct-ToF Laser SoC SiPM Laser time time SiPM Wrong ToF! Sunlight Laser Brighter-than-sunlight laser is required. Focused laser beam is necessary Slide 10

Only scanning LiDARs can afford long range requirement Automotive LiDAR’s topology Slide 11 Scanning LiDARs Laser SoC SiPM Flash LiDARs Laser SoC APD array ☺ Focused laser beam ⇒Long range  Low resolution  Bulky scanner ☺ High resolution ☺ Solid state  Unfocused laser beam ⇒Short range Scanner

1st Gen. LiDARs (or analog LiDARs) Slide 12 Further read: Why digital lidar is the future https://ouster.com/blog/why-digital-lidar-is-the-future/ Getting “points” for each component Dedicated focused lasers

1st Gen. LiDARs (or analog LiDARs) Slide 13 Further read: Why digital lidar is the future https://ouster.com/blog/why-digital-lidar-is-the-future/ Getting “points” for each component Dedicated laser paths scanning

1st Gen. LiDARs (or analog LiDARs) Slide 14 Further read: Why digital lidar is the future https://ouster.com/blog/why-digital-lidar-is-the-future/ Analog LiDARs (e.g. Velodyne) • Discrete Laser/RX • APD receivers • EELs Many components..  Expensive LiDARs  Difficult to scale Getting “points” for each component

Next Gen. LiDARs (or digital LiDARs) • Analog LiDARs – Discreate components – Unscalable resolution – Poor lasers/detectors (EEL, APD) • Next Gen. LiDARs – Integrated components • Higher performance, but lower price (as in Moore’s law) • Mass production LiDARs can be in $500~1000 range – Scalable performance – SoTA laser/detectors (VCSELs, SPADs) Slide 15

Ouster Integrated LiDAR Slide 16 Signal Processing Image Sensor VCSEL array Getting “2D surfaces” instead of “points” Surface scanning T[0] T[1] T[2]

Next Gen. LiDARs (or digital LiDARs) Slide 17 • 大きなゲームチェンジは？ – APDからSPADに – EELからVCSELに • Image sensor(SPAD array)+ デジタル回路を集積(ASIC) • よりリッチな信号処理がLiDAR上で 可能に • LiDARに特化した信号処理の発展 Image Sensor + ASIC

What can we do with Next Gen. LiDARs? • Laser power loss follows inverse square law – Not a big problem @50m – Severe SNR @ 200m LiDAR System SiPM Laser SoC @200m Laser time time SiPM Wrong ToF! Sunlight Laser Laser SoC SiPM Polygon mirror SNR: Num. laser photons Num. noise photons = Laser time time SiPM Wrong ToF! Sunlight Laser

What can we do with Next Gen. LiDARs? While sunlight photons are random events, Laser photons are periodic → Accumulation improves SNR by square root of acc. pixels Pix.1 Pix.2 Pix.3 Laser Accumulation of pix.1~3 ToF Sunlight • Accumulation, or oversampling for higher SNR – Strategies: Frame accumulation and Nearby pixel accumulation – If the pixel is “watching” the same object, SNR improves [Niclass, ISSCC2013]

Main Tradeoffs in LiDAR Design • Requirement extremely strict for self-driving LiDARs Importantly: Laser power limited by eye-safety but not for attackers.. Measurement Distance Sunlight Tolerance Image Quality (FPS, num.pixels) Image Quality High Low Measurement Distance [m] 100 200

Problem of Simple Accumulation Non-target reflection Target reflection Target object Non-target A D G B E H C F I Measuring pixel (MP) Pixel A,B,C Pixel E Pixel D,F,G,H,I time time time Target reflection Simple accumulation A+B+C+D+E+F+G+H+I time Detected peak Wrong!  Function similar to “blur” algorism

Concept of Smart Accumulation Technique (SAT) • Classify objects utilizing LiDAR raw-data characteristics • Accumulate pixels watching the same object only; SNR improves significantly – Peak level has high correlation with distance and reflectivity of the object – Floor level also has strong correlation with the object distance and reflectivity A D G B E H C F I Pixel A,B,C Pixel D,E,F,G,H,I time time Peak level Peak level Floor level Floor level

SAT Algorithm (3) Target object Non-target A D G B E H C F I Measuring pixel Pixel A,B,C Pixel E Pixel D,F,G,H,I PL High correlation! Peak level Floor level FL Correlation of PL & FL above threshold? Accumulate Skip SAT Algorithm D,F,G,H,I A,B,C Yes No Simple accumulation A+B+C+D+E+F+G+H+I time Detected peak Wrong!  Smart accumulation(SAT) D+E+F+G+H+I time Detected peak Correct! ☺ Accumulated waveforms Long meas. range + High image quality PL FL

Effect of Smart Accumulation Algorithm Imaged blurred Road detected Pedestrian undetected Road undetected Pedestrian Camera image “Simple” Accumulation Without Accumulation High image quality, but only short distance Poor image quality, Unable to small objects

Effect of Smart Accumulation Algorithm Pedestrian undetected Road undetected Pedestrian Camera image “Simple” Accumulation Without Accumulation Road detected Pedestrian detected “Smart” Accumulation High image quality, Able to detect small obj.

試作LiDAR • K.Yoshioka, ISSCC 2018 Slide 26 3D point cloud view ADC Image TDC Image Luminescence アナログ＋デジタル集積チップ

LiDARセンサセキュリティ Slide 27 • LIDARは測定原理上データ注入の脆弱性を抱える – ハッカー（攻撃者）がタイミング良く攻撃レーザを打ち込むことで 任意データを注入可能 – センサ幻惑攻撃 Y.Cao ACM CCS, 2019. 実車点群 虚偽データ 任意座標 虚偽データ注入 急ブレーキ誘発 搭乗者負傷

◼ LiDARは対象物に反射したレーザが帰還するまでの光の飛行時間より 距離を測定するToF(Time-of-Flight)方式で動作 ◆センサ幻惑はレーザパルスを特定タイミングで打ち込むことで虚偽データ注入 ⚫受光素子を使用しLiDARレーザタイミングと同期 LiDARとセンサ幻惑原理 レーザ ToF 計測 受光素子 レーザ 受光素子 ToF 攻撃レーザ レーザ ToF 計測 受光素子 同期用受光素子 センサ幻惑システム LiDAR LiDAR レーザ 受光素子 ToF 攻撃レーザ打ち込み

まとめ • より信頼できる高精度・高解像度LiDARのため、 集積LiDAR（次世代型LiDAR）の開発が盛ん – 性能向上＋コストダウン • LIDARは測定原理上データ注入の脆弱性を抱える – 攻撃capabilityを明らかにする段階 – センサレベルの防御手法を議論中 Slide 29