Unisim ௒ϦΞϧͳࣗಈӡస ηϯαγϛϡϨʔγϣϯ Michiya Abe @abemii_ Jul. 23, 2023. ίϯϐϡʔλϏδϣϯษڧձ@ؔ౦ CVPR 2023 ಡΈձʢલฤʣ

2 l Michiya Abe l ܦྺ l 2019 ɿम࢜ʢ৘ใཧ޻ֶʣ l 2019 ~ ɿࣗಈӡస޲͚ը૾ೝࣝͷݚڀ։ൃ l ෺ମݕग़ɾ૸࿏ೝࣝͷϞσϧͷ։ൃ l ϞσϧͷྔࢠԽɼΤοδͰͷߴ଎Խ l ޷͖ͳ΋ͷ l ςΩετΤσΟλʢNeovimʣ l Ϊλʔʢॳ৺ऀʣ ࣗݾ঺հ Twitter: @abemii_ Blog: https://abemii.hatenablog.com/ ˞ ൃද಺༰͸ॴଐػؔͱҰ੾ؔ܎͠·ͤΜ

3 l CVPR 2023 (Highlight) l ܭଌͨ͠ηϯασʔλʢը૾ɾLiDAR఺܈ʣ͔Βɼૢ࡞Մೳͳσδλϧπ ΠϯΛ࡞Δ l ৽͍͠γφϦΦΛ࡞Δ͜ͱ΋Ͱ͖ɼ closed-loop ධՁʹ΋࢖͑Δ ࠓ೔঺հ͢Δ࿦จ • Paper: https://openaccess.thecvf.com/content/CVPR2023/papers/Yang_UniSim_A_Neural_Closed-Loop_Sensor_Simulator_CVPR_2023_paper.pdf • Project page: https://waabi.ai/unisim/

·ͣ͸σϞΛ͝ཡ͍ͩ͘͞

5 Video: https://waabi.ai/unisim/ σϞɿΞΫλʔʢଞं྆ʣͷআڈ Recorded Simulated Α͘ݟΔͱͪΐͬͱΞʔςΟϑΝΫτ͕ੜ͍ͯ͡Δ͕ɼӈ Ϩʔϯ͸ं͕྆ଟ͘ɼશ͘؍ଌͰ͖͍ͯͳ͍ྖҬ͕͋Δͷ ͩΖ͏

6 ͜Μͳγʔϯ͸ك͕ͩɼ࣮֬ʹೝࣝͯࣗ͠ंͷߦಈΛม͑ ͳ͍ͱࣄނʹͳͬͯ͠·͏ σϞɿΞΫλʔΛૢ࡞ Recorded Simulated Video: https://waabi.ai/unisim/

7 σϞɿSDVʢࣗंʣͷηϯαҐஔΛૢ࡞ʢࠨӈɾ্Լʣ Video: https://waabi.ai/unisim/

8 ࣗಈӡసͷ҆શੑͷݕূ = ೉͍͠ l ΫϦςΟΧϧͳγʔϯΛݕূ͍ͨ͠ → ͦΜͳγʔϯ͸࣮ੈքͰ͸໓ଟʹಘΒΕͳ͍ l ࣮ੈքσʔλͷϩάϦϓϨΠ → ࣗंͷڍಈʹର͠ɼԠ౴తͰͳ͍ → ࣗंͷߦಈʹର͢Δ݁Ռ͕Ͳ͏ͳΔ͔Θ͔Βͳ͍ Ϟνϕʔγϣϯ

9 Closed-loop Simulation l ͋ΔγʔϯͰࣗं͕͋ΔߦಈΛͨ͠ͱ͖ɼ पғ͸ͲͷΑ͏ʹมԽ͢Δ͔ʢͲͷΑ͏ͳ݁ՌΛ΋ͨΒ͔͢ʣ: “what-if” l ҆શͳࣗ཯૸ߦγεςϜͷධՁͷͨΊʹઈରʹඞཁ Ϟνϕʔγϣϯ ੈքͷঢ়ଶ ݱ࣮తͳηϯα γϛϡϨʔγϣϯ ԿΒ͔ͷ੍ޚ ଞͷΞΫλʔͷৼΔ෣͍ͷγϛϡϨʔγϣϯ + ੈքͷঢ়ଶͷߋ৽

10 طଘͷ Closed-loop Simulation ͷ໰୊఺ l εέʔϥϒϧͰͳ͍ɿਓ͕ؒγφϦΦΛख࡞Γɼγϯϓϧͳ੍ޚ l ଟ༷ੑʹ͚ܽΔɿγφϦΦ΍Ξηοτ͕ݶΒΕ͍ͯΔ l ࣮ࣸతͰͳ͍ɿೝࣝʹ͓͚ΔυϝΠϯΪϟοϓ͕ੜ͡Δ Ϟνϕʔγϣϯ ͜Ε͸ Carla ͷྫ

11 ຊݚڀ Unisim: εέʔϥϒϧɾଟ༷ɾ࣮ࣸత l ϦΞϧσʔλΛऩू͢Δ͚ͩͳͷͰɼεέʔϥϒϧ l ΞΫλʔΛૢΔ͜ͱ͕Ͱ͖ΔͷͰɼଟ༷ੑ΋͋Δ l ࣮ࣸతͳֆΛ࡞Δ͜ͱ͕Ͱ͖ΔʢϦΞϧͳηϯαγϛϡϨʔγϣϯʣ Ϟνϕʔγϣϯ UniSim ࣗंͷߦಈʹैͬͯɼଞͷ ΞΫλʔͷҐஔ΋มԽ͢Δ ˠมԽͨ͋͠ͱͷ؍ଌ΋γ ϛϡϨʔτͰ͖Δ

12 ֓ཁ ಈతͳΞΫλʔ ੩తͳഎܠ εύʔεͳ άϦουͰ ϞσϧԽ ֶशՄೳͳજࡏද ݱ͔ΒϋΠύʔω οτϫʔΫͰදݱ Λੜ੒ NNF Λ߹੒ ηϯαத৺͔ΒϨΠΛඈ͹͠ɼ ϘϦϡʔϜϨϯμϦϯάΛߦ͍ɼ ֤ηϯαʹσίʔυ ͦΕͧΕผʑʹଟॏղ૾౓ಛ௃άϦου ͰϞσϧԽ 3࣍ݩͷγʔϯΛ • ੩తͳഎܠͱ • ಈతͳΞΫλʔͷू߹ ʹ෼཭ ਤ͸ [1] ಈըͷ34:00 ͔ΒҾ༻ͯ͠Ճ޻

13 ֓ཁ ಈతͳΞΫλʔ ੩తͳഎܠ εύʔεͳ άϦουͰ ϞσϧԽ ֶशՄೳͳજࡏද ݱ͔ΒϋΠύʔω οτϫʔΫͰදݱ Λੜ੒ NNF Λ߹੒ ηϯαத৺͔ΒϨΠΛඈ͹͠ɼ ϘϦϡʔϜϨϯμϦϯάΛߦ͍ɼ ֤ηϯαʹσίʔυ ͦΕͧΕผʑʹଟॏղ૾౓ಛ௃άϦου ͰϞσϧԽ 3࣍ݩͷγʔϯΛ • ੩తͳഎܠͱ • ಈతͳΞΫλʔͷू߹ ʹ෼཭ ਤ͸ [1] ಈըͷ34:00 ͔ΒҾ༻ͯ͠Ճ޻

14 ֓ཁ ಈతͳΞΫλʔ ੩తͳഎܠ εύʔεͳ άϦουͰ ϞσϧԽ ֶशՄೳͳજࡏද ݱ͔ΒϋΠύʔω οτϫʔΫͰදݱ Λੜ੒ NNF Λ߹੒ ηϯαத৺͔ΒϨΠΛඈ͹͠ɼ ϘϦϡʔϜϨϯμϦϯάΛߦ͍ɼ ֤ηϯαʹσίʔυ ͦΕͧΕผʑʹଟॏղ૾౓ಛ௃άϦου ͰϞσϧԽ 3࣍ݩͷγʔϯΛ • ੩తͳഎܠͱ • ಈతͳΞΫλʔͷू߹ ʹ෼཭ ਤ͸ [1] ಈըͷ34:00 ͔ΒҾ༻ͯ͠Ճ޻

15 ֓ཁ ಈతͳΞΫλʔ ੩తͳഎܠ εύʔεͳ άϦουͰ ϞσϧԽ ֶशՄೳͳજࡏද ݱ͔ΒϋΠύʔω οτϫʔΫͰදݱ Λੜ੒ NNF Λ߹੒ ηϯαத৺͔ΒϨΠΛඈ͹͠ɼ ϘϦϡʔϜϨϯμϦϯάΛߦ͍ɼ ֤ηϯαʹσίʔυ ͦΕͧΕผʑʹଟॏղ૾౓ಛ௃άϦου ͰϞσϧԽ 3࣍ݩͷγʔϯΛ • ੩తͳഎܠͱ • ಈతͳΞΫλʔͷू߹ ʹ෼཭ ਤ͸ [1] ಈըͷ34:00 ͔ΒҾ༻ͯ͠Ճ޻

ख๏

17 γʔϯΛ Neural Network Ͱදݱ l NFF ͸ NeRF ΍ Occupancy ؔ਺ͷ্Ґू߹ l ߹੒Մೳ = ͍͔ͭ͘ͷγϯϓϧͳ NFF Λ૊Έ߹Θͤͯɼෳࡶͳ৔Λ࡞Δ͜ͱ͕Ͱ͖Δ Neural Feature Field (NFF) NeRF ͳΒ… • s: ϘϦϡʔϜີ౓ • f: RGB radiance Occupancy Func. ͳΒ… • s: occupancy ͷ֬཰ 3࣍ݩ্ ͷҐஔ x ࢹ఺ ํ޲ d ҉໧తͳ δΦϝτϦ s ಛ௃ هड़ࢠ f

18 ֶशՄೳͳଟॏղ૾౓ಛ௃άϦουΛ NNFͱ૊Έ߹ΘͤΔ େҬಛ௃ͱہॴಛ௃Λཱ྆ͨ͠ޮ཰తͳಛ௃άϦου ΫΤϦϙΠϯτ ଟॏղ૾౓ಛ௃άϦου ֤ϨϕϧͰ τϦϦχΞิؒ ࢹ఺ϕΫτϧͱ݁߹ͯ͠ɼ .-1IFBEͰॲཧ MLP ジオ メト リ 特徴 記述 ⼦ l େҬతͳίϯςΫετͱɼࡉ͔͍ಛ௃ͷ྆ํΛΤϯίʔυͰ͖Δ l MLP ϔουͷେ͖͞Λখ͘͞Ͱ͖ɼਪ࿦࣌ؒΛݮΒͤΔ l ࣮ࡍʹ͸ɼ Instant-NGP [2] ͷΑ͏ͳɼଟॏղ૾౓ϋογϡΤϯίʔσΟϯάͰ࣮ ૷͞Ε͍ͯΔΒ͍͠ʢ͋·Γࡉ͔͍৘ใ͸ॻ͍͍ͯͳ͍ʣ [2] Thomas Muller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. In SIGGRAPH, 2022.

19 ܭଌं͕྆௨ա͖ͯͨ͠ྖҬΛ੩తͳഎܠͱಈతͳ ΞΫλʔʹ෼͚ͯϞσϧԽ͢Δ l ผʑͰ 3D ۭؒϘϦϡʔϜΛఆٛ l ੩తͳഎܠɿੈք࠲ඪܥͰදݱ l ಈతͳΞΫλʔɿͦΕͧΕͷ෺ମத৺࠲ඪܥͰදݱ l ͦΕͧΕผʑͷଟॏղ૾౓ಛ௃άϦουͱ NFF ͰϞσϧԽ͢Δ l ֤ΞΫλʔͷ 3D ϞʔγϣϯΛ੾Γ཭͠ɼ ܗঢ়ͱ֎؍ͷදݱʹूதͰ͖Δ l ܗঢ়ͷදݱʹ͸ɼූ߸෇͖ڑ཭ؔ਺ (SDF) Λ࢖͏ɽ ߹੒χϡʔϥϧදݱ Dynamic Actors Static Background B ݁ߏͳ͕͍ڑ཭૸ߦ → ޮ཰తͳදݱ͕ඞཁ SDV A1 A2 ɿA1 ͷي੻ ɿA2 ͷي੻

20 l ΄ͱΜͲϑϦʔεϖʔεͳͷͰɼಛ௃άϦουΛૄʹͰ͖ɼܭࢉίετΛݮΒͤΔ l Geometric Prior Λ༻͍Δ͜ͱͰɼγʔϯͷ 3D ߏ଄ΛΑΓద੾ʹϞσϧԽͰ͖Δ l େ͖ͳ֎ૠΛ൐͏৽͍͠ࢹ఺ΛγϛϡϨʔτ͢Δͱ͖ʹ΋ޮՌత ີͳߴղ૾౓ϘΫηϧάϦου͸อ࣋Ͱ͖ͳ͍ → ද໘ۙ๣ͷϘΫηϧͷಛ௃͚ͩΛ࠷దԽ ૄͳഎܠγʔϯϞσϧ t = 1 t = 4 t = 7 ఺܈ͷਤ: https://scale.com/open-av-datasets/pandaset ͔ΒҾ༻ ֤ϑϨʔϜ͔Β LiDAR ఺܈Λू໿͠ɼ ີͳ 3D γʔϯ఺܈Λͭ͘Δ ू໿ ͨ࢟͠ Voxelize Morphology Dilation & ෼ׂ ϑϦʔεϖʔε ີͳ Occ. Grid ද໘ۙ๣ۭؒ փ৭ͷ෦෼͕࡞ΒΕΔ

21 l ͦΕͧΕͷΞΫλʔͷಛ௃άϦουΛͲ͏΍ͬͯಘΕ͹ྑ͍ʁ l Կ͕خ͍͠ʁ l ҟͳΔΞΫλʔ͸ҟͳΔࢹ఺͔Β؍ଌ͞ΕΔ → ಛ௃άϦου͕ҟͳΔྖҬͰ΋༗ӹ l ࣄલ෼෍ֶ͕श͞ΕΔ͜ͱͰɼಛ௃ؒͷ૬͕ؔัଊ͞Εɼ ݟ͑Δ෦෼͔Βݟ͑ͳ͍෦෼͕ਪଌͰ͖Δ ҰൠԽΞΫλʔϞσϧ ֤ΞΫλʔΛผʑͷಛ௃άϦουͰ ύϥϝʔλԽ 😖 ϝϞϦ͕ͨ͘͞Μඞཁ 😖 Overfitɿະ஌ͷࢹ఺ʹ൚Խ͠ͳ͍ ֤ΞΫλʔΛ௿࣍ݩͷજࡏίʔυͰ ϞσϧԽ͠ɼϋΠύʔωοτϫʔΫ ʹΑΓಛ௃άϦουΛճؼ ಈతͳ ΞΫλʔ

22 ֤ΞΫλʔͷNFFΛੈք࠲ඪʹม׵ͯ͠ എܠͱஔ͖׵͑Δ l ֤ΞΫλʔΛॴ๬ͷϙʔζͰม׵ l ੩తͳഎܠ͸ૄͳಛ௃άϦουͳͷͰɼ୯ʹϑϦʔεϖʔεͱஔ͖׵͑Δ l ͜ΕʹΑΓɼγʔϯͷ࠶ߏ੒ɼΞΫλʔͷૠೖɾ࡟আɾૢ࡞ɼࣗंͷૢ࡞ ͕Ͱ͖ΔΑ͏ʹͳΔ NFF ͷ߹੒ SDV A1 SDV A1 A2 SDV ΦϦδφϧ ૠೖ ࡟আ SDV A1 ૢ࡞

23 l Χϝϥը૾ɿ2D ͷҐஔʢϐΫηϧʣ + ஋ʢRGBʣ l Hybrid volume and neural rendering ϚϧνϞʔμϧͳηϯαγϛϡϨʔγϣϯʢΧϝϥը૾ + LiDAR ఺܈ʣ 視点 多重解像度特徴 グリッド MLP 実際の画像より⼩さい = 効率的 ηϯαத৺ o Ray ํ޲ d 2D ͷಛ௃Ϛοϓ F ͔Βɼ 2D ͷ CNN Λ࢖ͬͯɼ RGB ը૾ΛϨϯμϦϯά αϯϓϧΛू໿͠ɼϘϦϡʔϜϨϯμ ϦϯάʹΑΓ 2D ͷϐΫηϧ͝ͱͷ ಛ௃هड़ࢠΛಘΔ Ray ʹԊͬͨ 3D ͷϙΠϯτΛ நग़͠ɼNNF ʹΑΓಛ௃ f ͱ δΦϝτϦ s ΛಘΔ Opacity (不透明度) SDF s の関数となっている 近似的なステップ関数 この重みは物体の表⾯付 近に集中するような分布 になっているはず

24 l LiDAR ఺܈ɿ3DͷҐஔʢਂ౓ʣ+ ڧ౓ʢ൓ࣹ཰ʣ ϚϧνϞʔμϧͳηϯαγϛϡϨʔγϣϯʢΧϝϥը૾ + LiDAR ఺܈ʣ ਂ౓ αϯϓϧ͞Εͨ 3D ఺ͷਂ౓ͷظ଴஋ Λܭࢉ͠ɼਂ౓ͷܭଌΛγϛϡϨʔτ ͢Δ MLP 視点 多重解像度特徴 グリッド MLP ηϯαத৺ o Ray ํ޲ d Ray ʹԊͬͨ 3D ͷϙΠϯτΛ நग़͠ɼNNF ʹΑΓಛ௃ f ͱ δΦϝτϦ s ΛಘΔ ڧ౓ ϘϦϡʔϜϨϯμϦϯάͨ݁͠ՌΛ MLPͷσίʔμͰॲཧͯ͠༧ଌ （同じ）

25 l ࠷దԽର৅ɿ͢΂ͯͷಛ௃άϦου ʢʴજࡏίʔυɼϋΠύʔωοτϫʔΫɼMLPϔουɼσίʔμʔʣ l ଛࣦؔ਺ ֶशํ๏ Χϝϥը૾ ͷ࠶ߏ੒ L2 photometric loss Perceptual loss ʢֶशࡁΈ VGG ͷग़ྗΛൺֱʣ LiDAR ఺܈ ͷ࠶ߏ੒ ఺܈͸ϊΠδʔͳͷͰɼਂ౓ͷޡ͕ࠩେ͖͍ 5 % Λ֎Ε஋ͱͯ͠আڈ͢Δ ਖ਼ଇԽ • ϘϦϡʔϜϨϯμϦϯάͷαϯϓϧͷॏΈ෼෍ w ͕ද໘पลʹूத͢ΔΑ͏ʹ • SDF s ͕ Eikonal ํఔࣜΛຬͨ͢Α͏ʹ ʢSmooth zero level set ʣ ఢରతଛࣦ • ؍ଌࢹ఺ͱະ؍ଌࢹ఺ͰͷγϛϡϨʔγϣϯը૾Λ۠ผͰ͖ͳ͘͢ΔΑ͏ʹ ʢະ؍ଌࢹ఺Ͱͷੜ੒ͷ඼࣭Λ্͍͛ͨʣ 外挿設定で 効果あり！

26 l ࠷దԽର৅ɿ͢΂ͯͷಛ௃άϦου ʢʴજࡏίʔυɼϋΠύʔωοτϫʔΫɼMLPϔουɼσίʔμʔʣ l ଛࣦؔ਺ ֶशํ๏ Χϝϥը૾ ͷ࠶ߏ੒ L2 photometric loss Perceptual loss ʢֶशࡁΈ VGG ͷग़ྗΛൺֱʣ LiDAR ఺܈ ͷ࠶ߏ੒ ఺܈͸ϊΠδʔͳͷͰɼਂ౓ͷޡ͕ࠩେ͖͍ 5 % Λ֎Ε஋ͱͯ͠আڈ͢Δ ਖ਼ଇԽ • ϘϦϡʔϜϨϯμϦϯάͷαϯϓϧͷॏΈ෼෍ w ͕ද໘पลʹूத͢ΔΑ͏ʹ • SDF s ͕ Eikonal ํఔࣜΛຬͨ͢Α͏ʹ ʢSmooth zero level set ʣ ఢରతଛࣦ • ؍ଌࢹ఺ͱະ؍ଌࢹ఺ͰͷγϛϡϨʔγϣϯը૾Λ۠ผͰ͖ͳ͘͢ΔΑ͏ʹ ʢະ؍ଌࢹ఺Ͱͷੜ੒ͷ඼࣭Λ্͍͛ͨʣ 外挿設定で 効果あり！

27 l ࠷దԽର৅ɿ͢΂ͯͷಛ௃άϦου ʢʴજࡏίʔυɼϋΠύʔωοτϫʔΫɼMLPϔουɼσίʔμʔʣ l ଛࣦؔ਺ ֶशํ๏ Χϝϥը૾ ͷ࠶ߏ੒ L2 photometric loss Perceptual loss ʢֶशࡁΈ VGG ͷग़ྗΛൺֱʣ LiDAR ఺܈ ͷ࠶ߏ੒ ఺܈͸ϊΠδʔͳͷͰɼਂ౓ͷޡ͕ࠩେ͖͍ 5 % Λ֎Ε஋ͱͯ͠আڈ͢Δ ਖ਼ଇԽ • ϘϦϡʔϜϨϯμϦϯάͷαϯϓϧͷॏΈ෼෍ w ͕ද໘पลʹूத͢ΔΑ͏ʹ • SDF s ͕ Eikonal ํఔࣜΛຬͨ͢Α͏ʹ ʢSmooth zero level set ʣ ఢରతଛࣦ • ؍ଌࢹ఺ͱະ؍ଌࢹ఺ͰͷγϛϡϨʔγϣϯը૾Λ۠ผͰ͖ͳ͘͢ΔΑ͏ʹ ʢະ؍ଌࢹ఺Ͱͷੜ੒ͷ඼࣭Λ্͍͛ͨʣ 外挿設定で 効果あり！

28 l ࠷దԽର৅ɿ͢΂ͯͷಛ௃άϦου ʢʴજࡏίʔυɼϋΠύʔωοτϫʔΫɼMLPϔουɼσίʔμʔʣ l ଛࣦؔ਺ ֶशํ๏ Χϝϥը૾ ͷ࠶ߏ੒ L2 photometric loss Perceptual loss ʢֶशࡁΈ VGG ͷग़ྗΛൺֱʣ LiDAR ఺܈ ͷ࠶ߏ੒ ఺܈͸ϊΠδʔͳͷͰɼਂ౓ͷޡ͕ࠩେ͖͍ 5 % Λ֎Ε஋ͱͯ͠আڈ͢Δ ਖ਼ଇԽ • ϘϦϡʔϜϨϯμϦϯάͷαϯϓϧͷॏΈ෼෍ w ͕ද໘पลʹूத͢ΔΑ͏ʹ • SDF s ͕ Eikonal ํఔࣜΛຬͨ͢Α͏ʹ ʢSmooth zero level set ʣ ఢରతଛࣦ • ؍ଌࢹ఺ͱະ؍ଌࢹ఺ͰͷγϛϡϨʔγϣϯը૾Λ۠ผͰ͖ͳ͘͢ΔΑ͏ʹ ʢະ؍ଌࢹ఺Ͱͷੜ੒ͷ඼࣭Λ্͍͛ͨʣ 外挿設定で 効果あり！

࣮ݧ

30 طଘख๏Λ྇կ ࣮ݧɿ৽͍͠Ϗϡʔͷੜ੒ ఏҊख๏ • Interpolation: 1 ϑϨʔϜ͓͖ʹαϯϓϧֶͯ͠शɼ࢒ΓͰݕূ • Lane shift: ԣʹ 2 ~ 3 m ఔ౓ͣΕͨࢹ఺͔Βੜ੒

31 طଘख๏Λ྇կ ࣮ݧɿ৽͍͠Ϗϡʔͷੜ੒ • Interpolation: 1 ϑϨʔϜ͓͖ʹαϯϓϧֶͯ͠शɼ࢒ΓͰݕূ • Lane shift: ԣʹ 2 ~ 3 m ఔ౓ͣΕͨࢹ఺͔Βੜ੒ • ं྆ͷੜ੒඼࣭ • ϨʔϯϥΠϯ΍࿏໘ඪࣔͷੜ੒඼࣭ • ੜ੒͞ΕΔը૾ͷߴਫ਼ࡉ͞ • FVS [6] ͸ͲͪΒ΋ΘΓͱྑͦ͞͏ʁ [6] Gernot Riegler and Vladlen Koltun. Free view synthesis. In ECCV, 2020.

32 طଘख๏Λ྇կ ࣮ݧɿ৽͍͠Ϗϡʔͷੜ੒ ఏҊख๏ • Interpolation: 1 ϑϨʔϜ͓͖ʹαϯϓϧֶͯ͠शɼ࢒ΓͰݕূ • Lane shift: ԣʹ 2 ~ 3 m ఔ౓ͣΕͨࢹ఺͔Βੜ੒ • ະ஌ࢹ఺ͩͱɼഎܠͷੜ੒඼࣭ʹ͔ͳΓͷ͕ࠩ͋Γͦ͏ • طଘख๏͸΄΅ੜ੒Ͱ͖͍ͯͳ͍

33 طଘख๏Λ྇կ ࣮ݧɿ৽͍͠Ϗϡʔͷੜ੒ • ผͷྫͰ΋ಉ༷ͷ܏޲͕ΈΒΕΔ • Interpolation: 1 ϑϨʔϜ͓͖ʹαϯϓϧֶͯ͠शɼ࢒ΓͰݕূ • Lane shift: ԣʹ 2 ~ 3 m ఔ౓ͣΕͨࢹ఺͔Βੜ੒

34 ఆྔతʹ΋طଘख๏ΑΓ΋ྑ͍ ࣮ݧɿ ৽͍͠Ϗϡʔͷੜ੒ • Interpolation: 1 フレームおきにサンプルして学習，残りで検証 • Lane shift: 横に 2 ~ 3 m 程度ずれた視点から⽣成

35 ϊΠζ͕গͳ͘ɼΑΓ࿈ଓతͳϏʔϜϦϯάΛ΋ͭ LiDAR ఺܈͕ੜ੒ ࣮ݧɿLiDAR఺܈ͷ߹੒

36 Perception ͷධՁ΍ֶशʹ࢖͑Δ͔ʁ Domain gap は少ない？ ϦΞϧ σʔλ ֶश ධՁ ߹੒ σʔλ ߹੒ σʔλ ֶश ධՁ ϦΞϧ σʔλ ͜ͷ͕ࠩখ͍͞ͱ͍͍ͳ • -PH3FQMBZ -BOF4IJGUͷ྆ઃఆͰ΄΅ࠩͳ͠ • 3FBMˠ3FBM͔ΒͷੑೳྼԽ΋খ͍͞ Data Augmentation としても使える？ Real → Real は 40.9% Real → Real は 40.9% ߹੒σʔλ͚ͩΛֶशʹ࢖͏৔߹ • طଘख๏͸ੑೳྼԽ͕େ͖͍͕ɼఏҊख๏Ͱ͸ Ή͠Ζੑೳ͕ྑ͘ͳͬͨ ϦΞϧ ߹੒σʔλͷ৔߹ • ఏҊख๏͸ੑೳ޲্͕େ͖͍ → Domain gap は⼗分⼩さい ˠ %BUB"VHNFOUBUJPOͱͯ͠΋༗ޮ

37 Perception ͷධՁ΍ֶशʹ࢖͑Δ͔ʁ Domain gap は少ない？ ϦΞϧ σʔλ ֶश ධՁ ߹੒ σʔλ ߹੒ σʔλ ֶश ධՁ ϦΞϧ σʔλ ͜ͷ͕ࠩখ͍͞ͱ͍͍ͳ • -PH3FQMBZ -BOF4IJGUͷ྆ઃఆͰ΄΅ࠩͳ͠ • 3FBMˠ3FBM͔ΒͷੑೳྼԽ΋খ͍͞ Data Augmentation としても使える？ Real → Real は 40.9% Real → Real は 40.9% ߹੒σʔλ͚ͩΛֶशʹ࢖͏৔߹ • طଘख๏͸ੑೳྼԽ͕େ͖͍͕ɼఏҊख๏Ͱ͸ Ή͠Ζੑೳ͕ྑ͘ͳͬͨ ϦΞϧ ߹੒σʔλͷ৔߹ • ఏҊख๏͸ੑೳ޲্͕େ͖͍ → Domain gap は⼗分⼩さい ˠ %BUB"VHNFOUBUJPOͱͯ͠΋༗ޮ

38 Perception ͷධՁ΍ֶशʹ࢖͑Δ͔ʁ Domain gap は少ない？ ϦΞϧ σʔλ ֶश ධՁ ߹੒ σʔλ ߹੒ σʔλ ֶश ධՁ ϦΞϧ σʔλ ͜ͷ͕ࠩখ͍͞ͱ͍͍ͳ • -PH3FQMBZ -BOF4IJGUͷ྆ઃఆͰ΄΅ࠩͳ͠ • 3FBMˠ3FBM͔ΒͷੑೳྼԽ΋খ͍͞ Data Augmentation としても使える？ Real → Real は 40.9% Real → Real は 40.9% ߹੒σʔλ͚ͩΛֶशʹ࢖͏৔߹ • طଘख๏͸ੑೳྼԽ͕େ͖͍͕ɼఏҊख๏Ͱ͸ Ή͠Ζੑೳ͕ྑ͘ͳͬͨ ϦΞϧ ߹੒σʔλͷ৔߹ • ఏҊख๏͸ੑೳ޲্͕େ͖͍ → Domain gap は⼗分⼩さい ˠ %BUB"VHNFOUBUJPOͱͯ͠΋༗ޮ

·ͱΊ

40 l ࣮ੈքͷγφϦΦΛ׆༻ͯ͠ɼࣗ཯γεςϜͷςετʹ΋࢖༻Ͱ͖Δຊ෺ ͦͬ͘ΓͷԾ૝ੈքΛߏங l ΧϝϥɾLiDAR ͷγʔέϯεΛೖྗͱ͠ɼಈతͳΞΫλʔͱ੩తͳഎܠΛ ෼ղɾ࠶ߏ੒Ͱ͖ΔχϡʔϥϧηϯαγϛϡϨʔλ l υϝΠϯΪϟοϓ͕খ͘͞ɼੜ੒͞Εͨ৽͍͠γφϦΦʹΑΔ Closed- loop ςετʹ΋࢖༻Մೳ l ࠓޙͷ՝୊ɾݶք l র໌৚݅ɾఱީ৚݅΁ͷରԠ l ଟؔઅΞΫλʔ΁ͷରԠ ·ͱΊ [4] Jingkang Wang, Siva Manivasagam, Yun Chen, Ze Yang , Ioan Andrei Bârsan, Anqi Joyce Yang, Wei–Chiu Ma, Raquel Urtasun. CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation. In CoRL, 2022. ଟؔઅΞΫλʔͷྫ [4] ΑΓҾ༻

41 1. Raquel Urtasun. Next Generation Simulation for the Safe Development and Deployment of Self-Driving Technology. In CVPR Vision- Centric Autonomous Driving workshop, 2023. https://www.youtube.com/watch?v=0RjF9xbkiAY 2. Thomas Muller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. In SIGGRAPH, 2022. 3. Pengchuan Xiao, Zhenlei Shao, Steven Hao, Zishuo Zhang, Xiaolin Chai, Judy Jiao, Zesong Li, Jian Wu, Kai Sun, Kun Jiang, et al. Pandaset: Advanced sensor suite dataset for autonomous driving. In ITSC, 2021 https://scale.com/open-av-datasets/pandaset 4. Jingkang Wang, Siva Manivasagam, Yun Chen, Ze Yang , Ioan Andrei Bârsan, Anqi Joyce Yang, Wei–Chiu Ma, Raquel Urtasun. CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation. In CoRL, 2022. 5. Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In CVPR, 2021. 6. Gernot Riegler and Vladlen Koltun. Free view synthesis. In ECCV, 2020. ࢀߟจݙ