Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[ECCV2020 論文読み会] MotionSqueeze: Neural Motion Feature Learning for Video Understanding

C39ef030c6c350a571ac7058f527fb4b?s=47 tomoyukun
December 12, 2020

[ECCV2020 論文読み会] MotionSqueeze: Neural Motion Feature Learning for Video Understanding

C39ef030c6c350a571ac7058f527fb4b?s=128

tomoyukun

December 12, 2020
Tweet

Transcript

  1. ECCV࿦จಡΈձˏશ೔ຊCVษڧձ Motion Squeeze: 
 Neural Motion Feature Learning for Video

    Understanding ླ໦ ஐ೭ @tomoyukun
  2. ࣗݾ঺հ ླ໦ஐ೭!UPNPZVLVO ˙ ུྺ wdɿܚጯ੨໦ݚ wdɿ %F/".PCJMJUZ5FDIOPMPHJFT wdɿ$ZCFS"HFOU"*-BC 2 )1ɿIUUQTUPNPZVLVOHJUIVCJPCJPHSBQIZ

  3. ঺հ࿦จ 3 ˙ ಈըೝࣝʹ͓͚ΔNPUJPOಛ௃நग़ͷͨΊͷ৽͍͠ϞδϡʔϧͷఏҊ ˙ બΜͩཧ༝ɿ ಈըೝࣝ෼໺ͷڞ௨ೝࣝతͳ՝୊ʹγϯϓϧͳख๏ͰΞϓϩʔν͠ɺ ͔ͬ͠Γੑೳ޲্͕ୡ੒͞Ε͍͔ͯͨΒ &$$7 <1BQFS><1SPKFDU><$PEF>

  4. 7JEFPSFDPHOJUJPOVOEFSTUBOEJOH ˙ ಈը͔Βಛఆͷ৘ใΛநग़͢Δ໰୊શൠ ˙ ൚༻తͳಈըೝࣝϞσϧͱͯ͠ͷධՁ͸5SJNNFEBDUJPOSFDPHOJUJPOʹ͓͍ ͯߦΘΕΔ͜ͱ͕ଟ͍ 4 "DUJPOSFDPHOJUJPO ಈըதʹؚ·ΕΔߦಈΫϥεͷ෼ྨ໰୊ "DUJPOMPDBMJ[BUJPO

    7JEFPDBQUJPOJOH 7JEFPTVNNBSJ[BUJPO ʜ ˙ 5SJNNFEɿ ಈըதʹ୯ҰͷಈըͷΈ ؚ·ΕΔ ˙ 6OUSJNNFEɿ ೖྗಈըதʹෳ਺ͷߦಈؚ͕ ·Ε͍ͯͨΓɺCBDLHSPVOE DMBTTؚ͕·ΕΔ <> <> <> <>
  5. 1SFWJPVTXPSLT 5 ը૾ೝࣝʹ͓͚Δ %$//ͷ੒ޭ

  6. 1SFWJPVTXPSLT 6 %$// UFNQPSBMDPOW 5XP4USFBN$// ը૾ೝࣝʹ͓͚Δ %$//ͷ੒ޭ <> <>

  7. 5XP4USFBN$// 4JNPOZBO /*14` <> 3(#ೖྗͷ%$//ͱ0QUJDBMqPXTೖྗͷ%$//ͷΞϯαϯϒϧ ˙ ೖྗ࣌఺Ͱݟ͑৘ใ BQQFBSBODF ͱಈ͖৘ใ NPUJPO

    ʹ໌ࣔతʹ෼ׂ ˙ )BOEDSBGUFEGFBUVSFΛ௒͑Δਫ਼౓Λୡ੒ 7 0QUJDBMqPXɿ ϑϨʔϜؒͷϐΫηϧͷҠಈϕΫτϧ ˙ ݩͷً౓஋ͱҠಈઌͷً౓஋͕Ұக͢Δͱ͍͏੍໿ͱ ਖ਼ଇԽ߲Λݩʹ࠷దԽ໰୊Λղ͘͜ͱͰಘΒΕΔ ˙ ܭࢉίετ͕େ͖͍ͷ͕೉఺
  8. %$// UFNQPSBMDPOW ը૾ೝࣝͰ੒ޭͨ͠%DPOWΛ࣌ؒํ޲ʹ֦ு 8 $% 5SBO *$$7` <> 4MPX'BTU 'FJDIUFOIPGFS

    *$$7` <> 54. -JO *$$7` <> % YZ DPOWΛ % YZU DPOWʹ֦ு %DPOWΛ% %DPOWͰ දݱ͠ɺύϥϝʔλΛޮ཰Խ ۭؒతղ૾౓ॏࢹͷ4MPXQBUIXBZͱ ࣌ؒతղ૾౓ॏࢹͷ'BTUQBUIXBZʹ෼ׂ ຊ࿦จͷϕʔεϥΠϯ ˙ %$//ͷҰ෦ͷಛ௃ϚοϓΛ࣌ؒํ޲ ʹγϑτ ˙ γϑτޙͷ%DPOW͕νϟωϧ਺ͷ গͳ͍%DPOWʹରԠ͓ͯ͠Γɺ ࿦จதͰ΋l1TFVEP%NPEFMzͱݺশ ˙ %$//ͱಉ౳ͷύϥϝʔλ਺Ͱޮ཰త ʹۭ࣌ؒಛ௃Λநग़ ΞʔΩςΫνϟ؍఺Ͱͷ ༷ʑͳվྑͷऔΓ૊Έ 3   % 5SBO $713` <>
  9. 1SFWJPVTXPSLT 9 5XP4USFBN$// 5XP4USFBN%$// ը૾ೝࣝʹ͓͚Δ %$//ͷ੒ޭ <> <> <> %$//

    UFNQPSBMDPOW
  10. 5XP4USFBN%$// $BSSFJSB $713` <> 3(#ೖྗͷ%$//ͱ0QUJDBMqPXTೖྗͷ%$//ͷΞϯαϯϒϧ 10 5XPTUSFBN%$// ˙ %$//Λ࢖༻ͯ͠΋σʔληοτʹΑͬͯ͸'MPX3(# ˙

    3(# 'MPXͷΞϯαϯϒϧͰେ෯ʹਫ਼౓্͕͕Δ BDDVSBDZPOBDUJPOSFDPHOJUJPO 0QUJDBMqPX͕༗ޮͳΒ ྡ઀͢Δ3(#ϑϨʔϜը૾͔ΒܭࢉՄೳͳ ͷͰ FFʹ%$//Λֶश͢ΔதͰͦ͏͍ͬͨಛ௃Λ௫ΜͰཉ͍͠ %DPOWͷදݱೳྗ͸NPUJPOΛଊ͑Δͷʹෆे෼ͳͷͰ͸ʁ % %
  11. $//XJUIPQUJDBMqPXLOPXMFEHF 0QUJDBMqPXΛಉ࣌ʹਪఆ 11 'MPX4USFBNΛৠཹ "DUJPO'MPX/FU .PUJPO"VHNFOUFE3(#4USFBN .PUJPOΛແཧ΍Γֶशͤ͞Δ͜ͱͰҰఆͷޮՌ͸͋ΔΑ͏ ֶश࣌ʹ͸ґવͱͯ͠0QUJDBMqPXΛܭࢉ͢Δඞཁ͕͋Δ ωοτϫʔΫߏ଄؍఺ͰࠜຊతʹղܾͰ͖ͳ͍͔ʁ :VF)FJ/H

    8"$7` <> $SBTUP $713` <>
  12. 3FQSFTFOUBUJPO'MPX 1JFSHJPWBOOJ $713` <> 12 0QUJDBMqPXͷ࠷దԽܭࢉΛ໛ͨ͠FFʹֶशՄೳͳϞδϡʔϧΛఏҊ தؒ૚ʹೖΕΒΕΔͷͰ ಛ௃ϚοϓͷNPUJPO΋ܭࢉͰ͖Δ ඍ෼ՄೳͳॲཧΛJUFSBUJWFʹߦ͏ͷΈ *UFSBUJPOPG57-PQUJNJ[BUJPO

    'MPXMBZFS தؒ૚ʹೖΕΕ͹ղ૾౓͸མͪΔ͕ ґવͱͯ͠'MPXMBZFSͷܭࢉίετ͸େ͖͍
  13. 1SPQPTJUJPO 13 .PUJPOநग़ͷͨΊͷl.PUJPO4RVFF[F .4 NPEVMFzͷఏҊ ˙ ྡ઀ϑϨʔϜͷಛ௃ϚοϓͷDPSSFMBUJPO͔ΒରԠ఺Λநग़͠NPUJPOΛܭࢉ ˙ ඍ෼ՄೳͳॲཧͰߏ੒͞ΕɺFOEUPFOEʹֶशՄೳ ˙

    ਫ਼౓ͱܭࢉίετͷτϨʔυΦϑΛେ෯ʹվળ
  14. .PUJPO4RVFF[F ྡ઀ϑϨʔϜؒͷରԠ͢Δ࠲ඪΛ֬཰஋ͱͯ͠ٻΊͯɺมҟͷظ଴஋ΛͱΔ 14

  15. .PUJPO4RVFF[Fc$PSSFMBUJPODPNQVUBUJPO ˙ ྡ઀ϑϨʔϜͷ࠲ඪରͷ૬ؔ஋ͱͯ͠ ಛ௃Ϛοϓͷ஋ͷ಺ੵΛܭࢉ ˙ ஫໨࠲ඪ ͔ΒͷมҐ ͸ Λ த৺ͱͨ͠

    ͷۙ๣ύονͷൣғʹ੍ݶ ˙ ԋࢉ਺ '-01T ͸ x p x P × P H × W × C × P2 15 P × P ʜ ஫໨࠲ඪ ஫໨࠲ඪ͔Βͷมҟ ࣌ࠁ
  16. .PUJPO4RVFF[Fc%JTQMBDFNFOUDBMDVMBUJPO ˙ ֤஫໨࠲ඪͷ૬ؔ஋ΛTPGUNBYؔ਺ʹೖྗ͠ ஫໨࠲ඪͷରԠ఺Λ֬཰஋ͱͯ͠ಘΔ ˙ ૬ؔ஋ʹ(BVTTJBOLFSOFMΛద༻ͯ͠PVUMJFSΛ཈੍ 16   

          (BVTTJBOLFSOFM Թ౓ύϥϝʔλ
  17. .PUJPO4RVFF[Fc%JTQMBDFNFOUDBMDVMBUJPO ˙ ֤஫໨࠲ඪͷ૬ؔ஋ΛTPGUNBYؔ਺ʹೖྗ͠ ஫໨࠲ඪͷରԠ఺Λ֬཰஋ͱͯ͠ಘΔ ˙ ૬ؔ஋ʹ(BVTTJBOLFSOFMΛద༻ͯ͠PVUMJFSΛ཈੍ ˙ ֬཰஋Λݩʹมҟͷظ଴஋ΛNPUJPOͱͯ͠ܭࢉ w)º8º YZํ޲

    ͷϚοϓͱͳΔ ˙ Թ౓ύϥϝʔλͰNFBONBYΛௐ੔ 17 d Թ౓ύϥϝʔλ (BVTTJBOLFSOFM
  18. .PUJPO4RVFF[Fc$POpEFODFNBQ ˙ ֤஫໨࠲ඪͷۙ๣ύον಺ͷ֬཰஋ͷ࠷େ஋Λ NPUJPOͷDPOpEFODFͱͯ͠ಛ௃ϚοϓʹDPODBU w࠷ऴతʹ)º8º YZํ޲ͷNPUJPO DPOpEFODF ʹ 18 

           
  19. $//΁ͷಋೖํ๏ ˙ .4NPEVMFͰಘΒΕͨNPUJPOಛ௃ͱ.4NPEVMFҎલͷಛ௃ͷཁૉ࿨ΛͱΔ ˙ ࣍ϑϨʔϜ͕ଘࡏ͠ͳ͍࠷ޙͷϑϨʔϜͷNPUJPO͸[FSPQBEEJOH 19

  20. &YQFSJNFOUT ˙ %BUBTFUT w4PNFUIJOH4PNFUIJOH77ɿ  DMBTTFTXJUI WJEFPT  TFDPOET w,JOFUJDTɿ

     DMBTTFTXJUIPWFS WJEFPT  BSPVOETFDPOET ˙ %BUBQSPDFTT w5SBJO  SBOEPNDMJQTBNQMF SBOEPNDSPQͰݻఆ௕ͷDMJQΛநग़͠DMJQ୯ҐͰֶश w5FTU  TJOHMFDMJQQSFEJDUJPOɿ ڪΒ͘DFOUFS DMJQTBNQMF DFOUFSDSPQͰݻఆ௕ͷDMJQΛநग़͠ਪ࿦  BWFSBHFQSFEJDUJPOɿ ,JOFUJDTɿSBOEPNɺ44ɿVOJGPSN SBOEPNDMJQTTBNQMF DFOUFSDSPQ Ͱݻఆ௕ͷDMJQTΛநग़͠ɺͦΕΒͷਪ࿦஋ͷฏۉΛWJEFP୯Ґͷਪ࿦ͱ͢Δ 20 IUUQTXXXTMJEFTIBSFOFUYBWJHJSPEFFQMFBSOJOHGPS WJEFPBDUJPOSFDPHOJUJPOVQD GSPN@BDUJPOTBWF ,JOFUJDT<> 4PNFUIJOH4PNFUIJOH<> IUUQTCODPNEBUBTFUTTPNFUIJOHTPNFUIJOH
  21. "CMBUJPOTUVEJFT 54.3FT/FUʹಋೖɺ4PNFUIJOH4PNFUIJOH7ͰධՁ 21 4ɿ(BVTTJBOLFSOFMͳ͠ ,4ɿ(BVTTJBOLFSOFM͋Γ $.ɿ$POpEFODF͋Γ #%ɿ#BDLXBSEͷNPUJPO΋௥Ճ ˙ .4NPEVMFͷ֤ཁૉͷޮՌ w(BVTTJBOLFSOFMɺ$POpEFODFͷ༗ޮੑ

    Λ֬ೝ w૒ํ޲ͷNPUJPOநग़΋༗ޮ͕ͩɺ ਫ਼౓޲্෯͸େ͖͘ͳ͍ͱͯ͠جຊతʹ͸ GPSXBSEͷΈ࢖༻ BDDVSBDZ
  22. "CMBUJPOTUVEJFT 54.3FT/FUʹಋೖɺ4PNFUIJOH4PNFUIJOH7ͰධՁ 22 ˙ ۙ๣ύοναΠζͷൺֱ wۙ๣ύοναΠζ͸େ͖͍΄Ͳਫ਼౓͕ྑ͍ ܏޲ʹ͋Δ͕ɺ౰વܭࢉίετ΋૿͑Δ ಛ௃ϚοϓͷαΠζ͸Y

  23. "CMBUJPOTUVEJFT 23 ˙ .4NPEVMFΛೖΕΔҐஔͷൺֱ wઙ͗͢ΔͱରԠ఺ΛͱΔͷʹे෼ͳಛ௃நग़ ͕͞Ε͍ͯͳ͍ wਂ͗͢Δͱۭؒํ޲ʹந৅Խ͞Ε͗ͯ͢ NPUJPOͷղ૾౓΋Լ͕Δ w3FT/FUͷCMPDL͋ͨΓ͕ͪΐ͏Ͳྑ͍ wෳ਺ೖΕͯ΋ޮՌ͸খ͍͞

    SFT@Y͸3FT/FUͷY൪໨ͷCMPDLͷ ௚ޙʹ.4NPEVMFΛೖΕΔ͜ͱΛҙຯ͢Δ 54.3FT/FUʹಋೖɺ4PNFUIJOH4PNFUIJOH7ͰධՁ BDDVSBDZ
  24. "CMBUJPOTUVEJFT 54.3FT/FUʹಋೖɺ4PNFUIJOH4PNFUIJOH7ͰධՁ 24 ˙ .PUJPOಛ௃ͷϚʔδํ๏ wBEE ཁૉ࿨ ͕࠷΋ྑ͍ w.4POMZͩͱ͕͢͞ʹਫ਼౓͸Լ͕Δ BDDVSBDZ

  25. "CMBUJPOTUVEJFT ෳ਺ͷϞσϧʹಋೖɺ4PNFUIJOH4PNFUIJOH7ͰධՁ 25 ۭؒ৘ใͷΈΛߟྀͨ͠ϞσϧͷΈͳΒͣ ۭ࣌ؒ৘ใΛߟྀͨ͠Ϟσϧʹಋೖͯ͠΋ ޮՌ͋Γ TQBDF UJNF TQBDFPOMZ

  26. $PNQBSJTPOXJUI4P5" 26 ਫ਼౓ͱܭࢉίετͷτϨʔυΦϑΛେ෯ʹվળ 54.T %$//T PO4PNFUIJOH4PNFUIJOH7 Θ͔ͣͳܭࢉίετͷ૿ՃͰ 5XP4USFBNʹͨ͠ͷͱಉ͘͡Β͍ͷਫ਼౓ʹͳ͍ͬͯΔ

  27. 7JTVBMJ[BUJPO 27 NPUJPO DPOpEFODF NPUJPO DPOpEFODF NPUJPODPMPS DPEJOH ௚઀తͳڭࢣͳ͠ͰNPUJPOΒ͖͠΋ͷΛநग़Ͱ͖͍ͯΔ

  28. ·ͱΊ FOEUPFOEʹֶशՄೳͳNPUJPOநग़ϞδϡʔϧͷఏҊ ˙ ߦಈೝࣝʹ͓͚Δਫ਼౓ͱܭࢉίετͷτϨʔυΦϑΛେ͖͘վળ ˙ நग़͞ΕΔಛ௃͸ϐΫηϧͷҠಈʹରԠ͓ͯ͠Γɺ௚઀తͳڭࢣΛඞཁͱͤͣʹ NPUJPOΛநग़Ͱ͖͍ͯΔ͜ͱΛ֬ೝ ݸਓతײ૝ɿ ˙ ࣗવಈըͷಛੑ΍աڈͷݚڀͷྲྀΕ͔Βग़ͯ͘Δ՝୊ײʹج͍ͮͨఏҊख๏Ͱ

    ೲಘײ͕͋ͬͨ ˙ ͍͔ʹNPUJPOΛநग़͠΍͍͢ܭࢉߏ଄ʹͰ͖Δ͔ͱ͍͏ҙຯͰ͸ɺPQUJDBM qPXਪఆʹ͓͍ͯ༗ޮͳΞʔΩςΫνϟͱ྆ྠͰࠓޙͷվળ͕ظ଴Ͱ͖ͦ͏ ˙ 7JEFPVOEFSTUBOEJOH͕͍ͩͿl"DUJPOSFDPHOJUJPOzʹಛԽ͖ͯͯ͠Δײ͡ ͕͋ΔɻଞͷυϝΠϯ΍λεΫͰ΋༗ޮͳϞσϧ͸มΘΒͳ͍ͩΖ͏͔ʁ 28
  29. 3FGFSFODFT <>,4PPNSP FUBM l6$'"EBUBTFUPGIVNBOBDUJPOTDMBTTFTGSPNWJEFPTJOUIFXJME zJO$P33 <>$(V FUBM z"7""WJEFPEBUBTFUPGTQBUJPUFNQPSBMMZMPDBMJ[FEBUPNJDWJTVBMBDUJPOT zJO$713 <>.(ZHMJ

    FUBM l$SFBUJOHTVNNBSJFTGSPNVTFSWJEFPT zJO&$$7 <>3,SJTIOB FUBM l%FOTF$BQUJPOJOH&WFOUTJO7JEFPT JO*$$7 <>%5SBO FUBM l-FBSOJOHTQBUJPUFNQPSBMGFBUVSFTXJUIEDPOWPMVUJPOBMOFUXPSLT zJO*$$7 <>,4JNPOZBO FUBM l5XPTUSFBNDPOWPMVUJPOBMOFUXPSLTGPSBDUJPOSFDPHOJUJPOJOWJEFPT zJO/*14 <>%5SBO FUBM l"DMPTFSMPPLBUTQBUJPUFNQPSBMDPOWPMVUJPOTGPSBDUJPOSFDPHOJUJPO zJO*$$7 <>$'FJDIUFOIPGFS FUBM l4MPX'BTU/FUXPSLTGPS7JEFP3FDPHOJUJPO zJO*$$7 <>+-JO FUBM l54.5FNQPSBM4IJGU.PEVMFGPS&⒏DJFOU7JEFP6OEFSTUBOEJOH zJO*$$7 <>+$BSSFJSB FUBM l2VPWBEJT BDUJPOSFDPHOJUJPO BOFXNPEFMBOEUIFLJOFUJDTEBUBTFU zJO*$$7 <>+:VF)FJ/H FUBM l"DUJPOqPXOFU-FBSOJOHNPUJPOSFQSFTFOUBUJPOGPSBDUJPOSFDPHOJUJPO zJO8"$7 <>/$SBTUP FUBM l."34.PUJPO"VHNFOUFE3(#4USFBNGPS"DUJPO3FDPHOJUJPO zJO$713 <>"1JFSHJPWBOOJ FUBM l3FQSFTFOUBUJPOqPXGPSBDUJPOSFDPHOJUJPO zJO$713 <>8,BZ FUBM l5IFLJOFUJDTIVNBOBDUJPOWJEFPEBUBTFU zJOBS9JWQSFQSJOU <>3(PZBM FUBM l5IFTPNFUIJOHTPNFUIJOHWJEFPEBUBCBTFGPSMFBSOJOHBOEFWBMVBUJOHWJTVBMDPNNPOTFOTF zJO*$$7 29