画像、その先へ 〜モーション解析への誘い〜 2024.06.13 藤原 研⼈（LINEヤフー株式会社-⼤阪⼤学VHL）

7JSUVBM)VNBO-BC ࠷ઌ୺ͷݚڀ͔Β੡඼·Ͱ

7)-ઓུ 7)- Ϣʔβʔ ੡࡞ऀ ٕज़ࢧԉ ίϯςϯπఏڙ Τϯήʔδϝϯτ

2ϚϧνϞʔμϧج൫Ϟσϧͷొ৔Ͱ ը૾ೝࣝ͸ऴΘΔͷ͔ʁ

"ऴΘͬͪΌ͑ऴΘͬͪΌ͑ʂ

ʜͱ͍͏ͷ΋ ͜Ε·Ͱ͋·Γը૾ͱ޲͖߹ͬͯ͜ͳ͔ͬͨ fixed weights trainable weights !! " !# " !! !# ""(! # ) ""(! ! ) ""(!$ ) !$ !$ " !! " !# " !$ " !! !# !$ GT NN SLR GSLR (ours) <'VKJXBSB)BTIJNPUP$713> <-J*$$7> <-J*$$7>

7)-ͷ׆ಈ ਓؒͷಈ࡞σʔλʹண໨͠ݚڀ։ൃ׆ಈΛ࣮ࢪ

Ϟʔγϣϯσʔλղੳ

എܠͳͥϞʔγϣϯσʔλʁ 2 %ͷεέϧτϯʁͳͥ̎%ಈը͡Όͳ͍ͷʁಈըͩͬͨΒج൫7-.ͰࡁΈͦ͏͡Όͳ͍ʁ "Ͱ͖Ε͹ͦΕ͕͍͍ʂ͚Ͳ ·ͩ Ͱ͖ͳ͍͜ͱͨ͘͞Μ • ಈը͔Βͷ࢟੎ͷԞߦ͖ɾΧϝϥҐஔ౳ͷਪఆͷࠔ೉͞ • Ұ؏ͨ͠ಈ࡞͔൱͔ͷ൑ఆ • FUDʜ <%XJWFEJFUBM %7>

എܠͳͥϞʔγϣϯσʔλʁ 2 %ͷεέϧτϯʁͳͥʢ̎%ʣಈը͡Όͳ͍ͷʁಈըͩͬͨΒج൫7-.ͰࡁΈͦ͏͡Όͳ͍ʁ "ग़ྗͷ࢖͍উख

ಈ࡞ͷऔಘํ๏ ϞʔγϣϯΩϟϓνϟʔσόΠε • ޫֶࣜ • ׳ੑࣜ • ࣓ؾࣜ ಈը͔Βਪఆ͢Δݚڀ΋׆ൃ ՄࢹԽIUUQTMPUIHJUIVCJPPMZNQF#7)@QMBZFSIUNM ޫֶࣜ<$.6.PDBQ%BUBCBTF IUUQNPDBQDTDNVFEV>

ಈ࡞ͷදݱํ๏ ֶश༻ʹ౷Ұදݱʹม׵͢Δඞཁੑ • ,*5εέϧτϯ<1MBQQFSU> • 4.1-εέϧτϯ<-PQFS> • ࠊɺؔઅͷؔઅճసɾ଎౓౳ΛؚΉߴ࣍ݩදݱʹม׵͢Δ͜ͱ΋ • σʔλϑϨʔϜ਺Y ؔઅ਺Yදݱ࣍ݩ 0 1 2 3 5 4 6 9 8 7 11 10 14 13 12 15 17 16 18 19 20 21 0 11 2 5 8 Frame i Frame i+1 Frame i+(N-1) … … SMPL Skeleton Motion Sequence of Right Leg Extract Interpolate 0 1 2 3 4 8 9 10 5 6 7 16 17 18 19 20 11 12 13 14 15 Frame i Frame i+1 Frame i+(N-1) … … … N Frames N Points Motion Patch of Right Leg Motion Sequence of Right Leg Interpolate Extract KIT Skeleton 0 14 11 12 13 15

λεΫ Ξχϝʔγϣϯ౳ΞϓϦέʔγϣϯΛݟਾ͑ͨݚڀ͕׆ൃ ςΩετʹΑΔ Ϟʔγϣϯݕࡧ ςΩετʹΑΔ Ϟʔγϣϯੜ੒ <1FUSPWJDI*$$7> <;IBOH$713>

՝୊ Ξϊςʔγϣϯ͕໘౗͍͘͞ ϑϨʔϜ͝ͱʹهड़͕ඞཁ ϥϕϧ৘ใ σʔλ͕ͦ΋ͦ΋গͳ͍ σʔλܗ͕༷ࣜʑ σʔλ਺ ΋ͬͱଟ༷ͳಈ࡞͕ཉ͍͠ Ұਓͷಈ࡞ʹয఺ ଟ༷ͳಈ͖

7)-ͷݚڀ׆ಈ :V 'VKJXBSB l'SBNF-FWFM-BCFM3FGJOFNFOUGPS4LFMFUPO#BTFE8FBLMZ4VQFSWJTFE5FNQPSBM "DUJPO-PDBMJ[BUJPOz """* :V 5BOBLB 'VKJXBSB l&YQMPSJOH7JTJPO5SBOTGPSNFSTGPS%)VNBO.PUJPO-BOHVBHF.PEFMT XJUI.PUJPO1BUDIFT z$713 UPBQQFBS 5BOBLB 'VKJXBSB l3PMFBXBSF*OUFSBDUJPO(FOFSBUJPOGSPN5FYUVBM%FTDSJQUJPO z*$$7

՝୊̍ ϥϕϧ৘ใ ໰΋͏গ͠ΞϊςʔγϣϯΛָʹ͢Δ͜ͱ͸Ͱ͖ͳ͍͔ʁ ˠऑڭࢣ͋Γઃఆͷ໰୊Λఏএ 4LFMFUBM8FBLMZ4VQFSWJTFE5FNQPSBM"DUJPO-PDBMJ[BUJPO 485"- Ϟʔγϣϯσʔλͱؚ·ΕΔߦಈϥϕϧ͕༩͑ΒΕͨࡍɺͲ͕͜Ͳ͜ʹରԠ͢Δ͔Λ༧ଌ

485"- ఏҊϞσϧ<:V"""*>

໰୊ͱରࡦ ֶश͕ਐΉʹͭΕɺಛ௃తͳ࢟੎ͷϑϨʔϜ͚͔ͩ͠ݕग़Ͱ͖ͳ͘ͳΔ ˠഎܠͳͲͷ৘ใ͕օແͰ͋ΔͨΊ ϊΠζ͋ΓϥϕϧͰͷֶशͷ஌ݟ<5BOBLB$713>Λར༻͠ɺॳظͷ༧ଌΛٙࣅϥϕϧʹ ֶशޙظ -BCFM $BU %PH $BS 1SPCBCJMJU Z $// $BU %PH $BS *NBHF

໰୊ͱରࡦ ֶश͕ਐΉʹͭΕɺಛ௃తͳ࢟੎ͷϑϨʔϜ͚͔ͩ͠ݕग़Ͱ͖ͳ͘ͳΔ ˠഎܠͳͲͷ৘ใ͕օແͰ͋ΔͨΊ ϊΠζ͋ΓϥϕϧͰͷֶशͷ஌ݟ<5BOBLB$713>Λར༻͠ɺॳظͷ༧ଌΛٙࣅϥϕϧʹ ֶशޙظ -BCFM $BU %PH $BS 1SPCBCJMJU Z $// $BU %PH $BS *NBHF 6QEBUF-BCFM

݁Ռ

՝୊̎ σʔλ਺ ໰σʔλ਺ͷগͳ͞Λ৐Γӽ͑Δํ๏͸ͳ͍͔ʁ ˠϞʔγϣϯͷը૾ԽΛఏএ 0 1 2 3 5 4 6 9 8 7 11 10 14 13 12 15 17 16 18 19 20 21 0 11 2 5 8 Frame i Frame i+1 Frame i+(N-1) … … … N Frames N Points Motion Patch of Right Leg SMPL Skeleton Motion Sequence of Right Leg Extract Interpolate

.PUJPO1BUDIFT ֤෦ҐΛ7J5ͷύονͱͯ͠ѻ͑Δ.PUJPO1BUDI<:V$713> ˠಈ͖Λ৭ͷάϥσʔγϣϯͰදݱ ܇࿅ࡁΈ7J5͕ར༻Մೳʹ #Body Parts #Frames Right Leg Left Leg Torso Right Arm Left Arm a person brings their arms down Motion Patches #Body Parts #Frames a person takes a small step forward Motion Patches Right Leg Left Leg Torso Right Arm Left Arm #Fram es Transformer Motion Features … #Body Parts #Frames #Fram es Vision Transformer Motion Features … #Frames #Joints Existing Methods Proposed Method … … … Motion Patches (Shared by skeleton) Skeleton Sequence (Differed by skeleton) Pre-trained weights for 2D images available N Frames N Points

݁Ռ ը૾υϝΠϯ͔Βͷ஌ࣝΛར༻͠ਫ਼౓޲্ୡ੒ ҟͳΔεέϧτϯߏ଄ͷσʔλΛNPUJPOQBUDIʹม׵͠ɺసҠֶश΋Մೳ Rank #1 Rank #2 Rank #1 Person runs in a zigzag motion and ducks under an invisible object halfway through then returns to full height. Person walks six steps to side with right arm up. A person walks slowly towards the right while slowly raising a hand and then walking backwards. The man moves to his right. Person walks six steps to side with right arm up. Rank #2 TMR Ours A person who seems to evade something from their left side and run at a insane pace. A person strafes to the right. Person runs in a zigzag motion and ducks under an invisible object halfway through then returns to full height. A man crouches down while quickly walking forward and then stands up straight. A person walking and helping maintain their balance and support, from holding onto a side rail or wall. A person runs diagonally across a room with their arms swinging hands down. A person who seems to evade something from their left side and run at a insane pace. A person running then quickly taking a step to their right. A person runs forward with one leg crossing in front of the other repetitively before coming to a stop. A person runs diagonally across a room with their arms swinging hands down.

՝୊̏ ଟ༷ͳಈ͖ ໰ଞʹͲΜͳಈ͖͕࡞ΕΔͩΖ͏ʁ ˠೋਓͷަྲྀͷੜ੒ΛఏҊ "JTXIJTQFSJOH JO#`TFBS "BOE#BSF TIBLJOHIBOET

໾ׂʹ߹ͬͨจ΁l຋༁z ೋਓͷߦಈͷࡍɺͦΕͧΕʹ໾ׂ͕͋Δ͜ͱΛఏএ<5BOBLB*$$7> • ରশతަྲྀˠςΩετ͸ͦͷ·· • ඇରশަྲྀˠೳಈɾडಈͷจষʹ຋༁ɺରԠ͢Δߦಈʹ෇༩ֶ͠श 5SBOTGPSNFS QBSBNFUFSTIBSF 5SBOTGPSNFS QBSBNFUFSTIBSF "DUJWFWPJDF "QFSTPOJT QVTIJOH OPJTF $SPTT"UUFOUJPO 1BTTJWFWPJDF "QFSTPOJT QVTIFECZ OPJTF -BOHVBHF &ODPEFS -BOHVBHF &ODPEFS "DUPS 3FDFJWFS

݁Ռ ໾ׂͷจষΛ໌֬Խ͢Δ͜ͱʹΑΓɺੜ੒ਫ਼౓͕޲্ ΞςϯγϣϯػߏͷಋೖʹΑΓ௰᧒ͷ͋͏ಈ࡞Λ࣮ݱ 0VST 5XP5SBOTGPSNFS "DUPS "DUJWF7PJDF "QFSTPOJTLJDLJOH UIFPUIFSQFSTPO 3FDFJWFS 1BTTJWF7PJDF "QFSTPOJTLJDLFECZ UIFPUIFSQFSTPO "DUPS "DUJWF7PJDF "QFSTPOJTQVTIJOH UIFPUIFSQFSTPO 3FDFJWFS 1BTTJWF7PJDF "QFSTPOJTQVTIFECZ UIFPUIFSQFSTPO આ໌จ

͓ΘΓʹ

Ϟʔγϣϯ΍Ζ͏Αʂ ༷ʑͳΞϓϦέʔγϣϯʹͭͳ͕ΔϞʔγϣϯσʔλ͕೤͍ʂ • ղੳΛࠔ೉ʹ͢Δ༷ʑͳཁҼ໰୊ઃఆ͕໘ന͍ʂ • σʔλগͳ͍ଞऀͷࢀೖোน͕ߴ͍ʂ ҰํͰɺج൫ϞσϧΛ༻͍ͨݚڀ΋͔ͳΓ׆ൃʹ • .PUJPO(15'JOFUVOFE--.TBSF(FOFSBM1VSQPTF.PUJPO(FOFSBUPST<;IBOH"""*> ˠ--.ʹهड़ʹରԠ͢Δಈ͖τʔΫϯྻΛग़ྗͤ͞ΔࢼΈ

˜-:$PSQPSBUJPO 5IBOLZPV

ؔ࿈จݙ • 1MBQQFSU .BUUIJBT $ISJTUJBO.BOEFSZ BOE5BNJN"TGPVS5IFLJUNPUJPOMBOHVBHFEBUBTFU #JH EBUB OP • -PQFS .BUUIFX /BVSFFO .BINPPE +BWJFS3PNFSP (FSBSE1POT.PMM BOE.JDIBFM+#MBDL4.1- "TLJOOFENVMUJQFSTPOMJOFBSNPEFM*O 4FNJOBM(SBQIJDT1BQFST1VTIJOHUIF#PVOEBSJFT 7PMVNF QQ • 1FUSPWJDI .BUIJT .JDIBFM+#MBDL BOE(ÛM7BSPM5.35FYUUPNPUJPOSFUSJFWBMVTJOHDPOUSBTUJWF %IVNBONPUJPOTZOUIFTJT*O 1SPDFFEJOHTPGUIF*&&&$7'*OUFSOBUJPOBM$POGFSFODFPO$PNQVUFS 7JTJPO QQ • ;IBOH +JBOSPOH :BOHTPOH ;IBOH 9JBPEPOH $VO 4IBPMJ )VBOH :POH;IBOH )POHXFJ;IBP )POHUBP -V BOE9J4IFO5NHQU(FOFSBUJOHIVNBONPUJPOGSPNUFYUVBMEFTDSJQUJPOTXJUI EJTDSFUFSFQSFTFOUBUJPOT BS9JW QSFQSJOUBS9JW • 5BOBLB %BJLJ %BJLJ*LBNJ 5PTIJIJLP:BNBTBLJ BOE,JZPIBSV "J[BXB+PJOUPQUJNJ[BUJPOGSBNFXPSL GPSMFBSOJOHXJUIOPJTZMBCFMT*O 1SPDFFEJOHTPGUIF*&&&DPOGFSFODFPODPNQVUFSWJTJPOBOEQBUUFSO SFDPHOJUJPO QQ • $.6(SBQIJDT-BC.PUJPO$BQUVSF%BUBCBTF IUUQNPDBQDTDNVFEV