SSII2024 [OS2] 画像、その先へ 〜モーション解析への誘い〜

1. 画像、その先へ 〜モーション解析への誘い〜 2024.06.13 藤原 研⼈（LINEヤフー株式会社-⼤阪⼤学VHL）
2.  7JSUVBM
   )VNBO
   -BC ࠷ઌ୺ͷݚڀ͔Β੡඼·Ͱ
3.  7)-ઓུ 7)- Ϣʔβʔ ੡࡞ऀ ٕज़ࢧԉ ίϯςϯπఏڙ Τϯήʔδϝϯτ
4. 2
   ϚϧνϞʔμϧج൫Ϟσϧͷొ৔Ͱ ը૾ೝࣝ͸ऴΘΔͷ͔ʁ 
5. "
   ऴΘͬͪΌ͑ऴΘͬͪΌ͑ʂ 
6. ʜͱ͍͏ͷ΋ ͜Ε·Ͱ͋·Γը૾ͱ޲͖߹ͬͯ͜ͳ͔ͬͨ fixed weights trainable weights !! " !# " !! !# ""(! # ) ""(! ! ) ""(!$ ) !$ !$ " !! " !# " !$ " !! !# !$ GT NN SLR GSLR (ours) <'VKJXBSB )BTIJNPUP
   $713
   > <-J
   *$$7
   > <-J
   *$$7
   >
7.  7)-ͷ׆ಈ ਓؒͷಈ࡞σʔλʹண໨͠ݚڀ։ൃ׆ಈΛ࣮ࢪ
8. Ϟʔγϣϯσʔλղੳ 
9.  എܠ
   ͳͥϞʔγϣϯσʔλʁ 2 %ͷεέϧτϯʁͳͥ̎%ಈը͡Όͳ͍ͷʁಈըͩͬͨΒج൫7-.ͰࡁΈͦ͏͡Όͳ͍ʁ "Ͱ͖Ε͹ͦΕ͕͍͍ʂ͚Ͳ ·ͩ Ͱ͖ͳ͍͜ͱͨ͘͞Μ • ಈը͔Βͷ࢟੎ͷԞߦ͖ɾΧϝϥҐஔ౳ͷਪఆͷࠔ೉͞ • Ұ؏ͨ͠ಈ࡞͔൱͔ͷ൑ఆ • FUDʜ <%XJWFEJ
   FU
   BM
   %7>
10.  എܠ
    ͳͥϞʔγϣϯσʔλʁ 2 %ͷεέϧτϯʁͳͥʢ̎%ʣಈը͡Όͳ͍ͷʁಈըͩͬͨΒج൫7-.ͰࡁΈͦ͏͡Όͳ͍ʁ "
    ग़ྗͷ࢖͍উख
11.  ಈ࡞ͷऔಘํ๏ ϞʔγϣϯΩϟϓνϟʔσόΠε • ޫֶࣜ • ׳ੑࣜ • ࣓ؾࣜ ಈը͔Βਪఆ͢Δݚڀ΋׆ൃ ՄࢹԽIUUQTMPUIHJUIVCJPPMZNQF#7)@QMBZFSIUNM ޫֶࣜ
    <$.6
    .PDBQ
    %BUBCBTF IUUQNPDBQDTDNVFEV>
12.  ಈ࡞ͷදݱํ๏ ֶश༻ʹ౷Ұදݱʹม׵͢Δඞཁੑ • ,*5εέϧτϯ
    <1MBQQFSU > • 4.1-εέϧτϯ
    <-PQFS > • ࠊɺؔઅͷؔઅճసɾ଎౓౳ΛؚΉߴ࣍ݩදݱʹม׵͢Δ͜ͱ΋ • σʔλ
    
    ϑϨʔϜ਺
    Y
    ؔઅ਺
    Y
    දݱ࣍ݩ 0 1 2 3 5 4 6 9 8 7 11 10 14 13 12 15 17 16 18 19 20 21 0 11 2 5 8 Frame i Frame i+1 Frame i+(N-1) … … SMPL Skeleton Motion Sequence of Right Leg Extract Interpolate 0 1 2 3 4 8 9 10 5 6 7 16 17 18 19 20 11 12 13 14 15 Frame i Frame i+1 Frame i+(N-1) … … … N Frames N Points Motion Patch of Right Leg Motion Sequence of Right Leg Interpolate Extract KIT Skeleton 0 14 11 12 13 15
13.  λεΫ Ξχϝʔγϣϯ౳ΞϓϦέʔγϣϯΛݟਾ͑ͨݚڀ͕׆ൃ ςΩετʹΑΔ Ϟʔγϣϯݕࡧ ςΩετʹΑΔ Ϟʔγϣϯੜ੒ <1FUSPWJDI *$$7> <;IBOH $713>
14.  ՝୊ Ξϊςʔγϣϯ͕໘౗͍͘͞ ϑϨʔϜ͝ͱʹهड़͕ඞཁ ϥϕϧ৘ใ σʔλ͕ͦ΋ͦ΋গͳ͍ σʔλܗ͕༷ࣜʑ σʔλ਺ ΋ͬͱଟ༷ͳಈ࡞͕ཉ͍͠ Ұਓͷಈ࡞ʹয఺ ଟ༷ͳಈ͖
15. 7)-ͷݚڀ׆ಈ  :V
    'VKJXBSB
    l'SBNF-FWFM
    -BCFM
    3FGJOFNFOU
    GPS
    4LFMFUPO#BTFE
    8FBLMZ4VQFSWJTFE
    5FNQPSBM
    "DUJPO
    -PDBMJ[BUJPOz
    """*
     :V
    5BOBLB
    'VKJXBSB
    l&YQMPSJOH
    7JTJPO
    5SBOTGPSNFST
    GPS
    %
    )VNBO
    .PUJPO-BOHVBHF
    .PEFMT XJUI
    .PUJPO
    1BUDIFT z
    $713
    
    UP
    BQQFBS 5BOBLB
    'VKJXBSB
    l3PMFBXBSF
    *OUFSBDUJPO
    (FOFSBUJPO
    GSPN
    5FYUVBM
    %FTDSJQUJPO z
    *$$7
    
    
16.  ՝୊̍ ϥϕϧ৘ใ ໰΋͏গ͠ΞϊςʔγϣϯΛָʹ͢Δ͜ͱ͸Ͱ͖ͳ͍͔ʁ ˠऑڭࢣ͋Γઃఆͷ໰୊Λఏএ 4LFMFUBM
    8FBLMZ
    4VQFSWJTFE
    5FNQPSBM
    "DUJPO
    -PDBMJ[BUJPO
    485"- Ϟʔγϣϯσʔλͱؚ·ΕΔߦಈϥϕϧ͕༩͑ΒΕͨࡍɺͲ͕͜Ͳ͜ʹରԠ͢Δ͔Λ༧ଌ
17.  485"- ఏҊϞσϧ<:V """*>
18.  ໰୊ͱରࡦ ֶश͕ਐΉʹͭΕɺಛ௃తͳ࢟੎ͷϑϨʔϜ͚͔ͩ͠ݕग़Ͱ͖ͳ͘ͳΔ ˠഎܠͳͲͷ৘ใ͕օແͰ͋ΔͨΊ ϊΠζ͋ΓϥϕϧͰͷֶशͷ஌ݟ<5BOBLB $713>Λར༻͠ɺॳظͷ༧ଌΛٙࣅϥϕϧʹ ֶशޙظ -BCFM $BU %PH $BS      1SPCBCJMJU Z  $// $BU %PH $BS *NBHF
19.  ໰୊ͱରࡦ ֶश͕ਐΉʹͭΕɺಛ௃తͳ࢟੎ͷϑϨʔϜ͚͔ͩ͠ݕग़Ͱ͖ͳ͘ͳΔ ˠഎܠͳͲͷ৘ใ͕օແͰ͋ΔͨΊ ϊΠζ͋ΓϥϕϧͰͷֶशͷ஌ݟ<5BOBLB $713>Λར༻͠ɺॳظͷ༧ଌΛٙࣅϥϕϧʹ ֶशޙظ -BCFM $BU %PH $BS      1SPCBCJMJU Z  $// $BU %PH $BS *NBHF 6QEBUF
    -BCFM
20.  ݁Ռ
21.  ՝୊̎ σʔλ਺ ໰σʔλ਺ͷগͳ͞Λ৐Γӽ͑Δํ๏͸ͳ͍͔ʁ ˠϞʔγϣϯͷը૾ԽΛఏএ 0 1 2 3 5 4 6 9 8 7 11 10 14 13 12 15 17 16 18 19 20 21 0 11 2 5 8 Frame i Frame i+1 Frame i+(N-1) … … … N Frames N Points Motion Patch of Right Leg SMPL Skeleton Motion Sequence of Right Leg Extract Interpolate
22.  .PUJPO
    1BUDIFT ֤෦ҐΛ7J5ͷύονͱͯ͠ѻ͑Δ.PUJPO
    1BUDI<:V $713
    > ˠಈ͖Λ৭ͷάϥσʔγϣϯͰදݱ ܇࿅ࡁΈ7J5͕ར༻Մೳʹ
      #Body Parts #Frames Right Leg Left Leg Torso Right Arm Left Arm a person brings their arms down Motion Patches #Body Parts #Frames a person takes a small step forward Motion Patches Right Leg Left Leg Torso Right Arm Left Arm #Fram es Transformer Motion Features … #Body Parts #Frames #Fram es Vision Transformer Motion Features … #Frames #Joints Existing Methods Proposed Method … … … Motion Patches (Shared by skeleton) Skeleton Sequence (Differed by skeleton) Pre-trained weights for 2D images available N Frames N Points
23.  ݁Ռ ը૾υϝΠϯ͔Βͷ஌ࣝΛར༻͠ਫ਼౓޲্ୡ੒ ҟͳΔεέϧτϯߏ଄ͷσʔλΛNPUJPO
    QBUDIʹม׵͠ɺసҠֶश΋Մೳ Rank #1 Rank #2 Rank #1 Person runs in a zigzag motion and ducks under an invisible object halfway through then returns to full height. Person walks six steps to side with right arm up. A person walks slowly towards the right while slowly raising a hand and then walking backwards. The man moves to his right. Person walks six steps to side with right arm up. Rank #2 TMR Ours A person who seems to evade something from their left side and run at a insane pace. A person strafes to the right. Person runs in a zigzag motion and ducks under an invisible object halfway through then returns to full height. A man crouches down while quickly walking forward and then stands up straight. A person walking and helping maintain their balance and support, from holding onto a side rail or wall. A person runs diagonally across a room with their arms swinging hands down. A person who seems to evade something from their left side and run at a insane pace. A person running then quickly taking a step to their right. A person runs forward with one leg crossing in front of the other repetitively before coming to a stop. A person runs diagonally across a room with their arms swinging hands down.
24.  ՝୊̏ ଟ༷ͳಈ͖ ໰ଞʹͲΜͳಈ͖͕࡞ΕΔͩΖ͏ʁ ˠೋਓͷަྲྀͷੜ੒ΛఏҊ "
    JT
    XIJTQFSJOH
    JO
    #`T
    FBS "
    BOE
    #
    BSF
    TIBLJOH
    IBOET
25.  ໾ׂʹ߹ͬͨจ΁l຋༁z ೋਓͷߦಈͷࡍɺͦΕͧΕʹ໾ׂ͕͋Δ͜ͱΛఏএ<5BOBLB *$$7> • ରশతަྲྀˠςΩετ͸ͦͷ·· • ඇରশަྲྀˠೳಈɾडಈͷจষʹ຋༁ɺରԠ͢Δߦಈʹ෇༩ֶ͠श 5SBOTGPSNFS QBSBNFUFS
    TIBSF 5SBOTGPSNFS QBSBNFUFS
    TIBSF "DUJWF
    WPJDF "
    QFSTPO
    JT
    QVTIJOH
     OPJTF $SPTT
    "UUFOUJPO 1BTTJWF
    WPJDF
    
    "
    QFSTPO
    JT
    QVTIFE
    CZ
     OPJTF -BOHVBHF
    &ODPEFS -BOHVBHF
    &ODPEFS "DUPS 3FDFJWFS
26.  ݁Ռ ໾ׂͷจষΛ໌֬Խ͢Δ͜ͱʹΑΓɺੜ੒ਫ਼౓͕޲্ ΞςϯγϣϯػߏͷಋೖʹΑΓ௰᧒ͷ͋͏ಈ࡞Λ࣮ݱ 0VST 5XP
    5SBOTGPSNFS "DUPS
    "DUJWF
    7PJDF  "
    QFSTPO
    JT
    LJDLJOH
    UIF
    PUIFS
    QFSTPO 3FDFJWFS
    1BTTJWF
    7PJDF 
    "
    QFSTPO
    JT
    LJDLFE
    CZ
    UIF
    PUIFS
    QFSTPO
    "DUPS
    "DUJWF
    7PJDF  "
    QFSTPO
    JT
    QVTIJOH
    UIF
    PUIFS
    QFSTPO 3FDFJWFS
    1BTTJWF
    7PJDF 
    "
    QFSTPO
    JT
    QVTIFE
    CZ
    UIF
    PUIFS
    QFSTPO
    આ໌จ
27. ͓ΘΓʹ 
28.  Ϟʔγϣϯ΍Ζ͏Αʂ ༷ʑͳΞϓϦέʔγϣϯʹͭͳ͕ΔϞʔγϣϯσʔλ͕೤͍ʂ • ղੳΛࠔ೉ʹ͢Δ༷ʑͳཁҼ
    
    ໰୊ઃఆ͕໘ന͍ʂ • σʔλগͳ͍
    
    ଞऀͷࢀೖোน͕ߴ͍ʂ ҰํͰɺج൫ϞσϧΛ༻͍ͨݚڀ΋͔ͳΓ׆ൃʹ • .PUJPO(15'JOFUVOFE
    --.T
    BSF
    (FOFSBM1VSQPTF
    .PUJPO
    (FOFSBUPST
    <;IBOH """*>
    ˠ--.ʹهड़ʹରԠ͢Δಈ͖τʔΫϯྻΛग़ྗͤ͞ΔࢼΈ
29. ˜
    -:
    $PSQPSBUJPO 5IBOL
    ZPV 
30.  ؔ࿈จݙ • 1MBQQFSU
    .BUUIJBT
    $ISJTUJBO
    .BOEFSZ
    BOE
    5BNJN
    "TGPVS
    5IF
    LJU
    NPUJPOMBOHVBHF
    EBUBTFU #JH
    EBUB 
    OP
    
     
    
    • -PQFS
    .BUUIFX
    /BVSFFO .BINPPE
    +BWJFS
    3PNFSP
    (FSBSE
    1POT.PMM
    BOE
    .JDIBFM
    +
    #MBDL
    4.1-
    "
    TLJOOFE
    NVMUJQFSTPO
    MJOFBS
    NPEFM
    *O 4FNJOBM
    (SBQIJDT
    1BQFST
    1VTIJOH
    UIF
    #PVOEBSJFT
    7PMVNF
    
    QQ
    
     • 1FUSPWJDI
    .BUIJT
    .JDIBFM
    +
    #MBDL
    BOE
    (ÛM
    7BSPM
    5.3
    5FYUUPNPUJPO
    SFUSJFWBM
    VTJOH
    DPOUSBTUJWF
    %
    IVNBO
    NPUJPO
    TZOUIFTJT
    *O 1SPDFFEJOHT
    PG
    UIF
    *&&&$7'
    *OUFSOBUJPOBM
    $POGFSFODF
    PO
    $PNQVUFS
    7JTJPO
    QQ
    
     • ;IBOH
    +JBOSPOH
    :BOHTPOH ;IBOH
    9JBPEPOH $VO
    4IBPMJ )VBOH
    :POH
    ;IBOH
    )POHXFJ
    ;IBP
    )POHUBP -V
    BOE
    9J
    4IFO
    5NHQU
    (FOFSBUJOH
    IVNBO
    NPUJPO
    GSPN
    UFYUVBM
    EFTDSJQUJPOT
    XJUI
    EJTDSFUF
    SFQSFTFOUBUJPOT BS9JW QSFQSJOU
    BS9JW   • 5BOBLB
    %BJLJ
    %BJLJ
    *LBNJ
    5PTIJIJLP
    :BNBTBLJ
    BOE
    ,JZPIBSV "J[BXB
    +PJOU
    PQUJNJ[BUJPO
    GSBNFXPSL
    GPS
    MFBSOJOH
    XJUI
    OPJTZ
    MBCFMT
    *O 1SPDFFEJOHT
    PG
    UIF
    *&&&
    DPOGFSFODF
    PO
    DPNQVUFS
    WJTJPO
    BOE
    QBUUFSO
    SFDPHOJUJPO
    QQ
    
     • $.6
    (SBQIJDT
    -BC
    .PUJPO
    $BQUVSF
    %BUBCBTF
    IUUQNPDBQDTDNVFEV