Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Leveraging Temporal Joint Depths for Improving 3D Human Pose Estimation in Video

Cd6e67775ba3275c466e9927ede23080?s=47 Naoki Kato
November 04, 2020

Leveraging Temporal Joint Depths for Improving 3D Human Pose Estimation in Video

Naoki Kato, Hiroto Honda, Yusuke Uchida, "Leveraging Temporal Joint Depths for Improving 3D Human Pose Estimation in Video," In GCCE 2020.

https://arxiv.org/abs/2011.02172

Cd6e67775ba3275c466e9927ede23080?s=128

Naoki Kato

November 04, 2020
Tweet

Transcript

  1. !]DQYQjs0IEP[]Y]OQIh ] jG /HYHUDJLQJ7HPSRUDO-RLQW'HSWKVIRU ,PSURYLQJ'+XPDQ3RVH(VWLPDWLRQLQ9LGHR 1DRNL.DWR+LURWR+RQGD<XVXNH8FKLGD 0RELOLW\7HFKQRORJLHV&R/WG

  2. !]DQYQjs0IEP[]Y]OQIh ] jG 'KXPDQSRVHHVWLPDWLRQ Ŷ /RFDOL]HKXPDQMRLQWVLQD'FRRUGLQDWHV\VWHP Ŷ $SSOLFDWLRQV Ŷ 0RWLRQFDSWXUHV\VWHP

    Ŷ $595 Ŷ +XPDQDFWLRQDQDO\VLV Ŷ ,QYHKLFOHFDPHUDYLGHRDQDO\VLV Ŷ 'LIILFXOWLHV Ŷ +DUGWREXLOGKLJKO\GLYHUVHGDWDVHW Ŷ $PELJXLW\LQMRLQWGHSWK %DFNJURXQG
  3. !]DQYQjs0IEP[]Y]OQIh ] jG /LIWDVHTXHQFHRI'SRVHVWR'SRVHV —$EOHWRUHGXFHRYHUILWWLQJ —([SORLWWHPSRUDOLQIRUPDWLRQZLWKORZFRPSXWDWLRQDOFRVW 䘣/DUJHUDPELJXLW\LQMRLQWGHSWK ([LVWLQJ:RUNV 'SRVHVĺ5HILQHG'SRVHVĺ'SRVHV>&KHQJ,&&9¶@ '&RQY1HW>3DYOOR&935¶@

  4. !]DQYQjs0IEP[]Y]OQIh ] jG ,PSURYHH[LVWLQJDSSURDFKZKLOHUHWDLQLQJWKHEHQHILWV Ŷ $YRLGRYHUILWWLQJ Ŷ ([SORLWWHPSRUDOLQIRUPDWLRQ Ŷ 5HGXFHDPELJXLW\LQMRLQWGHSWKV

    0RWLYDWLRQ
  5. !]DQYQjs0IEP[]Y]OQIh ] jG (VWLPDWH'SRVHVIURPDVHTXHQFHRI'SRVHVDQGGHSWKV 2XU$SSURDFK —$EOHWRUHGXFHDPELJXLW\LQMRLQWGHSWK

  6. !]DQYQjs0IEP[]Y]OQIh ] jG Ŷ )LUVWVWDJH䠖HVWLPDWH'SRVHDQGMRLQWGHSWKVIURPHDFKIUDPH ĺ,QWHJUDO5HJUHVVLRQ 0RGHO

  7. !]DQYQjs0IEP[]Y]OQIh ] jG Ŷ )LUVWVWDJH䠖HVWLPDWH'SRVHDQGMRLQWGHSWKVIURPHDFKIUDPH ĺ,QWHJUDO5HJUHVVLRQ Ŷ 6HFRQGVWDJH䠖HVWLPDWH'SRVHVIURPDVHTXHQFHRI'SRVHVDQGGHSWKV ĺ'&RQY1HW 0RGHO

  8. !]DQYQjs0IEP[]Y]OQIh ] jG 8VLQJWKHFHQWURLGRIWKHKHDWPDSDVWKHHVWLPDWHGMRLQWFRRUGLQDWH Ŷ (QGWRHQGLQIHUHQFHRQMRLQWFRRUGLQDWHVXVLQJKHDWPDSV Ŷ &DQEHWUDLQHGXVLQJERWK'DQG'GDWDVHWV )LUVW6WDJH,QWHJUDO5HJUHVVLRQ>6XQ(&&9¶@ Ŷ

    $UFKLWHFWXUH Ŷ %DFNERQH5HV1HW Ŷ +HDGGHFRQYROXWLRQOD\HUVWRSURGXFH'KHDWPDSV ;6XQ%;LDR):HL6/LDQJ<:HL³,QWHJUDO+XPDQ3RVH5HJUHVVLRQ´,Q(&&9
  9. !]DQYQjs0IEP[]Y]OQIh ] jG (VWLPDWH'SRVHLQWKHFHQWHUIUDPHIURPDQLQSXWVHTXHQFHXVLQJ'&RQY1HW ZLWKWHPSRUDOGLODWHGFRQYROXWLRQV Ŷ 6WDFNRIUHVLGXDOEORFNV FRQY%15H/8GURSRXWî Ŷ $SSO\WHPSRUDOGLODWHGFRQYROXWLRQRQWKHLQSXWOD\HUDQGWKHILUVWOD\HURIHDFKUHVLGXDOEORFN

    Ŷ 7HPSRUDOUHFHSWLYHILHOGV:% :NHUQHOVL]H%WKHQXPEHURIUHVLGXDOEORFNV Ŷ ([WHQGLQSXWGLPHQVLRQVIRUGHSWK>WHPSRUDOOHQJWK@î>WKHQXPEHURIMRLQWV@î 6HFRQG6WDJH'&RQY1HW>3DYOOR&935¶@ '3DYOOR&)HLFKWHQKRIHU'*UDQJLHU0$XOL³'KXPDQSRVHHVWLPDWLRQLQYLGHRZLWKWHPSRUDO FRQYROXWLRQVDQGVHPLVXSHUYLVHGWUDLQLQJ´,Q&935
  10. !]DQYQjs0IEP[]Y]OQIh ] jG 2XUDSSURFK Ŷ )LUVWVWDJH Ŷ ,QSXWLPDJH Ŷ 2XWSXW'SRVH

    'SRVHGHSWK Ŷ 6HFRQGVWDJH Ŷ ,QSXW'SRVHGHSWKVHTXHQFH Ŷ 2XWSXW'SRVH ([LVWLQJDSSURDFK Ŷ )LUVWVWDJH Ŷ ,QSXWLPDJH Ŷ 2XWSXW'SRVH Ŷ 6HFRQGVWDJH Ŷ ,QSXW'SRVHVHTXHQFH Ŷ 2XWSXW'SRVH &RPSDULVRQRIRXUDSSURDFKZLWKH[LVWLQJDSSURDFKHV 0RGHO6XPPDU\ —5HGXFHGGHSWKDPELJXLW\LQWKHVHFRQGVWDJH 7KHGHSWKSUHGLFWHGE\WKHILUVWVWDJHPLJKWEHHUURQHRXV
  11. !]DQYQjs0IEP[]Y]OQIh ] jG Ŷ 7UDLQLQJVHFRQGVWDJHXVLQJWKHSUHGLFWLRQRIWKHILUVWVWDJHPD\GHFUHDVHWKHSHUIRUPDQFH Ŷ 3UHGLFWLRQRQXQVHHQGDWDKDYHDGLIIHUHQWGLVWULEXWLRQ Ŷ :HSURSRVHWRDGGJDXVVLDQQRLVHWRWKHLQSXWRIWKHVHFRQGVWDJHLQWUDLQLQJWRPLPLFWKH WHVWWLPHSUHGLFWLRQVRIWKHILUVWVWDJH

    'DWD$XJPHQWDWLRQ6WUDWHJ\
  12. !]DQYQjs0IEP[]Y]OQIh ] jG Ŷ )LUVWVWDJH 'DWDDXJPHQWDWLRQ Ŷ +RUL]RQWDOIOLSRQSRVH Ŷ 6FDOLQJ

    Ŷ $GGJDXVVLDQQRLVHWR'SRVH Ŷ $GGJDXVVLDQQRLVHWRMRLQWGHSWK 'DWDDXJPHQWDWLRQ Ŷ +RUL]RQWDOIOLSRQLPDJH Ŷ 5RWDWLRQ Ŷ 6FDOLQJ Ŷ %%R[6KLIWLQJ Ŷ &RORUMLWWHULQJ 7UDLQLQJGHWDLOV Ŷ 2SWLPL]HU$GDP: ZHLJKWGHFD\H Ŷ %DWFKVL]H Ŷ 'HFD\OHDUQLQJUDWHIURPHWRHZLWK FRVLQHIXQFWLRQIRUHSRFKV 7UDLQLQJGHWDLOV Ŷ 6L]HRILQSXWLPDJHî Ŷ 2SWLPL]HU$GDP Ŷ %DWFKVL]H Ŷ 'HFD\OHDUQLQJUDWHIURPHWRHZLWK FRVLQHIXQFWLRQIRUHSRFKV ,PSOHPHQWDWLRQGHWDLOV Ŷ 6HFRQGVWDJH
  13. !]DQYQjs0IEP[]Y]OQIh ] jG Ŷ 'DWDVHW+XPDQ0 Ŷ VHTXHQFHV Ŷ VXEMHFWV IRUWUDLQIRUYDOLGDWLRQ

    Ŷ FDPHUDV Ŷ DFWLYLWLHVîSDWWHUQV Ŷ (YDOXDWLRQPHWULFV Ŷ 0HDQ3HU-RLQW3RVLWLRQ(UURU 03-3(  $YHUDJHGGLVWDQFHVEHWZHHQWKHSUHGLFWHGDQG JURXQGWUXWKFRRUGLQDWHV Ŷ 3$03-3( 03-3(DIWHUULJLGDOLJQPHQW ([SHULPHQWDO6HWXS 'DWDH[DPSOHRI+XPDQ0
  14. !]DQYQjs0IEP[]Y]OQIh ] jG 0RGHOSHUIRUPDQFHZLWKGLIIHUHQWLQSXWVDQGWHPSRUDOUHFHSWLYHILHOGV 7KHUHVSHFWLYHPRGHOVKDYH NHUQHOVL]HUHVLGXDOEORFNV RI   

      DQG  Ŷ ,QSXWWLQJMRLQWGHSWKVGLUHFWO\LQWUDLQLQJLQFUHDVHVWKHHUURU Ŷ $GGLQJJDXVVLDQQRLVHWRGHSWKVLQWUDLQLQJLQFUHDVHVWKHPRGHOSHUIRUPDQFH Ŷ ,QFUHDVHGSHUIRUPDQFHZLWKODUJHUHFHSWLYHILHOG 03-3(RIWKHILUVWVWDJHLVPP 6IWYPXW ,QSXW 5HFHSWLYHILHOG     'SRVH     'SRVHGHSWK ZRQRLVH     'SRVHGHSWK QRLVHı      03-3( PP RIHDFKPRGHO
  15. !]DQYQjs0IEP[]Y]OQIh ] jG 'SQTEVMWSR[MXLXLI)\MWXMRK1IXLSHW ᡭἲ 03-3( PP 3$03-3( PP 0DUWLQH]HWDO

    ,&&9¶   6XQHWDO (&&9¶   &DLHWDO ,&&9¶ ‚   3DYOORHWDO &935¶ ‚   2XUV‚   &RPSDULVRQZLWKWKHH[LVWLQJPHWKRGV 䠄‚H[SORLWVWHPSRUDOLQIRUPDWLRQ䠅 Ŷ $FKLHYHWKHKLJKHVWSHUIRUPDQFHLQERWKPHWULFV Ŷ PPORZHU03-3(DQGPPORZHU3$03-3(WKDQWKHPHWKRGRI3DYOORHWDO
  16. !]DQYQjs0IEP[]Y]OQIh ] jG Ŷ &RQFOXVLRQ Ŷ 3URSRVHWRXVHMRLQWGHSWKVDVDQLQWHUPHGLDWHUHSUHVHQWDWLRQIRUHVWLPDWLQJ'KXPDQ SRVHIURPYLGHRV Ŷ 3URSRVHWRDGGJDXVVLDQQRLVHWRWKHLQSXWVRIWKHVHFRQGVWDJHWRPLPLFWKHWHVWWLPH

    SUHGLFWLRQVRIWKHILUVWVWDJH Ŷ 'HPRQVWUDWHHIIHFWLYHQHVVRIWKHSURSRVHGDSSURDFKRQ+XPDQ0 Ŷ )XWXUH:RUNV Ŷ $PRUHHIIHFWLYHGDWDDXJPHQWDWLRQPHWKRGIRUWUDLQLQJWZRVWDJHPRGHO Ŷ $SLSHOLQHWRH[SOLFLWO\GHDOZLWKRFFOXGHGIUDPHV &RQFOXVLRQDQG)XWXUH:RUNV