Leveraging Temporal Joint Depths for Improving 3D Human Pose Estimation in Video

Naoki Kato, Hiroto Honda, Yusuke Uchida, "Leveraging Temporal Joint Depths for Improving 3D Human Pose Estimation in Video," In GCCE 2020.

https://arxiv.org/abs/2011.02172

Transcript

1. !]DQYQjs0IEP[]Y]OQIh ] jG /HYHUDJLQJ7HPSRUDO-RLQW'HSWKVIRU ,PSURYLQJ'+XPDQ3RVH(VWLPDWLRQLQ9LGHR 1DRNL.DWR+LURWR+RQGD<XVXNH8FKLGD 0RELOLW\7HFKQRORJLHV&R/WG
2. !]DQYQjs0IEP[]Y]OQIh ] jG 'KXPDQSRVHHVWLPDWLRQ Ŷ /RFDOL]HKXPDQMRLQWVLQD'FRRUGLQDWHV\VWHP Ŷ $SSOLFDWLRQV Ŷ 0RWLRQFDSWXUHV\VWHP Ŷ $595 Ŷ +XPDQDFWLRQDQDO\VLV Ŷ ,QYHKLFOHFDPHUDYLGHRDQDO\VLV Ŷ 'LIILFXOWLHV Ŷ +DUGWREXLOGKLJKO\GLYHUVHGDWDVHW Ŷ $PELJXLW\LQMRLQWGHSWK %DFNJURXQG
3. !]DQYQjs0IEP[]Y]OQIh ] jG /LIWDVHTXHQFHRI'SRVHVWR'SRVHV —$EOHWRUHGXFHRYHUILWWLQJ —([SORLWWHPSRUDOLQIRUPDWLRQZLWKORZFRPSXWDWLRQDOFRVW 䘣/DUJHUDPELJXLW\LQMRLQWGHSWK ([LVWLQJ:RUNV 'SRVHVĺ5HILQHG'SRVHVĺ'SRVHV>&KHQJ,&&9¶@ '&RQY1HW>3DYOOR&935¶@
4. !]DQYQjs0IEP[]Y]OQIh ] jG ,PSURYHH[LVWLQJDSSURDFKZKLOHUHWDLQLQJWKHEHQHILWV Ŷ $YRLGRYHUILWWLQJ Ŷ ([SORLWWHPSRUDOLQIRUPDWLRQ Ŷ 5HGXFHDPELJXLW\LQMRLQWGHSWKV 0RWLYDWLRQ
5. !]DQYQjs0IEP[]Y]OQIh ] jG (VWLPDWH'SRVHVIURPDVHTXHQFHRI'SRVHVDQGGHSWKV 2XU$SSURDFK —$EOHWRUHGXFHDPELJXLW\LQMRLQWGHSWK
6. !]DQYQjs0IEP[]Y]OQIh ] jG Ŷ )LUVWVWDJH䠖HVWLPDWH'SRVHDQGMRLQWGHSWKVIURPHDFKIUDPH ĺ,QWHJUDO5HJUHVVLRQ 0RGHO
7. !]DQYQjs0IEP[]Y]OQIh ] jG Ŷ )LUVWVWDJH䠖HVWLPDWH'SRVHDQGMRLQWGHSWKVIURPHDFKIUDPH ĺ,QWHJUDO5HJUHVVLRQ Ŷ 6HFRQGVWDJH䠖HVWLPDWH'SRVHVIURPDVHTXHQFHRI'SRVHVDQGGHSWKV ĺ'&RQY1HW 0RGHO
8. !]DQYQjs0IEP[]Y]OQIh ] jG 8VLQJWKHFHQWURLGRIWKHKHDWPDSDVWKHHVWLPDWHGMRLQWFRRUGLQDWH Ŷ (QGWRHQGLQIHUHQFHRQMRLQWFRRUGLQDWHVXVLQJKHDWPDSV Ŷ &DQEHWUDLQHGXVLQJERWK'DQG'GDWDVHWV )LUVW6WDJH,QWHJUDO5HJUHVVLRQ>6XQ(&&9¶@ Ŷ $UFKLWHFWXUH Ŷ %DFNERQH5HV1HW Ŷ +HDGGHFRQYROXWLRQOD\HUVWRSURGXFH'KHDWPDSV ;6XQ%;LDR):HL6/LDQJ<:HL³,QWHJUDO+XPDQ3RVH5HJUHVVLRQ´,Q(&&9
9. !]DQYQjs0IEP[]Y]OQIh ] jG (VWLPDWH'SRVHLQWKHFHQWHUIUDPHIURPDQLQSXWVHTXHQFHXVLQJ'&RQY1HW ZLWKWHPSRUDOGLODWHGFRQYROXWLRQV Ŷ 6WDFNRIUHVLGXDOEORFNV FRQY%15H/8GURSRXWî Ŷ $SSO\WHPSRUDOGLODWHGFRQYROXWLRQRQWKHLQSXWOD\HUDQGWKHILUVWOD\HURIHDFKUHVLGXDOEORFN Ŷ 7HPSRUDOUHFHSWLYHILHOGV:% :NHUQHOVL]H%WKHQXPEHURIUHVLGXDOEORFNV Ŷ ([WHQGLQSXWGLPHQVLRQVIRUGHSWK>WHPSRUDOOHQJWK@î>WKHQXPEHURIMRLQWV@î 6HFRQG6WDJH'&RQY1HW>3DYOOR&935¶@ '3DYOOR&)HLFKWHQKRIHU'*UDQJLHU0$XOL³'KXPDQSRVHHVWLPDWLRQLQYLGHRZLWKWHPSRUDO FRQYROXWLRQVDQGVHPLVXSHUYLVHGWUDLQLQJ´,Q&935
10. !]DQYQjs0IEP[]Y]OQIh ] jG 2XUDSSURFK Ŷ )LUVWVWDJH Ŷ ,QSXWLPDJH Ŷ 2XWSXW'SRVH 'SRVHGHSWK Ŷ 6HFRQGVWDJH Ŷ ,QSXW'SRVHGHSWKVHTXHQFH Ŷ 2XWSXW'SRVH ([LVWLQJDSSURDFK Ŷ )LUVWVWDJH Ŷ ,QSXWLPDJH Ŷ 2XWSXW'SRVH Ŷ 6HFRQGVWDJH Ŷ ,QSXW'SRVHVHTXHQFH Ŷ 2XWSXW'SRVH &RPSDULVRQRIRXUDSSURDFKZLWKH[LVWLQJDSSURDFKHV 0RGHO6XPPDU\ —5HGXFHGGHSWKDPELJXLW\LQWKHVHFRQGVWDJH 7KHGHSWKSUHGLFWHGE\WKHILUVWVWDJHPLJKWEHHUURQHRXV
11. !]DQYQjs0IEP[]Y]OQIh ] jG Ŷ 7UDLQLQJVHFRQGVWDJHXVLQJWKHSUHGLFWLRQRIWKHILUVWVWDJHPD\GHFUHDVHWKHSHUIRUPDQFH Ŷ 3UHGLFWLRQRQXQVHHQGDWDKDYHDGLIIHUHQWGLVWULEXWLRQ Ŷ :HSURSRVHWRDGGJDXVVLDQQRLVHWRWKHLQSXWRIWKHVHFRQGVWDJHLQWUDLQLQJWRPLPLFWKH WHVWWLPHSUHGLFWLRQVRIWKHILUVWVWDJH 'DWD$XJPHQWDWLRQ6WUDWHJ\
12. !]DQYQjs0IEP[]Y]OQIh ] jG Ŷ )LUVWVWDJH 'DWDDXJPHQWDWLRQ Ŷ +RUL]RQWDOIOLSRQSRVH Ŷ 6FDOLQJ Ŷ $GGJDXVVLDQQRLVHWR'SRVH Ŷ $GGJDXVVLDQQRLVHWRMRLQWGHSWK 'DWDDXJPHQWDWLRQ Ŷ +RUL]RQWDOIOLSRQLPDJH Ŷ 5RWDWLRQ Ŷ 6FDOLQJ Ŷ %%R[6KLIWLQJ Ŷ &RORUMLWWHULQJ 7UDLQLQJGHWDLOV Ŷ 2SWLPL]HU$GDP: ZHLJKWGHFD\H Ŷ %DWFKVL]H Ŷ 'HFD\OHDUQLQJUDWHIURPHWRHZLWK FRVLQHIXQFWLRQIRUHSRFKV 7UDLQLQJGHWDLOV Ŷ 6L]HRILQSXWLPDJHî Ŷ 2SWLPL]HU$GDP Ŷ %DWFKVL]H Ŷ 'HFD\OHDUQLQJUDWHIURPHWRHZLWK FRVLQHIXQFWLRQIRUHSRFKV ,PSOHPHQWDWLRQGHWDLOV Ŷ 6HFRQGVWDJH
13. !]DQYQjs0IEP[]Y]OQIh ] jG Ŷ 'DWDVHW+XPDQ0 Ŷ VHTXHQFHV Ŷ VXEMHFWV IRUWUDLQIRUYDOLGDWLRQ Ŷ FDPHUDV Ŷ DFWLYLWLHVîSDWWHUQV Ŷ (YDOXDWLRQPHWULFV Ŷ 0HDQ3HU-RLQW3RVLWLRQ(UURU 03-3(  $YHUDJHGGLVWDQFHVEHWZHHQWKHSUHGLFWHGDQG JURXQGWUXWKFRRUGLQDWHV Ŷ 3$03-3( 03-3(DIWHUULJLGDOLJQPHQW ([SHULPHQWDO6HWXS 'DWDH[DPSOHRI+XPDQ0
14. !]DQYQjs0IEP[]Y]OQIh ] jG 0RGHOSHUIRUPDQFHZLWKGLIIHUHQWLQSXWVDQGWHPSRUDOUHFHSWLYHILHOGV 7KHUHVSHFWLYHPRGHOVKDYH NHUQHOVL]HUHVLGXDOEORFNV RI      DQG  Ŷ ,QSXWWLQJMRLQWGHSWKVGLUHFWO\LQWUDLQLQJLQFUHDVHVWKHHUURU Ŷ $GGLQJJDXVVLDQQRLVHWRGHSWKVLQWUDLQLQJLQFUHDVHVWKHPRGHOSHUIRUPDQFH Ŷ ,QFUHDVHGSHUIRUPDQFHZLWKODUJHUHFHSWLYHILHOG 03-3(RIWKHILUVWVWDJHLVPP 6IWYPXW ,QSXW 5HFHSWLYHILHOG     'SRVH     'SRVHGHSWK ZRQRLVH     'SRVHGHSWK QRLVHı      03-3( PP RIHDFKPRGHO
15. !]DQYQjs0IEP[]Y]OQIh ] jG 'SQTEVMWSR[MXLXLI)\MWXMRK1IXLSHW ᡭἲ 03-3( PP 3$03-3( PP 0DUWLQH]HWDO ,&&9¶   6XQHWDO (&&9¶   &DLHWDO ,&&9¶ ‚   3DYOORHWDO &935¶ ‚   2XUV‚   &RPSDULVRQZLWKWKHH[LVWLQJPHWKRGV 䠄‚H[SORLWVWHPSRUDOLQIRUPDWLRQ䠅 Ŷ $FKLHYHWKHKLJKHVWSHUIRUPDQFHLQERWKPHWULFV Ŷ PPORZHU03-3(DQGPPORZHU3$03-3(WKDQWKHPHWKRGRI3DYOORHWDO
16. !]DQYQjs0IEP[]Y]OQIh ] jG Ŷ &RQFOXVLRQ Ŷ 3URSRVHWRXVHMRLQWGHSWKVDVDQLQWHUPHGLDWHUHSUHVHQWDWLRQIRUHVWLPDWLQJ'KXPDQ SRVHIURPYLGHRV Ŷ 3URSRVHWRDGGJDXVVLDQQRLVHWRWKHLQSXWVRIWKHVHFRQGVWDJHWRPLPLFWKHWHVWWLPH SUHGLFWLRQVRIWKHILUVWVWDJH Ŷ 'HPRQVWUDWHHIIHFWLYHQHVVRIWKHSURSRVHGDSSURDFKRQ+XPDQ0 Ŷ )XWXUH:RUNV Ŷ $PRUHHIIHFWLYHGDWDDXJPHQWDWLRQPHWKRGIRUWUDLQLQJWZRVWDJHPRGHO Ŷ $SLSHOLQHWRH[SOLFLWO\GHDOZLWKRFFOXGHGIUDPHV &RQFOXVLRQDQG)XWXUH:RUNV