Upgrade to Pro — share decks privately, control downloads, hide ads and more …

【CVPR 2024】VTimeLLM: Empower LLM to Grasp Video...

mori yuichiro
July 09, 2024
0
13

【CVPR 2024】VTimeLLM: Empower LLM to Grasp Video Moments 論文読み会発表資料

mori yuichiro

July 09, 2024
Tweet

More Decks by mori yuichiro

See All by mori yuichiro
【EMNLP 2023】Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs 論文読み会発表資料
zaemon1251hesty
0
33

Featured

See All Featured
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
38
1.9k
A Philosophy of Restraint
colly
203
16k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.1k
Designing for Performance
lara
604
68k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.5k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
191
16k
Building Better People: How to give real-time feedback that sticks.
wjessup
365
19k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
132
33k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
A better future with KSS
kneath
238
17k
Fontdeck: Realign not Redesign
paulrobertlloyd
82
5.3k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
365
25k

Transcript

  1. 75JNF--.&NQPXFS--.UP (SBTQ7JEFP.PNFOUT #JO)VBOH 9JO8BOH )POH$IFO ;JIBO4POH 8FOXV;IV 5TJOHIVB6OJWFSTJUZ $713 :VJDIJSP.PSJ

    1BQFS3FBEJOH 1

  2. 4VNNBSZ • -BSHFMBOHVBHFNPEFMT --.T IBWFCFFOBEBQUFEUPIBOEMFWJEFP EBUBCVUTUSVHHMFXJUIQSFDJTFFWFOUUJNJOH • 75JNF--. BOFX7JEFP--. BEESFTTFTUIJTCZVTJOHBUISFF

    TUBHFUSBJOJOHTUSBUFHZUPJNQSPWFUFNQPSBMVOEFSTUBOEJOH • &YQFSJNFOUTTIPX75JNF--.FYDFMTJOGJOFHSBJOFEWJEFP DPNQSFIFOTJPOBOEPVUQFSGPSNTFYJTUJOH7JEFP--.TJOWBSJPVT UBTLT 2

  3. #BDLHSPVOE • 7JEFP--. • VUJMJ[JOH-BSHF-BOHVBHF.PEFMUPVOEFSTUBOEWJEFP • BOBHFOUGPSJOUFSBDUJOHXJUIIVNBOCBTFEPOBWJEFP • FH 

    7JEFP--B." <;IBOH &./-1> 7JEFP$IBU <-J BS9JW> 3 7JEFP--B."

  4. 1SPCMFN • &YJTUJOH7JEFP--.TGPDVTPOHFOFSBUJPOPGHFOFSJDWJEFP DBQUJPOT UIVTGBJMUPDBQUVSFUIFSFMBUJPOTIJQTCFUXFFOTQFDJGJD CPVOEFEFWFOUT 4 7JEFP$IBU  7JEFP--B."DBOOPU

    MPDBMJ[FUIFFWFOU UJNJOH 7JEFP--.TBSF GBSGSPNQSFDJTFMZ VOEFSTUBOEJOH WJEFP

  5. 3FTFBSDI0CKFDUJWF 5 5IFZCVJME75JNF--.5IF'JSTUCPVOEBSZBXBSF7JEFP--. 5BDLMJOHUXPDIBMMFOHFTGPS7JEFP--.TUPDBQUVSFFWFOUCPVOEBSJFTJO WJEFP  4DBSDJUZPGMBSHFTDBMFCPVOEBSZBXBSFWJEFPEBUBTFUT Ø 5IFZDSFBUFEBUBTFUTJODPSQPSBUJOHFYJTUJOHUFNQPSBMSFMBUFE 7JEFP6OEFSTUBOEJOHUBTLT

     %FTJHOFGGFDUJWFUFNQPSBMSFMBUFEWJEFPUBTLTGPSUSBJOJOH7JEFP--. Ø 5IFZQSPQPTF#PVOEBSZBXBSF5SBJOJOH4USBUFHZ

  6. 1SPQPTFE.FUIPE • 1SFMJNJOBSZ5XPUFNQPSBMSFMBUFE7JEFP6OEFSTUBOEJOHUBTLT 6 5FNQPSBM7JEFP (SPVOEJOH 57( %FOTF7JEFP $BQUJPOJOH %7$

    • UFNQPSBMMPDBMJ[BUJPODPSSFTQPOEJOHWJEFP TFHNFOUTGPSHJWFOUFYUVBMJOQVUT • UFNQPSBMMPDBMJ[BUJPOBOEDBQUJPOJOH GPSBMMFWFOUT 5FNQPSBMMZ(SPVOEJOH/BUVSBM4FOUFODFJO7JEFP <+JOHZVBO &./-1> %FOTF$BQUJPOJOHJOFWFOUT <,SJTIOB *$$7>

  7. 1SPQPTFE.FUIPE • #PVOEBSZBXBSF5SBJOJOH4USBUFHZ DPOTJTUT5SBJOJOH4UBHFT 7

  8. 1SPQPTFE.FUIPE • 4UBHF'FBUVSF"MJHONFOU • "JNTUPUSBJOUIFWJTVBMBEBQUFS UPBMJHOWJEFP GFBUVSFTXJUI--.`TTFNBOUJDTQBDF • %BUBTFU-$4, JNBHFUFYUQBJSVTFEGPS

    QSFUSBJOJOH--B7"<-JV /FVS*14`> • /PUF • 4JODFDPOUFNQPSBSZMBSHFTDBMFWJEFPUFYUEBUBTFUTDPOUBJO TVCTUBOUJBMUFYUVBMOPJTF UIFZVTFUIFJNBHFUFYUEBUBTFU 8

  9. 1SPQPTFE.FUIPE • 4UBHF#PVOEBSZ1FSDFQUJPO • "JNTUPGBDJMJUBUJOHUIFVOEFSTUBOEJOHPG WBSJPVTFWFOUTPDDVSSJOHXJUIJOUIFWJEFP • %BUBTFUTJOHMFUVSOBOENVMUJUVSO RVFTUJPOBOTXFSJOH 2"

     • GSPN*OUFSO7JE.'-5<8BOH *$-3>  CPVOEBSZBXBSFEBUBTFU • *O*OUFSO7JE.'-5 1Z4DFOF%FUFDU BVUPNBUJDBMMZTQMJUVOUSJNNFEWJEFPJOUPNVMUJQMF DMJQT 9

  10. 1SPQPTFE.FUIPE • 4UBHF*OTUSVDUJPO5VOJOH • "JNTUPBMJHOXJUIIVNBOJOUFOU • %JBMPHVFNBEFGSPNIJHIRVBMJUZUFNQPSBM SFMBUFEEBUBTFUTFOBCMF75JNF--.NPSF QSFDJTFFWFOUMPDBMJ[BUJPOBOEEFTDSJQUJPO •

    %BUBTFU--.HFOFSBUFEWJEFPCBTFE EJBMPHVFTGSPN"DUJWJUZ/FU$BQUJPO <,SJTIOB *$$7> %7$ BOE%J%F.P <)FOESJDLT *$$7> 75( 10 Trained on Stage 2

  11. 1SPQPTFE.FUIPE • 1SPNQUTPGUSBJOJOH4UBHF 11 WJEFPGFBUVSF FNCFEEJOHMJTU XJMMCFJOTFSUFEIFSF TZTUFN QSPNQU

  12. 1SPQPTFE.FUIPE • 1SPNQUTPGUSBJOJOH4UBHF 12 WJEFPGFBUVSF FNCFEEJOHMJTU XJMMCFJOTFSUFEIFSF TZTUFN QSPNQU Q1

    A1 Q2 A2 Q3 A3 Q4 A4 #PUI2BOE"BSF HFOFSBUFECZ--.

  13. &YQFSJNFOUT • *NQMFNFOUBUJPO • --.7JDVOBW # # <;IFOH /FVS*14> •

    -PTTGVODUJPOPSJHJOBMBVUPSFHSFTTJWFUSBJOJOHPCKFDUJWFTPGUIF--. DSPTT FOUSPQZ • *OTFDPOEBOEUIJSETUBHFT UIFZPOMZUSBJO-P3"QBSBNFUFST XJUIGSFF[JOH QSFUSBJOFEQBSBNFUFST • 5SBJOJOH359(16 • #BTFMJOFT • &YJTUJOH7JEFP--.T 7JEFP$IBU#<-J BS9JW> 7JEFP--B." #<;IBOH  &./-1> 7JEFP$IBU(15 #<.BB[ "$-> 13

  14. &YQFSJNFOUT • &WBMVBUFPO5XP Ћ UBTLT • 57( • %BUBTFU"DUJWJUZ/FU$BQUJPOTBOE$IBSBEFT45" •

    .FUSJDNFBO*06 N*06 SFDBMM!N 3!N • %7$ • %BUBTFU"DUJWJUZ/FU$BQUJPOT • .FUSJD MPDBMJ[JOH U*P6  DBQUJPOJOH $*%&Sɾ.&5&03 DPNCJOBUJPO 40%"@D • 7JEFP%JBMPHVF • %BUBTFU7JEFP$IBU(15 CFODINBSL • .FUSJD(15BTTJHOTTDPSFT d UPBHFOFSBUFEUFYUXJUIBTQFDUT 14 7JEFP%JBMPHVF&WBMVBUJPO

  15. &YQFSJNFOUT 75( %7$ 7JEFP%JBMPHVF 15 3FTVMUTTIPXT75JNF--.PVUQFSGPSNTCBTFMJOFTBDSPTT WBSJPVTWJEFPVOEFSTUBOEJOHUBTLT

  16. &YQFSJNFOUT "CMBUJPO4UVEZ  *NBHF5FYUQBJSJTNPSFTVJUBCMFSBUIFSUIBO7JEFP5FYUQBJSGPS4UBHF  7JTVBM"EBQUFSTIPVMECFGSFF[FBGUFS4UBHF  -P3" QBSBNFUFSUSBJOFEPO4UBHFTIPVMECFBEEFEPO4UBHF 16

  17. &YQFSJNFOUT 2VBMJUBUJWFSFTVMUTPOWJEFPEJBMPHVF • 75JNF--. JMMVTUSBUFTHPPEEJBMPHVFQFSGPSNBODF 17

  18. $PODMVTJPO • 5IFZJOUSPEVDF75JNF--. B7JEFP--.DBQBCMFPGDPNQSFIFOEJOH NVMUJQMFFWFOUTXJUIJOBWJEFPBOEQSPWJEJOHQSFDJTFUFNQPSBM CPVOEBSJFT • 5IFZQSPQPTFBUISFFTUBHFUFNQPSBMBXBSFUSBJOJOHGSBNFXPSLXJUI UFNQPSBMSFMBUFE2VFTUJPO"OTXFSJOH MFWFSBHJOHNVMUJFWFOUEBUBTFUTGPS

    5FNQPSBM7JEFP(SPVOEJOHBOE%FOTF7JEFP$BQUJPOJOH • &YUFOTJWFFYQFSJNFOUTEFNPOTUSBUFUIBU75JNF--.PVUQFSGPSNT FYJTUJOH7JEFP--.TTJHOJGJDBOUMZBDSPTTWBSJPVTWJEFPVOEFSTUBOEJOH UBTLT 18

  19. 5IBOLZPV 19

  20. .ZDPODFSOT • $PNQBSJTPOXJUI%7$ 57(POMZNPEFMT • 1FSGPSNBODFPOPUIFSEPNBJO%7$EBUBTFUT • &WBMVBUJPOWBMJEJUZGPSQFSGPSNBODFPO7JEFP%JBMPHVF 20