Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

【CVPR 2024】VTimeLLM: Empower LLM to Grasp Video...

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for mori yuichiro mori yuichiro
July 09, 2024
84
0

【CVPR 2024】VTimeLLM: Empower LLM to Grasp Video Moments 論文読み会発表資料

Avatar for mori yuichiro

mori yuichiro

July 09, 2024

More Decks by mori yuichiro

See All by mori yuichiro
【EMNLP 2023】Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs 論文読み会発表資料
Avatar for mori yuichiro zaemon1251hesty
0
160

Featured

See All Featured
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
32
3.1k
Taking LLMs out of the black box: A practical guide to human-in-the-loop distillation
Avatar for Ines Montani inesmontani
PRO
3
2.5k
Marketing Yourself as an Engineer | Alaka | Gurzu
Avatar for Gurzu gurzu
0
370
Impact Scores and Hybrid Strategies: The future of link building
Avatar for Tamara Novitovic tamaranovitovic
0
480
Mozcon NYC 2025: Stop Losing SEO Traffic
Avatar for Sam Torres samtorres
1
600
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
Avatar for Kenny Baas-Schwegler baasie
1
600
Darren the Foodie - Storyboard
Avatar for Kevinho.art khoart
PRO
4
4.1k
Six Lessons from altMBA
Avatar for Skipper Chong Warson skipperchong
29
4.6k
Building an army of robots
Avatar for Kyle Neath kneath
307
47k
Technical Leadership for Architectural Decision Making
Avatar for Kenny Baas-Schwegler baasie
3
640
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
31
6.4k
The Director’s Chair: Orchestrating AI for Truly Effective Learning
Avatar for Mike Taylor tmiket
1
350

Transcript

  1. 75JNF--.&NQPXFS--.UP (SBTQ7JEFP.PNFOUT #JO)VBOH 9JO8BOH )POH$IFO ;JIBO4POH 8FOXV;IV 5TJOHIVB6OJWFSTJUZ $713 :VJDIJSP.PSJ

    1BQFS3FBEJOH 1

  2. 4VNNBSZ • -BSHFMBOHVBHFNPEFMT --.T IBWFCFFOBEBQUFEUPIBOEMFWJEFP EBUBCVUTUSVHHMFXJUIQSFDJTFFWFOUUJNJOH • 75JNF--. BOFX7JEFP--. BEESFTTFTUIJTCZVTJOHBUISFF

    TUBHFUSBJOJOHTUSBUFHZUPJNQSPWFUFNQPSBMVOEFSTUBOEJOH • &YQFSJNFOUTTIPX75JNF--.FYDFMTJOGJOFHSBJOFEWJEFP DPNQSFIFOTJPOBOEPVUQFSGPSNTFYJTUJOH7JEFP--.TJOWBSJPVT UBTLT 2

  3. #BDLHSPVOE • 7JEFP--. • VUJMJ[JOH-BSHF-BOHVBHF.PEFMUPVOEFSTUBOEWJEFP • BOBHFOUGPSJOUFSBDUJOHXJUIIVNBOCBTFEPOBWJEFP • FH 

    7JEFP--B." <;IBOH &./-1> 7JEFP$IBU <-J BS9JW> 3 7JEFP--B."

  4. 1SPCMFN • &YJTUJOH7JEFP--.TGPDVTPOHFOFSBUJPOPGHFOFSJDWJEFP DBQUJPOT UIVTGBJMUPDBQUVSFUIFSFMBUJPOTIJQTCFUXFFOTQFDJGJD CPVOEFEFWFOUT 4 7JEFP$IBU  7JEFP--B."DBOOPU

    MPDBMJ[FUIFFWFOU UJNJOH 7JEFP--.TBSF GBSGSPNQSFDJTFMZ VOEFSTUBOEJOH WJEFP

  5. 3FTFBSDI0CKFDUJWF 5 5IFZCVJME75JNF--.5IF'JSTUCPVOEBSZBXBSF7JEFP--. 5BDLMJOHUXPDIBMMFOHFTGPS7JEFP--.TUPDBQUVSFFWFOUCPVOEBSJFTJO WJEFP  4DBSDJUZPGMBSHFTDBMFCPVOEBSZBXBSFWJEFPEBUBTFUT Ø 5IFZDSFBUFEBUBTFUTJODPSQPSBUJOHFYJTUJOHUFNQPSBMSFMBUFE 7JEFP6OEFSTUBOEJOHUBTLT

     %FTJHOFGGFDUJWFUFNQPSBMSFMBUFEWJEFPUBTLTGPSUSBJOJOH7JEFP--. Ø 5IFZQSPQPTF#PVOEBSZBXBSF5SBJOJOH4USBUFHZ

  6. 1SPQPTFE.FUIPE • 1SFMJNJOBSZ5XPUFNQPSBMSFMBUFE7JEFP6OEFSTUBOEJOHUBTLT 6 5FNQPSBM7JEFP (SPVOEJOH 57( %FOTF7JEFP $BQUJPOJOH %7$

    • UFNQPSBMMPDBMJ[BUJPODPSSFTQPOEJOHWJEFP TFHNFOUTGPSHJWFOUFYUVBMJOQVUT • UFNQPSBMMPDBMJ[BUJPOBOEDBQUJPOJOH GPSBMMFWFOUT 5FNQPSBMMZ(SPVOEJOH/BUVSBM4FOUFODFJO7JEFP <+JOHZVBO &./-1> %FOTF$BQUJPOJOHJOFWFOUT <,SJTIOB *$$7>

  7. 1SPQPTFE.FUIPE • #PVOEBSZBXBSF5SBJOJOH4USBUFHZ DPOTJTUT5SBJOJOH4UBHFT 7

  8. 1SPQPTFE.FUIPE • 4UBHF'FBUVSF"MJHONFOU • "JNTUPUSBJOUIFWJTVBMBEBQUFS UPBMJHOWJEFP GFBUVSFTXJUI--.`TTFNBOUJDTQBDF • %BUBTFU-$4, JNBHFUFYUQBJSVTFEGPS

    QSFUSBJOJOH--B7"<-JV /FVS*14`> • /PUF • 4JODFDPOUFNQPSBSZMBSHFTDBMFWJEFPUFYUEBUBTFUTDPOUBJO TVCTUBOUJBMUFYUVBMOPJTF UIFZVTFUIFJNBHFUFYUEBUBTFU 8

  9. 1SPQPTFE.FUIPE • 4UBHF#PVOEBSZ1FSDFQUJPO • "JNTUPGBDJMJUBUJOHUIFVOEFSTUBOEJOHPG WBSJPVTFWFOUTPDDVSSJOHXJUIJOUIFWJEFP • %BUBTFUTJOHMFUVSOBOENVMUJUVSO RVFTUJPOBOTXFSJOH 2"

     • GSPN*OUFSO7JE.'-5<8BOH *$-3>  CPVOEBSZBXBSFEBUBTFU • *O*OUFSO7JE.'-5 1Z4DFOF%FUFDU BVUPNBUJDBMMZTQMJUVOUSJNNFEWJEFPJOUPNVMUJQMF DMJQT 9

  10. 1SPQPTFE.FUIPE • 4UBHF*OTUSVDUJPO5VOJOH • "JNTUPBMJHOXJUIIVNBOJOUFOU • %JBMPHVFNBEFGSPNIJHIRVBMJUZUFNQPSBM SFMBUFEEBUBTFUTFOBCMF75JNF--.NPSF QSFDJTFFWFOUMPDBMJ[BUJPOBOEEFTDSJQUJPO •

    %BUBTFU--.HFOFSBUFEWJEFPCBTFE EJBMPHVFTGSPN"DUJWJUZ/FU$BQUJPO <,SJTIOB *$$7> %7$ BOE%J%F.P <)FOESJDLT *$$7> 75( 10 Trained on Stage 2

  11. 1SPQPTFE.FUIPE • 1SPNQUTPGUSBJOJOH4UBHF 11 WJEFPGFBUVSF FNCFEEJOHMJTU XJMMCFJOTFSUFEIFSF TZTUFN QSPNQU

  12. 1SPQPTFE.FUIPE • 1SPNQUTPGUSBJOJOH4UBHF 12 WJEFPGFBUVSF FNCFEEJOHMJTU XJMMCFJOTFSUFEIFSF TZTUFN QSPNQU Q1

    A1 Q2 A2 Q3 A3 Q4 A4 #PUI2BOE"BSF HFOFSBUFECZ--.

  13. &YQFSJNFOUT • *NQMFNFOUBUJPO • --.7JDVOBW # # <;IFOH /FVS*14> •

    -PTTGVODUJPOPSJHJOBMBVUPSFHSFTTJWFUSBJOJOHPCKFDUJWFTPGUIF--. DSPTT FOUSPQZ • *OTFDPOEBOEUIJSETUBHFT UIFZPOMZUSBJO-P3"QBSBNFUFST XJUIGSFF[JOH QSFUSBJOFEQBSBNFUFST • 5SBJOJOH359(16 • #BTFMJOFT • &YJTUJOH7JEFP--.T 7JEFP$IBU#<-J BS9JW> 7JEFP--B." #<;IBOH  &./-1> 7JEFP$IBU(15 #<.BB[ "$-> 13

  14. &YQFSJNFOUT • &WBMVBUFPO5XP Ћ UBTLT • 57( • %BUBTFU"DUJWJUZ/FU$BQUJPOTBOE$IBSBEFT45" •

    .FUSJDNFBO*06 N*06 SFDBMM!N 3!N • %7$ • %BUBTFU"DUJWJUZ/FU$BQUJPOT • .FUSJD MPDBMJ[JOH U*P6  DBQUJPOJOH $*%&Sɾ.&5&03 DPNCJOBUJPO 40%"@D • 7JEFP%JBMPHVF • %BUBTFU7JEFP$IBU(15 CFODINBSL • .FUSJD(15BTTJHOTTDPSFT d UPBHFOFSBUFEUFYUXJUIBTQFDUT 14 7JEFP%JBMPHVF&WBMVBUJPO

  15. &YQFSJNFOUT 75( %7$ 7JEFP%JBMPHVF 15 3FTVMUTTIPXT75JNF--.PVUQFSGPSNTCBTFMJOFTBDSPTT WBSJPVTWJEFPVOEFSTUBOEJOHUBTLT

  16. &YQFSJNFOUT "CMBUJPO4UVEZ  *NBHF5FYUQBJSJTNPSFTVJUBCMFSBUIFSUIBO7JEFP5FYUQBJSGPS4UBHF  7JTVBM"EBQUFSTIPVMECFGSFF[FBGUFS4UBHF  -P3" QBSBNFUFSUSBJOFEPO4UBHFTIPVMECFBEEFEPO4UBHF 16

  17. &YQFSJNFOUT 2VBMJUBUJWFSFTVMUTPOWJEFPEJBMPHVF • 75JNF--. JMMVTUSBUFTHPPEEJBMPHVFQFSGPSNBODF 17

  18. $PODMVTJPO • 5IFZJOUSPEVDF75JNF--. B7JEFP--.DBQBCMFPGDPNQSFIFOEJOH NVMUJQMFFWFOUTXJUIJOBWJEFPBOEQSPWJEJOHQSFDJTFUFNQPSBM CPVOEBSJFT • 5IFZQSPQPTFBUISFFTUBHFUFNQPSBMBXBSFUSBJOJOHGSBNFXPSLXJUI UFNQPSBMSFMBUFE2VFTUJPO"OTXFSJOH MFWFSBHJOHNVMUJFWFOUEBUBTFUTGPS

    5FNQPSBM7JEFP(SPVOEJOHBOE%FOTF7JEFP$BQUJPOJOH • &YUFOTJWFFYQFSJNFOUTEFNPOTUSBUFUIBU75JNF--.PVUQFSGPSNT FYJTUJOH7JEFP--.TTJHOJGJDBOUMZBDSPTTWBSJPVTWJEFPVOEFSTUBOEJOH UBTLT 18

  19. 5IBOLZPV 19

  20. .ZDPODFSOT • $PNQBSJTPOXJUI%7$ 57(POMZNPEFMT • 1FSGPSNBODFPOPUIFSEPNBJO%7$EBUBTFUT • &WBMVBUJPOWBMJEJUZGPSQFSGPSNBODFPO7JEFP%JBMPHVF 20