이미지까지 이해하는 Multimodal LLM의 학습 방법 밝혀내기

.VMUJNPEBM--.੉ۆ 5SBJOJOH3FDJQFTGPS.VMUJNPEBM--.)POFZCFF $VSSFOU1SPHSFTT'VUVSF%JSFDUJPO

द੼ U ө૑ ઱য૓ ױযٜਸ ӝ߈ਵ۽ ׮਺ U ױযܳ
৘ஏೞب۾ ೟णػ BVUPSFHSFTTJWFNPEFM -BOHVBHF.PEFM -. 𝑚𝑎𝑥 $ !"# $ log 𝑝 𝑤!%& 𝑤'! )

द੼ U ө૑ ઱য૓ ױযٜਸ ӝ߈ਵ۽ ׮਺ U ױযܳ
৘ஏೞب۾ ೟णػ BVUPSFHSFTTJWFNPEFM -BOHVBHF.PEFM -. t߄աաח ઱۽ ޖट ࢝੉ঠ u t߄աաח ઱۽ ֢ۆ࢝ ੑפ׮u 𝑚𝑎𝑥 $ !"# $ log 𝑝 𝑤!%& 𝑤'! )

೟ण ؘ੉ఠ৬ ݽ؛ ௼ӝ ҙ੼ীࢲ੄ TDBMFVQ -BSHF-BOHVBHF.PEFM --.

೟ण ؘ੉ఠ৬ ݽ؛ ௼ӝ ҙ੼ীࢲ੄ TDBMFVQ -BSHF-BOHVBHF.PEFM --. t
j׮਺ী ৢ ं੗ח ޖ঺ੋо u t੉ ࣻৌ਷ п ं੗о ੗োࣻ੄ ઁғੋ ಁఢਸ ٮܵפ׮ ٮۄࢲju

7BSJPVTBQQMJDBUJPOTPG--.T ୁࠈ ѐੋ ࠺ࢲ ੹ޙо ࢲ࠺झ ޙࢲ ߣ৉ ਃড ௏٘
ࢤࢿ ٣ߡӦ j

.VMUJNPEBM --. tఫझ౟ܳ ֈয ੉޷૑ য়٣য় ١ ׮ܲ NPEBMJUZ ੿ࠁܳ
੉೧ೞҊ ࢚ഐ੘ਊ ೡ ࣻ ੓ח --.u

.VMUJNPEBM--. ׮ܲ NPEBMJUZ ৘੉޷૑ ৬ U द੼ө૑ ઱য૓ ױযٜਸ ӝ߈ਵ۽
׮਺ ױযܳ ৘ஏ 𝑚𝑎𝑥 $ !"# $ log 𝑝 𝑤!%& 𝑣, 𝑤'! ) .--.

੗োযܳ ాೠ ੄ࢎࣗా --. tݾҗ ׮ܻо ӡҊ ੹߈੸ਵ۽ ࠙ഘࡄਸ ڸݴju
t݈ॹೞन زޛ਷ ೒ۄ߁Ҋ۽ju --.

.VMUJNPEBM--. ׮ܲ NPEBMJUZ ৘੉޷૑ ৬ U द੼ө૑ ઱য૓ ױযٜਸ ӝ߈ਵ۽
׮਺ ױযܳ ৘ஏ 𝑚𝑎𝑥 $ !"# $ log 𝑝 𝑤!%& 𝑣, 𝑤'! ) tࢎ૓ী ੓ח زޛী ؀೧ ࢸݺ೧઻u tࢎ૓ী ੓ח زޛ਷ ೒ۄ߁Ҋ۽ ju MLLM

7BSJPVTNVMUJNPEBMVO EFSTUBOEJOH DBQBCJMJUJFT .VMUJNPEBM--. 0$3 t8IJDIEJSFDUJPOju

7BSJPVTNVMUJNPEBMVOEFSTUBOEJOHDBQBCJMJUJFT .VMUJNPEBM--. $IBSU2" t8IBUJTUIFBWFSBHFPGju 0$3 t8IJDIEJSFDUJPOju

7BSJPVTNVMUJNPEBMVOEFSTUBOEJOHDBQBCJMJUJFT .VMUJNPEBM--. %PDVNFOUSFBEJOH $IBSU2" “What is the average of …”
“What is the quantity of…” 0$3 “Which direction …”

7BSJPVTNVMUJNPEBMVOEFSTUBOEJOHDBQBCJMJUJFT .VMUJNPEBM--. %PDVNFOUSFBEJOH $IBSU2" t8IBUJTUIFBWFSBHFPGju $PEFHFOFSBUJPO t(FOFSBUFB1ZUIPODPEFju t8IBUJTUIFRVBOUJUZPGju 0$3 t8IJDIEJSFDUJPOju

*NQSPWJOH--.BQQMJDBUJPOTXJUIWJTJPO ୁࠈ ѐੋ ࠺ࢲ ੹ޙо ࢲ࠺झ ޙࢲ ߣ৉ ਃড ௏٘
ࢤࢿ ٣ߡӦ … .VMUJNPEBM--.

.VMUJNPEBM-BSHF-BOHVBHF.PEFMT .--.T 'MBNJOHP #-*1 ,PTNPT 1B-.& --B.""EBQUFS
*OTUSVDU#-*1 .JOJ(15 --B7" $IBNFMFPO 7JTJPO--. -ZOY 2XFO7- (15W ,PTNPT )POFZCFF --B7" /&Y5(15 9DPNQPTFS $PH7-. (FNJOJ 9DPNQPTFS 2XFO7-.BY --B7"/F95 (FNJOJ .. $PH7-. 2XFO7- 1BMJ(FNNB --B7"0OFWJTJPO $MBVEF /7-. (15P

"DDFQUFEBTB)JHIMJHIU BU$713 _PGBDDFQUFEQBQFST

*OTVGGJDJFOUUSBJOJOHSFDJQFTj Flamingo BLIP2 Kosmos-1 PaLM-E LLaMA-Adapter InstructBLIP
MiniGPT-4 LLaVA Chameleon VisionLLM Lynx Qwen-VL GPT-4v Kosmos-2 Honeybee LLaVA1.5 NExT-GPT Xcomposer CogVLM Gemini Xcomposer2 Qwen-VL-Max LLaVA-NeXT Gemini1.5 MM1 CogVLM2 Qwen2-VL PaliGemma LLaVA-Onevision Claude 3.5 NVLM GPT-4o ݆਷ ݽ؛ٜ੉ ॔ই૑Ҋ ੓঻૑݅ ҳઑա ೟ण ߑߨۿ ҙ੼ীࢲ੄ ׮নೠ ఐ࢝਷ ࠗ઒೮؍ दӝ

)PX ই ӒѢ $-*1-JOFBS7JDVOB ҳઑ ӝ߈ਵ۽ 72" (2" $$.
5FYU$BQTjؘ੉ఠ۽ ೟णदఃҊ MFBSOJOHSBUFח j ೞݶ ظਃ_ …

*OTVGGJDJFOUUSBJOJOHSFDJQFTj ޙઁ ੋध оࢸ ࣻ݀ Ѿۿ प೷ Ѩૐ

ݽ؛ ҳઑ ೟ण ߑߨ tݽ؛ ҳઑীࢲ ޥо ֬஖Ҋ ੓ח Ѫ਷
হա u tযڃ ؘ੉ఠٜਸ о૑Ҋ যڌѱ ೟णदெঠ ೞ૑ u

#BTJDBSDIJUFDUVSFPG.--.T 7JTJPO &ODPEFS -BSHF -BOHVBHF.PEFM 1SPKFDUPS

#BTJDBSDIJUFDUVSFPG.--.T 7JTJPO &ODPEFS -BSHF -BOHVBHF.PEFM 1SPKFDUPS $-*1 4JH-*1 .P$P %*/0
4". j

#BTJDBSDIJUFDUVSFPG.--.T 7JTJPO &ODPEFS -BSHF -BOHVBHF.PEFM 1SPKFDUPS $-*1 4JH-*1 .P$P %*/0
4". j --B." 7JDVOB 2XFO *OUFSO-. j

Basic architecture of MLLMs Vision Encoder Large Language Model Projector
$-*1 4JH-*1 .P$P %*/0 4". j --B." 7JDVOB 2XFO *OUFSO-. j

#BTJDBSDIJUFDUVSFPG.--.T 7JTJPO &ODPEFS -BSHF -BOHVBHF.PEFM $-*1 4JH-*1 .P$P %*/0 4".
j --B." 7JDVOB 2XFO *OUFSO-. j 1SPKFDUPS

#BTJDBSDIJUFDUVSFPG.--.T 7JTJPO &ODPEFS -BSHF -BOHVBHF.PEFM 1SPKFDUPS -JOFBS 3FTBNQMFS 2'PSNFS

#BTJDBSDIJUFDUVSFPG.--.T 7JTJPO &ODPEFS -BSHF -BOHVBHF.PEFM 1SPKFDUPS t೒ۄ߁Ҋח ؀୓۽ju t੉ زޛ੄
ࢲध૑ח u 1SPKFDUPS QMBZTBDSVDJBMSPMFJOCSJEHJOH QSFUSBJOFEWJTJPO FODPEFSTXJUI--.T

#BTJDBSDIJUFDUVSFPG.--.T 7JTJPO &ODPEFS -BSHF -BOHVBHF.PEFM 1SPKFDUPS -JOFBS 3FTBNQMFS 2'PSNFS
$"CTUSBDUPS %"CTUSBDUPS

$VSSFOUQSPKFDUPST-JOFBSWT3FTBNQMFS 7JTJPO &ODPEFS -JOFBS ✘ 'MFYJCJMJUZ 0OFUPPOF NBQQJOH 7JTVBMGFBUVSFT 7JTVBMUPLFOT
-JOFBS 1SPKFDUPS

$VSSFOUQSPKFDUPST-JOFBSWT3FTBNQMFS 7JTJPO &ODPEFS "CTUSBDUPS 3FTBNQMFS 7JTJPO &ODPEFS -JOFBS ✘ 'MFYJCJMJUZ
0OFUPPOF NBQQJOH ✓ 'MFYJCJMJUZ "CTUSBDUFE NBQQJOH 3FTBNQMFS 7JTVBMGFBUVSFT $SPTT "UUFOUJPO -FBSOBCMF RVFSJFT 7JTVBMUPLFOT -JOFBS 1SPKFDUPS

3FTBNQMFSJTGMFYJCMF CVUj 3FTBNQMFSUFOETUPCFCJBTFEXJUIBTBMJFOUSFHJPO -FBSOBCMFRVFSJFT -JOFBS -JOFBS 7JTVBM GFBUVSFT

3FTBNQMFSJTGMFYJCMF CVUj 3FTBNQMFSUFOETUPCFCJBTFEXJUIBTBMJFOUSFHJPO -FBSOBCMFRVFSJFT -JOFBS -JOFBS "UUFOUJPO NBQ 7JTVBM GFBUVSFT

4PGUNBY e!! ∑ " e!"

4PGUNBY e!! ∑ " e!" × × ×

3FTBNQMFSJTGMFYJCMF CVUj 3FTBNQMFSUFOETUPCFCJBTFEXJUIBTBMJFOUSFHJPO "WFSBHFQFSGPSNBODFPGTQBUJBMSFMBUJPOTIJQUBTLT #JBTFEBUUFOUJPOUPXBSEBTBMJFOUSFHJPO

$VSSFOUQSPKFDUPST-JOFBSWT3FTBNQMFS 7JTJPO &ODPEFS 7JTJPO &ODPEFS -JOFBS 3FTBNQMFS 7JTVBMGFBUVSFT $SPTT "UUFOUJPO
-FBSOBCMF RVFSJFT 7JTVBMUPLFOT ✘ 'MFYJCJMJUZ ✓ -PDBMJUZ ✓ 'MFYJCJMJUZ ✘ -PDBMJUZ "CTUSBDUPS 3FTBNQMFS -JOFBS 1SPKFDUPS

4PMVUJPOJOKFDUJOHMPDBMJUZUPBCTUSBDUPST 5XPEFTJHOQSJODJQMFT 'MFYJCJMJUZ JONBOBHJOHUIFOVNCFSPGWJTVBMUPLFOT 1SFTFSWBUJPOPGMPDBMDPOUFYUTJOWJTVBMGFBUVSFT 5XPBCTUSBDUPSTVOEFSUIFQSJODJQMFT $POWPMVUJPOCBTFE$"CTUSBDUPS
%FGPSNBCMFBUUFOUJPOCBTFE%"CTUSBDUPS

4PMVUJPOJOKFDUJOHMPDBMJUZUPBCTUSBDUPST 5XPEFTJHOQSJODJQMFT 'MFYJCJMJUZ JONBOBHJOHUIFOVNCFSPGWJTVBMUPLFOT 1SFTFSWBUJPOPGMPDBMDPOUFYUTJOWJTVBMGFBUVSFT (a) Linear Projector
✗ Flexibility ✓ Locality Preservation Linear Projector (b) Abstractor ✓ Flexibility ✗ Locality Preservation (c) Locality-enhanced Abstractor ✓ Flexibility ✓ Locality Preservation Vision Encoder Projector Large Language Model Visual Tokens Text Tokens Visual Features Resampler C-Abstractor D-Abstractor

$PNQBSJTPOXJUIQSFWJPVTQSPKFDUPST &GGFDUJWFOFTTPGPVSMPDBMJUZFOIBODFEQSPKFDUPS #FUUFSTQBUJBMVOEFSTUBOEJOHDBQBCJMJUZ #FUUFSQFSGPSNBODFFGGJDJFODZCBMBODF 1SPKFDUPS . TTUFQ ..&
..# 4&&% "WH 104 43 0- 13 43 *- -JOFBS 6OBWBJMBCMFEVFUPJOGMFYJCJMJUZ 3FTBNQMFS $"CTUSBDUPS -JOFBS 3FTBNQMFS $"CTUSBDUPS

ݽ؛ ҳઑ ೟ण ߑߨ tݽ؛ ҳઑীࢲ ޥо ֬஖Ҋ ੓ח Ѫ਷
হա u tযڃ ؘ੉ఠٜਸ о૑Ҋ যڌѱ ೟णदெঠ ೞ૑ u

7JTVBMJOTUSVDUJPOUVOJOH 1VSQPTFPGWJTVBMJOTUSVDUJPOUVOJOH &OIBODJOHWJTJPOMBOHVBHFVOEFSTUBOEJOH )VNBOQSFGFSFODFBMJHONFOU *OTUSVDUJ[BUJPO 5SBOTGPSNJOHFYJTUJOHEBUBTFUTJOUPJOTUSVDUJPOGPMMPXJOHGPSNBUWJB(15BTTJTUFE PS
UFNQMBUFCBTFE XBZ

7JTVBMJOTUSVDUJPOUVOJOHJOTUSVDUJ[BUJPO $BQUJPOt5IFSFJTBHSPVQPGNFOBSPVOEB USVDLu t5IFDPMPSPGBWJOUBHFUSVDLJTju j 0CKFDUT5SVDL< > j
72"t8IBUJTUIFDPMPSPGUIFUSVDL 3FEu t)PXNBOZQFPQMFBSFUIFSF u'JWFu j (15 )VNBO)FMQNFXSJUFBOBEWFSUJTFNFOU GPSUIFWJOUBHFSFEUSVDL "*$MBTTJDT3FE7JOUBHF1JDLVQ 5SVDLm "5JNFMFTT#FBVUZ-PPLJOHUP PXOBQJFDFPGBVUPNPUJWFIJTUPSZ 5IJT NFUJDVMPVTMZSFTUPSFETWJOUBHFSFE QJDLVQUSVDLJTj 1SFEFGJOFE UFNQMBUFT )VNBO"OTXFSVTJOHBTJOHMFXPSEPS QISBTF8IBUDPMPSJTUIFUSVDLPOUIFMFGU "*3FE )VNBO)PXNBOZQFPQMFBSFUIFSF "*GJWF 5FNQMBUFCBTFE (15BTTJTUFE

7JTVBMJOTUSVDUJPOUVOJOHJOTUSVDUJ[BUJPO 7BSJPVTWJTJPOMBOHVBHFEBUBTFUTBSFVTFEGPSUIFUFNQMBUFCBTFEWJTVBMJOTUSVDUJPO 5BTL %BUBTFUT TBNQMFT $BQUJPOJOH #MJQ$BQ'JMU $0:0. . 72"
0QFO 72"W (2" 0$372" 743 . 72" .$ 4DJFODF2" "0,72" . 3&$ 3FG$0$0 3FG$0$0 3FG$0$0H 7JTVBM(FOPNF . *OTUSVDUJPO --B7"L 4IBSF(15 .

5FNQMBUFTGPSJOEJWJEVBMEBUBTFUT

)JEEFOSFDJQFGPSWJTVBMJOTUSVDUJPOUVOJOH %BUBTFU .JYUVSF %BUBTFU #BMBODF 5FNQMBUF %FTJHO .VMUJUVSO %FEVQMJDBUJPO

)JEEFOSFDJQF EBUBTFUNJYUVSF (15BTTJTUFE %WT% à IFMQGVMGPSEFTDSJQUJWFSFTQPOTFT 5FNQMBUFCBTFE %WT% à IFMQGVMJOQFSDFQUVBMVOEFSTUBOEJOH
*OTUSVDUJPO5BTL5ZQF .--.#FODINBSL 5FNQMBUFCBTFE (15BTTJTUFE .VMUJQMFDIPJDF #JOBSZZFTOP (15FWBM 72" 0QFO 72" .$ 3&$ $BQ 7*OTU 5*OTU ..# 4&&%J ..&Q ..& --B7"X % ✓ ✓ ✓ ✓ ✓ ✓ % ✓ ✓ ✓ ✓ % ✓ ✓

)JEEFOSFDJQF EBUBTFUNJYUVSF %JWFSTJUZPGEBUBTFUTBOEUBTLTJTJNQPSUBOUGPSMFBSOJOHQSPGJDJFODZJOSFMBUFEUBTLT *OTUSVDUJPO5BTL5ZQF .--.#FODINBSL 5FNQMBUFCBTFE (15BTTJTUFE .VMUJQMFDIPJDF #JOBSZZFTOP (15FWBM
72" 0QFO 72" .$ 3&$ $BQ 7*OTU 5*OTU ..# 4&&%J ..&Q ..& --B7"X % ✓ ✓ ✓ ✓ ✓ ✓ % ✓ ✓ ✓ ✓ ✓ ✓ % ✓ ✓ ✓ ✓ ✓ % ✓ ✓ ✓ ✓ ✓

)JEEFOSFDJQF EBUBTFUCBMBODF %BUBTFUMFWFMVOJGPSNTBNQMJOHJTFGGFDUJWF .BOVBMUVOJOHPGTBNQMJOHXFJHIUTJTUIFCFTU .JYUVSFUZQF ..# 4&&%J ..&Q
"WH 1FSEBUBTFU 1FSUBTL 1FSTBNQMFL 1FSEBUBTFUUVOFE

)JEEFOSFDJQF UFNQMBUFEFTJHO %BUBTFUMFWFM GJOF UBTLMFWFM DPBSTF 0OFUFNQMBUFQFSEBUBTFUJTTVGGJDJFOU (SBOVMBSJUZ
%JWFSTJUZ ..# 4&&%J ..&Q "WH 'JOF 4JOHMF $PBSTF 4JOHMF 'JOF .VMUJ 'JOF .VMUJ GMJQ

)JEEFOSFDJQF UFNQMBUFEFTJHO %BUBTFUMFWFM GJOF UBTLMFWFM DPBSTF 0OFUFNQMBUFQFSEBUBTFUJTTVGGJDJFOU Granularity
Diversity MMB SEED-i MME-p Avg Fine Single Coarse Single Fine Multi Fine Multi + flip

)JEEFOSFDJQF UFNQMBUFEFTJHO %BUBTFUMFWFM GJOF UBTLMFWFM DPBSTF 0OFUFNQMBUFQFSEBUBTFUJTTVGGJDJFOU (SBOVMBSJUZ
%JWFSTJUZ ..# 4&&%J ..&Q "WH 'JOF 4JOHMF $PBSTF 4JOHMF 'JOF .VMUJ 'JOF .VMUJ GMJQ

)JEEFOSFDJQF NVMUJUVSOEFEVQMJDBUJPO $POWFSTBUJPOMJLFNVMUJUVSOUFNQMBUFJTFGGFDUJWF %FEVQMJDBUJPOJTSFRVJSFEUPQSFWFOUTIPSUDVUMFBSOJOH .VMUJUVSO %FEVQMJDBUJPO ..# 4&&%J
..&Q "WH ✓ ✓ ✓

$PNQBSJTPOXJUIQSFWJPVT4P5" Method LLM Projector Vision Enc. Res. MMB MME-p MME
SEED-i LLaVA-w "QQSPBDIFTVTJOH#--. MiniGPT-4 7JDVOB# 3FTBNQMFS &7"$-*17J5( InstructBLIP 7JDVOB# 2GPSNFS &7"$-*17J5( IDEFICS --B."# 'MBNJOHP 0QFO$-*1 7J5) Qwen-VL-Chat 2XFO# 3FTBNQMFS 0QFO$-*1 7J5CJH( LLaVA-1.5 7JDVOB# -JOFBS $-*17J5- Honeybee 7JDVOB# $"CT $-*17J5- %"CT "QQSPBDIFTVTJOH#--. MiniGPT-4 7JDVOB# 3FTBNQMFS &7"$-*17J5( BLIP-2 7JDVOB# 2GPSNFS &7"$-*17J5( InstructBLIP 7JDVOB# 2GPSNFS &7"$-*17J5( LLaVA-1.5 7JDVOB# -JOFBS $-*17J5- Honeybee 7JDVOB# $"CT $-*17J5- %"CT

$VSSFOUQSPHSFTT,BOBOBW ,BOBOB 7JTJPO,BOBOBW "VEJP,BOBOBB 6OJGJFE,BOBOBP ঱যݽ؛ ݣ౭ݽ׳ ঱যݽ؛ ࠺઱঴ ࢤࢿݽ؛
਺ࢿ ݽ؛

$VSSFOUQSPHSFTT,BOBOBW ୭Ӕ .--.ٜ੄ ౟۪٘ܳ ࢓ಝࠁݶ ؊਌ ׮নೞҊ ௾ झாੌ੄
ؘ੉ఠࣇ ೟ण à NBDIJOFHFOFSBUFEؘ੉ఠ ഝਊ ߂ ௬ܻ౭ ஶ౟܀੉ ҙѤ ؊਌ ௾ ೧࢚ب੄ ੑ۱ ੉޷૑ ഝਊà --.੄ ੑ۱ਵ۽ ٜযоח WJTVBMUPLFOࣻܳ ബҗ੸ਵ۽ ઴ੌ ࣻ ੓যঠ ೣ )POFZCFF .PSFEBUBTFUT #JHHFSSFTPMVUJPO "EWBODFESFDJQF j ,BOBOBW

$VSSFOUQSPHSFTT,BOBOBW ୭Ӕ .--.ٜ੄ ౟۪٘ܳ ࢓ಝࠁݶ ؊਌ ׮নೞҊ ௾ झாੌ੄
ؘ੉ఠࣇ ೟ण à NBDIJOFHFOFSBUFEؘ੉ఠ ഝਊ ߂ ௬ܻ౭ ஶ౟܀੉ ҙѤ ؊਌ ௾ ೧࢚ب੄ ੑ۱ ੉޷૑ ഝਊà --.੄ ੑ۱ਵ۽ ٜযоח WJTVBMUPLFOࣻܳ ബҗ੸ਵ۽ ઴ੌ ࣻ ੓যঠ ೣ ..# $IBSU2" %PD72" *OGP72" .BUI7JTUB 4&&%J ...6 (157 (FNJOJ1SP )POFZCFF ,BOBOBW

੉ ੉޷૑ח ৈ۞ թࢿ੉ ࡈр ௿ېध ౟۟ ઱ਤী ݽৈ ੓ח
ݽणਸ ࠁৈસפ׮੉ ౟۟਷ ҟఖ੉ աח ௼܁ Ӓܾҗ ೻٘ۄ੉౟ ӒܻҊ ࣁ۲ػ ٣੗ੋਵ۽ ੋ೧ ֙؀ ژח ֙؀੄ ର۝ਵ۽ ࠁੑפ׮౟۟਷ ੜ ҙܻغҊ ੓ਵݴ ӵՖೠ ࢚క۽ ࠁੑפ׮թࢿٜ਷ ؀ࠗ࠙ ౭࣊எ৬ ୒߄૑ܳ ੑҊ ੓णפ׮ੌࠗח ࢶӖۄझܳ ଱ਊೞҊ ੓Ҋ ೠ ݺ਷ ݽ੗ܳ ॳҊ ੓णפ׮ Ӓٜ਷ ౟۟੄ ٣੗ੋҗ ࢚కী ؀೧ ੉ঠӝೞח Ѫ୊ۢ ࠁ੉ݴ ই݃ب ର۝ী ؀ೠ ҙब੉ա хࢎܳ աఋղҊ ੓ਸ Ѫੑפ׮ߓ҃ীח TVQFSGSFTIۄח р౸੉ ੓ח ࢚੼੉ ੓णפ׮߄ׯ਷ ನ੢غয ੓ਵݴ Ӓܿ੗о ӡѱ ܻ٘ਕઉ ੓য కন੉ ೞטীࢲ ࢚؀੸ਵ۽ ծ਷ пبী ੓਺ਸ աఋշפ׮ 2VBMJUBUJWFFYBNQMFTEFUBJMFEDBQUJPOJOH t੉޷૑ী ؀೧ ੗ࣁ൤ ࢸݺ೧઻u Kanana-v

#FFG৬ 1PSL੄ ױߔ૕ ബਯ ಣӐ஖ܳ ҅࢑ೞӝ ਤ೧ ݢ੷ пп੄
чਸ ഛੋ೧ঠ ೤פ׮ #FFG੄ ױߔ૕ ബਯ਷ ੑפ׮ 1PSL੄ ױߔ૕ ബਯ਷ ੑפ׮ ಣӐਸ ҳೞӝ ਤ೧ ੉ ف чਸ ؊ೠ റ ۽ աׄפ׮ ಣӐ #FFG੄ ױߔ૕ ബਯ 1PSL੄ ױߔ૕ ബਯ ಣӐ ಣӐ ٮۄࢲ t#FFGu৬ t1PSLu੄ ױߔ૕ ബਯ ಣӐ஖ח ੑפ׮ 2VBMJUBUJWFFYBNQMFTEFUBJMFEDBQUJPOJOH t#FFG৬ 1PSL੄ ױߔ૕ ബਯ ಣӐ஖ܳ ҅࢑೧઻u Kanana-v

𝑥 о о੢ ӟ ߸੄ ӡ੉੉Ҋ ࢖пഋ੉ غӝ ਤೠ ઑѤী
੄ೞৈ 𝑥 ਸ ݅઒೧ঠ ೤פ׮ ژೠ كп࢖пഋ੉ غ۰ݶ 𝑥# > 7# + 11#, ∴ 𝑥# > 170ਸ ݅઒೧ঠ ೤פ׮ ف ઑѤਸ ݅઒दఃח ੗োࣻ 𝑥ח ੉޲۽ ҳೞח ೤਷ ੑפ׮ 2VBMJUBUJWFFYBNQMFTEFUBJMFEDBQUJPOJOH tӒܿਸ ࠁҊ Ӓܿ ࣘ ޙઁ੄ ׹ਸ ݏ୶য ࠁࣁਃu Kanana-v

ࢎ૓ ࣘ ਺ध਷ ӣߏੑפ׮ӣߏਸ ݅٘ח ߑߨ਷ ׮਺җ эणפ׮ वਸ অҊ
੸׼۝੄ ޛ۽ ߏਸ ૑਷ റ ҅ۆ ׼Ӕ য়੉ যޗ ೫ ݍ࢓ ױޖ૑ ਋়ਸ ળ࠺೤פ׮ ҅ۆҗ ׼Ӕ਷ ࠃইࢲ ળ࠺ೞҊ য়੉ח ଻ ेয ޛী ׸о نפ׮ যޗҗ ೫ ݍ࢓਷ ࠃইࢲ ળ࠺ೞҊ ਋়਷ ࢕ইࢲ ળ࠺೤פ׮ ӣ ਤী ߏਸ তѱ ಝҊ ળ࠺ೠ ੤ܐٜਸ ৢ۰ࢲ ӣߏਸ ݈ইસפ׮ ӣߏਸ ੜۄࢲ ઁҕೞݶ ؾפ׮ 2VBMJUBUJWFFYBNQMFTEFUBJMFEDBQUJPOJOH tࢎ૓ ࣘ ਺ध੄ ੉ܴҗ ݅٘ח ߨਸ ঌ۰઻u Kanana-v

'VUVSFEJSFDUJPO ୭Ӕ ॔ই૑Ҋ ੓ח .--.ٜ੉ ֥ۄ਍ ࢿמਸ ࠁৈ઱Ҋ ੓૑݅ ৈ੹൤
ӓࠂ೧ঠ ೡ դઁٜ੉ ݆਺

'VUVSFEJSFDUJPODBUBTUSPQIJDGPSHFUUJOH --. QBSBNFUFS੄ সؘ੉౟۽ ੋೠ LOPXMFEHFGPSHFUUJOH੉गо ઁ؀۽ ׮ܞ૑૑ ঋҊ ੓਺

'VUVSFEJSFDUJPOWJTVBMIBMMVDJOBUJPO --.࠙ঠ ীࢲ৬ ݃ଲо૑۽ .--.ب WJTVBMIBMMVDJOBUJPO ੉गо ઓ੤ %FTDSJCFUIJT JNBHFJOEFUBJM
5IFJNBHFTIPXTB SFE GJSFIZESBOU MPDBUFEPOB TJEFXBMLj

'VUVSFEJSFDUJPOTVQQPSUNVDIMPOHFSDPOUFYUT --.ী ׮নೠ ݽ׳ܻ౭ FH ਺ࢿ ࠺٣য় ܳ ୶оೡࣻ۾ --.੄
ੑ۱ द௫झ ӡ੉ ߂ ҅࢑۝੉ ௼ѱ ૐо à ؊਌ ബҗ੸ੋ NPEBMJUZBCTUSBDUPS߂ --.ো࢑ োҳ ೙ਃ -BSHF-BOHVBHF.PEFM 4FRVFODF-FOHUI $PNQVUBUJPODPTU

'VUVSFEJSFDUJPONPSFDPNQMFYWJTVBMSFBTPOJOH ؊਌ ࠂ੟ೞҊ ੉޷૑ܳ ੜ ੉೧೧ঠ ೞח tWJTVBMuSFBTPOJOH מ۱ਸ ؊਌
ఃਕঠ ೣ 28IBUTIPVMEZPVQVUJO UIFTRVBSFUPHFUB DPSSFDUEJBHSBN 2#JSE#PCCJFKVNQTPOBGFODFGSPNUIFQPTUPO UIFMFGUFOEUPUIFPUIFSFOE&BDIKVNQUBLFTIJN TFDPOET)FNBLFTKVNQTBIFBEBOEUIFO KVNQCBDL5IFOIFBHBJONBLFTKVNQTBIFBE BOEKVNQCBDL BOETPPO*OIPXNBOZTFDPOET DBO#PCCJFHFUGSPNPOFFOEUPUIFPUIFSFOE 24FWFOTUJDLTMJFPOUPQPG FBDIPUIFS4UJDLJTBU UIFCPUUPNUPQ8IJDI TUJDLJTJOUIFNJEEMF

참고 문헌 IUUQTXXXGMJDLSDPNQIPUPTTIBOLBSPOMJOF IUUQTPVSXPSMEJOEBUBPSHHSBQIFSFOFSHZFGGJDJFODZPGNFBUBOEEBJSZQSPEVDUJPO IUUQTXXXJOEVTUSZEPDVNFOUTVDTGFEVEPDTJENUHK $IB
+ ,BOH 8 .VO + 3PI # )POFZCFF-PDBMJUZFOIBODFEQSPKFDUPSGPSNVMUJNPEBMMMN*O 1SPDFFEJOHTPGUIF*&&&$7'$POGFSFODFPO$PNQVUFS7JTJPOBOE 1BUUFSO3FDPHOJUJPO QQ IUUQTXXXGMJDLSDPNQIPUPTTJLLJOQIPUPTUSFBN IUUQTDPNNPOTXJLJNFEJBPSHXJLJ'JMFӣߏKQH IUUQTDPNNPOTXJLJNFEJBPSHXJLJ'JMF0QFO@GJSF@IZESBOU@ KQH IUUQTTNBSUEBUBTFUHJUIVCJPTNBSU

이미지까지 이해하는 Multimodal LLM의 학습 방법 밝혀내기

이미지까지 이해하는 Multimodal LLM의 학습 방법 밝혀내기

More Decks by kakao

Other Decks in Programming

Featured

Transcript