카카오브레인의 텍스트 기반 이미지 생성 기술

#GenerativeModel

카카오브레인에서 개발한 텍스트 기반 이미지 생성 기술을 소개합니다

발표자 : sam.ksh
카카오브레인에서 생성 모델을 개발하고 있는 연구원입니다.

Transcript

1. $PQZSJHIU ,BLBP $PSQ"MMSJHIUTSFTFSWFE3FEJTUSJCVUJPOPSQVCMJDEJTQMBZJTOPUQFSNJUUFEXJUIPVUXSJUUFOQFSNJTTJPOGSPN,BLBP
   ஠஠য়࠳ۨੋ੄ ఫझ౟ ӝ߈
   ੉޷૑ ࢤࢿ ӝࣿ
   JG LBLBP
   
   ஠஠য়࠳ۨੋ
   ӣࣁള TBNLTI
   

   View Slide

2. ஠஠য়࠳ۨੋ੄ 5*ࢤࢿ ݽ؛ ࣗѐ
   അ੤ ݽ؛ ѐࢶਸ ਤೠ োҳ ೐۽ં౟
   

   View Slide

3. ஠஠য়࠳ۨੋ੄ 5*ࢤࢿ ݽ؛ ࣗѐ
   5PXBSET&GGJDJFOU5FYU$POEJUJPOBM%JGGVTJPO.PEFMT
   CZ"EWFSTBSJBM5SBJOJOH
   

   View Slide

4. %FOPJTJOH%JGGVTJPO1SPCBCJMJTUJD.PEFMT
   
   "QIPUPPGBUFEEZCFBSPOUIFSPBE
   "IJHIRVBMJUZ QJDUVSFPGBNFEJFWBMLOJHIUXJUIHPMEFOBSNPS
   
   +)P "+BJO BOE1"CCFFMt%FOPJTJOH%JGGVTJPO1SPCBCJMJTUJD.PEFMTu /FVS*14s
   

   View Slide

5. 0WFSBMM"SDIJUFDUVSF
   VO$-*1 BLB%"--&
   
   ਬࢎೞѱ 1SJPSm %FDPEFSm 43
   ݽٕ۽ ҳࢿغয ੓णפ׮
   
   "3BNFTI 1%IBSJXBM "/JDIPM $$IV BOE.$IFOu)JFSBSDIJDBM5FYU$POEJUJPOBM*NBHF(FOFSBUJPOXJUI$-*1-BUFOUTuBS9JWW
   

   View Slide

6. 0WFSBMM"SDIJUFDUVSF
   A high quality picture of a medieval
   knight with golden armor
   Prior
   Y/
   Y
   Y
   CLIP (text encoder)
   Y/
   CLIP (image feature)
   

   View Slide

7. 0WFSBMM"SDIJUFDUVSF
   A high quality picture of a medieval
   knight with golden armor
   Y/
   Y
   Y
   CLIP (text encoder)
   Y/
   CLIP (image feature)
   Prior
   

   View Slide

8. 0WFSBMM"SDIJUFDUVSF
   A high quality picture of a medieval
   knight with golden armor
   Y/
   Y
   Y
   CLIP (text encoder)
   Y/
   CLIP (image feature)
   Prior
   

   View Slide

9. 0WFSBMM"SDIJUFDUVSF 1SJPS m %FDPEFSm 43
   

   View Slide

10. 0WFSBMM"SDIJUFDUVSF 1SJPS m %FDPEFSm 43
    $-*1UFYUGFBUVSF QPPMJOH੹ ੿ࠁب ನೣ
    
    

    View Slide

11. 0WFSBMM"SDIJUFDUVSF 1SJPS m %FDPEFSm 43
    $-*1UFYUGFBUVSF UJNF
    

    View Slide

12. 0WFSBMM"SDIJUFDUVSF 1SJPSm %FDPEFS m 43
    A high quality picture of a medieval
    knight with golden armor
    Causal Transformer
    Y/
    Y
    Y/
    CLIP (text encoder)
    CLIP (image feature)
    

    View Slide

13. 0WFSBMM"SDIJUFDUVSF 1SJPSm %FDPEFSm 43
    Y
    Y/
    

    View Slide

14. 0WFSBMM"SDIJUFDUVSF 1SJPSm %FDPEFSm *NQSPWFE43
    Y
    Y/ Y
    1SFUSBJOFE 'SFF[F
    5SBJOBCMF JOJUJBMJ[FEGSPNUIF
    QSFUSBJOFEOFU
    MSE loss
    Perceptual loss
    Adversarial loss
    

    View Slide

15. &YQFSJNFOU4FUVQ
    1SJPS %FDPEFS 43
    ೟ण ؘ੉ఠ .JNBHFUFYUQBJST $0:0ؘ੉ఠࣇ ನೣ
    
    ೟ण ߑध झ௼ې஖ ೟ण
    $-*1ݽ؛ 7J5- 7J5-
    QBSBNFUFST # . ..
    PQUJNJ[BUJPO TUFQT . . ..
    TBNQMJOHTUFQT TUFQT TUFQT TUFQT
    

    View Slide

16. 2VBOUJUBUJWF3FTVMUT
    $$. $0$0 ,GSPNWBM
    
    $-*1T 7J5#
    
    '*%
    

    View Slide

17. 2VBOUJUBUJWF3FTVMUT
    $$. $0$0 ,GSPNWBM
    
    $-*1T 7J5#
    
    '*%
    
    "3BNFTI 1%IBSJXBM "/JDIPM $$IV BOE.$IFOu)JFSBSDIJDBM5FYU$POEJUJPOBM*NBHF(FOFSBUJPOXJUI$-*1-BUFOUTuBS9JWW
    WT GSPNVO$-*1
    
    

    View Slide

18. 2VBMJUBUJWF3FTVMUT
    ࠂ੟ೠ ஶࣆী ؀ೠ ؀਽ מ۱ ಣо
    

    View Slide

19. 2VBMJUBUJWF3FTVMUT
    ࠂ੟ೠ ஶࣆী ؀ೠ ؀਽ מ۱ ಣо (PSZFP DFMBEPOJOUIFTIBQFPGEBSUI WBEFS IPMEJOHBTXPSE
    (PSZFP DFMBEPOJOUIFTIBQFPGEBSUI WBEFS
    

    View Slide

20. 2VBMJUBUJWF3FTVMUT
    ࠂ੟ೠ ஶࣆী ؀ೠ ؀਽ מ۱ ಣо
    (PSZFP DFMBEPOJOUIFTIBQFPGEBSUI WBEFS IPMEJOHBTXPSE
    "TVQFSNBUIXJ[BSEDBU SJDIMZUFYUVSFEPJMQBJOUJOH
    

    View Slide

21. 2VBMJUBUJWF3FTVMUT
    ࠂ੟ೠ ஶࣆী ؀ೠ ؀਽ מ۱ ಣо "OBTUSPOBVUSJEJOHBIPSTF DIJMESFOESBXJOHTUZMF
    "OBTUSPOBVUSJEJOHBIPSTF DIJMESFOESBXJOHTUZMF
    

    View Slide

22. 2VBMJUBUJWF3FTVMUT
    1IPUPSFBMJTUJDࢤࢿ оמ ৈࠗ ಣо
    

    View Slide

23. 2VBMJUBUJWF3FTVMUT
    1IPUPSFBMJTUJDࢤࢿ оמ ৈࠗ ಣо "OBTUSPOBVUSJEJOHBIPSTF DIJMESFOESBXJOHTUZMF
    BQIPUPPGBCBCZQVQQZXBJUJOHGPSIFSNPN
    

    View Slide

24. 2VBMJUBUJWF3FTVMUT
    1IPUPSFBMJTUJDࢤࢿ оמ ৈࠗ ಣо
    1IPUPPGBZFBSPME XIJUFNBO TJMWFSIBJS OFBUCFBSE XFBSJOHBSFEHJMFU UIJDLSJNNFEHMBTTFT
    

    View Slide

25. 2VBMJUBUJWF3FTVMUT
    1IPUPSFBMJTUJDࢤࢿ оמ ৈࠗ ಣо "OBTUSPOBVUSJEJOHBIPSTF DIJMESFOESBXJOHTUZMF
    BQPSUSBJUPGBOPMENPOL
    

    View Slide

26. 2VBMJUBUJWF3FTVMUT
    *NBHFWBSJBUJPOоמ ৈࠗ ಣо
    

    View Slide

27. 2VBMJUBUJWF3FTVMUT
    *NBHFWBSJBUJPOоמ ৈࠗ ಣо
    

    View Slide

28. അ੤ ݽ؛ ѐࢶਸ ਤೠ োҳ ೐۽ં౟
    0VS "QQSPBDIFT UP *NQSPWF 5XPTUBHF *NBHF
    (FOFSBUJPO 'SBNFXPSL
    

    View Slide

29. 5XP4UBHF*NBHF(FOFSBUJPO'SBNFXPSL
    
    "0PSE 07JOZBMT BOE,,BWVLDVPHMVt/FVSBM%JTDSFUF3FQSFTFOUBUJPO-FBSOJOHu/FVS*14s
    

    View Slide

30. 5XP4UBHF*NBHF(FOFSBUJPO'SBNFXPSL
    
    "0PSE 07JOZBMT BOE,,BWVLDVPHMVt/FVSBM%JTDSFUF3FQSFTFOUBUJPO-FBSOJOHu/FVS*14s
    

    View Slide

31. 5XP4UBHF*NBHF(FOFSBUJPO'SBNFXPSL
    
    "0PSE 07JOZBMT BOE,,BWVLDVPHMVt/FVSBM%JTDSFUF3FQSFTFOUBUJPO-FBSOJOHu/FVS*14s
    

    View Slide

32. 5XP4UBHF*NBHF(FOFSBUJPO'SBNFXPSL
    
    "0PSE 07JOZBMT BOE,,BWVLDVPHMVt/FVSBM%JTDSFUF3FQSFTFOUBUJPO-FBSOJOHu/FVS*14s
    4UBHF
    

    View Slide

33. 5XP4UBHF*NBHF(FOFSBUJPO'SBNFXPSL
    
    "0PSE 07JOZBMT BOE,,BWVLDVPHMVt/FVSBM%JTDSFUF3FQSFTFOUBUJPO-FBSOJOHu/FVS*14s
    4UBHF"3PSOPO"3NPEFMJOH
    

    View Slide

34. അ੤ ݽ؛ ѐࢶਸ ਤೠ োҳ ೐۽ં౟
    
    %-FF $,JN 4,JN .$IP BOE8)BOt"VUPSFHSFTTJWF*NBHF(FOFSBUJPOVTJOH3FTJEVBM2VBOUJ[BUJPOt$713s
    
    5:V 4,JN $,JN %-FF BOE#)BOt-PDBMMZ)JFSBSDIJDBM"VUPSFHSFTTJWF.PEFMJOHGPS*NBHF(FOFSBUJPOt/FVS*14s
    
    %-FF $,JN 4,JN .$IP BOE8)BOt%SBGUBOE3FWJTF&GGFDUJWF*NBHF(FOFSBUJPOXJUI$POUFYUVBM325SBOTGPSNFSt/FVS*14s
    327"&
    
    $713s
    )27"&
    
    /FVS*14s
    %SBGUBOE3FWJTF
    
    /FVS*14s
    

    View Slide

35. 4FRVFODF-FOHUIBOE"3.PEFM
    

    View Slide

36. 4FRVFODF-FOHUIBOE"3.PEFM
    

    View Slide

37. 3FTJEVBM2VBOUJ[FE7"& 327"&
    
    
    %-FF $,JN 4,JN .$IP BOE8)BOt"VUPSFHSFTTJWF*NBHF(FOFSBUJPOVTJOH3FTJEVBM2VBOUJ[BUJPOt$713s
    

    View Slide

38. 3FTJEVBM2VBOUJ[FE7"& 327"&
    
    
    %-FF $,JN 4,JN .$IP BOE8)BOt"VUPSFHSFTTJWF*NBHF(FOFSBUJPOVTJOH3FTJEVBM2VBOUJ[BUJPOt$713s
    

    View Slide

39. 3FTJEVBM2VBOUJ[FE7"& 327"&
    
    
    %-FF $,JN 4,JN .$IP BOE8)BOt"VUPSFHSFTTJWF*NBHF(FOFSBUJPOVTJOH3FTJEVBM2VBOUJ[BUJPOt$713s
    

    View Slide

40. 3FTJEVBM2VBOUJ[FE7"& 327"&
    
    
    %-FF $,JN 4,JN .$IP BOE8)BOt"VUPSFHSFTTJWF*NBHF(FOFSBUJPOVTJOH3FTJEVBM2VBOUJ[BUJPOt$713s
    

    View Slide

41. 3FTJEVBM2VBOUJ[FE7"& 327"&
    
    
    %-FF $,JN 4,JN .$IP BOE8)BOt"VUPSFHSFTTJWF*NBHF(FOFSBUJPOVTJOH3FTJEVBM2VBOUJ[BUJPOt$713s
    

    View Slide

42. 3FTJEVBM2VBOUJ[FE7"& 327"&
    
    
    %-FF $,JN 4,JN .$IP BOE8)BOt"VUPSFHSFTTJWF*NBHF(FOFSBUJPOVTJOH3FTJEVBM2VBOUJ[BUJPOt$713s
    $PEF3FTPMVUJPO
    ) Y 8 Y %
    
    $PEFCPPL4J[F S'*%
    72("/
    YY m QBQFS
    
    YY
    
    
    
    327"&
    YY
    YY
    

    View Slide

43. 3FTJEVBM2VBOUJ[FE5SBOTGPSNFS 325SBOTGPSNFS
    
    
    %-FF $,JN 4,JN .$IP BOE8)BOt"VUPSFHSFTTJWF*NBHF(FOFSBUJPOVTJOH3FTJEVBM2VBOUJ[BUJPOt$713s
    

    View Slide

44. 325SBOTGPSNFSز੘ ߑध
    

    View Slide

45. 3FTJEVBM2VBOUJ[FE5SBOTGPSNFS 325SBOTGPSNFS
    
    
    %-FF $,JN 4,JN .$IP BOE8)BOt"VUPSFHSFTTJWF*NBHF(FOFSBUJPOVTJOH3FTJEVBM2VBOUJ[BUJPOt$713s
    

    View Slide

46. 3FTJEVBM2VBOUJ[FE5SBOTGPSNFS 325SBOTGPSNFS
    
    
    %-FF $,JN 4,JN .$IP BOE8)BOt"VUPSFHSFTTJWF*NBHF(FOFSBUJPOVTJOH3FTJEVBM2VBOUJ[BUJPOt$713s
    

    View Slide

47. )JFSBSDIJDBM2VBOUJ[FE7"& )27"&
    
    
    5:V 4,JN $,JN %-FF BOE#)BOt-PDBMMZ)JFSBSDIJDBM"VUPSFHSFTTJWF.PEFMJOHGPS*NBHF(FOFSBUJPOt/FVS*14s
    

    View Slide

48. )JFSBSDIJDBM2VBOUJ[FE7"& )27"&
    
    
    5:V 4,JN $,JN %-FF BOE#)BOt-PDBMMZ)JFSBSDIJDBM"VUPSFHSFTTJWF.PEFMJOHGPS*NBHF(FOFSBUJPOt/FVS*14s
    

    View Slide

49. )JFSBSDIJDBM2VBOUJ[FE7"& )27"&
    
    
    5:V 4,JN $,JN %-FF BOE#)BOt-PDBMMZ)JFSBSDIJDBM"VUPSFHSFTTJWF.PEFMJOHGPS*NBHF(FOFSBUJPOt/FVS*14s
    

    View Slide

50. $POUFYUVBM325SBOTGPSNFS
    
    %-FF $,JN 4,JN .$IP BOE8)BOt%SBGUBOE3FWJTF&GGFDUJWF*NBHF(FOFSBUJPOXJUI$POUFYUVBM325SBOTGPSNFSt/FVS*14s
    

    View Slide

51. $POUFYUVBM325SBOTGPSNFS
    
    %-FF $,JN 4,JN .$IP BOE8)BOt%SBGUBOE3FWJTF&GGFDUJWF*NBHF(FOFSBUJPOXJUI$POUFYUVBM325SBOTGPSNFSt/FVS*14s
    

    View Slide

52. $POUFYUVBM325SBOTGPSNFS
    
    %-FF $,JN 4,JN .$IP BOE8)BOt%SBGUBOE3FWJTF&GGFDUJWF*NBHF(FOFSBUJPOXJUI$POUFYUVBM325SBOTGPSNFSt/FVS*14s
    

    View Slide

53. $POUFYUVBM325SBOTGPSNFS
    
    %-FF $,JN 4,JN .$IP BOE8)BOt%SBGUBOE3FWJTF&GGFDUJWF*NBHF(FOFSBUJPOXJUI$POUFYUVBM325SBOTGPSNFSt/FVS*14s
    

    View Slide

54. ߊ಴ܳ ݃஖ݶࢲ
    

    View Slide

55. ଵҊ ޙ೴
    
    +)P "+BJO BOE1"CCFFMt%FOPJTJOH%JGGVTJPO1SPCBCJMJTUJD.PEFMTu /FVS*14s
    
    "3BNFTI 1%IBSJXBM "/JDIPM $$IV BOE.$IFOu)JFSBSDIJDBM5FYU$POEJUJPOBM*NBHF(FOFSBUJPOXJUI$-*1-BUFOUTuBS9JWW
    
    "0PSE 07JOZBMT BOE,,BWVLDVPHMVt/FVSBM%JTDSFUF3FQSFTFOUBUJPO-FBSOJOHu/FVS*14s
    
    %-FF $,JN 4,JN .$IP BOE8)BOt"VUPSFHSFTTJWF*NBHF(FOFSBUJPOVTJOH3FTJEVBM2VBOUJ[BUJPOt$713s
    
    5:V 4,JN $,JN %-FF BOE#)BOt-PDBMMZ)JFSBSDIJDBM"VUPSFHSFTTJWF.PEFMJOHGPS*NBHF(FOFSBUJPOt/FVS*14s
    
    %-FF $,JN 4,JN .$IP BOE8)BOt%SBGUBOE3FWJTF&GGFDUJWF*NBHF(FOFSBUJPOXJUI$POUFYUVBM325SBOTGPSNFSt/FVS*14s
    

    View Slide

56. &0%
    

    View Slide