Upgrade to Pro — share decks privately, control downloads, hide ads and more …

LLM에서 배우는 이미지 생성 모델 ZERO부터 학습하기

LLM에서 배우는 이미지 생성 모델 ZERO부터 학습하기

3개월 전부터 Diffusion 모델을 zero부터 학습하는 프로젝트를 시작했습니다.

매 실험마다 천만원 이상의 GPU를 쓰는 경험을 몇 년만에 했기 때문에, 큰 스케일의 학습 과정이 활발하게 공유되고 있는 LLM 논문을 많이 참고하며 실험을 해 왔습니다.

LLM에 많은 관심이 쏠린 지금, Large-scale diffusion model 학습은

- 시각적이기 때문에 global scale이 용이하고
- 시장에 충분한 기회가 있으나 관심이 적고
- 큰 모델 학습에 관련된 경험이 거의 없기 때문에

그 과정에서 수많은 엔지니어링 문제를 푸는 것이 도전적이고 즐거운 것 같습니다!

저희와 함께 Domain-specific 지식으로 전문가가 쓸 수 있는 fine-grained 이미지 생성 모델을 만들고 싶으신 분은 언제든 편하게 연락주세요!

SHIFT UP AI Labs: https://bit.ly/shiftup-ai

* 이미지 생성 모델에 대한 Discussion이나 저희 팀에 관심이 있는 분이 계신 곳이라면 언제든 가서 Talk을 할 의향이 있으니 편하게 연락주세요!

Avatar for carpedm20

carpedm20

March 29, 2024

More Decks by carpedm20

Other Decks in Programming

Transcript

  1. ӣకള ೧ఊ गಌ ஹೊ౴ ೟ࠗ ઔস റ 0QFO"* ੑࢎ ূ૑פয

    ݺਵ۽ Ӗ۽ߥ ݅ݺ ਬ੷ܳ ݽ਷ ହস 4)*'561 carpedm20
  2. ӣకള carpedm20 ௿ۄ੉߁ 🧗 ࢟۞٘ 🥗 ؋റ ѐߊ ೠ׮Ҋ ١ೠद

    ೮؍ ࣁ࢚ী ؀೧ ঌҊ रযࢲ ࢜۽਍ ݅թ੉ۆ ݽ੐ ֙ р ೧ழ ೞ਋झীࢲ ૑ղݶࢲ ଼ ੍Ҋ Ӗ ॳҊ ࢎۈ ݅աҊ ֙ ੉࢚ ਬ੄޷೮؍ ৈݺҗ੄ ؀ചܳ ӝ۾
  3. $POUFOU  അসীࢲ וՙח ੉޷૑ ࢤࢿ ݽ؛੄ ೠ҅  --.

    ೟णীࢲ ߓ਋ӝ  #%JGGVTJPO.PEFM೟णਸ ೞݴ ߓ਍ ੼
  4.  ৔য੄ য۰਑  ৮ࢿب ੓ח Ӓܿ੉ ੜ աয়૑ ঋח׮

     ౠ੿ झఋੌ۽ ੜ ӒܻҊ र׮  э਷ நܼఠ ঴ҷ ௏झౝ ੄ ׮ܲ ੗ࣁ Ӓܻӝ  ੜ Ӓܻ૑ ޅೞҊ ݈۽ ಴അೞӝ য۰਍ Ѫٜ • ঈࣁࢲܻ • ׮নೠ ੄࢚ ੤૕ • ߓ҃ ހझఠ • ࡄ੄ ߑೱ ҳب • ޛ୓ р ҙ҅ ୨ਸ ١ী ݒҊ ੓Ѣա নࣚਵ۽ ੟Ҋ ੓ח
  5. п ࢚ടী ݏѱ -P3"ܳ ٜ݅Ҋ $POUSPM/FUܳ ࢚ട ߹۽ ଺ח ߑߨূ

    ೠ҅о ੓׮ ঈࣁࢲܻ झఋੌ ׮নೠ ੄࢚ ੤૕ ߓ҃ ހझఠ ࡄ੄ ߑೱ ҳب ޛ୓ р ҙ҅
  6. OpenAIী ੓؍ ஘ҳо ա৬ࢲ द੘ೠ ೐۽ં౟ ࢎप࢚ पಁ೮૑݅ Compute੉ ࠗ઒ೠ

    ౱ীѱ ݆਷ ߓ਎ ੼ਸ ҕਬೣ *ؘ੉ఠী ցޖ ૘઺ਸ ೞ૑ ঋও਺
  7. • ؘ੉ఠо ݆ই૓ ݅ఀ ݽ؛ ௼ӝب ழઉঠ ୭ࢶ੄ ࢿמਸ ࠁ੢ೡ

    ࣻ ੓׮ • $PNQVUF੉ ઁೠػ ࢚ടীࢲ ୭੸੄ ݽ؛ ߂ ؘ੉ఠ ௼ӝܳ ҅࢑ೡ ࣻ ੓׮ ؊ ݆਷ ؘ੉ఠ۽ જ਷ ࢿמਸ ղח ੘਷ ݽ؛ਸ ೟णೣ
  8. • ؘ੉ఠо ݆ই૓ ݅ఀ ݽ؛ ௼ӝب ழઉঠ ୭ࢶ੄ ࢿמਸ ࠁ੢ೡ

    ࣻ ੓׮ • $PNQVUF੉ ઁೠػ ࢚ടীࢲ ୭੸੄ ݽ؛ ߂ ؘ੉ఠ ௼ӝܳ ৘ஏೡ ࣻ ੓׮
  9. • 110۽ ࢎۈٜ੉ ؊ ࢶഐೞח ݽ؛ਸ ೟ण • ࠺Үо ࢤࢿࠁ׮

    ܻࣗझо ੸ѱ ٜӝ ٸޙ • 3FXBSE NPEFMਸ ೟ण೧ঠ ೞա %10ܳ ॳݶ ࠂ੟بо ઴য ٝ MFTTGMFYJCMF
  10. • જ਷ ௬ܻ౭੄ 4ZOUIFUJDؘ੉ఠח 8FC$SBXM ࠁ׮ ഻ঁ ੸਷ নਵ۽ જ਷

    ݽ؛ਸ ೟ण दఆ ࣻ ੓׮ • EBZTPO"(16T ୌ݅ਗ ੉ೞ ChatGPT & GPT-4
  11. LLM োҳীࢲ ଵҊೡ Ѫ -FBSOJOHSBUF TDIFEVMF ݽ؛ ߓ஖ ࢎ੉ૉ উ੿੸ੋ

    ݽ؛ ೟णਸ ਤ೧ ࠊঠೞח NFUSJD $POUJOVBM -FBSOJOH ۄ૑ ݽ؛ ೟ण द ߊࢤೞח ߡӒ
  12. LLM োҳীࢲ ଵҊೡ Ѫ -FBSOJOHSBUF TDIFEVMF ݽ؛ ߓ஖ ࢎ੉ૉ উ੿੸ੋ

    ݽ؛ ೟णਸ ਤ೧ ࠊঠೞח NFUSJD $POUJOVBM -FBSOJOH ۄ૑ ݽ؛ ೟ण द ߊࢤೞח ߡӒ જ਷ 4ZOUIFUJD ؘ੉ఠ ݅٘ח ߨ ݽ؛ ٣੗ੋ
  13. ੉ ৻ীب Gemini: A Family of Highly Capable Multimodal Models

    5SBJOFEPOMBSHFDPMMFDUJPOPG 8FCEPDVNFOUT CPPLT BOEDPEF JNBHF BVEJP BOEWJEFP 2VBMJUZGJMUFSJOHGPSBMMEBUBTFUT )FVSJTUJDT NPEFMCBTFEDMBTTJGJFST 'JOBMEBUBNJYUVSFTXFJHIUTEFUFSNJOFEUISPVHIBCMBUJPOPOTNBMMFSNPEFMT *ODSFBTFEXFJHIUPGEPNBJOSFMFWBOUEBUB UPXBSETUIFFOEPGUSBJOJOH
  14. ੉ ৻ীب ؘ੉ఠо ઁೠػ ࢚ടীࢶ • ؘ੉ఠܳ ߈ࠂ > ݽ؛

    ௼ӝ ¬ • ௏٘ܳ ୶о > ӝઓ ؘ੉ఠ ߈ࠂ • 1FSQMFYJUZ ೙ఠ݂ > %FEVQMJDBUJPO Scaling Data-Constrained Language Models
  15. • striking red hair • horned black headpiece • piercing

    blue eyes • crimson lipstick • sleek bodysuit • black leather belts • silver buckles • gunmetal gray rifle • imposing stance • flowing red cape • knee-high footwear • dark background • powerful posture • detailed armor • black fingerless gloves • red accents • light skin tone • animated style • sharp facial features • glossy lip shine • aggressive expression • asymmetrical design • thigh strap holster • mechanical arm pieces • crimson thigh-highs • bright red scarf • scarlet cape interior • high-heeled boots • compact shoulder pads • reflective surfaces
  16. • striking red hair • horned black headpiece • piercing

    blue eyes • crimson lipstick • sleek bodysuit • black leather belts • silver buckles • gunmetal gray rifle • imposing stance • flowing red cape • knee-high footwear • dark background • powerful posture • detailed armor • black fingerless gloves • red accents • light skin tone • animated style • sharp facial features • glossy lip shine • aggressive expression • asymmetrical design • thigh strap holster • mechanical arm pieces • crimson thigh-highs • bright red scarf • scarlet cape interior • high-heeled boots • compact shoulder pads • reflective surfaces
  17. • striking red hair • horned black headpiece • piercing

    blue eyes • crimson lipstick • sleek bodysuit • black leather belts • silver buckles • gunmetal gray rifle • imposing stance • flowing red cape • knee-high footwear • dark background • powerful posture • detailed armor • black fingerless gloves • red accents • light skin tone • animated style • sharp facial features • glossy lip shine • aggressive expression • asymmetrical design • thigh strap holster • mechanical arm pieces • crimson thigh-highs • bright red scarf • scarlet cape interior • high-heeled boots • compact shoulder pads • reflective surfaces
  18. • striking red hair • horned black headpiece • piercing

    blue eyes • crimson lipstick • sleek bodysuit • black leather belts • silver buckles • gunmetal gray rifle • imposing stance • flowing red cape • knee-high footwear • dark background • powerful posture • detailed armor • black fingerless gloves • red accents • light skin tone • animated style • sharp facial features • glossy lip shine • aggressive expression • asymmetrical design • thigh strap holster • mechanical arm pieces • crimson thigh-highs • bright red scarf • scarlet cape interior • high-heeled boots • compact shoulder pads • reflective surfaces
  19. DALL·E 3  ੉޷૑ ஭࣌ ࢿמ੉ ݒ਋ ઺ਃ  ੄

    ૣ਷ ஭࣌ ੄ ٣పੌೠ ஭࣌ਵ۽ ೟ण  5ఫझ౟ ੋ௏؊ ࢎਊ  (15۽ ࢎਊ੗੄ ૣ਷ ೐܁೐౟ܳ VQTBNQMFೣ Improving Image Generation with Better Captions
  20. • White naval cap • Cerulean blue hair • Military-inspired

    uniform • Golden epaulettes • Stern gaze • Light blue eyes
  21. • White naval cap • Cerulean blue hair • Military-inspired

    uniform • Golden epaulettes • Stern gaze • Light blue eyes
  22. • White naval cap • Cerulean blue hair • Military-inspired

    uniform • Golden epaulettes • Stern gaze • Light blue eyes
  23. GPT4-Vision ShareGPT4V: Improving Large Multi-Modal Models with Better Captions You

    are a powerful image captioner. Create detailed captions describing the contents of the given image. Include the object types and colors, counting the objects, object actions, precise object locations, texts, doublechecking relative positions between objects, etc. Instead of describing the imaginary content, only describing the content one can determine confidently from the image. Do not describe the contents by itemizing them in list form. Minimize aesthetic descriptions as much as possible.
  24. GPT4-Vision ShareGPT4V: Improving Large Multi-Modal Models with Better Captions You

    are a powerful image captioner. Create detailed captions describing the contents of the given image. Include the object types and colors, counting the objects, object actions, precise object locations, texts, doublechecking relative positions between objects, etc. Instead of describing the imaginary content, only describing the content one can determine confidently from the image. Do not describe the contents by itemizing them in list form. Minimize aesthetic descriptions as much as possible. • Cerulean blue hair • Military-inspired uniform • Naval officer's cap • Golden epaulettes • Stern gaze • Light blue eyes • White thigh-highs • Naval insignia pins • Anchored naval theme • Golden sword handle • High-waisted shorts • Blue-striped white jacket • Nautical skirt flap • Battleship turret platforms • Metallic heel decorations • Flowing hair motion • Immaculate white gloves • Pearly skin tone • Soft facial features • White naval cap • Silver belt buckle • Delicate lace trim • Floating hair ribbon • Detailed jacket buttons • Crisp ironed seams • Intricate gold braiding • Platform shoe design • Elegant standing pose • Sword sheath hanging • Glossy shoe finish • Contrasting shadows • Softly lit contours • Pastel color palette • Graceful hand placement • Ship deck floor • Transparent background • High visual fidelity • Stylized anime character • Minimalist backdrop • Polished artistic rendering <image, caption> dataset
  25. Orca: Progressive Learning from Complex Explanation Traces of GPT-4 You

    are a powerful image captioner. Create detailed captions describing the contents of the given image. Include the object types and colors, counting the objects, object actions, precise object locations, texts, doublechecking relative positions between objects, etc. Instead of describing the imaginary content, only describing the content one can determine confidently from the image. Do not describe the contents by itemizing them in list form. Minimize aesthetic descriptions as much as possible. ৈ۞ ѐ੄ system instruction
  26. Orca: Progressive Learning from Complex Explanation Traces of GPT-4 You

    are a powerful image captioner. Create detailed captions describing the contents of the given image. Include the object types and colors, counting the objects, object actions, precise object locations, texts, doublechecking relative positions between objects, etc. Instead of describing the imaginary content, only describing the content one can determine confidently from the image. Do not describe the contents by itemizing them in list form. Minimize aesthetic descriptions as much as possible. ৈ۞ ѐ੄ system instruction ݽ؛੉ যڃ о੉٘ۄੋ ഋधҗ హਵ۽ ׹ਸ ೡ Ѫੋоܳ ੿ೣ
  27. Orca: Progressive Learning from Complex Explanation Traces of GPT-4 You

    are a powerful image captioner. Create detailed captions describing the contents of the given image. Include the object types and colors, counting the objects, object actions, precise object locations, texts, doublechecking relative positions between objects, etc. Instead of describing the imaginary content, only describing the content one can determine confidently from the image. Do not describe the contents by itemizing them in list form. Minimize aesthetic descriptions as much as possible. ৈ۞ ѐ੄ system instruction ݽ؛੉ যڃ о੉٘ۄੋ ഋधҗ హਵ۽ ׹ਸ ೡ Ѫੋоܳ ੿ೣ Instruction Tuning <User query, LFM response> Explanation Tuning <System message, User query, LFM response> ׮নࢿ ױ҅߹ ୶ۿਸ ਬب
  28. Orca: Progressive Learning from Complex Explanation Traces of GPT-4 You

    are a powerful image captioner. Create detailed captions describing the contents of the given image. Include the object types and colors, counting the objects, object actions, precise object locations, texts, doublechecking relative positions between objects, etc. Instead of describing the imaginary content, only describing the content one can determine confidently from the image. Do not describe the contents by itemizing them in list form. Minimize aesthetic descriptions as much as possible. ৈ۞ ѐ੄ system instruction ݽ؛੉ যڃ о੉٘ۄੋ ഋधҗ హਵ۽ ׹ਸ ೡ Ѫੋоܳ ੿ೣ
  29. Orca: Progressive Learning from Complex Explanation Traces of GPT-4 You

    are a powerful image captioner. Create detailed captions describing the contents of the given image. Include the object types and colors, counting the objects, object actions, precise object locations, texts, doublechecking relative positions between objects, etc. Instead of describing the imaginary content, only describing the content one can determine confidently from the image. Do not describe the contents by itemizing them in list form. Minimize aesthetic descriptions as much as possible. ਗೞח ஭࣌ਸ ਬبೡ ࣻ ੓ח User instruction
  30. Orca: Progressive Learning from Complex Explanation Traces of GPT-4 You

    are a powerful image captioner. Create detailed captions describing the contents of the given image. Include the object types and colors, counting the objects, object actions, precise object locations, texts, doublechecking relative positions between objects, etc. Instead of describing the imaginary content, only describing the content one can determine confidently from the image. Do not describe the contents by itemizing them in list form. Minimize aesthetic descriptions as much as possible. ਗೞח ஭࣌ਸ ਬبೡ ࣻ ੓ח User instruction • Cerulean blue hair • Military-inspired uniform • Naval officer's cap • Golden epaulettes • Stern gaze • Light blue eyes • White thigh-highs • Naval insignia pins • Anchored naval theme • Golden sword handle • High-waisted shorts • Blue-striped white jacket • Nautical skirt flap • Battleship turret platforms • Metallic heel decorations • Flowing hair motion • Immaculate white gloves • Pearly skin tone • Soft facial features • White naval cap
  31. GPT4-Vision Orca: Progressive Learning from Complex Explanation Traces of GPT-4

    Model Name Prompt Cost (USD) Completion Cost (USD) Max Prompt Tokens gpt-4-vision-preview $0.00001000 $0.00003000 128000 gemini-pro-vision $0.00000020 $0.00000050 30720 Gemini-Vision : 1 : 2 1 : 4 1 : 6 =
  32. ArtBook-50M Dataset Dataset Volume Caption Misc ArtBook-50M-2D 30M - ArtBook-50M-3D

    25M - ArtBook-50M-AI 5M - This is not JourneyDB ArtBook-GPT 4M GPT-4V 12 System Instruction ArtBook-Gem 24M Gemini 12 System Instruction ArtBook-50M 50M Raw
  33. ArtBook-50M Dataset Dataset Volume Caption Misc ArtBook-50M-2D 30M - ArtBook-50M-3D

    25M - ArtBook-50M-AI 5M - This is not JourneyDB ArtBook-GPT 4M GPT-4V 12 System Instruction ArtBook-Gem 24M Gemini 12 System Instruction ArtBook-50M 50M Raw
  34. %BUBQSFQSPDFTTJOH  0$3۽ Ӗ੗ ੓ח ࢎ૓ ઁѢ  $BQUJPO UPLFOӡ੉о

    ੉ೞੋ ੉޷૑ ઁѢ  *NBHFTJ[F BTQFDUSBUJP ೙ఠ݂  ౠ੿ ஶࣆ੉ ನೣػ ੉޷૑ ઁѢ  %VQMJDBUJPO SFNPWBMVTJOH.JMWVT  "FTUIFUJDTDPSF
  35. %BUBQSFQSPDFTTJOH  0$3۽ Ӗ੗ ੓ח ࢎ૓ ઁѢ  $BQUJPO UPLFOӡ੉о

    ੉ೞੋ ੉޷૑ ઁѢ  *NBHFTJ[F BTQFDUSBUJP ೙ఠ݂  ౠ੿ ஶࣆ੉ ನೣػ ੉޷૑ ઁѢ  %VQMJDBUJPO SFNPWBMVTJOH.JMWVT  "FTUIFUJDTDPSF
  36. ӝࠄ੸ੋ )೟ण ࢿמ ѐࢶ • $6%"7FSTJPO • .JYFE1SFDJTJPO • 1Z5PSDI

    UPSDIDPNQJMF • 'VMMZ4IBSEFE%BUB1BSBMMFM • ೟ण ؘ੉ఠী ݏח $VTUPN5PLFOJ[FS • 1Z5PSDI 1SPGJMFS /TJHIU$PNQVUF
  37. 5SBJOJOH4QFFE&SSPS from torch.profiler import profile, record_function, ProfilerActivity with profile( activities=[ProfilerActivity.CPU,

    ProfilerActivity.CUDA], record_shapes=True ) as prof: # 문제가 있던 코드 optimizer.step() prof.export_chrome_trace("trace.json")
  38. 7BMJEBUFUIFNPEFMQFSGPSNBODF • 7BMJEBUJPO-PTT X&." • '*% • ೟ण ؘ੉ఠ੄ EJTUSJCVUJPOҗ

    ਬࢎೞա MFBLBHF ݄ӝ • $-*14DPSF • 6TJOHGPT-4 (or LLaVA)BTKVEHF
  39. 7BMJEBUFUIFNPEFMQFSGPSNBODF • 7BMJEBUJPO-PTT • '*% • ೟ण ؘ੉ఠ੄ EJTUSJCVUJPOҗ ਬࢎೞա

    MFBLBHF ݄ӝ • $-*14DPSF • 6TJOHGPT-4 (or LLaVA) BTKVEHF • LLMীࢲ ݆੉ ॳ੉ח ߑध PIXART-Σ:Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image GeneraMon
  40. GPT4-Vision As an AI visual assistant, you are analyzing two

    specific images. Given a specific caption, you need to judge which image aligns with the caption more closely. Please pay attention to the key information, including object identities, properties, spatial relationships, object numbers and image style, etc. The caption for the two images is: {caption} Please respond me strictly in the following format: <the first image is better> or <the second image is better> or <The two images are tied.>. The reason is <give your reason here>.
  41. GPT4-Vision As an AI visual assistant, you are analyzing two

    specific images. Given a specific caption, you need to judge which image aligns with the caption more closely. Please pay attention to the key information, including object identities, properties, spatial relationships, object numbers and image style, etc. The caption for the two images is: {caption} Please respond me strictly in the following format: <the first image is better> or <the second image is better> or <The two images are tied.>. The reason is <give your reason here>.
  42. GPT4-Vision As an AI visual assistant, you are analyzing two

    specific images. Given a specific caption, you need to judge which image aligns with the caption more closely. Please pay attention to the key information, including object identities, properties, spatial relationships, object numbers and image style, etc. The caption for the two images is: {caption} Please respond me strictly in the following format: <the first image is better> or <the second image is better> or <The two images are tied.>. The reason is <give your reason here>. <the first image is better>
  43. ֎

  44. 회사 소개 : 개 요 회사개요 기업명 대표 설립일 게임

    임직원 홈페이지 사업영역 소재지 시프트업 김형태 2013년 12월 2일 데스티니차일드 / 승리의 여신 : 니케 스텔라 블레이드 290여명 shiftup.co.kr 모바일게임 개발 및 서비스 / 콘솔 게임 개 발 서울시 강남구 서초동
  45. 4)*'561"*-BCT ௏झೖ ࢚੢दز Ѥ द೐౟সjӝসо஖ ઑਗী ॕܻח ׀ 창업 10년

    만에 코스피 상장을 추진하는 게임사 시프트업이 ‘제2의 크래프톤’이 될 수 있을지 업계의 관심이 쏠리고 있다. 시프트업 '니케', 누적 매출 1조원 돌파… 시프트업의 서브컬처 게임 ‘승리의 여신:니케’가 누적 매출 1조 원을 넘겼다. 2022년 11월 출시된 지 약 1년 4개월 만이다.
  46. 프로젝트 : 스텔라 블레이드 미국 경제지 포브스 “플레이스테이션 쇼케이스 작품

    중 매우 놀라운 신작”으로 소개 회사소개 프로젝트 게임산업 시프트업채용 소니의 적극적인 구애로 국내 게임사 최초로 PS5 독점 3N도 도전하지 못했던 트리플 A급 액션 게임 2024 플레이스테이션 최고의 기대작 1위 출시일 발표 후 예약 구매 1위
  47. 시프트업의 성장 전략 NEXT <승리의 여신 : 니케> 글로벌 점유율

    확대 + 국내 최초의 AAA급 콘솔 IP <스텔라 블레이드> 개발 + 끊임 없는 신규 IP 제작 Project 글로벌 장기 흥행작과 AAA게임 IP를 보유한 게 임사 회사소개 프로젝트 게임산업 시프트업채용
  48. જ਷ ֤ޙ ॳӝо ݾ಴ ࢑೟ োҳ ӝഥ KAIST 인공지능 대학원

    KAIST에서 전액 출자하여 기업에서 필요한 연구과제를 수행하기 위한 CCC회원사 전용 프로그램 포항공과대학교 산학협력단 OpenAI에서의 GPT-3, 강화 학습, BlockSparse 연구 경험