AI application systems realized by HyperCLOVA, a large-scale general-purpose language model for Japanese and its challenges

日本語の大規模汎用言語モデル「HyperCLOVA」が実現できるAI応用システムとその課題
佐藤 敏紀（LINE株式会社 AI開発室 NLP開発チーム マネージャー）

CCSE2021での発表資料です
https://ccse.jp/2021/

Transcript

1. ೔ຊޠͷେن໛൚༻ݴޠϞσϧ 
 ʮHyperCLOVAʯ͕࣮ݱͰ͖Δ 
 AIԠ༻γεςϜͱͦͷ՝୊ Toshinori Sato (@overlast) LINE / AI Company 
2. ࠤ౻ හل (@overlast) • γχΞιϑτ΢ΣΞΤϯδχΞ / Ϛωʔδϟʔ • ࣗવݴޠॲཧ • ৘ใݕࡧ • LINE CLOVA • ೔ຊޠͷࣗવݴޠཧղ(NLU)γεςϜ • HyperCLOVA • ೔ຊޠ൛ͷ։ൃ੹೚ऀ • ೔ຊޠίʔύε / AIϑΟϧλʔ / Ԡ༻ٕज़ͷ։ൃɾධՁ • OSS: NEologd ϓϩδΣΫτͷϝΠϯ։ൃऀ • mecab-ipadic-NEologd 2
3. Attention, please ! ࠷ॳʹ͓఻͍͑ͨ͠·͢ɻ - ͜ͷηογϣϯ͸20෼ͱ΍΍ۦ͚଍Ͱ͢ͷͰɺ͢͜͠ूதྗ͕ඞཁͰ͢ ! - Ͳ͏ͯ͠΋ࣗવݴޠॲཧͷઐ໳༻ޠ͕ଟ͘ͳΓ·͕͢ɺ෼͔Βͳ͍༻ޠ͸ 
 Θ͔Γқ͘ղઆͰ͖·͢ͷͰɺTwitterͰ @overlast ʹmention͍ͯͩ͘͠͞ - େن໛൚༻ݴޠϞσϧ͕ ”ະདྷΛͲ͏ม͑Δ͔” ༧૝͠ͳ͕Β͓ฉ͖͍ͩ͘͞
5. ໰୊: ਓؒͱHyperCLOVA͕νϟοτΛ͍ͯ͠·͢
6. פ͍Ͱ͢Ͷʔɻ
 ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ Ͳ͕ͬͪ HyperCLOVA ͔෼͔Γ·͔͢?
7. פ͍Ͱ͢Ͷʔɻ
 ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ
 ۝भͷग़਎ͳͷͰژ౎ͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ Ͳ͕ͬͪ HyperCLOVA ͔෼͔Γ·͔͢?
8. פ͍Ͱ͢Ͷʔɻ
 ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ
 ۝भͷग़਎ͳͷͰژ౎ͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ෱ԬݝͰ͚ͨͬ͠?
 തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ Ͳ͕ͬͪ HyperCLOVA ͔෼͔Γ·͔͢?
9. פ͍Ͱ͢Ͷʔɻ
 ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ
 ۝भͷग़਎ͳͷͰژ౎ͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ෱ԬݝͰ͚ͨͬ͠?
 തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ෱Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़਎ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ޷͖Ͱ͔͢ʁ Ͳ͕ͬͪ HyperCLOVA ͔෼͔Γ·͔͢?
10. פ͍Ͱ͢Ͷʔɻ
 ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ
 ۝भͷग़਎ͳͷͰژ౎ͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ෱ԬݝͰ͚ͨͬ͠?
 തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ΋ͪΖΜͰ͢!
 ຊ৔ͷಲࠎϥʔϝϯ৯΂ͯΈ͍ͨͰ͢Ͷɻ ෱Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़਎ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ޷͖Ͱ͔͢ʁ Ͳ͕ͬͪ HyperCLOVA ͔෼͔Γ·͔͢?
11. פ͍Ͱ͢Ͷʔɻ
 ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ
 ۝भͷग़਎ͳͷͰژ౎ͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ෱ԬݝͰ͚ͨͬ͠?
 തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ΋ͪΖΜͰ͢!
 ຊ৔ͷಲࠎϥʔϝϯ৯΂ͯΈ͍ͨͰ͢Ͷɻ ෱Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़਎ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ޷͖Ͱ͔͢ʁ ͍͍Ͱ͢ΑͶɻ
 ͜ͷפ͍࣌ظ͸ಛʹ͍͍Ͱ͢Αɻ Ͳ͕ͬͪ HyperCLOVA ͔෼͔Γ·͔͢?
12. פ͍Ͱ͢Ͷʔɻ
 ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ
 ۝भͷग़਎ͳͷͰژ౎ͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ෱ԬݝͰ͚ͨͬ͠?
 തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ΋ͪΖΜͰ͢!
 ຊ৔ͷಲࠎϥʔϝϯ৯΂ͯΈ͍ͨͰ͢Ͷɻ ෱Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़਎ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ޷͖Ͱ͔͢ʁ ͍͍Ͱ͢ΑͶɻ
 ͜ͷפ͍࣌ظ͸ಛʹ͍͍Ͱ͢Αɻ ͬͪ͜Ͱ͢!ࠇ͍࿮Ͱғͬͨํ͕HyperCLOVAͰ͢
13. େن໛൚༻ݴޠϞσϧ + α ͳγεςϜ Automatic evaluation with 39B JP Model for a QA task
14. HyperCLOVAͷΞʔΩςΫνϟ Eco System Infra Model Data 
15. HyperCLOVA ΛԠ༻ͨ͠γεςϜʹΑΔग़ྗͷڧΈ ਓؒͷ༷ʹ׈Β͔ͳ 
 ςΩετΛੜ੒Ͱ͖Δ ಺༰ͷධՁ ೚ҙͷτϐοΫʹ 
 ௥ैͰ͖Δ ඼࣭ͷධՁ ৽͍͠࿩୊Λ 
 ఏڙͰ͖Δ ද૚తͳධՁ
16. פ͍Ͱ͢Ͷʔɻ
 ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ
 ۝भͷग़਎ͳͷͰژ౎ͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ෱ԬݝͰ͚ͨͬ͠?
 തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ΋ͪΖΜͰ͢!
 ຊ৔ͷಲࠎϥʔϝϯ৯΂ͯΈ͍ͨͰ͢Ͷɻ ෱Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़਎ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ޷͖Ͱ͔͢ʁ ͍͍Ͱ͢ΑͶɻ
 ͜ͷפ͍࣌ظ͸ಛʹ͍͍Ͱ͢Αɻ ͬͪ͜Ͱ͢!ࠇ͍࿮Ͱғͬͨํ͕HyperCLOVAͰ͢
17. פ͍Ͱ͢Ͷʔɻ
 ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ
 ۝भͷग़਎ͳͷͰژ౎ͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ෱ԬݝͰ͚ͨͬ͠?
 തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ΋ͪΖΜͰ͢!
 ຊ৔ͷಲࠎϥʔϝϯ৯΂ͯΈ͍ͨͰ͢Ͷɻ ෱Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़਎ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ޷͖Ͱ͔͢ʁ ͍͍Ͱ͢ΑͶɻ
 ͜ͷפ͍࣌ظ͸ಛʹ͍͍Ͱ͢Αɻ ͬͪ͜Ͱ͢!ࠇ͍࿮Ͱғͬͨํ͕HyperCLOVAͰ͢
18. פ͍Ͱ͢Ͷʔɻ
 ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ
 ۝भͷग़਎ͳͷͰژ౎ͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ෱ԬݝͰ͚ͨͬ͠?
 തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ΋ͪΖΜͰ͢!
 ຊ৔ͷಲࠎϥʔϝϯ৯΂ͯΈ͍ͨͰ͢Ͷɻ ෱Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़਎ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ޷͖Ͱ͔͢ʁ ͍͍Ͱ͢ΑͶɻ
 ͜ͷפ͍࣌ظ͸ಛʹ͍͍Ͱ͢Αɻ ͬͪ͜Ͱ͢!ࠇ͍࿮Ͱғͬͨํ͕HyperCLOVAͰ͢ ਓؒͷ༷ʹ׈Β͔ͳςΩετΛੜ੒Ͱ͖͍ͯ·͢Ͷ !
19. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ - ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
20. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ - ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
21. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ - ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
23. Title 80pt - ༧ଌ͍ͨ͠ςΩετͷτʔΫϯ਺NͱɺίʔύεΛଌఆͯ͠ಘͨط஌ͷτʔΫϯྻछ਺Vͷͱ͖ - ςΩετͷ֬཰෼෍ΛٻΊΔܭࢉ࣌ʹอ࣋͢Δύϥϝλ਺͕VͷN৐ʹͳΔ => ίεύ͕ѱ͍ ֬཰తݴޠϞσϧͷ࣍ݩͷढ͍ χϡʔϥϧݴޠϞσϧ - LSTM΍Transformer͕جʹͳ͍ͬͯͯѥछ͕ࢁఔ͋ΔɻBERT΍HyperCLOVA͸Transformerͷ೿ੜ - ಛ௃ϕΫτϧͷ࣍ݩ਺N(e.g. GPT3͸12288) x ޠኮ਺VͷߦྻΛอ࣋͢ΔͷͰVͷnഒ => ίεύ͕ྑ͍ - ֬཰తݴޠϞσϧͷ༷ʹ֬཰෼෍ͷ௿͍ܥྻͷ౷ܭ஋Λཅʹ࣋ͨͳͯ͘ࡁΉ ֬཰తݴޠϞσϧ ݴޠϞσϧ͸֬཰తͱχϡʔϥϧͷ2λΠϓʹ෼͔ΕΔ - ྫ: ୯ޠn-gramϞσϧ = ίʔύεதͷ୯ޠn-gramग़ݱ֬཰ʹج͖ͮɺ೚ҙͷςΩετͷ֬཰෼෍ΛਪఆͰ͖Δ - n-gram = ܥྻΛ୯Ґ௕ͷτʔΫϯྻʹ۠੾Δ͜ͱͰಘΒΕΔɺ͢΂ͯͷ௕͞nͷτʔΫϯྻ - ྫ: ݴޠϞσϧͷจࣈ1-gram: ݴ/ޠ/Ϟ/σ/ϧɺ ݴޠϞσϧͷจࣈ2-gram: ݴޠ/ޠϞ/Ϟσ/σϧ
24. Title 80pt - ༧ଌ͍ͨ͠ςΩετͷτʔΫϯ਺NͱɺίʔύεΛଌఆͯ͠ಘͨط஌ͷτʔΫϯྻछ਺Vͷͱ͖ - ςΩετͷ֬཰෼෍ΛٻΊΔܭࢉ࣌ʹอ࣋͢Δύϥϝλ਺͕VͷN৐ʹͳΔ => ίεύ͕ѱ͍ ֬཰తݴޠϞσϧͷ࣍ݩͷढ͍ χϡʔϥϧݴޠϞσϧ - LSTM΍Transformer͕جʹͳ͍ͬͯͯѥछ͕ࢁఔ͋ΔɻBERT΍HyperCLOVA͸Transformerͷ೿ੜ - ಛ௃ϕΫτϧͷ࣍ݩ਺N(e.g. GPT3͸12288) x ޠኮ਺VͷߦྻΛอ࣋͢ΔͷͰVͷnഒ => ίεύ͕ྑ͍ - ֬཰తݴޠϞσϧͷ༷ʹ֬཰෼෍ͷ௿͍ܥྻͷ౷ܭ஋Λཅʹ࣋ͨͳͯ͘ࡁΉ ֬཰తݴޠϞσϧ HyperCLOVA͸χϡʔϥϧݴޠϞσϧΛ಺แ - ྫ: ୯ޠn-gramϞσϧ = ίʔύεதͷ୯ޠn-gramग़ݱ֬཰ʹج͖ͮɺ೚ҙͷςΩετͷ֬཰෼෍ΛਪఆͰ͖Δ - n-gram = ܥྻΛ୯Ґ௕ͷτʔΫϯྻʹ۠੾Δ͜ͱͰಘΒΕΔɺ͢΂ͯͷ௕͞nͷτʔΫϯྻ - ྫ: ݴޠϞσϧͷจࣈ1-gram: ݴ/ޠ/Ϟ/σ/ϧɺ ݴޠϞσϧͷจࣈ2-gram: ݴޠ/ޠϞ/Ϟσ/σϧ
25. NLPͷٕज़τϨϯυͷมભ Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2 
 (2019) GPT-3 
 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) MT-NLG (2021)
26. Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2 
 (2019) GPT-3 
 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) ʻBreak Through 1ʼ Attentionٕज़͕ൃలɺ ֶश͕ߴ଎Խ MT-NLG (2021) NLPͷٕज़τϨϯυͷมભ
27. Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2 
 (2019) GPT-3 
 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) ʻBreak Through 1ʼ Attentionٕज़͕ൃలɺ ֶश͕ߴ଎Խ ʻBreak Through 2ʼ Generative pre-trainingʹΑΔ ϥϕϧ෇σʔλͷେྔੜ੒ MT-NLG (2021) NLPͷٕज़τϨϯυͷมભ
28. Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2 
 (2019) GPT-3 
 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) ʻBreak Through 1ʼ Attentionٕज़͕ൃలɺ ֶश͕ߴ଎Խ ʻBreak Through 2ʼ Generative pre-trainingʹΑΔ ϥϕϧ෇σʔλͷେྔੜ੒ ʻBreak Through 3ʼ Prompting : ̍ͭͷݴޠϞσϧͰ 
 ༷ʑͳλεΫΛॲཧͰ͖ΔՄೳੑ MT-NLG (2021) NLPͷٕज़τϨϯυͷมભ
29. 3. Prompting ֶश ֶश Ϟσϧ ֶश σʔλ ਪ࿦ σʔλ ਪଌ σʔλ ֶश ࣄલֶश Ϟσϧ ֶश σʔλ ਪ࿦ σʔλ ਪଌ σʔλ ֶश ಛԽܕֶश Ϟσϧ λεΫ ಛԽֶश σʔλ ֶश େن໛൚༻ ݴޠϞσϧ ֶश σʔλ ਪ࿦ ਪଌ σʔλ Prompt ≠Program ैདྷ: ֶशϞσϧΛλεΫʹಛԽͤ͞Δ͜ͱͰੑೳΛ֬อ͖ͯͨ͠ ࠓޙ: ࠶ֶश΍ɺϓϩάϥϜͷ࣮૷Λͤͣʹ ໨తλεΫͷਪ࿦Λ࣮ࢪͰ͖Δ
30. ࣄલֶशࡁΈେن໛൚༻ݴޠϞσϧͷ޿ൣͳ׆༻ 
31. ࣄલֶशࡁΈେن໛൚༻ݴޠϞσϧͷ޿ൣͳ׆༻ 
32. ͻͱͭͷେن໛൚༻ݴޠϞσϧͰɺଟ༷ͳλεΫΛղ͚ΔՄೳੑ •ର࿩γεςϜ •จষཁ໿ •આ໌จੜ੒ •຋༁ ɾɾɾ •ܭࢉ •ϓϩάϥϜੜ੒ Prompt(ϓϩϯϓτ)ͷ༩͑ํ࣍ୈͰ ༷ʑͳλεΫ͕࣮ݱͰ͖ΔՄೳੑ͕ Prompting ʹΑΔϞσϧͷ࠶ར༻ͷଅਐ
33. Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2 
 (2019) GPT-3 
 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) ʻBreak Through 1ʼ Attentionٕज़͕ൃలɺ ֶश͕ߴ଎Խ ʻBreak Through 2ʼ Generative pre-trainingʹΑΔ ϥϕϧ෇σʔλͷେྔੜ੒ ʻBreak Through 3ʼ Prompting : ̍ͭͷݴޠϞσϧͰ 
 ༷ʑͳλεΫΛॲཧͰ͖ΔՄೳੑ ʻBreak Through 4ʼ Scaling Lawͷൃݟ ੑೳ͸ύϥϝʔλ਺ɺσʔληοτ MT-NLG (2021) NLPͷٕज़τϨϯυͷมભ
34. 4. Scaling Law • ύϥϝʔλ਺Λ૿΍͢ͱϞσϧͷੑೳ͸૿͑Δ • ͜ͷݶք஋͕ݱঢ়ݟ͔͍ͭͬͯͳ͍ େن໛൚༻ݴޠϞσϧͷ ։ൃڝ૪͕ܹԽ
35. ڝ߹ଞ͕ࣾެද͍ͯ͠ΔϞσϧͷαΠζ OpenAI Google GPT: Generative Pre-trained Transformer T5: Text-To-Text Transfer Transformer MT-NLG: Megatron-Turing Natural Language Generation GPT-3(2020)ɿ1.3B, 2.7B, 6.7B, 13B, 175B GPT-2(2019)ɿ117M, 345M, 762M, 1.5B T5(2019)ɿ60M, 220M, 770M, 2.8B, 11B Microsoft MT-NLG(2021)ɿ530B
36. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ - ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
37. େن໛൚༻ݴޠϞσϧ + α ͳγεςϜ Automatic evaluation with 39B JP Model for a QA task
38. HyperCLOVAͷΞʔΩςΫνϟ Eco System Infra Model Data 
39. HyperCLOVA ͷ೔ຊޠϞσϧߏஙͷݱঢ় 1.3B → 6.7B → 13B → 39B 13B → 39B 82B 204B ʙ (2022೥த) ଟݴޠϞσϧ େن໛Ϟσϧ 
 ೔ຊޠ / ଟݴޠ ௒େن໛ 
 ೔ຊޠϞσϧ ೔ຊޠϞσϧ ߏங࡞ۀ͕ਐߦத 
40. HyperCLOVAͷϞσϧߏஙͷख๏ EMNLP 2021 ͷNAVERͷ࿦จ*΋͝ࢀর͍ͩ͘͞ https://arxiv.org/abs/2109.04650 ࠷௿ͷֶश཰Λ Megatron-LMͷඪ४஋ͷ 1/10ʹ͢Δඞཁ͕͋ͬͨ NVIDIA Superpod ্Ͱֶश 
 Superpod͸Ԇ΂1024ຕͷA100 Λࢗͨ͠128୆ͷDGXΫϥελ ֶशʹ࢖͏ίʔύεͷτʔΫ ϯ਺͸ɺߏங͢ΔϞσϧͷύ ϥϝλ਺ͷ3ഒҎ্͕๬·͍͠ ByteϨϕϧͷBPE tokenizerʹΑΔίʔύε ࣄྫͷτʔΫϯԽ Transformer Decoder architecture Λ࠾༻ Megatron-LM Λ࢖༻ (Shoeybi et al., 2019)  https://arxiv.org/abs/2109.04650 * What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers, Boseop Kim et.al, EMNLP 2021
41. HyperCLOVAͷϞσϧߏஙͷख๏ EMNLP 2021 ͷNAVERͷ࿦จ΋͝ࢀর͍ͩ͘͞ ࠷௿ͷֶश཰Λ Megatron-LMͷඪ४஋ͷ 1/10ʹ͢Δඞཁ͕͋ͬͨ NVIDIA Superpod ্Ͱֶश 
 Superpod͸Ԇ΂1024ຕͷA100 Λࢗͨ͠128୆ͷDGXΫϥελ ֶशʹ࢖͏ίʔύεͷτʔΫ ϯ਺͸ɺߏங͢ΔϞσϧͷύ ϥϝλ਺ͷ3ഒҎ্͕๬·͍͠ ByteϨϕϧͷBPE tokenizerʹΑΔίʔύε ࣄྫͷτʔΫϯԽ Transformer Decoder architecture Λ࠾༻ Megatron-LM Λ࢖༻ (Shoeybi et al., 2019) 
42. 
43. LINE LM Corpus ͷݱࡏͷঢ়گ For 82B JP Model αϯϓϧ 10B τʔΫϯ 500B σʔλαΠζ 1.8TB
44. ίʔύεͷͨΊͷσʔλऩूͷํ਑ LINEͷαʔϏεͷձ࿩σʔλΛ࢖Θͳ͍ - LINEͷ͢΂ͯͷϝοηʔδ
    - OpenChatͷ͢΂ͯͷ౤ߘ
    ݖརॲཧΛਖ਼͓͘͜͠ͳ͔ͬͯΒίʔύεʹ௥Ճ͢Δ  ൚༻ੑΛ޲্͢ΔίϯςϯπΛ
    - LINEͷ֎ʹίʔύεͷαϒηοτΛఏڙ͢ΔՄೳੑΛ࢒͢ 
45. LINE LM Corpus(for HyperCLOVA’s LMs) LINE ΍ LINE OpenChat ͷσʔλ͸ຊίʔύεʹ࠾࿥͠·ͤΜ - BERTͷϞσϧΛߏங͢ΔͨΊͷίʔύεͱͯ͠2019೥͔Β։ൃΛ։࢝ - LINE ݕࡧػೳͷͨΊʹΫϩʔϧ͞Εͨσʔλ΋ར༻ͨ͠ - "ඇެ։ͳݸਓ৘ใ"ͱͯ͠༰қʹநग़Ͱ͖ͨσʔλ͸আ֎͍ͯ͠Δ - ೔ຊޠදݱΛϞσϧʹؚΊΔ͏͑Ͱେ੾ͦ͏ͳαΠτΛඃ෴ͨ͠ - ͍͔ͭ͘ͷ֎෦ίϯςϯπΛߪೖͯ͠ɺݖར໰୊Λղܾͨ͠͏͑Ͱ࢖༻ ! 
46. LINE LM Corpus for 39Bͷ಺༁ 82B Ҏ্ͷϞσϧߏஙʹ͍ͭͯ͸ɺߋʹ޿ൣͳൣғ͔ΒίϯςϯπΛऩू͍ͯ͠·͢ ղઆ τʔΫϯ਺ Blog ೔ຊޠͷϒϩά 105.8B News ೔ຊޠͷχϡʔεهࣄ 12.4B Q&A ೔ຊޠͷ࣭໰Ԡ౴αΠτ 10.5B WikiJa/En/Ko Wikipedia ͷ dump data 4.7B Novel ೔ຊޠͷখઆαΠτ 1.0B Shopping ೔ຊޠͷγϣοϐϯάαΠτ 20.9B Others ͦͷଞ 0.1B Total 155.4B 
47. ςΩετੜ੒ ςΩετͷੜ੒తͳཁ໿ ର࿩γεςϜ
48. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ - ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
49. - ঎඼ύοέʔδʹ͸આ໌จ͕ॻ͔Ε͍ͯΔ͜ͱ͕ଟ͍͕ɺ͋ͷઆ໌จΛߟ ͑ͯॻ͍͍ͯΔํ͕֤ࣾʹ͍Βͬ͠ΌΔ - ͜ͷσϞͰ͸ɺCLOVA Studio(Play-ground) ͷύϥϝλΛௐ੔ͯ͠ɺಈతʹ આ໌จΛੜ੒͢ΔྲྀΕΛ͓ݟͤ͠·͢ - ঎඼ͷλΠτϧͱ੒෼ͷϦετ͔ΒɺCLOVA Studio͸ͦͷ঎඼Λએ఻͢Δ ͨΊͷઆ໌จΛੜ੒͠·͢ HyperCLOVAͷσϞ: ঎඼ͷ֓ཁͷઆ໌จΛੜ੒
50. Demo Movie 60sec
51. 1. ਓ͕ؒͰ͖Δॲཧʹม׵ (ෳ਺ͷ)ೖྗςΩετ (ෳ਺ͷ)ग़ྗςΩετ #### (ෳ਺ͷ)ೖྗςΩετ (ෳ਺ͷ)ग़ྗςΩετ #### ɾɾɾ ͷ༷ʹShotͷ܁Γฦ͠Λॻ͘ɻ ྫ֎తͳ৘ใ͸ͳΔ΂্͘ͷํʹຒΊࠐΉ 2. ॲཧͷݟຊΛோΊΔ ʮಓ୺Ͱ୭͔ʹϓϩϯϓτΛݟͤ ͨޙͰɺϓϩϯϓτʹؚ·Ε͍ͯ ͳ͍(ෳ਺ͷ)ೖྗςΩετΛݟͤ ͨΒɺ(ෳ਺ͷ)ग़ྗςΩετΛ࡞ ΕΔՄೳੑ͕͋Δ͔ʯΛߟ͑Δɻ 
 
 Shot͕଍Γͳ͍͜ͱ͕໰୊ͳΒɺ ͞Βʹ଍͢͜ͱͰରॲͰ͖Δ 3. ΫΤϦΛ௥Ճ ϓϩϯϓτͷ຤ඌͷ####ͷ࣍ͷߦʹ ΫΤϦͱͳΔ(ෳ਺ͷ)ೖྗςΩετΛ ೖྗͨ͠Βɺ(ෳ਺ͷ)ग़ྗςΩετͷ ࠷ॳͷه߸෦෼Λ௥هͯ͠ɺ࣮ߦͯ͠ ΈΔ ग़ྗ͕ඍົͳΒύϥϝλͷௐ੔Λ͢Δ ବ໨ͳΒ1΍2ʹ΋ͲΔ ΍ͬͯΈ͍ͨ͜ͱ to ϓϩϯϓτܕͷࢦྩॻ
52. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ - ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
53. HyperCLOVAΛ༻͍ͨର࿩γεςϜͷओ؍ධՁ with 6.7B/13B/39B JP Model for 4 tasks ͢΂ͯͷλεΫͱϞσϧͷ૊Έ߹ΘͤͰΞϊςʔγϣϯΛ࣮ࢪ ಉ͡5ਓͷΞϊςʔλʔʹΑΔओ؍ධՁ ֤ηογϣϯ͸Mԟ෮ͷձ࿩ϖΞ Ϣʔβʔ͸ධՁ༻ͷNݸͷτϐοΫͷϦετΛड͚औΔ ֤ηογϣϯͰ͸ɼϦετ͔Β1ͭͷޠኮΛফඅ͢Δ Play-groundͰ࣮ࢪ 4. ࡶஊΛ͢Δ 3. τϐοΫ΁ͷϢʔβײ৘΁ͷରԠ 2. ҟͳΔτϐοΫ΁ͷભҠʹରԠ 1. جૅతͳޠኮͷཧղ 
54. ྫ֎: ࡶஊͷλεΫ͸ΰʔϧͷୡ੒౓ΛධՁ͍ͯ͠ͳ͍(ෆཁ͔ͩΒ) ԁ׈ͳԠ౴ Qɿࣗવͳ൓ԠͰ͔ͨ͠ʁ ձ࿩ͷܦҢʹ୼ͼ΍ໃ६͸͋Γ·ͤΜ͔ʁ τϐοΫ΁ͷ௥ै Q: Did it stay on topic? τϐοΫΛݟࣦ͍ͬͯͳ͔͔ͬͨʢ͜ͷ৔߹ɺԿʹ͍ͭͯฉ͔Ε͍ͯΔͷ͔Λݟࣦ͍ͬͯͳ͍͔ʁ) ࿩୊ͷ੾Γସ͕͑Ͱ͖͔ͨʢ͜ͷ৔߹ɺલͷ࣭໰ʹҾ͖໭͔ͤͨʁ) τϐοΫͷఏڙ΍ 
 ࣭໰ͷ౤͔͚͛ Qɿ࿩୊Λఏڙ͔ͨ͠ʁ ճ౴தʹൃݴऀͷ࿩ΛҾ͖ग़͢͜ͱ͕Ͱ͖͔ͨʢͰ͖ͳ͍Մೳੑ͕ߴ͍ʣ ໨ඪͷୡ੒ Qɿ໨త͸ୡ੒Ͱ͖·͔ͨ͠ʁ શλεΫʹڞ௨ͳධՁ߲໨ ! 
55. 1. جૅతͳޠኮͷཧղ ॳ౳ޠኮ த౳ޠኮ খֶߍ ۚͮͪ தֶߍ Ԗච େਓ νϡʔϦοϓ ઌੜ ώϚϫϦ ϥΠΦϯ ص ΩϦϯ Ҝࢠ ిं ۺ ं αϯμϧ ηʔλʔ ΓΜ͝ εΧʔτ Έ͔Μ Ωϟϕπ αϯϚ ͖Ύ͏Γ Ϛάϩ εζϝ ϋʔϞχΧ Πϯί ϐΞϊ τϯϘ ΞϦ IUUQTSFQPTJUPSZOJOKBMBDKQ BDUJPOSFQPTJUPSZ@BDUJPO@DPNNPO@EPXOMPBEJUFN@JEJUFN@OPBUUSJCVUF@JE fi MF@OP ToDO: ֤ޠኮʹରͯ͠ɺҙຯ(Ϩϕϧ1)ͱ૝ى͢Δײ৘(Ϩϕϧ2)ʹ͍࣭ͭͯ໰Λ͢Δ 
56. Topic A Topic B Topic A Topic B ৽ܕίϩφ΢Πϧε Πϯό΢ϯυ Πνϩʔ େ୩ᠳฏ ۓٸࣄଶએݴ ৽ܕίϩφϫΫνϯ AR(֦ுݱ࣮) ࣗಈӡసٕज़ YouTuber VTuber ϨΦφϧυɾμɾϰΟϯν ΫϩʔυɾϞω ฏ੒ ྩ࿨ Πϯλʔωοτ 5G σϑϨܦࡁ ௒ߴྸԽࣾձ ւ֎ཱྀߦ ࠃ಺ཱྀߦ ిؾࣗಈं ϦχΞதԝ৽װઢ 2. ҟͳΔෳ਺ͷτϐοΫΛτϥοΩϯά ToDO: τϐοΫAͰձ࿩Λ࢝Ίɺ10ԟ෮͢ΔલʹτϐοΫBʹ੾Γସ͑Δ ฏ੒ 
57. 3. ͋ΔτϐοΫʹର͢ΔϢʔβʔͷײ৘ʹ൓Ԡ Topic sentiment A sentiment B ৽ܕίϩφ΢Πϧε ؤுΖ͏ ෆ҆ͩ Πϯό΢ϯυ ໭ͬͯ͘Δ ໭Βͳ͍ ৽ܕίϩφϫΫνϯ ଴ͱ͏ ͍ͭʹͳΔ YouTuber ΍Γ͍ͨ ΍Γͨ͘ͳ͍ େ୩ᠳฏ ׆༂ͯ͠ཉ͍͠ ࡾৼͯ͠ཉ͍͠ AR(֦ுݱ࣮) ໘ന͍ ๞͖ͨ ௒ߴྸԽࣾձ େৎ෉ ৺഑ ւ֎ཱྀߦ ߦ͖͍ͨ ߦ͖ͨ͘ͳ͍ ిؾࣗಈं ৐Γ͍ͨ ৐Γͨ͘ͳ͍ ϦχΞதԝ৽װઢ ৐Γ͍ͨ ৐Γͨ͘ͳ͍ ToDOɿTopicʹ͍ͭͯ15ԟ෮ͷձ࿩Λ͢Δɻ࠷ॳͷ15ԟ෮͸ײ৘Aɺ࣍ʹײ৘Bͷؾ࣋ͪͰ࿩͢ 
58. 4.ࡶஊΛ͢Δ 
59. 39B JP Modelͷओ؍ධՁͷ݁Ռͷཁ໿ 1. جૅతͳޠኮͷཧղ 2. ҟͳΔτϐοΫ΁ ͷભҠʹରԠ 3. τϐοΫ΁ͷલ ޲͖ͳײ৘΁ͷରԠ 3. τϐοΫ΁ͷޙΖ ޲͖ͳײ৘΁ͷରԠ 4. ࡶஊΛ͢Δ ԁ׈ͳԠ౴ 0.978(356/364) 0.908(1749/1926) 0.908(1198/1320) 0.872(1072/1229) 0.925(86/93) τϐοΫ΁ͷ௥ै 0.984(358/364) 0.952(1834/1926) 0.951(1255/1320) 0.93(1144/1229) 0.935(87/93) τϐοΫͷఏڙ΍ 
 ࣭໰ͷ౤͔͚͛ 0.003(1/364) 0.023(44/1926) 0.033(44/1320) 0.035(43/1229) 0.086(8/93) ໨ඪͷୡ੒ 0.835(304/364) 0.907(1746/1926) 0.899(1187/1320) 0.505(621/1229) - 
60. 39B JP Modelͷओ؍ධՁͷ݁Ռͷཁ໿ 1. جૅతͳޠኮͷཧղ 2. ҟͳΔτϐοΫ΁ ͷભҠʹରԠ 3. τϐοΫ΁ͷલ ޲͖ͳײ৘΁ͷରԠ 3. τϐοΫ΁ͷޙΖ ޲͖ͳײ৘΁ͷରԠ 4. ࡶஊΛ͢Δ ԁ׈ͳԠ౴ 0.978(356/364) 0.908(1749/1926) 0.908(1198/1320) 0.872(1072/1229) 0.925(86/93) τϐοΫ΁ͷ௥ै 0.984(358/364) 0.952(1834/1926) 0.951(1255/1320) 0.93(1144/1229) 0.935(87/93) τϐοΫͷఏڙ΍ 
 ࣭໰ͷ౤͔͚͛ 0.003(1/364) 0.023(44/1926) 0.033(44/1320) 0.035(43/1229) 0.086(8/93) ໨ඪͷୡ੒ 0.835(304/364) 0.907(1746/1926) 0.899(1187/1320) 0.505(621/1229) - 
61. ΞϓϦέʔγϣϯ: HyperCLOVA Friends ೚ҙͷௐ੔ՄೳͳΩϟϥΫλʔͱHyperCLOVAͰձ࿩͢Δ
62. Demo Movie 60sec
63. Application example: HyperCLOVA Friends Talk with any adjustable character using HyperCLOVA HyperCLOVA͸͋Δఔ౓ϩʔϧϓϨΠͰ͖Δ
64. HyperCLOVA͸൚༻తͳϩʔϧϓϨΠ͕Մೳ Challenge: εϜʔζͳձ࿩ͱτϐοΫͷ௥੻Ҏ֎ͷػೳ ൃݴͷਅِΛ֬ೝ͔ͯ͠ Βճ౴͢Δ͜ͱ͕ඞཁ ձ࿩͕εϜʔζͰɺݴͬͨ ͜ͱͷҙຯ͕ཧղͰ͖Δ ͍͔ͭ͘ͷᐆດͳճ౴ ྫɿચ୕࣌ͷ͓౬ͷԹ౓ σʔλͷภΓʹΑΔӨڹ ྫɿະઃఆ͕ͩͬͨঁੑ ʹͳͬͨ ϖϧιφͷҰ؏ੑ͸ 
 গٙ͠Θ͍͠ จࣈͷηοτͳ͠ʹ 
 ελʔτͰ͖Δ
65. LINEͷର࿩γεςϜ͕޷੒੷Λ࢒͠·ͨ͠ ਓ޻஌ೳֶձ ݴޠɾԻ੠ཧղͱର࿩ॲཧݚڀձ SIG-SLUD ୈ12ճର࿩γεςϜγϯϙδ΢Ϝ ର࿩γεςϜϥΠϒίϯϖςΟγϣϯ̐ https://dialog-system-live-competition.github.io/dslc4/
66. ର࿩γεςϜγϯϙδ΢Ϝʹؔ͢ΔϒϩάهࣄΛग़͠·ͨ͠ https://blog.clova.line.me/hyperclova-202112
67. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ - ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
68. Eco System Infra Model Data 39B JP model ͷࣗಈධՁ 
69. 39B JPϞσϧʹΑΔQAλεΫͷࣗಈධՁ Few-shot͸ɺਪ࿦͝ͱʹͷൃηοτ͔ΒϥϯμϜʹจষΛநग़ͯ͠࡞੒͠·ͨ͠ ίϯςΩετɺ࣭໰จɺճ౴ΛؚΉFew-shotͷ࡞੒ 
    ਪ࿦݁Ռͷதʹਖ਼ղؚ͕·Ε͍ͯͯɺ༰қʹநग़Ͱ͖Δ৔߹͸ਖ਼ղͱ൑அ͢Δ TASK: RCQA* possible only - ௨ৗͷ3$2"λεΫͷσʔληοτ͔Βճ౴ෆೳͷ໰୊Λ࡟আ * ղ౴Մೳੑ෇͖ಡղσʔληοτ: http://www.cl.ecei.tohoku.ac.jp/rcqa/ 
70. RCQA possible only task ͷͨΊͷ 
 39B JPϞσϧʹΑΔࣗಈධՁ݁Ռ model / few-shot shot temperature top_p answer match 6.7B / contextual 0 0.5 0.8 - 4 0.1 0.9 66.52 13B / contextual 0 0.5 0.8 - 4 0.4 0.1 70.28 39B / contextual 0 0.4 0.5 80.51 1 0.4 0.5 89.18 2 0.4 0.5 89.31 3 0.4 0.5 89.09 4 0.4 0.5 89.83 39B / non-contextual 0 0.4 0.5 69.50 1 0.4 0.5 76.97 2 0.4 0.5 79.08 3 0.4 0.5 79.38 4 0.4 0.5 80.51 
71. HyperCLOVA’s LM vs BERT-large TASK: RCQA possible only (௨ৗͷRCQAλεΫ͔Β౴͑ΒΕͳ͍࣭໰Λ࡟আ͠·ͨ͠) - ਅʹࠔ೉ͳ໰୊ΛHyperCLOVAͰղܾͰ͖ͳ͍Մೳੑ͕͋Γ·͢ - normal => possible only: train: 43610 => 21091, dev: 6863 => 3524, test: 6178 => 3079 - BERT͸ɺಛఆͷλεΫʹରͯ͠ඍௐ੔͢Δ͜ͱͰɺΑΓߴ͍݁ՌΛಘΒΕΔՄೳੑ͕͋Δ - HyperCLOVA͸PromptingͱϥϑͳύϥϝʔλݕࡧͰBERTͱಉ౳ͷύϑΥʔϚϯεΛ࣮ݱͰ͖Δ test acc test F1 memo HyperCLOVA 85.03 89.95 JP 39B 2-shots, 
 temperature=0.4, top_p= 0.5 BERT-jp-large 86.68 90.49 Using subset of LINE LM corpus 
72. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ - ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
73. ೔ຊޠͷ೉͠͞ शಘͷ೉͠͞ ೔ຊޠͷ࿩ऀ͸ - ͻΒ͕ͳ - ยԾ໊ - ׽ࣈ - ϩʔϚࣈ ͳͲͳͲΛ ҰͭͷจॻΛॻͨ͘Ίʹ࢖͏ େྔͷඞਢޠኮ ೔ৗձ࿩ʹඞཁͳޠኮ਺ - 8,000ޠҎ্ ଟ͘ͷ୯ޠΛ஌Δඞཁ͕͋Δ - ಉԻҟٛޠ - ܟޠ - ํݴͳͲ ୯ޠͷলུ ೔ຊޠ࿩ऀ͸ɺจॻதͷҎԼ ͷ୯ޠΛলུ͢Δ͜ͱ͕͋Δ - ओޠ - ໨తޠ লུ͞Εͨ୯ޠ͸Ұҙʹਪ࿦ Ͱ͖ͳ͍Մೳੑ͕͋Δ 
74. ςΩετੜ੒ͷ೉͠͞ ੜ੒͞ΕͨςΩετͷ 
 જࡏతͳϦεΫ ҎԼͷΑ͏ͳٕज़Λ։ൃ ͢Δඞཁ͕͋Γ·͢ɻ - ίʔύεͷ಺༰ͷภΓ΍ දهํ๏ͷվળ - ग़ྗ͞ΕͨςΩετͷਅ ࣮ੑͱ҆શੑͷ֬อ AIྙཧͷ࣮ફ ೖग़ྗςΩετʹ͸༷ʑͳྙཧత ഑ྀ͕ඞཁͰ͢ - ಟੑ - ੑతͳ΋ͷ - ߈ܸతͳ΋ͷ - ๯ಚత - ڴഭత - ΞΠσϯςΟςΟ΁ͷ߈ܸ ຊ࣭తͳධՁͷࣗಈԽ ಈతςΩετੜ੒ͷ݁Ռʹద ༻Ͱ͖ΔϝτϦΫε͕ඞཁ - ࿩୊ੑͷ͋Δίϯςϯπͷਫ਼౓ - ੜ੒͞ΕͨςΩετͷҰ؏ੑ - ໨తͷୡ੒౓ͷ൑ఆ 
75. NLPʹHyperCLOVA͸ຊ౰ʹඞཁͳͷ͔ʁ - YES !!  ΋͠΋༧ࢉࣥߦ͕ڐ͞ΕΔͳΒ
    ʜ
    - NLPͷྺ࢙͸AIؔ࿈ٕज़ͷൃలͱڧ݁͘ͼ͍͍ͭͯΔ - LINE͸ࣗ෼ͨͪͰϞσϧΛ࡞͓ͬͯ٬༷ʹ࢖ͬͯ΋Β͏ ํ޲ʹਐΈ͍ͨͱߟ͍͑ͯ·͢ Large-scale general-purpose LMs DNN Traditional only ML Rule only Small LM only
76. LINEͷݴޠϞσϧΛOSSͱͯ͠ຊ೥౓தʹ഑෍։࢝ ΋ͪΖΜHyperCLOVAҎ֎Ͱ͢ ੑೳ໨ඪ: LINEͷݴޠϞσϧ for OSS > other OSS ͳݴޠϞσϧ ՄೳͰ͋Ε͹ɺෳ਺೥ؒ͸ఆظతʹߋ৽͍ͨ͠ͱߟ͍͑ͯ·͢ HyperCLOVAʹ࢖͍ͬͯΔίʔύε(LINE LM Corpus)ͷαϒηοτΛ࢖༻͠·͢ !
77. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ - ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
78. ·ͱΊ ݴޠϞσϧͱ͸Կ͔ ɹɹֶशͨ͠ίʔύε಺ʹ͓͚Δ೚ҙͷςΩετͷ֬཰෼෍ ݴޠϞσϧ͸Կ͕Ͱ͖Δͷ͔ ɹɹ༩͑ΒΕͨจ຺ʹैͬͯΑΓྑ͍ςΩετΛੜ੒Ͱ͖Δ ݴޠϞσϧ͸ԿΛม͑Δͷ͔ ɹɹAI͕೔ຊޠΛ࢖͏͋ΒΏΔ࡞ۀΛखॿ͚͢ΔଘࡏʹͳΓ 
 ɹɹਓʑͷ࣌ؒΛઅ໿͢Δ͜ͱͰɺਓʑͷੜ׆Λ๛͔ʹ͍ͯ͘͠ )ZQFS$-07"
79. ೥౓ͷ৽ଔ࠾༻͕Φʔϓϯͯ͠·͢ͷͰɺ͝Ԡื͍ͩ͘͞