Share
Embed
AI application systems realized by HyperCLOVA, a large-scale general-purpose language model for Japanese and its challenges
日本語の大規模汎用言語モデル「HyperCLOVA」が実現できるAI応用システムとその課題
佐藤 敏紀(LINE株式会社 AI開発室 NLP開発チーム マネージャー)
CCSE2021での発表資料です
https://ccse.jp/2021/
LINE Developers
December 17, 2021
Transcript
79 slides
-
ࠤ౻ හل (@overlast) • γχΞιϑτΣΞΤϯδχΞ / Ϛωʔδϟʔ • ࣗવݴޠॲཧ •
ใݕࡧ • LINE CLOVA • ຊޠͷࣗવݴޠཧղ(NLU)γεςϜ • HyperCLOVA • ຊޠ൛ͷ։ൃऀ • ຊޠίʔύε / AIϑΟϧλʔ / Ԡ༻ٕज़ͷ։ൃɾධՁ • OSS: NEologd ϓϩδΣΫτͷϝΠϯ։ൃऀ • mecab-ipadic-NEologd 2 -
Attention, please ! ࠷ॳʹ͓͍͑ͨ͠·͢ɻ - ͜ͷηογϣϯ20ͱۦ͚Ͱ͢ͷͰɺ͢͜͠ूதྗ͕ඞཁͰ͢ ! - Ͳ͏ͯࣗ͠વݴޠॲཧͷઐ༻ޠ͕ଟ͘ͳΓ·͕͢ɺ͔Βͳ͍༻ޠ
Θ͔Γқ͘ղઆͰ͖·͢ͷͰɺTwitterͰ @overlast ʹmention͍ͯͩ͘͠͞ - େن൚༻ݴޠϞσϧ͕ ”ະདྷΛͲ͏ม͑Δ͔” ༧͠ͳ͕Β͓ฉ͖͍ͩ͘͞ -
פ͍Ͱ͢Ͷʔɻ ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ भͷग़ͳͷͰژͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ԬݝͰ͚ͨͬ͠? തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ͪΖΜͰ͢! ຊͷಲࠎϥʔϝϯ৯ͯΈ͍ͨͰ͢Ͷɻ Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ͖Ͱ͔͢ʁ
͍͍Ͱ͢ΑͶɻ ͜ͷפ͍࣌ظಛʹ͍͍Ͱ͢Αɻ Ͳ͕ͬͪ HyperCLOVA ͔͔Γ·͔͢? -
פ͍Ͱ͢Ͷʔɻ ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ भͷग़ͳͷͰژͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ԬݝͰ͚ͨͬ͠? തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ͪΖΜͰ͢! ຊͷಲࠎϥʔϝϯ৯ͯΈ͍ͨͰ͢Ͷɻ Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ͖Ͱ͔͢ʁ
͍͍Ͱ͢ΑͶɻ ͜ͷפ͍࣌ظಛʹ͍͍Ͱ͢Αɻ ͬͪ͜Ͱ͢!ࠇ͍Ͱғͬͨํ͕HyperCLOVAͰ͢ -
HyperCLOVA ΛԠ༻ͨ͠γεςϜʹΑΔग़ྗͷڧΈ ਓؒͷ༷ʹΒ͔ͳ ςΩετΛੜͰ͖Δ ༰ͷධՁ ҙͷτϐοΫʹ ैͰ͖Δ ࣭ͷධՁ
৽͍͠Λ ఏڙͰ͖Δ දతͳධՁ -
פ͍Ͱ͢Ͷʔɻ ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ भͷग़ͳͷͰژͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ԬݝͰ͚ͨͬ͠? തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ͪΖΜͰ͢! ຊͷಲࠎϥʔϝϯ৯ͯΈ͍ͨͰ͢Ͷɻ Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ͖Ͱ͔͢ʁ
͍͍Ͱ͢ΑͶɻ ͜ͷפ͍࣌ظಛʹ͍͍Ͱ͢Αɻ ͬͪ͜Ͱ͢!ࠇ͍Ͱғͬͨํ͕HyperCLOVAͰ͢ -
פ͍Ͱ͢Ͷʔɻ ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ भͷग़ͳͷͰژͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ԬݝͰ͚ͨͬ͠? തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ͪΖΜͰ͢! ຊͷಲࠎϥʔϝϯ৯ͯΈ͍ͨͰ͢Ͷɻ Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ͖Ͱ͔͢ʁ
͍͍Ͱ͢ΑͶɻ ͜ͷפ͍࣌ظಛʹ͍͍Ͱ͢Αɻ ͬͪ͜Ͱ͢!ࠇ͍Ͱғͬͨํ͕HyperCLOVAͰ͢ -
פ͍Ͱ͢Ͷʔɻ ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ भͷग़ͳͷͰژͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ԬݝͰ͚ͨͬ͠? തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ͪΖΜͰ͢! ຊͷಲࠎϥʔϝϯ৯ͯΈ͍ͨͰ͢Ͷɻ Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ͖Ͱ͔͢ʁ
͍͍Ͱ͢ΑͶɻ ͜ͷפ͍࣌ظಛʹ͍͍Ͱ͢Αɻ ͬͪ͜Ͱ͢!ࠇ͍Ͱғͬͨํ͕HyperCLOVAͰ͢ ਓؒͷ༷ʹΒ͔ͳςΩετΛੜͰ͖͍ͯ·͢Ͷ ! -
Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - આ໌จੜγεςϜͷԠ༻ -
ରγεςϜͷԠ༻ - BERTͱͷൺֱ - ࣭ԠλεΫͰ - - ߴ࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝ - ·ͱΊ -
Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - આ໌จੜγεςϜͷԠ༻ -
ରγεςϜͷԠ༻ - BERTͱͷൺֱ - ࣭ԠλεΫͰ - - ߴ࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝ - ·ͱΊ -
Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - આ໌จੜγεςϜͷԠ༻ -
ରγεςϜͷԠ༻ - BERTͱͷൺֱ - ࣭ԠλεΫͰ - - ߴ࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝ - ·ͱΊ -
Title 80pt - ༧ଌ͍ͨ͠ςΩετͷτʔΫϯNͱɺίʔύεΛଌఆͯ͠ಘͨطͷτʔΫϯྻछVͷͱ͖ - ςΩετͷ֬ΛٻΊΔܭࢉ࣌ʹอ࣋͢Δύϥϝλ͕VͷNʹͳΔ => ίεύ͕ѱ͍ ֬తݴޠϞσϧͷ࣍ݩͷढ͍ χϡʔϥϧݴޠϞσϧ
- LSTMTransformer͕جʹͳ͍ͬͯͯѥछ͕ࢁఔ͋ΔɻBERTHyperCLOVATransformerͷੜ - ಛϕΫτϧͷ࣍ݩN(e.g. GPT312288) x ޠኮVͷߦྻΛอ࣋͢ΔͷͰVͷnഒ => ίεύ͕ྑ͍ - ֬తݴޠϞσϧͷ༷ʹ֬ͷ͍ܥྻͷ౷ܭΛཅʹ࣋ͨͳͯ͘ࡁΉ ֬తݴޠϞσϧ ݴޠϞσϧ֬తͱχϡʔϥϧͷ2λΠϓʹ͔ΕΔ - ྫ: ୯ޠn-gramϞσϧ = ίʔύεதͷ୯ޠn-gramग़ݱ֬ʹج͖ͮɺҙͷςΩετͷ֬ΛਪఆͰ͖Δ - n-gram = ܥྻΛ୯ҐͷτʔΫϯྻʹ۠Δ͜ͱͰಘΒΕΔɺͯ͢ͷ͞nͷτʔΫϯྻ - ྫ: ݴޠϞσϧͷจࣈ1-gram: ݴ/ޠ/Ϟ/σ/ϧɺ ݴޠϞσϧͷจࣈ2-gram: ݴޠ/ޠϞ/Ϟσ/σϧ -
Title 80pt - ༧ଌ͍ͨ͠ςΩετͷτʔΫϯNͱɺίʔύεΛଌఆͯ͠ಘͨطͷτʔΫϯྻछVͷͱ͖ - ςΩετͷ֬ΛٻΊΔܭࢉ࣌ʹอ࣋͢Δύϥϝλ͕VͷNʹͳΔ => ίεύ͕ѱ͍ ֬తݴޠϞσϧͷ࣍ݩͷढ͍ χϡʔϥϧݴޠϞσϧ
- LSTMTransformer͕جʹͳ͍ͬͯͯѥछ͕ࢁఔ͋ΔɻBERTHyperCLOVATransformerͷੜ - ಛϕΫτϧͷ࣍ݩN(e.g. GPT312288) x ޠኮVͷߦྻΛอ࣋͢ΔͷͰVͷnഒ => ίεύ͕ྑ͍ - ֬తݴޠϞσϧͷ༷ʹ֬ͷ͍ܥྻͷ౷ܭΛཅʹ࣋ͨͳͯ͘ࡁΉ ֬తݴޠϞσϧ HyperCLOVAχϡʔϥϧݴޠϞσϧΛแ - ྫ: ୯ޠn-gramϞσϧ = ίʔύεதͷ୯ޠn-gramग़ݱ֬ʹج͖ͮɺҙͷςΩετͷ֬ΛਪఆͰ͖Δ - n-gram = ܥྻΛ୯ҐͷτʔΫϯྻʹ۠Δ͜ͱͰಘΒΕΔɺͯ͢ͷ͞nͷτʔΫϯྻ - ྫ: ݴޠϞσϧͷจࣈ1-gram: ݴ/ޠ/Ϟ/σ/ϧɺ ݴޠϞσϧͷจࣈ2-gram: ݴޠ/ޠϞ/Ϟσ/σϧ -
NLPͷٕज़τϨϯυͷมભ Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2
(2019) GPT-3 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) MT-NLG (2021) -
Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2
(2019) GPT-3 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) ʻBreak Through 1ʼ Attentionٕज़͕ൃలɺ ֶश͕ߴԽ MT-NLG (2021) NLPͷٕज़τϨϯυͷมભ -
Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2
(2019) GPT-3 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) ʻBreak Through 1ʼ Attentionٕज़͕ൃలɺ ֶश͕ߴԽ ʻBreak Through 2ʼ Generative pre-trainingʹΑΔ ϥϕϧσʔλͷେྔੜ MT-NLG (2021) NLPͷٕज़τϨϯυͷมભ -
Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2
(2019) GPT-3 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) ʻBreak Through 1ʼ Attentionٕज़͕ൃలɺ ֶश͕ߴԽ ʻBreak Through 2ʼ Generative pre-trainingʹΑΔ ϥϕϧσʔλͷେྔੜ ʻBreak Through 3ʼ Prompting : ̍ͭͷݴޠϞσϧͰ ༷ʑͳλεΫΛॲཧͰ͖ΔՄೳੑ MT-NLG (2021) NLPͷٕज़τϨϯυͷมભ -
3. Prompting ֶश ֶश Ϟσϧ ֶश σʔλ ਪ σʔλ ਪଌ
σʔλ ֶश ࣄલֶश Ϟσϧ ֶश σʔλ ਪ σʔλ ਪଌ σʔλ ֶश ಛԽܕֶश Ϟσϧ λεΫ ಛԽֶश σʔλ ֶश େن൚༻ ݴޠϞσϧ ֶश σʔλ ਪ ਪଌ σʔλ Prompt ≠Program ैདྷ: ֶशϞσϧΛλεΫʹಛԽͤ͞Δ͜ͱͰੑೳΛ֬อ͖ͯͨ͠ ࠓޙ: ࠶ֶशɺϓϩάϥϜͷ࣮Λͤͣʹ తλεΫͷਪΛ࣮ࢪͰ͖Δ -
Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2
(2019) GPT-3 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) ʻBreak Through 1ʼ Attentionٕज़͕ൃలɺ ֶश͕ߴԽ ʻBreak Through 2ʼ Generative pre-trainingʹΑΔ ϥϕϧσʔλͷେྔੜ ʻBreak Through 3ʼ Prompting : ̍ͭͷݴޠϞσϧͰ ༷ʑͳλεΫΛॲཧͰ͖ΔՄೳੑ ʻBreak Through 4ʼ Scaling Lawͷൃݟ ੑೳύϥϝʔλɺσʔληοτ MT-NLG (2021) NLPͷٕज़τϨϯυͷมભ -
ڝ߹ଞ͕ࣾެද͍ͯ͠ΔϞσϧͷαΠζ OpenAI Google GPT: Generative Pre-trained Transformer T5: Text-To-Text Transfer
Transformer MT-NLG: Megatron-Turing Natural Language Generation GPT-3(2020)ɿ1.3B, 2.7B, 6.7B, 13B, 175B GPT-2(2019)ɿ117M, 345M, 762M, 1.5B T5(2019)ɿ60M, 220M, 770M, 2.8B, 11B Microsoft MT-NLG(2021)ɿ530B -
Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - આ໌จੜγεςϜͷԠ༻ -
ରγεςϜͷԠ༻ - BERTͱͷൺֱ - ࣭ԠλεΫͰ - - ߴ࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝ - ·ͱΊ -
HyperCLOVA ͷຊޠϞσϧߏஙͷݱঢ় 1.3B → 6.7B → 13B → 39B 13B
→ 39B 82B 204B ʙ (2022த) ଟݴޠϞσϧ େنϞσϧ ຊޠ / ଟݴޠ େن ຊޠϞσϧ ຊޠϞσϧ ߏங࡞ۀ͕ਐߦத -
HyperCLOVAͷϞσϧߏஙͷख๏ EMNLP 2021 ͷNAVERͷจ*͝ࢀর͍ͩ͘͞ https://arxiv.org/abs/2109.04650 ࠷ͷֶशΛ Megatron-LMͷඪ४ͷ 1/10ʹ͢Δඞཁ͕͋ͬͨ NVIDIA Superpod
্Ͱֶश SuperpodԆ1024ຕͷA100 Λͨ͠128ͷDGXΫϥελ ֶशʹ͏ίʔύεͷτʔΫ ϯɺߏங͢ΔϞσϧͷύ ϥϝλͷ3ഒҎ্͕·͍͠ ByteϨϕϧͷBPE tokenizerʹΑΔίʔύε ࣄྫͷτʔΫϯԽ Transformer Decoder architecture Λ࠾༻ Megatron-LM Λ༻ (Shoeybi et al., 2019) https://arxiv.org/abs/2109.04650 * What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers, Boseop Kim et.al, EMNLP 2021 -
HyperCLOVAͷϞσϧߏஙͷख๏ EMNLP 2021 ͷNAVERͷจ͝ࢀর͍ͩ͘͞ ࠷ͷֶशΛ Megatron-LMͷඪ४ͷ 1/10ʹ͢Δඞཁ͕͋ͬͨ NVIDIA Superpod ্Ͱֶश
SuperpodԆ1024ຕͷA100 Λͨ͠128ͷDGXΫϥελ ֶशʹ͏ίʔύεͷτʔΫ ϯɺߏங͢ΔϞσϧͷύ ϥϝλͷ3ഒҎ্͕·͍͠ ByteϨϕϧͷBPE tokenizerʹΑΔίʔύε ࣄྫͷτʔΫϯԽ Transformer Decoder architecture Λ࠾༻ Megatron-LM Λ༻ (Shoeybi et al., 2019) -
LINE LM Corpus ͷݱࡏͷঢ়گ For 82B JP Model αϯϓϧ 10B
τʔΫϯ 500B σʔλαΠζ 1.8TB -
ίʔύεͷͨΊͷσʔλऩूͷํ LINEͷαʔϏεͷձσʔλΛΘͳ͍ - LINEͷͯ͢ͷϝοηʔδ - OpenChatͷͯ͢ͷߘ ݖརॲཧΛਖ਼͓͘͜͠ͳ͔ͬͯΒίʔύεʹՃ͢Δ ൚༻ੑΛ্͢ΔίϯςϯπΛ -
LINEͷ֎ʹίʔύεͷαϒηοτΛఏڙ͢ΔՄೳੑΛ͢ -
LINE LM Corpus(for HyperCLOVA’s LMs) LINE LINE OpenChat ͷσʔλຊίʔύεʹ࠾͠·ͤΜ
- BERTͷϞσϧΛߏங͢ΔͨΊͷίʔύεͱͯ͠2019͔Β։ൃΛ։࢝ - LINE ݕࡧػೳͷͨΊʹΫϩʔϧ͞Εͨσʔλར༻ͨ͠ - "ඇެ։ͳݸਓใ"ͱͯ͠༰қʹநग़Ͱ͖ͨσʔλআ֎͍ͯ͠Δ - ຊޠදݱΛϞσϧʹؚΊΔ͏͑Ͱେͦ͏ͳαΠτΛඃ෴ͨ͠ - ͍͔ͭ͘ͷ֎෦ίϯςϯπΛߪೖͯ͠ɺݖརΛղܾͨ͠͏͑Ͱ༻ ! -
LINE LM Corpus for 39Bͷ༁ 82B Ҏ্ͷϞσϧߏஙʹ͍ͭͯɺߋʹൣͳൣғ͔ΒίϯςϯπΛऩू͍ͯ͠·͢ ղઆ τʔΫϯ Blog
ຊޠͷϒϩά 105.8B News ຊޠͷχϡʔεهࣄ 12.4B Q&A ຊޠͷ࣭ԠαΠτ 10.5B WikiJa/En/Ko Wikipedia ͷ dump data 4.7B Novel ຊޠͷখઆαΠτ 1.0B Shopping ຊޠͷγϣοϐϯάαΠτ 20.9B Others ͦͷଞ 0.1B Total 155.4B -
Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - આ໌จੜγεςϜͷԠ༻ -
ରγεςϜͷԠ༻ - BERTͱͷൺֱ - ࣭ԠλεΫͰ - - ߴ࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝ - ·ͱΊ -
- ύοέʔδʹઆ໌จ͕ॻ͔Ε͍ͯΔ͜ͱ͕ଟ͍͕ɺ͋ͷઆ໌จΛߟ ͑ͯॻ͍͍ͯΔํ͕֤ࣾʹ͍Βͬ͠ΌΔ - ͜ͷσϞͰɺCLOVA Studio(Play-ground) ͷύϥϝλΛௐͯ͠ɺಈతʹ આ໌จΛੜ͢ΔྲྀΕΛ͓ݟͤ͠·͢ - ͷλΠτϧͱͷϦετ͔ΒɺCLOVA
StudioͦͷΛએ͢Δ ͨΊͷઆ໌จΛੜ͠·͢ HyperCLOVAͷσϞ: ͷ֓ཁͷઆ໌จΛੜ -
1. ਓ͕ؒͰ͖Δॲཧʹม (ෳͷ)ೖྗςΩετ (ෳͷ)ग़ྗςΩετ #### (ෳͷ)ೖྗςΩετ (ෳͷ)ग़ྗςΩετ #### ɾɾɾ ͷ༷ʹShotͷ܁Γฦ͠Λॻ͘ɻ
ྫ֎తͳใͳΔ্͘ͷํʹຒΊࠐΉ 2. ॲཧͷݟຊΛோΊΔ ʮಓͰ୭͔ʹϓϩϯϓτΛݟͤ ͨޙͰɺϓϩϯϓτʹؚ·Ε͍ͯ ͳ͍(ෳͷ)ೖྗςΩετΛݟͤ ͨΒɺ(ෳͷ)ग़ྗςΩετΛ࡞ ΕΔՄೳੑ͕͋Δ͔ʯΛߟ͑Δɻ Shot͕Γͳ͍͜ͱ͕ͳΒɺ ͞Βʹ͢͜ͱͰରॲͰ͖Δ 3. ΫΤϦΛՃ ϓϩϯϓτͷඌͷ####ͷ࣍ͷߦʹ ΫΤϦͱͳΔ(ෳͷ)ೖྗςΩετΛ ೖྗͨ͠Βɺ(ෳͷ)ग़ྗςΩετͷ ࠷ॳͷه߸෦Λهͯ͠ɺ࣮ߦͯ͠ ΈΔ ग़ྗ͕ඍົͳΒύϥϝλͷௐΛ͢Δ ବͳΒ12ʹͲΔ ͬͯΈ͍ͨ͜ͱ to ϓϩϯϓτܕͷࢦྩॻ -
Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - આ໌จੜγεςϜͷԠ༻ -
ରγεςϜͷԠ༻ - BERTͱͷൺֱ - ࣭ԠλεΫͰ - - ߴ࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝ - ·ͱΊ -
HyperCLOVAΛ༻͍ͨରγεςϜͷओ؍ධՁ with 6.7B/13B/39B JP Model for 4 tasks ͯ͢ͷλεΫͱϞσϧͷΈ߹ΘͤͰΞϊςʔγϣϯΛ࣮ࢪ ಉ͡5ਓͷΞϊςʔλʔʹΑΔओ؍ධՁ
֤ηογϣϯMԟ෮ͷձϖΞ ϢʔβʔධՁ༻ͷNݸͷτϐοΫͷϦετΛड͚औΔ ֤ηογϣϯͰɼϦετ͔Β1ͭͷޠኮΛফඅ͢Δ Play-groundͰ࣮ࢪ 4. ࡶஊΛ͢Δ 3. τϐοΫͷϢʔβײͷରԠ 2. ҟͳΔτϐοΫͷભҠʹରԠ 1. جૅతͳޠኮͷཧղ -
ྫ֎: ࡶஊͷλεΫΰʔϧͷୡΛධՁ͍ͯ͠ͳ͍(ෆཁ͔ͩΒ) ԁͳԠ QɿࣗવͳԠͰ͔ͨ͠ʁ ձͷܦҢʹͼໃ६͋Γ·ͤΜ͔ʁ τϐοΫͷै Q: Did it stay
on topic? τϐοΫΛݟࣦ͍ͬͯͳ͔͔ͬͨʢ͜ͷ߹ɺԿʹ͍ͭͯฉ͔Ε͍ͯΔͷ͔Λݟࣦ͍ͬͯͳ͍͔ʁ) ͷΓସ͕͑Ͱ͖͔ͨʢ͜ͷ߹ɺલͷ࣭ʹҾ͖͔ͤͨʁ) τϐοΫͷఏڙ ࣭ͷ͔͚͛ QɿΛఏڙ͔ͨ͠ʁ ճதʹൃݴऀͷΛҾ͖ग़͢͜ͱ͕Ͱ͖͔ͨʢͰ͖ͳ͍Մೳੑ͕ߴ͍ʣ ඪͷୡ QɿతୡͰ͖·͔ͨ͠ʁ શλεΫʹڞ௨ͳධՁ߲ ! -
1. جૅతͳޠኮͷཧղ ॳޠኮ தޠኮ খֶߍ ۚͮͪ தֶߍ Ԗච େਓ νϡʔϦοϓ
ઌੜ ώϚϫϦ ϥΠΦϯ ص ΩϦϯ Ҝࢠ ిं ۺ ं αϯμϧ ηʔλʔ ΓΜ͝ εΧʔτ Έ͔Μ Ωϟϕπ αϯϚ ͖Ύ͏Γ Ϛάϩ εζϝ ϋʔϞχΧ Πϯί ϐΞϊ τϯϘ ΞϦ IUUQTSFQPTJUPSZOJOKBMBDKQ BDUJPOSFQPTJUPSZ@BDUJPO@DPNNPO@EPXOMPBEJUFN@JEJUFN@OPBUUSJCVUF@JE fi MF@OP ToDO: ֤ޠኮʹରͯ͠ɺҙຯ(Ϩϕϧ1)ͱى͢Δײ(Ϩϕϧ2)ʹ͍࣭ͭͯΛ͢Δ -
Topic A Topic B Topic A Topic B ৽ܕίϩφΠϧε Πϯόϯυ
Πνϩʔ େ୩ᠳฏ ۓٸࣄଶએݴ ৽ܕίϩφϫΫνϯ AR(֦ுݱ࣮) ࣗಈӡసٕज़ YouTuber VTuber ϨΦφϧυɾμɾϰΟϯν ΫϩʔυɾϞω ฏ ྩ Πϯλʔωοτ 5G σϑϨܦࡁ ߴྸԽࣾձ ւ֎ཱྀߦ ࠃཱྀߦ ిؾࣗಈं ϦχΞதԝ৽װઢ 2. ҟͳΔෳͷτϐοΫΛτϥοΩϯά ToDO: τϐοΫAͰձΛ࢝Ίɺ10ԟ෮͢ΔલʹτϐοΫBʹΓସ͑Δ ฏ -
3. ͋ΔτϐοΫʹର͢ΔϢʔβʔͷײʹԠ Topic sentiment A sentiment B ৽ܕίϩφΠϧε ؤுΖ͏ ෆ҆ͩ
Πϯόϯυ ͬͯ͘Δ Βͳ͍ ৽ܕίϩφϫΫνϯ ͱ͏ ͍ͭʹͳΔ YouTuber Γ͍ͨ Γͨ͘ͳ͍ େ୩ᠳฏ ׆༂ͯ͠ཉ͍͠ ࡾৼͯ͠ཉ͍͠ AR(֦ுݱ࣮) ໘ന͍ ͖ͨ ߴྸԽࣾձ େৎ ৺ ւ֎ཱྀߦ ߦ͖͍ͨ ߦ͖ͨ͘ͳ͍ ిؾࣗಈं Γ͍ͨ Γͨ͘ͳ͍ ϦχΞதԝ৽װઢ Γ͍ͨ Γͨ͘ͳ͍ ToDOɿTopicʹ͍ͭͯ15ԟ෮ͷձΛ͢Δɻ࠷ॳͷ15ԟ෮ײAɺ࣍ʹײBͷؾ࣋ͪͰ͢ -
39B JP Modelͷओ؍ධՁͷ݁Ռͷཁ 1. جૅతͳޠኮͷཧղ 2. ҟͳΔτϐοΫ ͷભҠʹରԠ 3. τϐοΫͷલ
͖ͳײͷରԠ 3. τϐοΫͷޙΖ ͖ͳײͷରԠ 4. ࡶஊΛ͢Δ ԁͳԠ 0.978(356/364) 0.908(1749/1926) 0.908(1198/1320) 0.872(1072/1229) 0.925(86/93) τϐοΫͷै 0.984(358/364) 0.952(1834/1926) 0.951(1255/1320) 0.93(1144/1229) 0.935(87/93) τϐοΫͷఏڙ ࣭ͷ͔͚͛ 0.003(1/364) 0.023(44/1926) 0.033(44/1320) 0.035(43/1229) 0.086(8/93) ඪͷୡ 0.835(304/364) 0.907(1746/1926) 0.899(1187/1320) 0.505(621/1229) - -
39B JP Modelͷओ؍ධՁͷ݁Ռͷཁ 1. جૅతͳޠኮͷཧղ 2. ҟͳΔτϐοΫ ͷભҠʹରԠ 3. τϐοΫͷલ
͖ͳײͷରԠ 3. τϐοΫͷޙΖ ͖ͳײͷରԠ 4. ࡶஊΛ͢Δ ԁͳԠ 0.978(356/364) 0.908(1749/1926) 0.908(1198/1320) 0.872(1072/1229) 0.925(86/93) τϐοΫͷै 0.984(358/364) 0.952(1834/1926) 0.951(1255/1320) 0.93(1144/1229) 0.935(87/93) τϐοΫͷఏڙ ࣭ͷ͔͚͛ 0.003(1/364) 0.023(44/1926) 0.033(44/1320) 0.035(43/1229) 0.086(8/93) ඪͷୡ 0.835(304/364) 0.907(1746/1926) 0.899(1187/1320) 0.505(621/1229) - -
Application example: HyperCLOVA Friends Talk with any adjustable character using
HyperCLOVA HyperCLOVA͋ΔఔϩʔϧϓϨΠͰ͖Δ -
HyperCLOVA൚༻తͳϩʔϧϓϨΠ͕Մೳ Challenge: εϜʔζͳձͱτϐοΫͷҎ֎ͷػೳ ൃݴͷਅِΛ֬ೝ͔ͯ͠ Βճ͢Δ͜ͱ͕ඞཁ ձ͕εϜʔζͰɺݴͬͨ ͜ͱͷҙຯ͕ཧղͰ͖Δ ͍͔ͭ͘ͷᐆດͳճ ྫɿચ୕࣌ͷ͓౬ͷԹ σʔλͷภΓʹΑΔӨڹ
ྫɿະઃఆ͕ͩͬͨঁੑ ʹͳͬͨ ϖϧιφͷҰ؏ੑ গٙ͠Θ͍͠ จࣈͷηοτͳ͠ʹ ελʔτͰ͖Δ -
Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - આ໌จੜγεςϜͷԠ༻ -
ରγεςϜͷԠ༻ - BERTͱͷൺֱ - ࣭ԠλεΫͰ - - ߴ࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝ - ·ͱΊ -
39B JPϞσϧʹΑΔQAλεΫͷࣗಈධՁ Few-shotɺਪ͝ͱʹͷൃηοτ͔ΒϥϯμϜʹจষΛநग़ͯ͠࡞͠·ͨ͠ ίϯςΩετɺ࣭จɺճΛؚΉFew-shotͷ࡞ ਪ݁Ռͷதʹਖ਼ղؚ͕·Ε͍ͯͯɺ༰қʹநग़Ͱ͖Δ߹ਖ਼ղͱஅ͢Δ TASK: RCQA* possible only -
௨ৗͷ3$2"λεΫͷσʔληοτ͔ΒճෆೳͷΛআ * ղՄೳੑ͖ಡղσʔληοτ: http://www.cl.ecei.tohoku.ac.jp/rcqa/ -
RCQA possible only task ͷͨΊͷ 39B JPϞσϧʹΑΔࣗಈධՁ݁Ռ model /
few-shot shot temperature top_p answer match 6.7B / contextual 0 0.5 0.8 - 4 0.1 0.9 66.52 13B / contextual 0 0.5 0.8 - 4 0.4 0.1 70.28 39B / contextual 0 0.4 0.5 80.51 1 0.4 0.5 89.18 2 0.4 0.5 89.31 3 0.4 0.5 89.09 4 0.4 0.5 89.83 39B / non-contextual 0 0.4 0.5 69.50 1 0.4 0.5 76.97 2 0.4 0.5 79.08 3 0.4 0.5 79.38 4 0.4 0.5 80.51 -
HyperCLOVA’s LM vs BERT-large TASK: RCQA possible only (௨ৗͷRCQAλεΫ͔Β͑ΒΕͳ͍࣭Λআ͠·ͨ͠) -
ਅʹࠔͳΛHyperCLOVAͰղܾͰ͖ͳ͍Մೳੑ͕͋Γ·͢ - normal => possible only: train: 43610 => 21091, dev: 6863 => 3524, test: 6178 => 3079 - BERTɺಛఆͷλεΫʹରͯ͠ඍௐ͢Δ͜ͱͰɺΑΓߴ͍݁ՌΛಘΒΕΔՄೳੑ͕͋Δ - HyperCLOVAPromptingͱϥϑͳύϥϝʔλݕࡧͰBERTͱಉͷύϑΥʔϚϯεΛ࣮ݱͰ͖Δ test acc test F1 memo HyperCLOVA 85.03 89.95 JP 39B 2-shots, temperature=0.4, top_p= 0.5 BERT-jp-large 86.68 90.49 Using subset of LINE LM corpus -
Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - આ໌จੜγεςϜͷԠ༻ -
ରγεςϜͷԠ༻ - BERTͱͷൺֱ - ࣭ԠλεΫͰ - - ߴ࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝ - ·ͱΊ -
ຊޠͷ͠͞ शಘͷ͠͞ ຊޠͷऀ - ͻΒ͕ͳ - ยԾ໊ - ࣈ -
ϩʔϚࣈ ͳͲͳͲΛ ҰͭͷจॻΛॻͨ͘Ίʹ͏ େྔͷඞਢޠኮ ৗձʹඞཁͳޠኮ - 8,000ޠҎ্ ଟ͘ͷ୯ޠΛΔඞཁ͕͋Δ - ಉԻҟٛޠ - ܟޠ - ํݴͳͲ ୯ޠͷলུ ຊޠऀɺจॻதͷҎԼ ͷ୯ޠΛলུ͢Δ͜ͱ͕͋Δ - ओޠ - తޠ লུ͞Εͨ୯ޠҰҙʹਪ Ͱ͖ͳ͍Մೳੑ͕͋Δ -
ςΩετੜͷ͠͞ ੜ͞ΕͨςΩετͷ જࡏతͳϦεΫ ҎԼͷΑ͏ͳٕज़Λ։ൃ ͢Δඞཁ͕͋Γ·͢ɻ - ίʔύεͷ༰ͷภΓ දهํ๏ͷվળ -
ग़ྗ͞ΕͨςΩετͷਅ ࣮ੑͱ҆શੑͷ֬อ AIྙཧͷ࣮ફ ೖग़ྗςΩετʹ༷ʑͳྙཧత ྀ͕ඞཁͰ͢ - ಟੑ - ੑతͳͷ - ߈ܸతͳͷ - ಚత - ڴഭత - ΞΠσϯςΟςΟͷ߈ܸ ຊ࣭తͳධՁͷࣗಈԽ ಈతςΩετੜͷ݁Ռʹద ༻Ͱ͖ΔϝτϦΫε͕ඞཁ - ੑͷ͋Δίϯςϯπͷਫ਼ - ੜ͞ΕͨςΩετͷҰ؏ੑ - తͷୡͷఆ -
NLPʹHyperCLOVAຊʹඞཁͳͷ͔ʁ - YES !! ͠༧ࢉࣥߦ͕ڐ͞ΕΔͳΒʜ - NLPͷྺ࢙AIؔ࿈ٕज़ͷൃలͱڧ݁͘ͼ͍͍ͭͯΔ - LINEࣗͨͪͰϞσϧΛ࡞͓ͬͯ٬༷ʹͬͯΒ͏
ํʹਐΈ͍ͨͱߟ͍͑ͯ·͢ Large-scale general-purpose LMs DNN Traditional only ML Rule only Small LM only -
LINEͷݴޠϞσϧΛOSSͱͯ͠ຊதʹ։࢝ ͪΖΜHyperCLOVAҎ֎Ͱ͢ ੑೳඪ: LINEͷݴޠϞσϧ for OSS > other OSS ͳݴޠϞσϧ
ՄೳͰ͋Εɺෳؒఆظతʹߋ৽͍ͨ͠ͱߟ͍͑ͯ·͢ HyperCLOVAʹ͍ͬͯΔίʔύε(LINE LM Corpus)ͷαϒηοτΛ༻͠·͢ ! -
Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - આ໌จੜγεςϜͷԠ༻ -
ରγεςϜͷԠ༻ - BERTͱͷൺֱ - ࣭ԠλεΫͰ - - ߴ࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝ - ·ͱΊ