AI application systems realized by HyperCLOVA, a large-scale general-purpose language model for Japanese and its challenges

Slide 1

Slide 1 text

೔ຊޠͷେن໛൚༻ݴޠϞσϧ   ʮHyperCLOVAʯ͕࣮ݱͰ͖Δ   AIԠ༻γεςϜͱͦͷ՝୊ Toshinori Sato (@overlast) LINE / AI Company

Slide 2

Slide 2 text

ࠤ౻ හل (@overlast) ● γχΞιϑτ΢ΣΞΤϯδχΞ / Ϛωʔδϟʔ ● ࣗવݴޠॲཧ ● ৘ใݕࡧ ● LINE CLOVA ● ೔ຊޠͷࣗવݴޠཧղ(NLU)γεςϜ ● HyperCLOVA ● ೔ຊޠ൛ͷ։ൃ੹೚ऀ ● ೔ຊޠίʔύε / AIϑΟϧλʔ / Ԡ༻ٕज़ͷ։ൃɾධՁ ● OSS: NEologd ϓϩδΣΫτͷϝΠϯ։ൃऀ ● mecab-ipadic-NEologd 2

Slide 3

Slide 3 text

Attention, please ! ࠷ॳʹ͓఻͍͑ͨ͠·͢ɻ - ͜ͷηογϣϯ͸20෼ͱ΍΍ۦ͚଍Ͱ͢ͷͰɺ͢͜͠ूதྗ͕ඞཁͰ͢ ! - Ͳ͏ͯ͠΋ࣗવݴޠॲཧͷઐ໳༻ޠ͕ଟ͘ͳΓ·͕͢ɺ෼͔Βͳ͍༻ޠ͸   Θ͔Γқ͘ղઆͰ͖·͢ͷͰɺTwitterͰ @overlast ʹmention͍ͯͩ͘͠͞ - େن໛൚༻ݴޠϞσϧ͕ ”ະདྷΛͲ͏ม͑Δ͔” ༧૝͠ͳ͕Β͓ฉ͖͍ͩ͘͞

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

໰୊: ਓؒͱHyperCLOVA͕νϟοτΛ͍ͯ͠·͢

Slide 6

Slide 6 text

פ͍Ͱ͢Ͷʔɻ  ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ Ͳ͕ͬͪ HyperCLOVA ͔෼͔Γ·͔͢?

Slide 7

Slide 7 text

פ͍Ͱ͢Ͷʔɻ  ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ  ۝भͷग़਎ͳͷͰژ౎ͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ Ͳ͕ͬͪ HyperCLOVA ͔෼͔Γ·͔͢?

Slide 8

Slide 8 text

פ͍Ͱ͢Ͷʔɻ  ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ  ۝भͷग़਎ͳͷͰژ౎ͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ෱ԬݝͰ͚ͨͬ͠?  തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ Ͳ͕ͬͪ HyperCLOVA ͔෼͔Γ·͔͢?

Slide 9

Slide 9 text

פ͍Ͱ͢Ͷʔɻ  ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ  ۝भͷग़਎ͳͷͰژ౎ͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ෱ԬݝͰ͚ͨͬ͠?  തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ෱Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़਎ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ޷͖Ͱ͔͢ʁ Ͳ͕ͬͪ HyperCLOVA ͔෼͔Γ·͔͢?

Slide 10

Slide 10 text

פ͍Ͱ͢Ͷʔɻ  ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ  ۝भͷग़਎ͳͷͰژ౎ͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ෱ԬݝͰ͚ͨͬ͠?  തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ΋ͪΖΜͰ͢!  ຊ৔ͷಲࠎϥʔϝϯ৯΂ͯΈ͍ͨͰ͢Ͷɻ ෱Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़਎ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ޷͖Ͱ͔͢ʁ Ͳ͕ͬͪ HyperCLOVA ͔෼͔Γ·͔͢?

Slide 11

Slide 11 text

פ͍Ͱ͢Ͷʔɻ  ෩अͻ͔ͳ͍Α͏ʹ͠ͳ͖Ό͍͚·ͤΜͶɻ ͦ͏Ͱ͢Ͷɻ  ۝भͷग़਎ͳͷͰژ౎ͷפ͞ʹ׳Εͳ͍ΜͰ͢ΑͶɻ ෱ԬݝͰ͚ͨͬ͠?  തଟϥʔϝϯඒຯ͍͠Ͱ͢ΑͶɻ ΋ͪΖΜͰ͢!  ຊ৔ͷಲࠎϥʔϝϯ৯΂ͯΈ͍ͨͰ͢Ͷɻ ෱Ԭͷྡͷࠤլݝͱ͍͏ͱ͜Ζͷग़਎ͳΜͰ͚͢ͲͶɻ ಲࠎϥʔϝϯ޷͖Ͱ͔͢ʁ ͍͍Ͱ͢ΑͶɻ  ͜ͷפ͍࣌ظ͸ಛʹ͍͍Ͱ͢Αɻ Ͳ͕ͬͪ HyperCLOVA ͔෼͔Γ·͔͢?

Slide 12

Slide 12 text

Slide 13

Slide 13 text

େن໛൚༻ݴޠϞσϧ + α ͳγεςϜ Automatic evaluation with 39B JP Model for a QA task

Slide 14

Slide 14 text

HyperCLOVAͷΞʔΩςΫνϟ Eco System Infra Model Data

Slide 15

Slide 15 text

HyperCLOVA ΛԠ༻ͨ͠γεςϜʹΑΔग़ྗͷڧΈ ਓؒͷ༷ʹ׈Β͔ͳ   ςΩετΛੜ੒Ͱ͖Δ ಺༰ͷධՁ ೚ҙͷτϐοΫʹ   ௥ैͰ͖Δ ඼࣭ͷධՁ ৽͍͠࿩୊Λ   ఏڙͰ͖Δ ද૚తͳධՁ

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ - ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

Title 80pt - ༧ଌ͍ͨ͠ςΩετͷτʔΫϯ਺NͱɺίʔύεΛଌఆͯ͠ಘͨط஌ͷτʔΫϯྻछ਺Vͷͱ͖ - ςΩετͷ֬཰෼෍ΛٻΊΔܭࢉ࣌ʹอ࣋͢Δύϥϝλ਺͕VͷN৐ʹͳΔ => ίεύ͕ѱ͍ ֬཰తݴޠϞσϧͷ࣍ݩͷढ͍ χϡʔϥϧݴޠϞσϧ - LSTM΍Transformer͕جʹͳ͍ͬͯͯѥछ͕ࢁఔ͋ΔɻBERT΍HyperCLOVA͸Transformerͷ೿ੜ - ಛ௃ϕΫτϧͷ࣍ݩ਺N(e.g. GPT3͸12288) x ޠኮ਺VͷߦྻΛอ࣋͢ΔͷͰVͷnഒ => ίεύ͕ྑ͍ - ֬཰తݴޠϞσϧͷ༷ʹ֬཰෼෍ͷ௿͍ܥྻͷ౷ܭ஋Λཅʹ࣋ͨͳͯ͘ࡁΉ ֬཰తݴޠϞσϧ ݴޠϞσϧ͸֬཰తͱχϡʔϥϧͷ2λΠϓʹ෼͔ΕΔ - ྫ: ୯ޠn-gramϞσϧ = ίʔύεதͷ୯ޠn-gramग़ݱ֬཰ʹج͖ͮɺ೚ҙͷςΩετͷ֬཰෼෍ΛਪఆͰ͖Δ - n-gram = ܥྻΛ୯Ґ௕ͷτʔΫϯྻʹ۠੾Δ͜ͱͰಘΒΕΔɺ͢΂ͯͷ௕͞nͷτʔΫϯྻ - ྫ: ݴޠϞσϧͷจࣈ1-gram: ݴ/ޠ/Ϟ/σ/ϧɺ ݴޠϞσϧͷจࣈ2-gram: ݴޠ/ޠϞ/Ϟσ/σϧ

Slide 24

Slide 24 text

Title 80pt - ༧ଌ͍ͨ͠ςΩετͷτʔΫϯ਺NͱɺίʔύεΛଌఆͯ͠ಘͨط஌ͷτʔΫϯྻछ਺Vͷͱ͖ - ςΩετͷ֬཰෼෍ΛٻΊΔܭࢉ࣌ʹอ࣋͢Δύϥϝλ਺͕VͷN৐ʹͳΔ => ίεύ͕ѱ͍ ֬཰తݴޠϞσϧͷ࣍ݩͷढ͍ χϡʔϥϧݴޠϞσϧ - LSTM΍Transformer͕جʹͳ͍ͬͯͯѥछ͕ࢁఔ͋ΔɻBERT΍HyperCLOVA͸Transformerͷ೿ੜ - ಛ௃ϕΫτϧͷ࣍ݩ਺N(e.g. GPT3͸12288) x ޠኮ਺VͷߦྻΛอ࣋͢ΔͷͰVͷnഒ => ίεύ͕ྑ͍ - ֬཰తݴޠϞσϧͷ༷ʹ֬཰෼෍ͷ௿͍ܥྻͷ౷ܭ஋Λཅʹ࣋ͨͳͯ͘ࡁΉ ֬཰తݴޠϞσϧ HyperCLOVA͸χϡʔϥϧݴޠϞσϧΛ಺แ - ྫ: ୯ޠn-gramϞσϧ = ίʔύεதͷ୯ޠn-gramग़ݱ֬཰ʹج͖ͮɺ೚ҙͷςΩετͷ֬཰෼෍ΛਪఆͰ͖Δ - n-gram = ܥྻΛ୯Ґ௕ͷτʔΫϯྻʹ۠੾Δ͜ͱͰಘΒΕΔɺ͢΂ͯͷ௕͞nͷτʔΫϯྻ - ྫ: ݴޠϞσϧͷจࣈ1-gram: ݴ/ޠ/Ϟ/σ/ϧɺ ݴޠϞσϧͷจࣈ2-gram: ݴޠ/ޠϞ/Ϟσ/σϧ

Slide 25

Slide 25 text

NLPͷٕज़τϨϯυͷมભ Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2   (2019) GPT-3   (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) MT-NLG (2021)

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2   (2019) GPT-3   (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) ʻBreak Through 1ʼ Attentionٕज़͕ൃలɺ ֶश͕ߴ଎Խ ʻBreak Through 2ʼ Generative pre-trainingʹΑΔ ϥϕϧ෇σʔλͷେྔੜ੒ ʻBreak Through 3ʼ Prompting : ̍ͭͷݴޠϞσϧͰ   ༷ʑͳλεΫΛॲཧͰ͖ΔՄೳੑ MT-NLG (2021) NLPͷٕज़τϨϯυͷมભ

Slide 29

Slide 29 text

3. Prompting ֶश ֶश Ϟσϧ ֶश σʔλ ਪ࿦ σʔλ ਪଌ σʔλ ֶश ࣄલֶश Ϟσϧ ֶश σʔλ ਪ࿦ σʔλ ਪଌ σʔλ ֶश ಛԽܕֶश Ϟσϧ λεΫ ಛԽֶश σʔλ ֶश େن໛൚༻ ݴޠϞσϧ ֶश σʔλ ਪ࿦ ਪଌ σʔλ Prompt ≠Program ैདྷ: ֶशϞσϧΛλεΫʹಛԽͤ͞Δ͜ͱͰੑೳΛ֬อ͖ͯͨ͠ ࠓޙ: ࠶ֶश΍ɺϓϩάϥϜͷ࣮૷Λͤͣʹ ໨తλεΫͷਪ࿦Λ࣮ࢪͰ͖Δ

Slide 30

Slide 30 text

ࣄલֶशࡁΈେن໛൚༻ݴޠϞσϧͷ޿ൣͳ׆༻

Slide 31

Slide 31 text

ࣄલֶशࡁΈେن໛൚༻ݴޠϞσϧͷ޿ൣͳ׆༻

Slide 32

Slide 32 text

ͻͱͭͷେن໛൚༻ݴޠϞσϧͰɺଟ༷ͳλεΫΛղ͚ΔՄೳੑ •ର࿩γεςϜ •จষཁ໿ •આ໌จੜ੒ •຋༁ ɾɾɾ •ܭࢉ •ϓϩάϥϜੜ੒ Prompt(ϓϩϯϓτ)ͷ༩͑ํ࣍ୈͰ ༷ʑͳλεΫ͕࣮ݱͰ͖ΔՄೳੑ͕ Prompting ʹΑΔϞσϧͷ࠶ར༻ͷଅਐ

Slide 33

Slide 33 text

Slide 34

Slide 34 text

4. Scaling Law • ύϥϝʔλ਺Λ૿΍͢ͱϞσϧͷੑೳ͸૿͑Δ • ͜ͷݶք஋͕ݱঢ়ݟ͔͍ͭͬͯͳ͍ େن໛൚༻ݴޠϞσϧͷ ։ൃڝ૪͕ܹԽ

Slide 35

Slide 35 text

ڝ߹ଞ͕ࣾެද͍ͯ͠ΔϞσϧͷαΠζ OpenAI Google GPT: Generative Pre-trained Transformer T5: Text-To-Text Transfer Transformer MT-NLG: Megatron-Turing Natural Language Generation GPT-3(2020)ɿ1.3B, 2.7B, 6.7B, 13B, 175B GPT-2(2019)ɿ117M, 345M, 762M, 1.5B T5(2019)ɿ60M, 220M, 770M, 2.8B, 11B Microsoft MT-NLG(2021)ɿ530B

Slide 36

Slide 36 text

Slide 37

Slide 37 text

େن໛൚༻ݴޠϞσϧ + α ͳγεςϜ Automatic evaluation with 39B JP Model for a QA task

Slide 38

Slide 38 text

HyperCLOVAͷΞʔΩςΫνϟ Eco System Infra Model Data

Slide 39

Slide 39 text

HyperCLOVA ͷ೔ຊޠϞσϧߏஙͷݱঢ় 1.3B → 6.7B → 13B → 39B 13B → 39B 82B 204B ʙ (2022೥த) ଟݴޠϞσϧ େن໛Ϟσϧ   ೔ຊޠ / ଟݴޠ ௒େن໛   ೔ຊޠϞσϧ ೔ຊޠϞσϧ ߏங࡞ۀ͕ਐߦத

Slide 40

Slide 40 text

HyperCLOVAͷϞσϧߏஙͷख๏ EMNLP 2021 ͷNAVERͷ࿦จ*΋͝ࢀর͍ͩ͘͞ https://arxiv.org/abs/2109.04650 ࠷௿ͷֶश཰Λ Megatron-LMͷඪ४஋ͷ 1/10ʹ͢Δඞཁ͕͋ͬͨ NVIDIA Superpod ্Ͱֶश   Superpod͸Ԇ΂1024ຕͷA100 Λࢗͨ͠128୆ͷDGXΫϥελ ֶशʹ࢖͏ίʔύεͷτʔΫ ϯ਺͸ɺߏங͢ΔϞσϧͷύ ϥϝλ਺ͷ3ഒҎ্͕๬·͍͠ ByteϨϕϧͷBPE tokenizerʹΑΔίʔύε ࣄྫͷτʔΫϯԽ Transformer Decoder architecture Λ࠾༻ Megatron-LM Λ࢖༻ (Shoeybi et al., 2019) https://arxiv.org/abs/2109.04650 * What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers, Boseop Kim et.al, EMNLP 2021

Slide 41

Slide 41 text

HyperCLOVAͷϞσϧߏஙͷख๏ EMNLP 2021 ͷNAVERͷ࿦จ΋͝ࢀর͍ͩ͘͞ ࠷௿ͷֶश཰Λ Megatron-LMͷඪ४஋ͷ 1/10ʹ͢Δඞཁ͕͋ͬͨ NVIDIA Superpod ্Ͱֶश   Superpod͸Ԇ΂1024ຕͷA100 Λࢗͨ͠128୆ͷDGXΫϥελ ֶशʹ࢖͏ίʔύεͷτʔΫ ϯ਺͸ɺߏங͢ΔϞσϧͷύ ϥϝλ਺ͷ3ഒҎ্͕๬·͍͠ ByteϨϕϧͷBPE tokenizerʹΑΔίʔύε ࣄྫͷτʔΫϯԽ Transformer Decoder architecture Λ࠾༻ Megatron-LM Λ࢖༻ (Shoeybi et al., 2019)

Slide 42

Slide 42 text

Slide 43

Slide 43 text

LINE LM Corpus ͷݱࡏͷঢ়گ For 82B JP Model αϯϓϧ 10B τʔΫϯ 500B σʔλαΠζ 1.8TB

Slide 44

Slide 44 text

ίʔύεͷͨΊͷσʔλऩूͷํ਑ LINEͷαʔϏεͷձ࿩σʔλΛ࢖Θͳ͍ - LINEͷ͢΂ͯͷϝοηʔδ - OpenChatͷ͢΂ͯͷ౤ߘ ݖརॲཧΛਖ਼͓͘͜͠ͳ͔ͬͯΒίʔύεʹ௥Ճ͢Δ ൚༻ੑΛ޲্͢ΔίϯςϯπΛ - LINEͷ֎ʹίʔύεͷαϒηοτΛఏڙ͢ΔՄೳੑΛ࢒͢

Slide 45

Slide 45 text

LINE LM Corpus(for HyperCLOVA’s LMs) LINE ΍ LINE OpenChat ͷσʔλ͸ຊίʔύεʹ࠾࿥͠·ͤΜ - BERTͷϞσϧΛߏங͢ΔͨΊͷίʔύεͱͯ͠2019೥͔Β։ൃΛ։࢝ - LINE ݕࡧػೳͷͨΊʹΫϩʔϧ͞Εͨσʔλ΋ར༻ͨ͠ - "ඇެ։ͳݸਓ৘ใ"ͱͯ͠༰қʹநग़Ͱ͖ͨσʔλ͸আ֎͍ͯ͠Δ - ೔ຊޠදݱΛϞσϧʹؚΊΔ͏͑Ͱେ੾ͦ͏ͳαΠτΛඃ෴ͨ͠ - ͍͔ͭ͘ͷ֎෦ίϯςϯπΛߪೖͯ͠ɺݖར໰୊Λղܾͨ͠͏͑Ͱ࢖༻ !

Slide 46

Slide 46 text

LINE LM Corpus for 39Bͷ಺༁ 82B Ҏ্ͷϞσϧߏஙʹ͍ͭͯ͸ɺߋʹ޿ൣͳൣғ͔ΒίϯςϯπΛऩू͍ͯ͠·͢ ղઆ τʔΫϯ਺ Blog ೔ຊޠͷϒϩά 105.8B News ೔ຊޠͷχϡʔεهࣄ 12.4B Q&A ೔ຊޠͷ࣭໰Ԡ౴αΠτ 10.5B WikiJa/En/Ko Wikipedia ͷ dump data 4.7B Novel ೔ຊޠͷখઆαΠτ 1.0B Shopping ೔ຊޠͷγϣοϐϯάαΠτ 20.9B Others ͦͷଞ 0.1B Total 155.4B

Slide 47

Slide 47 text

ςΩετੜ੒ ςΩετͷੜ੒తͳཁ໿ ର࿩γεςϜ

Slide 48

Slide 48 text

Slide 49

Slide 49 text

- ঎඼ύοέʔδʹ͸આ໌จ͕ॻ͔Ε͍ͯΔ͜ͱ͕ଟ͍͕ɺ͋ͷઆ໌จΛߟ ͑ͯॻ͍͍ͯΔํ͕֤ࣾʹ͍Βͬ͠ΌΔ - ͜ͷσϞͰ͸ɺCLOVA Studio(Play-ground) ͷύϥϝλΛௐ੔ͯ͠ɺಈతʹ આ໌จΛੜ੒͢ΔྲྀΕΛ͓ݟͤ͠·͢ - ঎඼ͷλΠτϧͱ੒෼ͷϦετ͔ΒɺCLOVA Studio͸ͦͷ঎඼Λએ఻͢Δ ͨΊͷઆ໌จΛੜ੒͠·͢ HyperCLOVAͷσϞ: ঎඼ͷ֓ཁͷઆ໌จΛੜ੒

Slide 50

Slide 50 text

Demo Movie 60sec

Slide 51

Slide 51 text

1. ਓ͕ؒͰ͖Δॲཧʹม׵ (ෳ਺ͷ)ೖྗςΩετ (ෳ਺ͷ)ग़ྗςΩετ #### (ෳ਺ͷ)ೖྗςΩετ (ෳ਺ͷ)ग़ྗςΩετ #### ɾɾɾ ͷ༷ʹShotͷ܁Γฦ͠Λॻ͘ɻ ྫ֎తͳ৘ใ͸ͳΔ΂্͘ͷํʹຒΊࠐΉ 2. ॲཧͷݟຊΛோΊΔ ʮಓ୺Ͱ୭͔ʹϓϩϯϓτΛݟͤ ͨޙͰɺϓϩϯϓτʹؚ·Ε͍ͯ ͳ͍(ෳ਺ͷ)ೖྗςΩετΛݟͤ ͨΒɺ(ෳ਺ͷ)ग़ྗςΩετΛ࡞ ΕΔՄೳੑ͕͋Δ͔ʯΛߟ͑Δɻ     Shot͕଍Γͳ͍͜ͱ͕໰୊ͳΒɺ ͞Βʹ଍͢͜ͱͰରॲͰ͖Δ 3. ΫΤϦΛ௥Ճ ϓϩϯϓτͷ຤ඌͷ####ͷ࣍ͷߦʹ ΫΤϦͱͳΔ(ෳ਺ͷ)ೖྗςΩετΛ ೖྗͨ͠Βɺ(ෳ਺ͷ)ग़ྗςΩετͷ ࠷ॳͷه߸෦෼Λ௥هͯ͠ɺ࣮ߦͯ͠ ΈΔ ग़ྗ͕ඍົͳΒύϥϝλͷௐ੔Λ͢Δ ବ໨ͳΒ1΍2ʹ΋ͲΔ ΍ͬͯΈ͍ͨ͜ͱ to ϓϩϯϓτܕͷࢦྩॻ

Slide 52

Slide 52 text

Slide 53

Slide 53 text

HyperCLOVAΛ༻͍ͨର࿩γεςϜͷओ؍ධՁ with 6.7B/13B/39B JP Model for 4 tasks ͢΂ͯͷλεΫͱϞσϧͷ૊Έ߹ΘͤͰΞϊςʔγϣϯΛ࣮ࢪ ಉ͡5ਓͷΞϊςʔλʔʹΑΔओ؍ධՁ ֤ηογϣϯ͸Mԟ෮ͷձ࿩ϖΞ Ϣʔβʔ͸ධՁ༻ͷNݸͷτϐοΫͷϦετΛड͚औΔ ֤ηογϣϯͰ͸ɼϦετ͔Β1ͭͷޠኮΛফඅ͢Δ Play-groundͰ࣮ࢪ 4. ࡶஊΛ͢Δ 3. τϐοΫ΁ͷϢʔβײ৘΁ͷରԠ 2. ҟͳΔτϐοΫ΁ͷભҠʹରԠ 1. جૅతͳޠኮͷཧղ

Slide 54

Slide 54 text

ྫ֎: ࡶஊͷλεΫ͸ΰʔϧͷୡ੒౓ΛධՁ͍ͯ͠ͳ͍(ෆཁ͔ͩΒ) ԁ׈ͳԠ౴ Qɿࣗવͳ൓ԠͰ͔ͨ͠ʁ ձ࿩ͷܦҢʹ୼ͼ΍ໃ६͸͋Γ·ͤΜ͔ʁ τϐοΫ΁ͷ௥ै Q: Did it stay on topic? τϐοΫΛݟࣦ͍ͬͯͳ͔͔ͬͨʢ͜ͷ৔߹ɺԿʹ͍ͭͯฉ͔Ε͍ͯΔͷ͔Λݟࣦ͍ͬͯͳ͍͔ʁ) ࿩୊ͷ੾Γସ͕͑Ͱ͖͔ͨʢ͜ͷ৔߹ɺલͷ࣭໰ʹҾ͖໭͔ͤͨʁ) τϐοΫͷఏڙ΍   ࣭໰ͷ౤͔͚͛ Qɿ࿩୊Λఏڙ͔ͨ͠ʁ ճ౴தʹൃݴऀͷ࿩ΛҾ͖ग़͢͜ͱ͕Ͱ͖͔ͨʢͰ͖ͳ͍Մೳੑ͕ߴ͍ʣ ໨ඪͷୡ੒ Qɿ໨త͸ୡ੒Ͱ͖·͔ͨ͠ʁ શλεΫʹڞ௨ͳධՁ߲໨ !

Slide 55

Slide 55 text

1. جૅతͳޠኮͷཧղ ॳ౳ޠኮ த౳ޠኮ খֶߍ ۚͮͪ தֶߍ Ԗච େਓ νϡʔϦοϓ ઌੜ ώϚϫϦ ϥΠΦϯ ص ΩϦϯ Ҝࢠ ిं ۺ ं αϯμϧ ηʔλʔ ΓΜ͝ εΧʔτ Έ͔Μ Ωϟϕπ αϯϚ ͖Ύ͏Γ Ϛάϩ εζϝ ϋʔϞχΧ Πϯί ϐΞϊ τϯϘ ΞϦ IUUQTSFQPTJUPSZOJOKBMBDKQ BDUJPOSFQPTJUPSZ@BDUJPO@DPNNPO@EPXOMPBEJUFN@JEJUFN@OPBUUSJCVUF@JE fi MF@OP ToDO: ֤ޠኮʹରͯ͠ɺҙຯ(Ϩϕϧ1)ͱ૝ى͢Δײ৘(Ϩϕϧ2)ʹ͍࣭ͭͯ໰Λ͢Δ

Slide 56

Slide 56 text

Topic A Topic B Topic A Topic B ৽ܕίϩφ΢Πϧε Πϯό΢ϯυ Πνϩʔ େ୩ᠳฏ ۓٸࣄଶએݴ ৽ܕίϩφϫΫνϯ AR(֦ுݱ࣮) ࣗಈӡసٕज़ YouTuber VTuber ϨΦφϧυɾμɾϰΟϯν ΫϩʔυɾϞω ฏ੒ ྩ࿨ Πϯλʔωοτ 5G σϑϨܦࡁ ௒ߴྸԽࣾձ ւ֎ཱྀߦ ࠃ಺ཱྀߦ ిؾࣗಈं ϦχΞதԝ৽װઢ 2. ҟͳΔෳ਺ͷτϐοΫΛτϥοΩϯά ToDO: τϐοΫAͰձ࿩Λ࢝Ίɺ10ԟ෮͢ΔલʹτϐοΫBʹ੾Γସ͑Δ ฏ੒

Slide 57

Slide 57 text

3. ͋ΔτϐοΫʹର͢ΔϢʔβʔͷײ৘ʹ൓Ԡ Topic sentiment A sentiment B ৽ܕίϩφ΢Πϧε ؤுΖ͏ ෆ҆ͩ Πϯό΢ϯυ ໭ͬͯ͘Δ ໭Βͳ͍ ৽ܕίϩφϫΫνϯ ଴ͱ͏ ͍ͭʹͳΔ YouTuber ΍Γ͍ͨ ΍Γͨ͘ͳ͍ େ୩ᠳฏ ׆༂ͯ͠ཉ͍͠ ࡾৼͯ͠ཉ͍͠ AR(֦ுݱ࣮) ໘ന͍ ๞͖ͨ ௒ߴྸԽࣾձ େৎ෉ ৺഑ ւ֎ཱྀߦ ߦ͖͍ͨ ߦ͖ͨ͘ͳ͍ ిؾࣗಈं ৐Γ͍ͨ ৐Γͨ͘ͳ͍ ϦχΞதԝ৽װઢ ৐Γ͍ͨ ৐Γͨ͘ͳ͍ ToDOɿTopicʹ͍ͭͯ15ԟ෮ͷձ࿩Λ͢Δɻ࠷ॳͷ15ԟ෮͸ײ৘Aɺ࣍ʹײ৘Bͷؾ࣋ͪͰ࿩͢

Slide 58

Slide 58 text

4.ࡶஊΛ͢Δ

Slide 59

Slide 59 text

39B JP Modelͷओ؍ධՁͷ݁Ռͷཁ໿ 1. جૅతͳޠኮͷཧղ 2. ҟͳΔτϐοΫ΁ ͷભҠʹରԠ 3. τϐοΫ΁ͷલ ޲͖ͳײ৘΁ͷରԠ 3. τϐοΫ΁ͷޙΖ ޲͖ͳײ৘΁ͷରԠ 4. ࡶஊΛ͢Δ ԁ׈ͳԠ౴ 0.978(356/364) 0.908(1749/1926) 0.908(1198/1320) 0.872(1072/1229) 0.925(86/93) τϐοΫ΁ͷ௥ै 0.984(358/364) 0.952(1834/1926) 0.951(1255/1320) 0.93(1144/1229) 0.935(87/93) τϐοΫͷఏڙ΍   ࣭໰ͷ౤͔͚͛ 0.003(1/364) 0.023(44/1926) 0.033(44/1320) 0.035(43/1229) 0.086(8/93) ໨ඪͷୡ੒ 0.835(304/364) 0.907(1746/1926) 0.899(1187/1320) 0.505(621/1229) -

Slide 60

Slide 60 text

Slide 61

Slide 61 text

ΞϓϦέʔγϣϯ: HyperCLOVA Friends ೚ҙͷௐ੔ՄೳͳΩϟϥΫλʔͱHyperCLOVAͰձ࿩͢Δ

Slide 62

Slide 62 text

Demo Movie 60sec

Slide 63

Slide 63 text

Application example: HyperCLOVA Friends Talk with any adjustable character using HyperCLOVA HyperCLOVA͸͋Δఔ౓ϩʔϧϓϨΠͰ͖Δ

Slide 64

Slide 64 text

HyperCLOVA͸൚༻తͳϩʔϧϓϨΠ͕Մೳ Challenge: εϜʔζͳձ࿩ͱτϐοΫͷ௥੻Ҏ֎ͷػೳ ൃݴͷਅِΛ֬ೝ͔ͯ͠ Βճ౴͢Δ͜ͱ͕ඞཁ ձ࿩͕εϜʔζͰɺݴͬͨ ͜ͱͷҙຯ͕ཧղͰ͖Δ ͍͔ͭ͘ͷᐆດͳճ౴ ྫɿચ୕࣌ͷ͓౬ͷԹ౓ σʔλͷภΓʹΑΔӨڹ ྫɿະઃఆ͕ͩͬͨঁੑ ʹͳͬͨ ϖϧιφͷҰ؏ੑ͸   গٙ͠Θ͍͠ จࣈͷηοτͳ͠ʹ   ελʔτͰ͖Δ

Slide 65

Slide 65 text

LINEͷର࿩γεςϜ͕޷੒੷Λ࢒͠·ͨ͠ ਓ޻஌ೳֶձ ݴޠɾԻ੠ཧղͱର࿩ॲཧݚڀձ SIG-SLUD ୈ12ճର࿩γεςϜγϯϙδ΢Ϝ ର࿩γεςϜϥΠϒίϯϖςΟγϣϯ̐ https://dialog-system-live-competition.github.io/dslc4/

Slide 66

Slide 66 text

ର࿩γεςϜγϯϙδ΢Ϝʹؔ͢ΔϒϩάهࣄΛग़͠·ͨ͠ https://blog.clova.line.me/hyperclova-202112

Slide 67

Slide 67 text

Slide 68

Slide 68 text

Eco System Infra Model Data 39B JP model ͷࣗಈධՁ

Slide 69

Slide 69 text

39B JPϞσϧʹΑΔQAλεΫͷࣗಈධՁ Few-shot͸ɺਪ࿦͝ͱʹͷൃηοτ͔ΒϥϯμϜʹจষΛநग़ͯ͠࡞੒͠·ͨ͠ ίϯςΩετɺ࣭໰จɺճ౴ΛؚΉFew-shotͷ࡞੒ ਪ࿦݁Ռͷதʹਖ਼ղؚ͕·Ε͍ͯͯɺ༰қʹநग़Ͱ͖Δ৔߹͸ਖ਼ղͱ൑அ͢Δ TASK: RCQA* possible only - ௨ৗͷ3$2"λεΫͷσʔληοτ͔Βճ౴ෆೳͷ໰୊Λ࡟আ * ղ౴Մೳੑ෇͖ಡղσʔληοτ: http://www.cl.ecei.tohoku.ac.jp/rcqa/

Slide 70

Slide 70 text

RCQA possible only task ͷͨΊͷ   39B JPϞσϧʹΑΔࣗಈධՁ݁Ռ model / few-shot shot temperature top_p answer match 6.7B / contextual 0 0.5 0.8 - 4 0.1 0.9 66.52 13B / contextual 0 0.5 0.8 - 4 0.4 0.1 70.28 39B / contextual 0 0.4 0.5 80.51 1 0.4 0.5 89.18 2 0.4 0.5 89.31 3 0.4 0.5 89.09 4 0.4 0.5 89.83 39B / non-contextual 0 0.4 0.5 69.50 1 0.4 0.5 76.97 2 0.4 0.5 79.08 3 0.4 0.5 79.38 4 0.4 0.5 80.51

Slide 71

Slide 71 text

HyperCLOVA’s LM vs BERT-large TASK: RCQA possible only (௨ৗͷRCQAλεΫ͔Β౴͑ΒΕͳ͍࣭໰Λ࡟আ͠·ͨ͠) - ਅʹࠔ೉ͳ໰୊ΛHyperCLOVAͰղܾͰ͖ͳ͍Մೳੑ͕͋Γ·͢ - normal => possible only: train: 43610 => 21091, dev: 6863 => 3524, test: 6178 => 3079 - BERT͸ɺಛఆͷλεΫʹରͯ͠ඍௐ੔͢Δ͜ͱͰɺΑΓߴ͍݁ՌΛಘΒΕΔՄೳੑ͕͋Δ - HyperCLOVA͸PromptingͱϥϑͳύϥϝʔλݕࡧͰBERTͱಉ౳ͷύϑΥʔϚϯεΛ࣮ݱͰ͖Δ test acc test F1 memo HyperCLOVA 85.03 89.95 JP 39B 2-shots,   temperature=0.4, top_p= 0.5 BERT-jp-large 86.68 90.49 Using subset of LINE LM corpus

Slide 72

Slide 72 text

Slide 73

Slide 73 text

೔ຊޠͷ೉͠͞ शಘͷ೉͠͞ ೔ຊޠͷ࿩ऀ͸ - ͻΒ͕ͳ - ยԾ໊ - ׽ࣈ - ϩʔϚࣈ ͳͲͳͲΛ ҰͭͷจॻΛॻͨ͘Ίʹ࢖͏ େྔͷඞਢޠኮ ೔ৗձ࿩ʹඞཁͳޠኮ਺ - 8,000ޠҎ্ ଟ͘ͷ୯ޠΛ஌Δඞཁ͕͋Δ - ಉԻҟٛޠ - ܟޠ - ํݴͳͲ ୯ޠͷলུ ೔ຊޠ࿩ऀ͸ɺจॻதͷҎԼ ͷ୯ޠΛলུ͢Δ͜ͱ͕͋Δ - ओޠ - ໨తޠ লུ͞Εͨ୯ޠ͸Ұҙʹਪ࿦ Ͱ͖ͳ͍Մೳੑ͕͋Δ

Slide 74

Slide 74 text

ςΩετੜ੒ͷ೉͠͞ ੜ੒͞ΕͨςΩετͷ   જࡏతͳϦεΫ ҎԼͷΑ͏ͳٕज़Λ։ൃ ͢Δඞཁ͕͋Γ·͢ɻ - ίʔύεͷ಺༰ͷภΓ΍ දهํ๏ͷվળ - ग़ྗ͞ΕͨςΩετͷਅ ࣮ੑͱ҆શੑͷ֬อ AIྙཧͷ࣮ફ ೖग़ྗςΩετʹ͸༷ʑͳྙཧత ഑ྀ͕ඞཁͰ͢ - ಟੑ - ੑతͳ΋ͷ - ߈ܸతͳ΋ͷ - ๯ಚత - ڴഭత - ΞΠσϯςΟςΟ΁ͷ߈ܸ ຊ࣭తͳධՁͷࣗಈԽ ಈతςΩετੜ੒ͷ݁Ռʹద ༻Ͱ͖ΔϝτϦΫε͕ඞཁ - ࿩୊ੑͷ͋Δίϯςϯπͷਫ਼౓ - ੜ੒͞ΕͨςΩετͷҰ؏ੑ - ໨తͷୡ੒౓ͷ൑ఆ

Slide 75

Slide 75 text

NLPʹHyperCLOVA͸ຊ౰ʹඞཁͳͷ͔ʁ - YES !! ΋͠΋༧ࢉࣥߦ͕ڐ͞ΕΔͳΒʜ - NLPͷྺ࢙͸AIؔ࿈ٕज़ͷൃలͱڧ݁͘ͼ͍͍ͭͯΔ - LINE͸ࣗ෼ͨͪͰϞσϧΛ࡞͓ͬͯ٬༷ʹ࢖ͬͯ΋Β͏ ํ޲ʹਐΈ͍ͨͱߟ͍͑ͯ·͢ Large-scale general-purpose LMs DNN Traditional only ML Rule only Small LM only

Slide 76

Slide 76 text

LINEͷݴޠϞσϧΛOSSͱͯ͠ຊ೥౓தʹ഑෍։࢝ ΋ͪΖΜHyperCLOVAҎ֎Ͱ͢ ੑೳ໨ඪ: LINEͷݴޠϞσϧ for OSS > other OSS ͳݴޠϞσϧ ՄೳͰ͋Ε͹ɺෳ਺೥ؒ͸ఆظతʹߋ৽͍ͨ͠ͱߟ͍͑ͯ·͢ HyperCLOVAʹ࢖͍ͬͯΔίʔύε(LINE LM Corpus)ͷαϒηοτΛ࢖༻͠·͢ !

Slide 77

Slide 77 text

Slide 78

Slide 78 text

·ͱΊ ݴޠϞσϧͱ͸Կ͔ ɹɹֶशͨ͠ίʔύε಺ʹ͓͚Δ೚ҙͷςΩετͷ֬཰෼෍ ݴޠϞσϧ͸Կ͕Ͱ͖Δͷ͔ ɹɹ༩͑ΒΕͨจ຺ʹैͬͯΑΓྑ͍ςΩετΛੜ੒Ͱ͖Δ ݴޠϞσϧ͸ԿΛม͑Δͷ͔ ɹɹAI͕೔ຊޠΛ࢖͏͋ΒΏΔ࡞ۀΛखॿ͚͢ΔଘࡏʹͳΓ   ɹɹਓʑͷ࣌ؒΛઅ໿͢Δ͜ͱͰɺਓʑͷੜ׆Λ๛͔ʹ͍ͯ͘͠ )ZQFS$-07"

Slide 79

Slide 79 text

೥౓ͷ৽ଔ࠾༻͕Φʔϓϯͯ͠·͢ͷͰɺ͝Ԡื͍ͩ͘͞