BERTによる自然言語処理を学ぼう!【 Live!人工知能 #26】 #Live人工知能

by yuky_az

Slide 1

Slide 1 text

ʙ -JWFɛʈٝঐ 㘴㙖㙦㙀㙞೮࿥e㘈㘍㗗㗏㗊ᗴ㗄㗗㗾㗙

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

ΠϯτϩμΫγϣϯ

Slide 4

Slide 4 text

ߨࢣ঺հ զ࠺޾௕ :VLJOBHB"[VNB !ZVLZ@B[ 4"*-BCגࣜձࣾ୅දऔక໾ "*ؔ࿈ͷڭҭɺݚڀ։ൃʹैࣄ ཧֶത࢜ʢ෺ཧֶʣ 6EFNZͰ਺ສਓΛࢦಋ༗໊اۀͰ"*ݚमΛ୲౰ ஶॻʹʮ͸͡ΊͯͷσΟʔϓϥʔχϯάʯͳͲ

Slide 5

Slide 5 text

ίʔεͷಛ௃ • BERTͷجૅΛֶͿ  → TransformerɺBERTͳͲͷࣗવݴޠॲཧٕज़Λֶͼ·͢ • PyTorchʹΑΔ࣮૷  → Google Colaboratory؀ڥͰBERTΛಈ࡞ͤ͞·͢ • ߨ࠲ͷର৅  → ҰาਐΜͩࣗવݴޠॲཧٕज़Λ਎ʹ෇͚͍ͨํ  → σΟʔϓϥʔχϯάʹؔ͢Δجૅతͳ஌ࣝͷ͋Δํ  → ϓϩάϥϛϯάɺӳޠʹ఍߅ײ͕খ͍͞ํ

Slide 6

Slide 6 text

ߨ࠲ͷ಺༰ Section 1. ߨ࠲ͱBERTͷ֓ཁ Section 2. γϯϓϧͳBERTͷ࣮૷ Section 3. BERTͷ࢓૊Έ Section 4. ϑΝΠϯνϡʔχϯάͷ׆༻ Section 5. BERTͷԠ༻

Slide 7

Slide 7 text

ࠓճͷ಺༰ 1. ΠϯτϩμΫγϣϯ 2. ίʔεͷ֓ཁ 3. ࣗવݴޠॲཧͷ֓ཁ 4. Transformerͷ֓ཁ 5. BERTͷ֓ཁ 6. Google Colaboratoryͷ࢖͍ํ

Slide 8

Slide 8 text

ڭࡐͷ঺հ • Pythonͷجૅ

Slide 9

Slide 9 text

ίʔεͷ֓ཁ

Slide 10

Slide 10 text

Section 1. ߨ࠲ͱBERTͷ֓ཁ ࣗવݴޠॲཧɺTransformerɺBERTʹ͍ͭͯ֓ཁΛֶͼ·͢

Slide 11

Slide 11 text

Section 2. γϯϓϧͳBERTͷ࣮૷ ࠷খݶͷPythonͷίʔυͰBERTΛ࣮૷͠·͢

Slide 12

Slide 12 text

Section 3. BERTͷ࢓૊Έ TransformerɺBERTͳͲʹ͍ͭͯ࢓૊ΈΛৄֶ͘͠ͼ·͢

Slide 13

Slide 13 text

Section 4. ϑΝΠϯνϡʔχϯάͷ׆༻ ϑΝΠϯνϡʔχϯάͷ֓ཁɺͦͯ͠BERTʹ͓͚Δ׆༻ʹ͍ͭͯղઆ͠·͢

Slide 14

Slide 14 text

Section 5. BERTͷԠ༻ BERTΛ࢖ͬͯɺࣗવݴޠॲཧͷλεΫʹऔΓ૊Έ·͢

Slide 15

Slide 15 text

ࣗવݴޠॲཧͷ֓ཁ

Slide 16

Slide 16 text

ࣗવݴޠॲཧͱ͸ • ࣗવݴޠͱ͸ɺ೔ຊޠ΍ӳޠͳͲͷզʑ͕ීஈ࢖͏ݴޠͷ͜ͱ • ࣗવݴޠॲཧʢNatural Language ProcessingɺNLPʣͱ͸ɺ  ࣗવݴޠΛίϯϐϡʔλͰॲཧ͢Δٕज़ͷ͜ͱ

Slide 17

Slide 17 text

ࣗવݴޠॲཧͷԠ༻ • ݕࡧΤϯδϯ • ػց຋༁ • ༧ଌม׵ • εύϜϑΟϧλ • Ի੠Ξγελϯτ • খઆͷࣥච • ର࿩γεςϜ • etc…

Slide 18

Slide 18 text

ࣗવݴޠॲཧٕज़ͷཁૉ • ܗଶૉղੳ  → จॻΛ୯ޠʹ෼ׂ͢Δٕज़ • ୯ޠͷ෼ࢄදݱ  → จॻ಺Ͱͷؔ܎ੑΛ౿·͑ͯɺ୯ޠΛϕΫτϧԽ͢Δٕज़ • ࠶ؼܕχϡʔϥϧωοτϫʔΫʢRNNʣ  → ࣌ܥྻΛѻ͏ͷ͕ಘҙͳχϡʔϥϧωοτϫʔΫͷҰछ • Seq2Seq  → RNNΛϕʔεʹͨ͠ɺจষͳͲΛੜ੒ՄೳͳϞσϧ • etc…

Slide 19

Slide 19 text

೔ຊޠͷܗଶૉղੳ • ܗଶૉͱ͸ɺݴ༿͕ҙຯΛ࣋ͭ·ͱ·Γͷ୯ޠͷ࠷খ୯Ґͷ͜ͱ • ܗଶૉղੳͱ͸ɺࣗવݴޠΛܗଶૉʹ·Ͱ෼ׂ͢Δ͜ͱ • ೔ຊޠ΍தࠃޠɺλΠޠ͸୯ޠؒʹεϖʔε͕ແ͍ͷͰɺ  ܗଶૉղੳ͕ඞཁ • ҎԼ͸୅දతͳ೔ຊޠͷܗଶૉղੳϥΠϒϥϦ • MeCab → ஌໊౓͕ߴ͘ɺߴ଎ɺߴਫ਼౓ • Janome → ଎౓͸MeCabʹྼΔ͕ɺಋೖ͕؆୯ • etc…

Slide 20

Slide 20 text

one-hotදݱ ͢΋΋ ΋ ΋΋ ΋ ΋΋ ͷ ͏ͪ ͢΋΋ ΋ ΋΋ ͷ ͏ͪ *% ʮ͢΋΋ʯͷone-hotදݱ: [1 0 0 0 0] ʮ΋ʯͷone-hotදݱ: [0 1 0 0 0]

Slide 21

Slide 21 text

෼ࢄදݱ • ୯ޠؒͷؔ࿈ੑ΍ྨࣅ౓ʹجͮ͘ϕΫτϧͰɺ୯ޠΛදݱ͢Δ ʜ ʜ ʜ 200ཁૉఔ౓ உੑ ϩϯυϯ Python • word2vecͳͲΛ࢖͑͹ɺ଍͠ࢉ΍Ҿ͖ࢉ͕ՄೳͳϕΫτϧΛ࡞Δ͜ͱ͕Ͱ͖Δ  ྫ: ʮԦʯ-ʮஉʯ+ʮঁʯ=ʮঁԦʯ

Slide 22

Slide 22 text

word2vec • word2vec͸ɺ෼ࢄදݱΛ࡞੒͢ΔͨΊͷٕज़ • word2vecͰ͸ɺCBOWʢcontinuous bag-of-wordsʣ΋͘͠͸ɺ  skip-gramͱ͍͏χϡʔϥϧωοτϫʔΫ͕༻͍ΒΕΔ

Slide 23

Slide 23 text

ॏΈ CBOWʢcontinuous bag-of-wordsʣ • લޙͷ୯ޠ͔Βର৅ͷ୯ޠΛ༧ଌ͢ΔχϡʔϥϧωοτϫʔΫ • ֶशʹཁ͢Δ͕࣌ؒskip-gramΑΓ΋୹͍ ͢ ΋΋ ΋΋ ΋ ೖྗ૚ தؒ૚ ग़ྗ૚ one-hotදݱ ॏΈ ॏΈΛද͢ߦྻ Wmid Wout ֶशʹΑΓɺWmid ͸෼ࢄදݱͷϕΫτϧ͕ฒΜͩߦྻʹͳΔ ΋ ෼ࢄදݱͱಉ͡ ཁૉ਺ͷϕΫτϧ

Slide 24

Slide 24 text

skip-gram ॏΈ • ͋Δ୯ޠ͔Βɺલޙͷ୯ޠΛ༧ଌ͢ΔχϡʔϥϧωοτϫʔΫ • CBOWΑΓ΋ֶशʹ͕͔͔࣌ؒΔ͕ɺਫ਼౓͕Α͍ ͢ ΋΋ ΋΋ ΋ ೖྗ૚ தؒ૚ ग़ྗ૚ one-hotදݱ ॏΈ ॏΈΛද͢ߦྻ Wmid Wout ֶशʹΑΓɺWmid ͸෼ࢄදݱͷ ϕΫτϧ͕ฒΜͩߦྻʹͳΔ ΋ ෼ࢄදݱͱಉ͡ ཁૉ਺ͷϕΫτϧ

Slide 25

Slide 25 text

࠶ؼܕχϡʔϥϧωοτϫʔΫʢRNNʣͱ͸ʁ • ࠶ؼܕχϡʔϥϧωοτϫʔΫʢReccurent Neural Networkʣ  → ೖྗͱਖ਼ղ͕ʮ࣌ܥྻσʔλʯͱͳΔ  → தؒ૚͕ʮ࠶ؼʯͷߏ଄Λ࣋ͪɺલޙͷ࣌ࠁͷதؒ૚ͱͭͳ͕Δ ೖྗ૚ தؒ૚ ग़ྗ૚ ࠶ؼ

Slide 26

Slide 26 text

RNNͷॱ఻೻ͱٯ఻೻ ೖྗ ૚ தؒ ૚ ग़ྗ ૚ ೖྗ ૚ தؒ ૚ ग़ྗ ૚ ೖྗ ૚ தؒ ૚ ग़ྗ ૚ t = 0 t = 1 t = 2 ॱ఻೻  ٯ఻೻ • ࣌ؒΛḪͬͯٯ఻೻͕ߦΘΕΔ • શͯͷ࣌ࠁͷதؒ૚Ͱɺ  ύϥϝʔλ͸ڞ༗͞ΕΔ

Slide 27

Slide 27 text

࣌ܥྻσʔλͷྫ • จॻ • Ի੠σʔλ • ಈը • גՁ • ࢈ۀ༻ػثͷঢ়ଶ • etc…

Slide 28

Slide 28 text

Seq2Seqͱ͸ • Seq2Seq͸ɺܥྻ(sequence)Λ ड͚औΓɺผͷܥྻ΁ม׵͢ΔϞσϧ • ࣗવݴޠॲཧͰΑ͘ར༻͞ΕΔ • จষͳͲͷೖྗΛѹॖ͢Δencoderͱɺग़ྗΛల։͢Δdecoder͔ΒͳΔ • ҎԼ͸׆༻ྫ • ػց຋༁ʢྫ: ӳޠͷจষ → ϑϥϯεޠͷจষʣ • จষཁ໿ʢݩͷจষ → ཁ໿จʣ • ର࿩ʢࣗ෼ͷൃݴ → ૬खͷൃݴʣ • etc…

Slide 29

Slide 29 text

Seq2Seqͷߏ଄ 3// ૚ Seq2SeqʹΑΔ຋༁ͷྫ ޗഐ ͸ ೣ ͩ I am a cat encoder decoder

Slide 30

Slide 30 text

͞Βʹৄֶ͘͠ͼ͍ͨํ΁… 6EFNZίʔε ࣗવݴޠॲཧͱνϟοτϘοτ "*ʹΑΔจষੜ੒ͱձ࿩Τϯδϯ։ൃ 6EFNZίʔε ਓ޻஌ೳʢ"*ʣΛ౥ࡌͨ͠5XJUUFSϘοτΛ࡞Ζ͏ ʲ4FR4FR"UUFOUJPO$PMBCʳ

Slide 31

Slide 31 text

Transformerͷ֓ཁ

Slide 32

Slide 32 text

RNNʹΑΔࣗવݴޠॲཧͷ໰୊఺ • ֶश͕࣌ؒ௕͍  → σʔλΛฒྻͰॲཧͰ͖ͳ͍ͨΊɺ  ֶशʹ͸௕͍͕͔͔࣌ؒΔ • จ຺ΛͱΒ͑Δͷ͕೉͍͠  → ௕࣌ؒͷؔ܎ੑΛͱΒ͑Δͷ͕ۤख

Slide 33

Slide 33 text

Transformerͷ֓ཁ • Transformerͱ͸ʁ  → 2017೥ʹಋೖ͞ΕͨσΟʔϓϥʔχϯάϞσϧͰɺ  ओʹࣗવݴޠॲཧͷ෼໺Ͱ࢖༻͞ΕΔ   → RNNͱಉ༷ʹɺࣗવݴޠͳͲͷ࣌ܥྻσʔλΛॲཧ͢ΔΑ͏ʹઃܭ ͞Ε͍ͯΔ͕ɺRNNͰ༻͍Δ࠶ؼɺCNNͰ༻͍Δ৞ΈࠐΈ͸࢖Θͳ͍  → Attention૚ͷΈͰߏங͞ΕΔ  → ຋༁΍ςΩετͷཁ໿ͳͲɺ༷ʑͳλεΫͰར༻Մೳ  → ฒྻԽ͕༰қͰ͋Γɺ܇࿅࣌ؒΛେ͖͘࡟ݮͰ͖Δ

Slide 34

Slide 34 text

Transformerͷ࿦จ • Attention Is All You Need  → https://arxiv.org/abs/1706.03762  • ʮAttentionʯ͸࣌ܥྻσʔλͷಛఆͷ෦෼ʹ  ஫ҙΛ޲͚ΔΑ͏ʹֶश͍ͤͯ͘͞ํ๏

Slide 35

Slide 35 text

TransformerͷϞσϧ "UUFOUJPO*T"MM:PV/FFE "TIJTI 7FUBM ΑΓҾ༻ &ODPEFS %FDPEFS &ODPEFSͷߏ଄ ̍ɽ&NCFEEJOH૚ʹΑΓೖྗจষΛϕΫτϧʹѹॖ ̎ɽ1PTJUJPOBM&ODPEFS૚ʹΑͬͯҐஔ৘ใΛՃ͑Δ ̏ɽ.VMUJ)FBE"UUFOUJPO૚ ̐ɽOPSNBMJ[BUJPOʢਖ਼نԽʣͳͲ ̑ɽ1PTJUJPOXJTFGVMMZDPOOFDUFEGFFEGPSXBSEOFUXPSL ̒ɽOPSNBMJ[BUJPOʢਖ਼نԽʣͳͲ Λճ܁Γฦ͢

Slide 36

Slide 36 text

TransformerͷϞσϧ "UUFOUJPO*T"MM:PV/FFE "TIJTI 7FUBM ΑΓҾ༻ &ODPEFS %FDPEFS %FDPEFSͷߏ଄ ̍ɽ&NCFEEJOH૚ʹΑΓೖྗจষΛϕΫτϧʹѹॖ ̎ɽ1PTJUJPOBM&ODPEFS૚ʹΑͬͯҐஔ৘ใΛՃ͑Δ ̏ɽ.VMUJ)FBE"UUFOUJPO૚ ̐ɽOPSNBMJ[BUJPOʢਖ਼نԽʣͳͲ ̑.VMUJ)FBE"UUFOUJPO૚ʢ&ODPEFSͷೖྗΛ࢖༻ʣ ̒ɽOPSNBMJ[BUJPOʢਖ਼نԽʣͳͲ ̓ɽ1PTJUJPOXJTFGVMMZDPOOFDUFEGFFEGPSXBSEOFUXPSL ̔ɽOPSNBMJ[BUJPOʢਖ਼نԽʣͳͲ Λճ܁Γฦ͢

Slide 37

Slide 37 text

BERTͷ֓ཁ

Slide 38

Slide 38 text

BERTͷ֓ཁ • BERTʢBirdirectional Encoder Representation from Transformersʣͱ͸ʁ  → 2018೥ͷޙ൒ʹGoogle͔Βൃද͞Εͨɺ  ࣗવݴޠॲཧͷͨΊͷ৽ͨͳσΟʔϓϥʔχϯάͷϞσϧ   → Transformer͕ϕʔεͱͳ͍ͬͯΔ  → ༷ʑͳࣗવݴޠॲཧλεΫͰϑΝΠϯνϡʔχϯά͕Մೳ  → ैདྷͷࣗવݴޠॲཧλεΫͱൺֱͯ͠ɺߴ͍൚༻ੑ

Slide 39

Slide 39 text

BERTͷ࿦จ • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding  → https://arxiv.org/abs/1810.04805

Slide 40

Slide 40 text

BERTͷֶश #&351SFUSBJOJOHPG%FFQ#JEJSFDUJPOBM5SBOTGPSNFSTGPS-BOHVBHF6OEFSTUBOEJOH %FWMJO +FUBM ΑΓҾ༻

Slide 41

Slide 41 text

BERTͷֶश • ࣄલֶश  → Transformer͕ɺจষ͔Βจ຺Λ૒ํ޲ʢBidirectionalʣʹֶश͢Δ  → Masked Language Model͓ΑͼNext Sentence PredictionʹΑΔ૒ํ޲ֶश • ϑΝΠϯνϡʔχϯά  → ࣄલֶशʹΑΓಘΒΕͨύϥϝʔλΛॳظ஋ͱͯ͠ɺϥϕϧ෇͖ͷσʔλͰɹ ϑΝΠϯνϡʔχϯάΛߦ͏

Slide 42

Slide 42 text

Masked Language Model • Masked Language Model  → จষ͔Βಛఆͷ୯ޠΛ15ˋϥϯμϜʹબͼɺ[MASK]τʔΫϯʹஔ͖׵͑Δ  → ྫ: my dog is hairy → my dog is [MASK]  → [MASK]ͷ୯ޠΛɺલޙͷจ຺͔Β༧ଌ͢Δ

Slide 43

Slide 43 text

Next Sentence Prediction • Next Sentence Prediction  → 2ͭͷจষʹؔ܎͕͋Δ͔Ͳ͏͔Λ൑ఆ͢Δ  → ޙΖͷจষΛ50%ͷ֬཰Ͱແؔ܎ͳจষʹஔ͖׵͑Δ  → ޙΖͷจষ͕ҙຯతʹద੾Ͱ͋Ε͹IsNextɺͦ͏Ͱͳ͚Ε͹NotNextͷ൑ఆ • [CLS] the man went to [MASK] store [SEP] / he bought a gallon [MASK] milk [SEP]  ൑ఆɿIsNext  [CLS] the man went to [MASK] store [SEP] / penguin [MASK] are ﬂight #less birds [SEP]  ൑ఆɿNotNext

Slide 44

Slide 44 text

BERTͷੑೳ #&351SFUSBJOJOHPG%FFQ#JEJSFDUJPOBM5SBOTGPSNFSTGPS-BOHVBHF6OEFSTUBOEJOH %FWMJO +FUBM ΑΓҾ༻ • SQuAD  → ʮStanford Question Answering Datasetʯͷུ  → ελϯϑΥʔυେֶ͕Ұൠެ։͍ͯ͠Δ  ݴޠॲཧͷਫ਼౓ΛଌΔϕϯνϚʔΫ  → σʔλ͸໿10ສݸͷ࣭໰Ԡ౴ͷϖΞΛؚΉ

Slide 45

Slide 45 text

BERTͷੑೳ #&351SFUSBJOJOHPG%FFQ#JEJSFDUJPOBM5SBOTGPSNFSTGPS-BOHVBHF6OEFSTUBOEJOH %FWMJO +FUBM ΑΓҾ༻ • GLUE  → ࣗવݴޠॲཧͷͨΊͷ9छྨͷֶशσʔλΛؚΉσʔληοτ

Slide 46

Slide 46 text

೔ຊޠͷ܇࿅ࡁΈϞσϧ • ژ౎େֶ ࠇڮɾ㬗ɾଜ࿬ݚڀࣨ  → http://nlp.ist.i.kyoto-u.ac.jp/index.php?BERT೔ຊޠPretrainedϞσϧ

Slide 47

Slide 47 text

Google Colaboratoryͷ࢖͍ํ

Slide 48

Slide 48 text

Google Colaboratoryͱ͸? • Google Colaboratory  → Google͕ఏڙ͢Δɺϒϥ΢βͰPythonΛ࣮ߦͰ͖Δ؀ڥ  → GoogleΞΧ΢ϯτͰར༻Մೳ  → جຊతʹແྉ  → ؀ڥߏங͕؆୯  → ڞ༗͕؆୯  → etc…  https://colab.research.google.com/

Slide 49

Slide 49 text

ίʔυηϧͱςΩετηϧ • ίʔυηϧ  → PythonͷίʔυΛهड़͠ɺ࣮ߦ͢Δ • ςΩετηϧ  → จষ΍਺ࣜΛهड़͢Δ

Slide 50

Slide 50 text

Google Colaboratoryͷ༷ʑͳػೳ • εΫϥονίʔυηϧ • ίʔυεχϖοτ • ϑΥʔϜ • etc…

Slide 51

Slide 51 text

Section 1. ߨ࠲ͱBERTͷ֓ཁ Section 2. γϯϓϧͳBERTͷ࣮૷ Section 3. BERTͷ࢓૊Έ Section 4. ϑΝΠϯνϡʔχϯάͷ׆༻ Section 5. BERTͷԠ༻ ࣍ճͷ಺༰

Slide 52

Slide 52 text

ࡶஊλΠϜ -JWFɛʈٝঐ 㘴㙖㙦㙀㙞೮࿥e㘈㘍㗗㗏㗊ᗴ㗄㗗㗾㗙