Tweet
Tweet

Slide 1

Slide 1 text

θϩ͔Β࡞ΔDeep Learning 2 ̏ষ word2vec 
 3.1ʙ3.2 ota42y θϩ͔Β࡞ΔDeep Learning 2 ࣗવݴޠฤ ಡॻձ ୈ5ճ

Slide 2

Slide 2 text

͜ͷষͰ΍Δ͜ͱ • word2vecΛ࣮૷͢Δ • ਪ࿦ϕʔεͰ୯ޠΛϕΫτϧͰද͢ํ๏ • γϯϓϧ͕ͩແବ͸ଟ͍࣮૷ • ଎౓͸࣍ͷষͰରԠ

Slide 3

Slide 3 text

3.1 ਪ࿦ϕʔεͷख๏ͱ χϡʔϥϧωοτϫʔΫ

Slide 4

Slide 4 text

ਪ࿦ϕʔεͷϕΫτϧԽ • ୯ޠΛϕΫτϧʹ͢Δ̎ͭͷख๏ • Χ΢ϯτϕʔεʢ̎ষʣ • ਪ࿦ϕʔεʢ̏ষʣ • ͲͪΒ΋෼෍ԾઃΛϕʔεʹͯ͠Δ͕Ξϓϩʔν͸શ͘ผ • ෼෍Ծઃɿ୯ޠͷҙຯ͸पғͷ୯ޠ͔Βܗ੒͞ΕΔ (p.67)

Slide 5

Slide 5 text

3.1.1ɹΧ΢ϯτϕʔεͷख๏ͷ໰୊఺ • Χ΢ϯτϕʔε͸पғͷ୯ޠͷස౓Λܭࢉ͢Δ • ޠኮ਺͕nͩͱn*nͷڊେͳڞىߦྻ͕ඞཁʹͳΔ • ࣍ݩ࡟ݮͷͨΊͷSVD͸O(n^3)ͷܭࢉྔɺ΍͹͍

Slide 6

Slide 6 text

ਪ࿦ϕʔεͷར఺ • Χ΢ϯτϕʔε͸ίʔύεશମͷ౷ܭσʔλΛҰؾʹར༻͢Δ • ਪ࿦ϕʔε(χϡʔϥϧωοτ)͸ίʔύεͷҰ෦Ͱֶश͢Δ • GPUͷฒྻܭࢉ΋ฉ͘ • খ෼͚ʹͰ͖ɺߴ଎ʹฒྻॲཧͰ͖ΔͷͰڊେσʔλͰ΋ରԠͰ͖Δ • ଞʹ΋ັྗతͳ఺͕͋Δ(Β͍͠ɺৄ͘͠͸3.5.3)

Slide 7

Slide 7 text

3.1.2ɹਪ࿦ϕʔεͷख๏ͷ֓ཁ

Slide 8

Slide 8 text

पғͷ୯ޠ͔Β୯ޠΛʮਪ࿦ʯ͢Δ • `?`ʹ͸Կ͕ೖΔ͔Λલޙ͔Βਪ࿦ • ίϯςΩετ͔ΒλʔήοτΛਪ࿦ • ίϯςΩετɿपғͷ୯ޠ(you, goodby) • λʔήοτɿର৅ͷ୯ޠ(`?`)

Slide 9

Slide 9 text

ਪ࿦݁Ռ • ֤୯ޠ͕ͦ͜ʹݱΕΔ֬཰Λग़ྗ • ίϯςΩετΛϞσϧʹ༩͑Δͱ୯ޠͷ֬཰෼෍͕ಘΒΕΔ

Slide 10

Slide 10 text

3.1.3 χϡʔϥϧωοτϫʔΫʹ͓͚Δ୯ ޠͷॲཧํ๏ • χϡʔϥϧωοτϫʔΫ(NN)ͷೖྗ͸ݻఆ௕ϕΫτϧ • ୯ޠΛͦͷ··ೖΕΔͷ͸೉͍͠ • ୯ޠΛone-hotදݱ(one-hotϕΫτϧ)ʹม׵͢Δ

Slide 11

Slide 11 text

one-hotදݱ • ޠኮ਺ͷ௕͞Λ࣋ͪɺ୯ޠIDͱ֘౰͢Δ෦෼͕1ɺͦΕҎ֎͕0 ͷϕΫτϧ • ͢΂ͯͷ୯ޠΛಉ͡௕͞ͷϕΫτϧͱͯ͠දݱ

Slide 12

Slide 12 text

one-hotදݱ • શ݁߹૚Ͱม׵͢ΔͳΒ؆୯(ྫ͸தؒ૚=3)

Slide 13

Slide 13 text

αϯϓϧίʔυ(p.99) • np.dot(c, W)͸୯ޠʹରԠ͢ΔॏΈΛऔΓग़ͯ͠Δ͚ͩ • W[0]ͷσʔλΛऔΓग़ͯ͠Δ͚ͩ • ແବͬΆ͍͕࣍ͷষͰ࣏͢Β͍͠

Slide 14

Slide 14 text

ϨΠϠදݱ • MatMulϨΠϠ(p.30)Ͱ΋ಉ͜͡ͱ͕Ͱ͖Δ • np.dot͢Δ͚ͩͷϨΠϠͳͷͰ

Slide 15

Slide 15 text

3.2ɹγϯϓϧͳword2vec

Slide 16

Slide 16 text

word2vecΛ࣮૷͢Δ • word2vecͰ࢖ΘΕΔϞσϧ͸CROWϞσϧͱskip-gramϞσϧ • "word2vec"͕͜ΕΒͷϞσϧΛࢦ͢৔߹΋͋Δ • ຊདྷͷҙຯͱ͸ζϨͯΔ

Slide 17

Slide 17 text

3.2.1 CBOWϞσϧͷਪ࿦ॲཧ • ίϯςΩετ͔ΒλʔήοτΛਪଌ͢ΔNN • ίϯςΩετʹपғͷ୯ޠ • λʔήοτʹର৅ͷ୯ޠ

Slide 18

Slide 18 text

୯ޠͷ෼ࢄදݱ • CBOWϞσϧΛ܇࿅͢Δ͜ͱͰ୯ޠͷ෼ࢄදݱΛಘΒΕΔ • Ϟσϧͷύϥϝʔλ͕෼ࢄදݱʹରԠ͢Δ

Slide 19

Slide 19 text

CBOWϞσϧͷશମ૾ • ίϯςΩετʹ̎ɺӅΕ૚ʹ̏ͷ৔߹

Slide 20

Slide 20 text

CBOWϞσϧͷશମ૾ • ೖྗ͸ෳ਺ݸͷone-hotදݱͷ୯ޠ • ग़ྗ͸֤୯ޠͷείΞ • softmaxΛ࢖͏ͱ֬཰͕ಘΒΕΔ • தؒ૚͸ೖྗ૚͔Βͷ஋ͷฏۉ

Slide 21

Slide 21 text

• ෼ࢄදݱͷਖ਼ମ • [$ W_{in}]͸7*3ͷॏΈ • ͜Ε͕୯ޠͷ෼ࢄදݱ • ֶशʹΑͬͯྑ͍෼ࢄදݱʹ͍ͯ͘͠

Slide 22

Slide 22 text

CBOWϞσϧͷϨΠϠදݱ

Slide 23

Slide 23 text

CBOWϞσϧͷϨΠϠදݱ • ̎ͭͷMatMulϨΠϠ • ୯ޠʹରԠ͢ΔॏΈΛऔΓग़͢΍ͭ(P.99) • ̎ͭͷฏۉΛऔΔ(=଍ͯ͠0.5Λ͔͚Δ) • score΁ͷશ݁߹૚ • ׆ੑԽؔ਺͸ແ͍ͷͰΘΓͱγϯϓϧ

Slide 24

Slide 24 text

3.2.2 CBOWϞσϧͷֶश • χϡʔϥϧωοτϫʔΫͷηΦϦʔ௨Γ • CBOW͸ଞΫϥε෼ྨΛ͢ΔNN • Ϋϥεʹone-hotͰද͞Εͨ୯ޠ • είΞ͔Β֬཰ΛٻΊͯɺਖ਼ղͱͷࠩΛֶश͢Δ • Softmaxؔ਺ʹ͔͚ͯ֬཰ʹ͢Δ • ڭࢣϥϕϧ͔ΒަࠩΤϯτϩϐʔޡࠩΛٻΊΔ

Slide 25

Slide 25 text

ϨΠϠදݱ • Softmax with lossΛ෇͚Ճ͑Δ

Slide 26

Slide 26 text

ίʔυϦʔσΟϯά • ch03/cbow_predict.py • https://github.com/oreilly-japan/deep-learning-from- scratch-2/blob/master/ch03/cbow_predict.py

Slide 27

Slide 27 text

3.2.3 word2vecͷॏΈͱ෼ࢄදݱ • ͱɹɹͷҧ͍ • ྆ํͱ΋΋୯ޠͷҙຯ͕Τϯίʔυ͞Ε͍ͯΔ • ܗঢ়͕ҧ͏ • ɹɹ͸7x3 • ɹɹ͸3x7 Win Wout Win Wout

Slide 28

Slide 28 text

෼ࢄදݱ͸ɹɹΛ࢖͏ • ɹɹ ͸શ͘࢖Θͳ͍ɹ • ɹɹʹର͢Δskip-ngramͰͷ༗༻ੑ࣮ݧ • https://arxiv.org/abs/1611.01462 • ɹɹ΋࢖͏͜ͱͰΑ͍݁Ռ͕ಘΒΕΔͱ͍͏ใࠂ΋ • https://nlp.stanford.edu/projects/glove/ • word2vecͱࣅ͍ͯΔ΍ͭͷख๏ Win Win Wout Wout