Share
Embed
Transformerは何をやっているのか
Transformerが結局のところ何をやっているのかを図メインで説明しています。
研究室の論文読み会で用いた資料の一部です。
Yusuke-Takagi-Q
April 24, 2021
Transcript
42 slides
-
͡Ίʹ • ग़ͯ͘Δਤશͯࣗ࡞ or จ ”Attention Is All You Need”
͔ΒͷҾ༻ • จಡΈձʹͯ”Attention Is All You Need”Λհͨ͠ࡍͷҰ෦ͳͷͰ, આ໌ݩจʹԊ͍ͬͯ·͢ • Transformer ͕ԿΛ͍ͬͯΔͷ͔ΛϝΠϯͰઆ໌͍ͯ͠ΔͷͰ, ܇࿅ํ ๏ֶशޙͷਪํ๏, ػց༁Ҏ֎ͷΈͳͲղઆ͍ͯ͠·ͤΜ 1 / 41 -
Overview : Training flow • Transformer ͷߏΛઆ໌͢Δલʹ܇࿅ͷྲྀΕΛ֬ೝ • ػց༁ͷઃఆͰҎԼͷਤͷΑ͏ͳྲྀΕ &ODPEFS
%FDPEFS * IBWF B QFO ࢲ ϖϯ Λ ࣋ͭ ɻ &04 ࢲ ϖϯ Λ ࣋ͭ ɻ &04 <#04> ࢲ ϖϯ Λ ࣋ͭ ɻ &04 4IJGUˠ 1SFEJDU "OTXFS 'SPNEBUBTFU 5SBOTGPSNFS 2 / 41 -
Overview : Training flow • ྫ͑ Transformer ͕ઌ಄ͷޠΛ༧ଌ͢Δͱ͖ ”I have
a pen.” ͱ ”<BOS>”ͷใ͕༻Մೳ • <BOS> ಛผͳτʔΫϯͰจͷ࢝·ΓΛද͢ 3 / 41 &ODPEFS %FDPEFS * IBWF B QFO ࢲ ϖϯ Λ ࣋ͭ ɻ &04 ࢲ ϖϯ Λ ࣋ͭ ɻ &04 <#04> -
Overview : Training flow • Transformer ͕ 2 ͭͷޠΛ༧ଌ͢Δͱ͖ ”I
have a pen.” ͱ ”<BOS> ࢲ” ͷใ͕༻Մೳ 4 / 41 &ODPEFS %FDPEFS * IBWF B QFO ࢲ ϖϯ Λ ࣋ͭ ɻ &04 ࢲ ϖϯ Λ ࣋ͭ ɻ &04 <#04> -
Overview : Model architecture • Transformer ԼͷਤͷΑ͏ʹ self-attention ͱ fully
connected layers ΛελοΫͨ͠ߏͱͳ͍ͬͯΔ • Ҏ߱ 5 ͭͷύʔτʹؔͯ͠ৄ͘͠આ໌͢Δ Figure 1: The Transformer - model architecture. ᶃ ᶄ ᶅ ᶆ ᶇ 5 / 41 -
Part 1 : Overall • ෮श : Transformer ͷશମ૾ Figure
1: The Transformer - model architecture. ᶃ ᶄ ᶅ ᶆ ᶇ 6 / 41 -
Part 1 : Embedding and Positional Encoding • จΛ Transformer
ʹೖྗ͢ΔͨΊʹ୯ޠΛϕΫτϧʹม͢Δඞཁ͕ ͋Δ • ͜͜ͰֶशࡁΈͷ embedding Λ༻͍ͯ input ͱ output ͷ୯ޠΛ dmodel ࣍ݩͷϕΫτϧʹม • <EOS> จͷऴΘΓΛද͢ಛผͳτʔΫϯ • <pad> จͷ͞Λଗ͑ΔͨΊͷύσΟϯάΛද͢τʔΫϯ *IBWFBQFO IBWF B QFO * &04 QBE QBE 𝑑!"#$% 7 / 41 -
Part 1 : Embedding and Positional Encoding • จͷߦྻʹ૬ରత or
ઈରతͳ୯ޠͷҐஔใ ΛՃ͑Δඞཁ͕͋Δ • Transformer ʹ࠶ؼߏΈࠐΈͳͲॱংΛߟྀͰ͖Δॲཧ͕ଘࡏ͠ ͳ͍͔Β • ”I have a pen .” ͱ ”I pen . a have” ͷҧ͍͕ encode Ͱ͖ͳ͍ • ͜ͷจͰ Positional Encoding ʹҎԼͷࣜΛ༻͍͍ͯΔ PE(pos,2i) = sin ( pos/100002i/dmodel ) PE(pos,2i+1) = cos ( pos/100002i/dmoddl ) • pos ୯ޠҐஔͰ i ϕΫτϧͷ࣍ݩ 8 / 41 -
Part 1 : Embedding and Positional Encoding 𝑑!"#$% 𝑃𝐸(',)) =
sin (7/10000)/#!"#$% ) 𝑃𝐸(,,-) = cos (6/10000)/#!"#$% ) 1PTJUJPOBM&ODPEJOH IBWF B QFO * &04 QBE QBE 𝑑!"#$% 9 / 41 -
Part 1 : Embedding and Positional Encoding • ͳͥ Positional
Encoding ʹ͜ͷؔΛ༻͍͍ͯΔͷ͔ʁ ⇒ ஶऀͨͪΦϑηοτ k Λݻఆͨ࣌͠ʹ PEpos+k ͕ PEpos ͷઢܗؔ ͰදݱͰ͖Δ͜ͱ͔Β, Ϟσϧ͕؆୯ʹ૬ରతͳҐஔؔΛֶशՄೳͳ ͣͱͷԾઆΛཱ͍ͯͯΔ • ඞͣ͜͠ͷؔͰͳ͍ͱ͍͚ͳ͍༁Ͱͳ͍ ui = 1 100002i/dmodel PEpos,2i = sin (posui) PEpos,2i+1 = cos (posui) PEpos+k,2i = sin ((pos + k)ui) = PEpos,2i cos (kui) + PEpos,2i+1 sin (kui) 10 / 41 -
Part 2 : Overall • ෮श : Transformer ͷશମ૾ Figure
1: The Transformer - model architecture. ᶃ ᶄ ᶅ ᶆ ᶇ 11 / 41 -
Part 2 : Encoder self-attention • Attention ͱ query ͱ
keyɾvalue ϖΞͷηοτΛग़ྗʹϚοϐϯά͢ Δͷͱઆ໌Ͱ͖Δ • query, key, value, output શͯϕΫτϧ • output value ͷॏΈ͖Ͱܭࢉ͞ΕΔ • ֤ʹׂΓͯΒΕΔॏΈ, query ͱରԠ͢Δ key ʹΑͬͯܭࢉ͞ΕΔ • ݴ༿ͩͱΑ͔͘Βͳ͍ͷͰҎ߱ͷεϥΠυͰਤΛަ͑ͯ self-attention ͷॲཧΛઆ໌ 13 / 41 -
Part 2 : Encoder self-attention • ͡Ίʹ, query, key, value
Λ Source, Target ͔Βύϥϝʔλ W q , W k , W v Λ༻͍ͯܭࢉ • Source = Target ⇒ self-attention IBWF B QFO * &04 QBE QBE IBWF B QFO * &04 QBE QBE 4PVSDF 5BSHFU 4BNF JOQVU ˠ 4FMG"UUFOUJPO 7BMVF ,FZ 2VFSZ 𝑆𝑊! 𝑆𝑊" 𝑇𝑊# 14 / 41 -
Part 2 : Encoder self-attention • ࣍ʹ softmax(QK⊤) Λܭࢉ •
ײతղऍΛ͢Δͱ, ಘΒΕΔߦྻ input จͷ୯ޠؒͷؔੑΛදͨ͠ ͷʹͳΔ • ྫ͑, ”I” ͱ ”have” ͱ͍ͯؔͯͦ͠͏͕ͩ, ”I” ͱ ”a” ͦΕ΄ ͲͰͳͦ͞͏ ͷΑ͏ͳͷ 7BMVF ,FZ 2VFSZ 𝑄𝐾! 4PGUNBY IBWF B QFO * &04 QBE QBE IBWF * B QFO &04 QBE QBE 3FQSFTFOUBSFMBUJPOTIJQCFUXFFOXPSET 15 / 41 -
Part 2 : Encoder self-attention • ࠷ޙʹ, softmax(QK⊤)V Λܭࢉ͠, Target
ΛՃ͑Δ (residual connection). • ࠷ऴతʹಘΒΕΔߦྻ ݩͷ Target ͷใʹ֤୯ޠͱؔ࿈͢Δ୯ޠ ใΛՃ͑ͨͷ 7BMVF 𝑆𝑜𝑓𝑡𝑚𝑎𝑥(𝑄𝐾!)𝑉 IBWF B QFO * &04 QBE QBE 𝑆𝑜𝑓𝑡𝑚𝑎𝑥(𝑄𝐾!) IBWF B QFO * &04 QBE QBE 5BSHFU ʴ SFTJEVBMDPOOFDUJPO 16 / 41 -
Part 2 : Encoder self-attention • ͏গ͠ softmax(QK⊤)V ͷܭࢉΛৄ͘͠ݟΔͱ 7BMVF
𝑆𝑜𝑓𝑡𝑚𝑎𝑥 𝑄𝐾! IBWF B QFO * &04 QBE QBE * IBWF * B QFO &04QBEQBE IBWF B QFO * QBE QBE IBWF B QFO * QBE QBE 𝑆𝑜𝑓𝑡𝑚𝑎𝑥 𝑄𝐾! [0]𝑉 TVN 3FQSFTFOUBSFMBUJPOTIJQ CFUXFFOl*zBOEJOQVUXPSET 8FJHIUFEWBMVF 17 / 41 -
Part 2 : Encoder self-attention • Encoder ͷ self-attention ྲྀΕ·ͱΊ
IBWF B QFO * &04 QBE QBE IBWF B QFO * &04 QBE QBE 4PVSDF 5BSHFU 7BMVF ,FZ 2VFSZ 𝑆𝑊! 𝑆𝑊" 𝑇𝑊# 𝑄𝐾$ 4PGUNBY ʴ IBWF B QFO * &04 QBE QBE 𝑆𝑜𝑓𝑡𝑚𝑎𝑥(𝑄𝐾$)𝑉 18 / 41 -
Part 2 : Scaled Dot-Product Attention • આ໌Ͱ softmax(QK⊤)V Λ༻͍͖͕ͯͨ,
࣮༻Ͱ softmax(QK⊤ √ dk )V ͷΑ͏ʹεέʔϦϯά͢Δඞཁ͕͋Δ • dk : query ͱ key ͷ࣍ݩ • ͳͥεέʔϦϯά͢Δඞཁ͕ʁ ⇒ ޯফࣦΛආ͚ΔͨΊ • ϕΫτϧͷ࣍ݩ͕૿͑Δͱੵେ͖͘ͳΓ͕ͪͰ͋Γ, ͦͷ··Ͱ sonfmax ͕΄΅ max ͱಉ͡Α͏ͳԋࢉʹͳͬͯ͠·͏ 19 / 41 -
Part 2 : Multi-Head Attention • ͞ΒʹจͰ Multi-Head Attention ͕༻͞Ε͍ͯΔ
• Ҏ߱ͷεϥΠυͰ Multi-Head Attention Λઆ໌͢Δ 20 / 41 -
Part 2 : Multi-Head Attention • ͡Ίʹ, query, key, value
Λ Source, Target ͔Βύϥϝʔλ W q i , W k i , W v i Λ༻͍ͯܭࢉ • h-head attention Ͱ͋Ε, W q i , W k i ∈ Rdmodel×dk , W v i ∈ Rdmodel×dv • dk = dv = dmodel /h, i = 1, . . . , h • ͭ·Γ, h-head ͳΒ query, key, value ͷ࣍ݩ͕ 1-head ࣌ͷ 1/h ʹͳΔ IBWF B QFO * &04 QBE QBE IBWF B QFO * &04 QBE QBE 4PVSDF 5BSHFU 7BMVF ,FZ 2VFSZ 𝑆𝑊 ! " 𝑆𝑊 ! # 𝑇𝑊 ! $ 𝑇𝑊 % $ 𝑆𝑊 % " 𝑆𝑊 % # 21 / 41 -
Part 2 : Multi-Head Attention • ࣍ʹ, ֤ head Ͱ
softmax(QiK⊤ i ) Λܭࢉ 𝑄! 𝐾! " 4PGUNBY ,FZ 2VFSZ 𝑄# 𝐾# " ,FZ 2VFSZ 4PGUNBY )FBE )FBE 22 / 41 -
Part 2 : Multi-Head Attention • ࠷ޙʹ Concat(softmax(QiK⊤ i )Vi)W
O Λܭࢉ͠ Target ΛՃ͑Δ • W O ∈ Rhdv×dmodel ύϥϝʔλ 𝑆𝑜𝑓𝑡𝑚𝑎𝑥(𝑄! 𝐾! ")𝑉! 𝑆𝑜𝑓𝑡𝑚𝑎𝑥(𝑄! 𝐾! ") 7BMVF 7BMVF 𝑆𝑜𝑓𝑡𝑚𝑎𝑥(𝑄# 𝐾# ") 𝑆𝑜𝑓𝑡𝑚𝑎𝑥(𝑄# 𝐾# ")𝑉# 𝐶𝑜𝑛𝑐𝑎𝑡 (ℎ𝑒𝑎𝑑! ,ℎ𝑒𝑎𝑑# )𝑊$ IBWF B QFO * &04 QBE QBE 5BSHFU ʴ 23 / 41 -
Part 2 : Multi-Head Attention • جຊతʹ Multi-Head Attention ͰྲྀΕಉ͡
• ҧ͍ head ͷ͚ͩ Attention ͕ܭࢉ͞ΕͯͦΕΒΛ concat ͯ͠ݩͷ࣍ ݩʹ͢ͱ͜Ζ • Multi-Head Attention ʹ͢Δརෳͷ Attention ͕ܭࢉ͞ΕΔͷͰ ΑΓෳࡶͳؔੑ͕ encode ՄೳͱͳΔͱ͜Ζ 24 / 41 -
Part 2 : Layer Normalization • Transformer Ͱ residual connection
ޙʹ layer normalization Λ༻͍ͯ ਖ਼نԽ͍ͯ͠Δ • ֤୯ޠຖʹਖ਼نԽ • β, γ : ύϥϝʔλ • µi , σi : ֤୯ޠͷฏۉͱඪ४ภࠩ • ϵ : 0 আࢉΛආ͚ΔͨΊͷখ͍͞ఆ LayerNorm(xi) = xi − µi σi + ϵ γ + β ࢲ ϖϯ Λ <#04> ࣋ͭ ɻ <&04> 𝜇! ,𝜎! 𝜇" ,𝜎" 𝜇# ,𝜎# ・・・ 26 / 41 -
Part 2 : Position-wise Feed-Forward Networks • attention sub-layer ͷޙ
fully connected feed-forward network Λ༻ ͍Δ • W1 , W2 , b1 , b2 : ύϥϝʔλ FFN(x) = max(0, xW1 + b1)W2 + b2 28 / 41 -
Part 2 : Summary • ෮श : Part2 ͷશମਤ •
͜Ε͕ Transformer ͷجຊతͳϨΠϠʔͱͳ͍ͬͯΔ • encoder Ͱ͜ͷϒϩοΫ͕ N = 6 ݸελοΫ͞Ε͍ͯΔ 29 / 41 -
Part 3 : Overall • ෮श : Transformer ͷશମ૾ Figure
1: The Transformer - model architecture. ᶃ ᶄ ᶅ ᶆ ᶇ 30 / 41 -
Part 3 : Masked self-attention • ͡Ίʹ, query, key, value
Λ Source, Target ͔Βύϥϝʔλ W q, W k, W v Λ༻͍ͯܭࢉ ࢲ ϖϯ Λ #04 ࣋ͭ ɻ &04 4PVSDF 5BSHFU 7BMVF ,FZ 2VFSZ 𝑆𝑊! 𝑆𝑊" 𝑇𝑊# ࢲ ϖϯ Λ #04 ࣋ͭ ɻ &04 'SPN &NCFEEJOH 4IJGUFE 4IJGUFE 31 / 41 -
Part 3 : Masked self-attention • ࣍ʹ, softmax(QiK⊤ i −
Mask) Λܭࢉ • Mask কདྷͷใͷ༻Λ͙ͨΊʹ༻͍ΒΕΔ • n+1 ൪ͷ୯ޠͷ༧ଌΛ͢Δࡍ, Transformer 1 ൪͔Β n ൪ͷ୯ ޠͷΈ༻Մೳ • ༁จ ʮࢲ ϖϯ Λ ࣋ͭ ɻ ʯ ͷ ʮϖϯʯ Λ༧ଌ͢Δͱ͖ ʮ<BOS> ࢲ ʯ·Ͱ༻Մೳ ,FZ 2VFSZ 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 㱣 .BTL − 𝑄𝐾! QSFWFOUGSPNVTJOHGVUVSFJOGPSNBUJPO 32 / 41 -
Part 3 : Masked self-attention • ࣍ʹ, softmax(QK⊤ − Mask)
Λܭࢉ • Mask কདྷͷใͷ༻Λ͙ͨΊʹ༻͍ΒΕΔ • n+1 ൪ͷ୯ޠͷ༧ଌΛ͢Δࡍ, Transformer 1 ൪͔Β n ൪ͷ୯ ޠͷΈ༻Մೳ • ༁จ ʮࢲ ϖϯ Λ ࣋ͭ ɻ ʯ ͷ ʮϖϯʯ Λ༧ଌ͢Δͱ͖ ʮ<BOS> ࢲ ʯ·Ͱ༻Մೳ .BTL − 𝑄𝐾! 4PGUNBY ࢲ ϖϯ Λ #04 &04 ࣋ͭ ɻ ࢲ #04 ϖϯ Λ <&04> ࣋ͭ ɻ 33 / 41 -
Part 3 : Masked self-attention • ࠷ޙʹ, softmax(QK⊤ − Mask)V
Λܭࢉ͠ Target ΛՃ͑Δ (residual connection). 7BMVF 𝑆𝑜𝑓𝑡𝑚𝑎𝑥(𝑄𝐾! − 𝑀𝑎𝑠𝑘)𝑉 𝑆𝑜𝑓𝑡𝑚𝑎𝑥(𝑄𝐾! − 𝑀𝑎𝑠𝑘) ࢲ ϖϯ Λ <#04> ࣋ͭ ɻ <&04> 5BSHFU ࢲ ϖϯ Λ <#04> ࣋ͭ ɻ <&04> ʴ 34 / 41 -
Part 4 : Overall • ෮श : Transformer ͷશମ૾ Figure
1: The Transformer - model architecture. ᶃ ᶄ ᶅ ᶆ ᶇ 35 / 41 -
Part 4 : Decoder attention • ͡Ίʹ, query, key, value
Λ Source, Target ͔Βύϥϝʔλ W q, W k, W v Λ༻͍ͯܭࢉ • ͜͜ͷ attention Ͱ Source encoder ͔Β, Target part 3 (Masked self-attention layer) ͔Βདྷ͍ͯΔ IBWF B QFO * <&04> <QBE> <QBE> 4PVSDF 7BMVF ,FZ 2VFSZ 𝑆𝑊! 𝑆𝑊" 𝑇𝑊# ࢲ ϖϯ Λ <#04> ࣋ͭ ɻ <&04> 5BSHFU 'SPN&ODPEFS 'SPN.BTLFE TFMGBUUFOUJPOMBZFS 36 / 41 -
Part 4 : Decoder attention • ࣍ʹ, softmax(QK⊤) Λܭࢉ •
ײతղऍΛ͢Δͱ, ಘΒΕΔߦྻ Source Λ Target ʹ༁͢ΔͨΊͷ ؔੑΛදͨ͠ͷʹͳΔ • ྫ͑ ”I” ͱ ”ࢲ” ࣍ͷޠ ”” Λ༧ଌ͢ΔͨΊʹڧ͍ؔͯ͘͠Δ ͣ ͷΑ͏ͳͷ ,FZ 2VFSZ 𝑄𝐾! 4PGUNBY IBWF * B QFO <&04><QBE><QBE> ࢲ ϖϯ Λ <#04> <&04> ࣋ͭ ɻ 37 / 41 -
Part 4 : Decoder attention • ࠷ޙʹ, softmax(QK⊤)V Λܭࢉ͠, Target
ΛՃ͑Δ • ࠷ऴతʹಘΒΕΔߦྻ ݩͷ Target ͷใʹ ֤ Target ୯ޠͱؔ͢Δ Source ͷ୯ޠใΛՃ͑ͨͷ 7BMVF 𝑆𝑜𝑓𝑡𝑚𝑎𝑥(𝑄𝐾!)𝑉 𝑆𝑜𝑓𝑡𝑚𝑎𝑥(𝑄𝐾!) ࢲ ϖϯ Λ <#04> ࣋ͭ ɻ <&04> 5BSHFU ࢲ ϖϯ Λ <#04> ࣋ͭ ɻ <&04> ʴ 38 / 41 -
Part 3, 4 : Summary • ෮श : Part 3
and 4 ͷશମਤ 'SPN &ODPEFS • decoder Ͱ͜ͷϒϩοΫ͕ N = 6 ݸελοΫ͞Ε͍ͯΔ 39 / 41 -
Part 5 : Overall • ෮श : Transformer ͷશମ૾ Figure
1: The Transformer - model architecture. ᶃ ᶄ ᶅ ᶆ ᶇ 40 / 41 -
Part 5 : predict • decoder layer ͔Βग़ྗ͞Εͨߦྻͷ֤ߦ࣍ͷޠͷ༧ଌʹΘΕΔ ࢲ
ϖϯ Λ <#04> ࣋ͭ ɻ <&04> ࢲ ൴ ϖϯ <&04> ൴ঁ ɾɾɾ ɾɾɾ ࢲ <&04> ɾɾɾ ɾɾɾ ɾɾɾ <QBE> -JOFBS 4PGUNBY 1SFEJDU ɾɾɾ ɾɾɾ "MMXPSETJOEBUBTFU 1SPCBCJMJUJFTPGOFYUXPSET 41 / 41