20190515 RNNと自然言語処理 RNNの基礎からAttention〜BERTの概要まで

Kazuki Motohashi - Skymind K.K.
実践者向けディープラーニング勉強会第3回 - 15/May/2019
スカイマインド株式会社
本橋和貴
RNNと⾃然⾔語処理
RNNの基礎からAttention〜BERTの概要まで

View Slide

実践者向けディープラーニング勉強会第3回 - 15/May/2019 Kazuki Motohashi - Skymind K.K.
‣本橋和貴 @kmotohas
- スカイマインド株式会社
• Deep Learning Engineer (前職ではDL+ROS)
- 素粒⼦物理学実験（LHC-ATLAS実験）出⾝
• 博⼠（理学）
- 好きな定理︓ネーターの定理
2
ࣗݾ঺հ
ܥʹ࿈ଓతͳରশੑ͕͋Δ৔߹͸ͦΕʹରԠ͢Δอଘଇ͕ଘࡏ͢Δͱड़΂ΔఆཧͰ͋Δɻ(Wikipedia)

View Slide

3
͸͡Ίʹ
: L = W
!53
https://www.slideshare.net/ren4yu/ss-84282514

View Slide

4
*NBHF/FUσʔληοτͱ৞ΈࠐΈχϡʔϥϧωοτ
https://medium.com/@smallﬁshbigsea/a-walk-through-of-alexnet-6cbd137a5637
*NBHF/FUͷେྔը૾Ͱ܇࿅ࡁΈͷχϡʔϥϧωοτϫʔΫΛ
ಛ௃நग़ػͱͯ͠ผͷը૾ೝࣝܥλεΫʹ࠶ར༻Մೳ

View Slide

5
!3
15

View Slide

6
ίϯϐϡʔλʔϏδϣϯͱࣗવݴޠॲཧ
Computer
Vision
Natural
Language
Processing
ௗ
ΓΜ͝
෺ཧੈք ه߸ੈք
෺ཧతͳޫ͕ղੳର৅ ֓೦తͳݴ༿͕ղੳର৅

View Slide

7 (引⽤元) https://twitter.com/_Ryobot/status/1050925881894400000/
#&35ࣗવݴޠॲཧ༻ͷ൚༻ಛ௃நग़ػ

View Slide

‣ Bidirectional Encoder Representations from Transformers
- 双⽅向のTransformerを⽤いて⾔語表現をエンコードするモデル
8
#&35ͱ͸ʁ

View Slide

‣ Googleのニューラル機械翻訳モデル
- RNNを使わずAttention機構のみ利⽤
9
5SBOTGPSNFSͱ͸ʁ

View Slide

‣ Agenda
- ⾃然⾔語処理とRNN
- Attention機構
- Transformer/BERTの概要
10
ͱ͍͏Θ͚Ͱ#&35ͷงғؾͷཧղ͕໨ඪ
https://robomind.co.jp/aivskodomotati3/

View Slide

‣ ⽂書分類 (document categorization)
‣ 機械翻訳 (machine translation)
‣ ⽂章要約 (text summarization)
‣ 質問応答 (question answering)
11
ࣗવݴޠॲཧͷओཁͳԠ༻λεΫ
IUUQTMJWFTBMFTNBODPNXIZZPVDBOUEFQFOEPONBDIJOFUSBOTMBUJPOGPSNVMUJMJOHVBMDVTUPNFSTVQQPSU
IUUQTUPXBSETEBUBTDJFODFDPNNBDIJOFMFBSOJOHOMQUFYUDMBTTJpDBUJPOVTJOHTDJLJUMFBSOQZUIPOBOEOMULDCBDB
IUUQTUPXBSETEBUBTDJFODFDPNUFYUTVNNBSJ[BUJPOVTJOHEFFQMFBSOJOHFFEFD
IUUQTUPXBSETEBUBTDJFODFDPNBVUPNBUJDRVFTUJPOBOTXFSJOHBD

View Slide

課題
- ⼀般に、⼊⼒の⻑さ (Tx
) も出⼒の⻑さ (Ty
) もセンテンスごとに異なる
- ある単語の特徴量を学習しても、単語の出現位置が変わると対応できない
12
ී௨ͷશ݁߹χϡʔϥϧωοτϫʔΫͷ՝୊
ɾ
ɾ
ɾ ɾ
ɾ
ɾ
ɾ
ɾ
ɾ
x<1>
AAAClXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVMeADYkuSk+KrTbQMzExNzE01jHQMzcyNbU0ADIMTc3MjC1rK+KqbQztamvjBZQN9AzAQAGTYQhlKDNAQUC+wHKGGIYUhnyGZIZShlyGVIY8hhIgO4chkaEYCKMZDBkMGAqAYrEM1UCxIiArEyyfylDLwAXUWwpUlQpUkQgUzQaS6UBeNFQ0D8gHmVkM1p0MtCUHiIuAOhUYVA2uGqw0+GxwwmC1wUuDPzjNqgabAXJLJZBOguhNLYjn75II/k5QVy6QLmHIQOjC6+YShjQGC7BbM4FuLwCLgHyRDNFfVjX9c7BVkGq1msEig9dA9y80uGlwGOiDvLIvyUsDU4NmM3ABI8AQPbgxGWFGeobGekaBJsoOTtCo4GCQZlBi0ACGtzmDA4MHQwBDKNDeaQx7GI4yHGMSZ7JlcmFygyhlYoTqEWZAAUz+AJ/imIY=
x<2>
AAAClXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVMeADYkuSk+KrTbQMzExNzE01jHQMzcyNbU0ADIMTc3MjC1rK+KqbYzsamvjBZQN9AzAQAGTYQhlKDNAQUC+wHKGGIYUhnyGZIZShlyGVIY8hhIgO4chkaEYCKMZDBkMGAqAYrEM1UCxIiArEyyfylDLwAXUWwpUlQpUkQgUzQaS6UBeNFQ0D8gHmVkM1p0MtCUHiIuAOhUYVA2uGqw0+GxwwmC1wUuDPzjNqgabAXJLJZBOguhNLYjn75II/k5QVy6QLmHIQOjC6+YShjQGC7BbM4FuLwCLgHyRDNFfVjX9c7BVkGq1msEig9dA9y80uGlwGOiDvLIvyUsDU4NmM3ABI8AQPbgxGWFGeobGekaBJsoOTtCo4GCQZlBi0ACGtzmDA4MHQwBDKNDeaQx7GI4yHGMSZ7JlcmFygyhlYoTqEWZAAUz+AKHlmIc=
x
AAACl3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVceADYkuSk+KrTbQMzExNzE01jHQMzcyNbU0ADIMTc3MjC1rK+KqbULiK+xqa+MFlA30DMBAAZNhCGUoM0BBQL7AcoYYhhSGfIZkhlKGXIZUhjyGEiA7hyGRoRgIoxkMGQwYCoBisQzVQLEiICsTLJ/KUMvABdRbClSVClSRCBTNBpLpQF40VDQPyAeZWQzWnQy0JQeIi4A6FRhUDa4arDT4bHDCYLXBS4M/OM2qBpsBckslkE6C6E0tiOfvkgj+TlBXLpAuYchA6MLr5hKGNAYLsFszgW4vAIuAfJEM0V9WNf1zsFWQarWawSKD10D3LzS4aXAY6IO8si/JSwNTg2YzcAEjwBA9uDEZYUZ6hsZ6RoEmyg5O0KjgYJBmUGLQAIa3OYMDgwdDAEMo0N4ZDPsYjjOcYJJksmdyY/KAKGVihOoRZkABTIEA/NOZlA==
y
AAACl3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVceADYkuSk+KrTbQs7SwsDAz0THQMzYzsDQyBTJMTc0tLU1qK+OqbULiK+1qa+MFlA30DMBAAZNhCGUoM0BBQL7AcoYYhhSGfIZkhlKGXIZUhjyGEiA7hyGRoRgIoxkMGQwYCoBisQzVQLEiICsTLJ/KUMvABdRbClSVClSRCBTNBpLpQF40VDQPyAeZWQzWnQy0JQeIi4A6FRhUDa4arDT4bHDCYLXBS4M/OM2qBpsBckslkE6C6E0tiOfvkgj+TlBXLpAuYchA6MLr5hKGNAYLsFszgW4vAIuAfJEM0V9WNf1zsFWQarWawSKD10D3LzS4aXAY6IO8si/JSwNTg2YzcAEjwBA9uDEZYUZ6hsZ6RoEmyg5O0KjgYJBmUGLQAIa3OYMDgwdDAEMo0N4ZDPsYjjOcYJJksmdyY/KAKGVihOoRZkABTIEAN6OZsA==
y<1>
AAAClXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVMeADYkuSk+KrTbQs7SwsDAz0THQMzYzsDQyBTJMTc0tLU1qK+OqbQztamvjBZQN9AzAQAGTYQhlKDNAQUC+wHKGGIYUhnyGZIZShlyGVIY8hhIgO4chkaEYCKMZDBkMGAqAYrEM1UCxIiArEyyfylDLwAXUWwpUlQpUkQgUzQaS6UBeNFQ0D8gHmVkM1p0MtCUHiIuAOhUYVA2uGqw0+GxwwmC1wUuDPzjNqgabAXJLJZBOguhNLYjn75II/k5QVy6QLmHIQOjC6+YShjQGC7BbM4FuLwCLgHyRDNFfVjX9c7BVkGq1msEig9dA9y80uGlwGOiDvLIvyUsDU4NmM3ABI8AQPbgxGWFGeobGekaBJsoOTtCo4GCQZlBi0ACGtzmDA4MHQwBDKNDeaQx7GI4yHGMSZ7JlcmFygyhlYoTqEWZAAUz+ANhqmKE=
y<2>
AAAClXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVMeADYkuSk+KrTbQs7SwsDAz0THQMzYzsDQyBTJMTc0tLU1qK+OqbYzsamvjBZQN9AzAQAGTYQhlKDNAQUC+wHKGGIYUhnyGZIZShlyGVIY8hhIgO4chkaEYCKMZDBkMGAqAYrEM1UCxIiArEyyfylDLwAXUWwpUlQpUkQgUzQaS6UBeNFQ0D8gHmVkM1p0MtCUHiIuAOhUYVA2uGqw0+GxwwmC1wUuDPzjNqgabAXJLJZBOguhNLYjn75II/k5QVy6QLmHIQOjC6+YShjQGC7BbM4FuLwCLgHyRDNFfVjX9c7BVkGq1msEig9dA9y80uGlwGOiDvLIvyUsDU4NmM3ABI8AQPbgxGWFGeobGekaBJsoOTtCo4GCQZlBi0ACGtzmDA4MHQwBDKNDeaQx7GI4yHGMSZ7JlcmFygyhlYoTqEWZAAUz+ANptmKI=
%PHT
"SF
$VUF
ݘ
͸
͔Θ͍͍

View Slide

13
x<1>
y<1>
a<1>
AAAClXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVMeADYkuSk+KrTbQMzIxMDE10THQMzc2NrEwAjIsjMwtDC1rE+OqbQztamvjBZRhqhRgqhRgqhQM9QzAQJkBCgLyBZYzxDCkMOQzJDOUMuQypDLkMZQA2TkMiQzFQBjNYMhgwFAAFItlqAaKFQFZmWD5VIZaBi6g3lKgqlSgikSgaDaQTAfyoqGieUA+yMxisO5koC05QFwE1KnAoGpw1WClwWeDEwarDV4a/MFpVjXYDJBbKoF0EkRvakE8f5dE8HeCunKBdAlDBkIXXjeXMKQxWIDdmgl0ewFYBOSLZIj+sqrpn4OtglSr1QwWGbwGun+hwU2Dw0Af5JV9SV4amBo0m4ELGAGwUFbAzQgz0jM01jMKNFF2cIJGBQeDNIMSgwYwvM0ZHBg8GAIYQoH2TmPYw3CU4RiTOJMtkwuTG0QpEyNUjzADCmDyBwAM3JjA
Wx
AAACkXichVE9S8NQFD2N3/Wr6iK4iEVxkHJbBT+moovgYq21Qi0lic8aTJOQpEUN/QOOLg5OCiLiH3BVF/+AQ3+COCq4OHgTA6JFvSHvnXvePfed955i6ZrjEjUiUktrW3tHZ1e0u6e3rz82MLjhmFVbFTnV1E17U5EdoWuGyLmaq4tNyxZyRdFFXtlb8tfzNWE7mmmsuweWKFbksqHtaKrsMlWKTXlbQZOCXVaKHiVS5MdUE6jnS/v1UixOiSCl0WaQDEEcYayasUtsYRsmVFRRgYABl7EOGQ5/BSRBsJgrwmPOZqQF6wJ1RFlb5SrBFTKzezyWOSuErMG539MJ1CrvovNvs3IU4/RIV/RCD3RNT/T+ay8v6OF7OeBZ+dQKq9R/NJx9+1dV4dnF7pfqT88udjAXeNXYuxUw/inUT33t8OQlu7A27k3QOT2z/zNq0D2fwKi9qhcZsXaKKD9A8ud1N4ONVCI5nUhlZuLpxfApOjGCMUzyfc8ijWWsIsf7HuMGt7iThqR5KS2FtVIk1AzhW0grH6OMlsg=
Wy
AAACkXichVHLSsNAFD3GV62vajeCG7EoLqTcVsHHquhGcOOrtlBLSeJYg2kSkrRQQ3/ApRsXrhRExB9wq278ARd+grhUcOPCmzQgWtQbMnPumXvunJlRLF1zXKKnNqm9o7OrO9IT7e3rHxiMDQ1vO2bVVkVWNXXTziuyI3TNEFlXc3WRt2whVxRd5JSDZX89VxO2o5nGllu3RLEilw1tT1Nll6lSbNrbCZoU7LJS9CiZJj+mW0AjV6o3SrEEJYOUxlpBKgQJhLFmxi6xg12YUFFFBQIGXMY6ZDj8FZACwWKuCI85m5EWrAs0EGVtlasEV8jMHvBY5qwQsgbnfk8nUKu8i86/zcoxTNAjXdErPdA1PdPHr728oIfvpc6z0tQKqzR4NLL5/q+qwrOL/S/Vn55d7GE+8Kqxdytg/FOoTX3t8OR1c3Fjwpukc3ph/2f0RPd8AqP2pl6si41TRPkBUj+vuxVsp5OpmWR6fTaRWQqfIoJRjGOK73sOGaxgDVne9xg3uMWdFJcWpIwU1kptoSaObyGtfgKljZbJ
3// 3FDVSSFOU/FVSBM/FUXPSL

View Slide

14
x<2>
y<2>
a<1>
a<2>
AAAClXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVMeADYkuSk+KrTbQMzIxMDE10THQMzc2NrEwAjIsjMwtDC1rE+OqbYzsamvjBZRhqhRgqhRgqhQM9QzAQJkBCgLyBZYzxDCkMOQzJDOUMuQypDLkMZQA2TkMiQzFQBjNYMhgwFAAFItlqAaKFQFZmWD5VIZaBi6g3lKgqlSgikSgaDaQTAfyoqGieUA+yMxisO5koC05QFwE1KnAoGpw1WClwWeDEwarDV4a/MFpVjXYDJBbKoF0EkRvakE8f5dE8HeCunKBdAlDBkIXXjeXMKQxWIDdmgl0ewFYBOSLZIj+sqrpn4OtglSr1QwWGbwGun+hwU2Dw0Af5JV9SV4amBo0m4ELGAGwUFbAzQgz0jM01jMKNFF2cIJGBQeDNIMSgwYwvM0ZHBg8GAIYQoH2TmPYw3CU4RiTOJMtkwuTG0QpEyNUjzADCmDyBwAO35jB
Wa
AAACkXichVE9S8NQFD3G7/rRqovgIhbFQcptFfyYii6Ci7bWCrWUJD5raJqEJC3U0D/g6OLgpCAi/gFXdfEPOPgTxFHBxcGbGBAt6g1579zz7rnvvPcUS9ccl+ixTWrv6Ozq7umN9PUPDEZjQ8NbjlmzVZFTTd20txXZEbpmiJyrubrYtmwhVxVd5JXKir+erwvb0Uxj021YoliVy4a2p6myy1QpNuPtBE0KdlkpepRIkR8zLaCZL8nNUixOiSCl8VaQDEEcYaybsQvsYBcmVNRQhYABl7EOGQ5/BSRBsJgrwmPOZqQF6wJNRFhb4yrBFTKzFR7LnBVC1uDc7+kEapV30fm3WTmOSXqgS3qhe7qiJ3r/tZcX9PC9NHhWPrXCKkUPR7Nv/6qqPLvY/1L96dnFHhYCrxp7twLGP4X6qa8fHL9klzKT3hSd0TP7P6VHuuMTGPVX9XxDZE4Q4QdI/rzuVrCVSiRnE6mNuXh6OXyKHoxhAtN83/NIYxXryPG+R7jGDW6lEWlRSkthrdQWakbwLaS1D3V1lrE=
x<1>
y<1>
Wx
Wy
Wx
Wy

View Slide

15
x<1>
x<2>
y<1>
y<2>
a<1>
a<2>
a<3>
AAAClXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVMeADYkuSk+KrTbQMzIxMDE10THQMzc2NrEwAjIsjMwtDC1rE+OqbYztamvjBZRhqhRgqhRgqhQM9QzAQJkBCgLyBZYzxDCkMOQzJDOUMuQypDLkMZQA2TkMiQzFQBjNYMhgwFAAFItlqAaKFQFZmWD5VIZaBi6g3lKgqlSgikSgaDaQTAfyoqGieUA+yMxisO5koC05QFwE1KnAoGpw1WClwWeDEwarDV4a/MFpVjXYDJBbKoF0EkRvakE8f5dE8HeCunKBdAlDBkIXXjeXMKQxWIDdmgl0ewFYBOSLZIj+sqrpn4OtglSr1QwWGbwGun+hwU2Dw0Af5JV9SV4amBo0m4ELGAGwUFbAzQgz0jM01jMKNFF2cIJGBQeDNIMSgwYwvM0ZHBg8GAIYQoH2TmPYw3CU4RiTOJMtkwuTG0QpEyNUjzADCmDyBwAQ4pjC
x<3>
AAAClXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVMeADYkuSk+KrTbQM7YwMjQ30jHQM7MwNTY2BjIMDc0sjM1qK+KqbYztamvjBZQN9AzAQAGTYQhlKDNAQUC+wHKGGIYUhnyGZIZShlyGVIY8hhIgO4chkaEYCKMZDBkMGAqAYrEM1UCxIiArEyyfylDLwAXUWwpUlQpUkQgUzQaS6UBeNFQ0D8gHmVkM1p0MtCUHiIuAOhUYVA2uGqw0+GxwwmC1wUuDPzjNqgabAXJLJZBOguhNLYjn75II/k5QVy6QLmHIQOjC6+YShjQGC7BbM4FuLwCLgHyRDNFfVjX9c7BVkGq1msEig9dA9y80uGlwGOiDvLIvyUsDU4NmM3ABI8AQPbgxGWFGeobGekaBJsoOTtCo4GCQZlBi0ACGtzmDA4MHQwBDKNDeaQx7GI4yHGMSZ7JlcmFygyhlYoTqEWZAAUz+AJm0mIM=
y<3>
AAAClXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVMeADYkuSk+KrTbQszQ3MjUy1jHQMzKxtDA2BzJMLMwMLY1rK+OqbYztamvjBZQN9AzAQAGTYQhlKDNAQUC+wHKGGIYUhnyGZIZShlyGVIY8hhIgO4chkaEYCKMZDBkMGAqAYrEM1UCxIiArEyyfylDLwAXUWwpUlQpUkQgUzQaS6UBeNFQ0D8gHmVkM1p0MtCUHiIuAOhUYVA2uGqw0+GxwwmC1wUuDPzjNqgabAXJLJZBOguhNLYjn75II/k5QVy6QLmHIQOjC6+YShjQGC7BbM4FuLwCLgHyRDNFfVjX9c7BVkGq1msEig9dA9y80uGlwGOiDvLIvyUsDU4NmM3ABI8AQPbgxGWFGeobGekaBJsoOTtCo4GCQZlBi0ACGtzmDA4MHQwBDKNDeaQx7GI4yHGMSZ7JlcmFygyhlYoTqEWZAAUz+AL0EmJQ=
Wx
Wa
Wy
Wx
Wa
Wy
Wx
Wy

View Slide

16
x<1>
x<2>
x
y
y<1>
y<2>
a<1>
a<2>
a<3>
a
AAACmXichVHLTttAFD24vBpeadlUYmM1ArGg1rXjKmlVVRQ2qCsgBJBCGtlmCBaObdlOBLXyA/0BJLqiUlVV/AQSG9o9Cz4BsQSJDQtunESIBXBHM3PnzDl3zsyYvmOHEdF5j/Sit69/YPBlamh4ZHQs/er1aujVA0sULc/xgnXTCIVju6IY2ZEj1v1AGDXTEWvmznxrf60hgtD23JVozxflmlF17S3bMiKGKmk93kiKlIKqWY5J0XTS3+szpOSyWT2vcZLXcnn1Q9P4Fn9aqey+Uz83m5V0psuUu0y5y5RVhZLIoBOLXvoPNrAJDxbqqEHARcS5AwMhtxJUEHzGyogZCzizk32BJlKsrTNLMMNgdIfHKq9KHdTldatmmKgtPsXhHrBSxiSd0V+6olM6ogu6fbRWnNRoednj2WxrhV8Z+/GmcPOsqsZzhO171ZOeI2whn3i12bufIK1bWG194/v+VeHj8mQ8Rb/okv0f0jmd8A3cxrX1e0ks/0SKP6D7yvLjyaqmqFlFW9Izs3OdrxjEBN5imt87h1ksYBFFPvcAx/iH/9KE9EVakL62qVJPRzOOByEV7gCOA5pA
x<3>
y<3>
Wx
Wa
Wy
Wx
Wa
Wy
Wx
Wa
Wy
Wx
Wy
Wa
ʜ

View Slide

17
x<1>
x<2>
x
y
y<1>
y<2>
a<1>
a<2>
a<3>
a
x<3>
y<3>
y
AAAClXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVMeADYkuSk+KrTbQszQ3MjUy1jHQMzKxtDA2BzJMLMwMLY1rK+OqbUrsamvjBZQN9AzAQAGTYQhlKDNAQUC+wHKGGIYUhnyGZIZShlyGVIY8hhIgO4chkaEYCKMZDBkMGAqAYrEM1UCxIiArEyyfylDLwAXUWwpUlQpUkQgUzQaS6UBeNFQ0D8gHmVkM1p0MtCUHiIuAOhUYVA2uGqw0+GxwwmC1wUuDPzjNqgabAXJLJZBOguhNLYjn75II/k5QVy6QLmHIQOjC6+YShjQGC7BbM4FuLwCLgHyRDNFfVjX9c7BVkGq1msEig9dA9y80uGlwGOiDvLIvyUsDU4NmM3ABI8AQPbgxGWFGeobGekaBJsoOTtCo4GCQZlBi0ACGtzmDA4MHQwBDKNDeaQx7GI4yHGMSZ7JlcmFygyhlYoTqEWZAAUz+AD/WmNU=
x
AAAClXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVMeADYkuSk+KrTbQM7YwMjQ30jHQM7MwNTY2BjIMDc0sjM1qK+KqbUrsamvjBZQN9AzAQAGTYQhlKDNAQUC+wHKGGIYUhnyGZIZShlyGVIY8hhIgO4chkaEYCKMZDBkMGAqAYrEM1UCxIiArEyyfylDLwAXUWwpUlQpUkQgUzQaS6UBeNFQ0D8gHmVkM1p0MtCUHiIuAOhUYVA2uGqw0+GxwwmC1wUuDPzjNqgabAXJLJZBOguhNLYjn75II/k5QVy6QLmHIQOjC6+YShjQGC7BbM4FuLwCLgHyRDNFfVjX9c7BVkGq1msEig9dA9y80uGlwGOiDvLIvyUsDU4NmM3ABI8AQPbgxGWFGeobGekaBJsoOTtCo4GCQZlBi0ACGtzmDA4MHQwBDKNDeaQx7GI4yHGMSZ7JlcmFygyhlYoTqEWZAAUz+AByGmMQ=
Wx
Wa
Wy
Wx
Wa
Wy
Wx
Wa
Wy
Wx
Wa
Wy
Wx
Wa
Wy
ʜ ʹ

View Slide

18
x<1>
x<2>
x
y
y<1>
y<2>
a<1>
a<2>
a<3>
a
x<3>
y<3>
y
x
Wx
Wa
Wy
Wx
Wa
Wy
Wx
Wa
Wy
Wx
Wa
Wy
Wx
Wa
Wy
ʜ ʹ

ೖྗͷ࣌ܥྻσʔλΛந৅తͳจ຺৘ใ DPOUFYU
ͱͯ͠Τϯίʔυ

View Slide

19
3//Ϣχοτ
x
a
AAACl3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVceADYkuSk+KrTbQMzIxMDE10THQMzc2NrEwAjIsjMwtDC1rE+OqbUp0De1qa+MFlGHqFGDqFGDqFAz1DMBAmQEKAvIFljPEMKQw5DMkM5Qy5DKkMuQxlADZOQyJDMVAGM1gyGDAUAAUi2WoBooVAVmZYPlUhloGLqDeUqCqVKCKRKBoNpBMB/KioaJ5QD7IzGKw7mSgLTlAXATUqcCganDVYKXBZ4MTBqsNXhr8wWlWNdgMkFsqgXQSRG9qQTx/l0Twd4K6coF0CUMGQhdeN5cwpDFYgN2aCXR7AVgE5ItkiP6yqumfg62CVKvVDBYZvAa6f6HBTYPDQB/klX1JXhqYGjSbgQsYAbBQVsDNCDPSMzTWMwo0UXZwgkYFB4M0gxKDBjC8zRkcGDwYAhhCgfbOYNjHcJzhBJMkkz2TG5MHRCkTI1SPMAMKYAoEALaTmXU=
tanh
a
AAAClXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsYVMeADYkuSk+KrTbQMzIxMDE10THQMzc2NrEwAjIsjMwtDC1rE+OqbUrsamvjBZRhqhRgqhRgqhQM9QzAQJkBCgLyBZYzxDCkMOQzJDOUMuQypDLkMZQA2TkMiQzFQBjNYMhgwFAAFItlqAaKFQFZmWD5VIZaBi6g3lKgqlSgikSgaDaQTAfyoqGieUA+yMxisO5koC05QFwE1KnAoGpw1WClwWeDEwarDV4a/MFpVjXYDJBbKoF0EkRvakE8f5dE8HeCunKBdAlDBkIXXjeXMKQxWIDdmgl0ewFYBOSLZIj+sqrpn4OtglSr1QwWGbwGun+hwU2Dw0Af5JV9SV4amBo0m4ELGAGwUFbAzQgz0jM01jMKNFF2cIJGBQeDNIMSgwYwvM0ZHBg8GAIYQoH2TmPYw3CU4RiTOJMtkwuTG0QpEyNUjzADCmDyBwCTpZkD
a = tanh(W[a, x] + b)
AAACjHichVHLSsNAFD2N7/qquhHcFIsv1DJRQfFRioK49FUrtLUkcdRgmoRkWtTSHxBcu3ClICIu3erKjT/gwk8QlwpuXHibBkRFvcPM3HvuPXfOzKi2obuCsceAVFVdU1tX3xBsbGpuaQ21ta+5Vt7ReEKzDMtZVxWXG7rJE0IXBl+3Ha7kVIMn1d25cj5Z4I6rW+aq2Ld5Jqdsm/qWrimCoGyoT9koTotYaSYtFHOnP5ny4mE5VhpKD+1VcplBdSAbirAo8yz805F9JwLfFq3QBdLYhAUNeeTAYUKQb0CBSyMFGQw2YRkUCXPI0708RwlB4uapilOFQugurdsUpXzUpLjc0/XYGp1i0HSIGUYPe2CX7IXdsyv2xN5/7VX0epS17NOuVrjczrYedq68/cvK0S6w88n6U7PAFiY8rTpptz2kfAutwi8cHL+sTC73FHvZGXsm/afskd3RDczCq3a+xJdPEKQPkL8/909nbSQqj0ZHlsYi8Vn/K+rRhW7003uPI44FLCJB5x7hGje4lVqkMWlKmqmUSgGf04EvJs1/AJVIl3s=

View Slide

20
3//Ϣχοτ
x
a
tanh
a
Softmax y
a
a = tanh(W[a, x] + b)
AAACjHichVHLSsNAFD2N7/qquhHcFIsv1DJRQfFRioK49FUrtLUkcdRgmoRkWtTSHxBcu3ClICIu3erKjT/gwk8QlwpuXHibBkRFvcPM3HvuPXfOzKi2obuCsceAVFVdU1tX3xBsbGpuaQ21ta+5Vt7ReEKzDMtZVxWXG7rJE0IXBl+3Ha7kVIMn1d25cj5Z4I6rW+aq2Ld5Jqdsm/qWrimCoGyoT9koTotYaSYtFHOnP5ny4mE5VhpKD+1VcplBdSAbirAo8yz805F9JwLfFq3QBdLYhAUNeeTAYUKQb0CBSyMFGQw2YRkUCXPI0708RwlB4uapilOFQugurdsUpXzUpLjc0/XYGp1i0HSIGUYPe2CX7IXdsyv2xN5/7VX0epS17NOuVrjczrYedq68/cvK0S6w88n6U7PAFiY8rTpptz2kfAutwi8cHL+sTC73FHvZGXsm/afskd3RDczCq3a+xJdPEKQPkL8/909nbSQqj0ZHlsYi8Vn/K+rRhW7003uPI44FLCJB5x7hGje4lVqkMWlKmqmUSgGf04EvJs1/AJVIl3s=

View Slide

21
x
a
tanh sigmoid
(0, 1)
a
˜
a
AAACnXichVFNSxtBGH5cW7Wx1WgvhR66NCgeJLzxA5MiRSqFHkT8ihGSGHbXMQ5udpfdSUCX/QP9Az0UBIUi0p+hBy/25sGfIB4t9OLBN5sE6UF9h5l555nneeeZGdOzZaCIrrq07hcve3r7XiX6X78ZGEwODa8Hbt23RN5ybdffMI1A2NIReSWVLTY8Xxg10xYFc3e+uV9oCD+QrrOm9jxRrhlVR25Ly1AMVZLZsBQXKfpVsxxSeoo4ZsbjJEOTnORy2enpXFRS0t4SoRFthrPqcxRVkqkOW++w9Q5bz6QpjhTaseQmj1HCFlxYqKMGAQeKcxsGAm5FZEDwGCsjZMznTMb7AhESrK0zSzDDYHSXxyqvim3U4XWzZhCrLT7F5u6zUscIXdIJ3dI5/aZrunu0VhjXaHrZ49lsaYVXGfz+bvXfs6oazwo7D6onPStsIxt7lezdi5HmLayWvrH/43b108pIOEpHdMP+D+mKTvkGTuOv9WtZrPxEgj+g88r648n6RDozmZ5YnkrNfWl/RR/e4yPG+L1nMIdvWEKezz3AGS7wR/ugfdUWtMUWVetqa97iv9AK98frnJc=
u
AAAClnichVG7SgNBFD2u7/iK2gg2i0GxkHDXBxoLEUW0jI+okISwu45xcV/sbgK65Af8AQsrBRHxG6wE9Qcs/ASxVLCx8GaTIBbqHWbmzplz7pyZ0VzT8AOi5yapuaW1rb2jM9bV3dPbF+8f2PadkqeLjO6Yjrerqb4wDVtkAiMwxa7rCdXSTLGjHS5X93fKwvMNx94KjlyRt9SibewbuhowVIgrYS4qkvWKWj6k5DRxzE5EiUJTnKRSczMzqUpuVbUstVCqFOKJBk1u0OQGTVaSFEUC9Ug78SvksAcHOkqwIGAj4NyECp9bFgoILmN5hIx5nBnRvkAFMdaWmCWYoTJ6yGORV9k6avO6WtOP1DqfYnL3WCljlJ7omt7okW7ohT5/rRVGNapejnjWalrhFvpOhjY//lVZPAc4+Fb96TnAPuYirwZ7dyOkegu9pi8fn75tzm+MhmN0Qa/s/5ye6Y5vYJff9ct1sXGGGH9A45Xl35PtyaQylZxcn04sLtW/ogPDGME4v/csFrGGNDJ87ilucY8HaUhakFak1RpVaqprBvEjpPQXXdeZWQ==
˜
a = tanh(Wa[a, x] + ba)
AAACmHichVHBStxAGP6MturW1lUvohfpYlHUZaKCIiqLPdTedHVdYXcNkzjqYDYJyeyihn0BX0DEk4KU4kN48GIfoAcfofRowUsP/skGSivVP2Tmm+//v3++mTE9WwaKsbsWrbXt1ev2js7Um66377rTPb0bgVvzLVGwXNv1N00eCFs6oqCkssWm5wteNW1RNPc/RvliXfiBdJ11deiJSpXvOnJHWlwRZaSnykra2yLkja1wXi02FsqKO3sjRYOXeMRM6IuN8fL4QTNbGTMNPmqkMyzL4hh6CvQEZJDEipv+gjK24cJCDVUIOFCEbXAE9JWgg8EjroKQOJ+QjPMCDaRIW6MqQRWc2H0ad2lVSliH1lHPIFZbtItNv0/KIQyz7+wru2e37Ir9YL//2yuMe0ReDmk2m1rhGd3H/WsPL6qqNCvs/VE961lhB7OxV0nevZiJTmE19fWjk/u1ufxw+IFdsJ/k/5zdsRs6gVP/ZV2uivwZUvQA+r/X/RRsTGb1qezk6nQmt5Q8RQcG8R4jdN8zyGEZKyjQvqe4xi2+aQNaTvukfW6Wai2Jpg9/hZZ/BP+CnNk=
update
5IFDBU XIJDIBMSFBEZBUFʜ GVMM
5IFDBUT XIJDIBMSFBEZBUFʜ GVMM
ʬTJNQMJpFEʭ(36 (BUFE3FDVSSFOU6OJU
<>

View Slide

22
x
a
tanh sigmoid
(0, 1)
a
˜
a
u
˜
u = (Wu[a, x] + bu)
AAACknicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvoxrgn5uYmxpfaxhRnpucmaoTHl0YnxlXblOga2tXqxOhUgNh2tbHaSfGlmvECygZ6BmCggMkwhDKUGaAgIF9gOUMMQwpDPkMyQylDLkMqQx5DCZCdw5DIUAyE0QyGDAYMBUCxWIZqoFgRkJUJlk9lqGXgAuotBapKBapIBIpmA8l0IC8aKpoH5IPMLAbrTgbakgPERUCdCgyqBlcNVhp8NjhhsNrgpcEfnGZVg80AuaUSSCdB9KYWxPN3SQR/J6grF0iXMGQgdOF1cwlDGoMF2K2ZQLcXgEVAvkiG6C+rmv452CpItVrNYJHBa6D7FxrcNDgM9EFe2ZfkpYGpQbMZuIARYIge3JiMMCM9Q2M9o0ATZQcnaFRwMEgzKDFoAMPbnMGBwYMhgCEUaO9khp0MhxgOM4kwWTE5MjlDlDIxQvUIM6AAJh8Az8eaMw==
update
a = u ˜
a + (1 u) a
AAACpXichVHLShxBFD22muho4qgbwc3goDEEh2oVFFEZdBE3gq8ZBcc01T2lKewX3TUD2swP+AMuXCmoBD/DjQu3Cn6CuFRw48I7PR1CIklu01Wnzr3n1qkq07dlqBi7a9KaW1rfvW9rT3V0fvjYle7uKYZeJbBEwfJsL9gweShs6YqCksoWG34guGPaYt3cna/n16siCKXnrqk9X2w5fMeV29Liiigjneffomk1W5spfeWOw41KySt7qqSkXRYRrzWSX4b1kZ/5z3FBJpaN6LM1I51lORZH5i3QE5BFEkte+hwllOHBQgUOBFwowjY4Qvo2oYPBJ24LEXEBIRnnBWpIkbZCVYIqOLG7NO7QajNhXVrXe4ax2qJdbPoDUmYwyG7YD/bIrtgFu2cvf+0VxT3qXvZoNhta4RtdB32rz/9VOTQrfP+l+qdnhW1Mxl4lefdjpn4Kq6Gv7h8+rk6tDEZD7IQ9kP9jdscu6QRu9ck6XRYrR0jRA+h/XvdbUBzN6WO50eXxbH4ueYo29GMAw3TfE8hjAUso0L5nuMYNbrVP2qK2phUbpVpTounFb6EZrwtmomA=
‐
ཁૉ͝ͱͷੵ
<>

View Slide

23
x
a
tanh sigmoid
(0, 1)
a
˜
a
u
˜
u = (Wu[a, x] + bu)
AAACknicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvoxrgn5uYmxpfaxhRnpucmaoTHl0YnxlXblOga2tXqxOhUgNh2tbHaSfGlmvECygZ6BmCggMkwhDKUGaAgIF9gOUMMQwpDPkMyQylDLkMqQx5DCZCdw5DIUAyE0QyGDAYMBUCxWIZqoFgRkJUJlk9lqGXgAuotBapKBapIBIpmA8l0IC8aKpoH5IPMLAbrTgbakgPERUCdCgyqBlcNVhp8NjhhsNrgpcEfnGZVg80AuaUSSCdB9KYWxPN3SQR/J6grF0iXMGQgdOF1cwlDGoMF2K2ZQLcXgEVAvkiG6C+rmv452CpItVrNYJHBa6D7FxrcNDgM9EFe2ZfkpYGpQbMZuIARYIge3JiMMCM9Q2M9o0ATZQcnaFRwMEgzKDFoAMPbnMGBwYMhgCEUaO9khp0MhxgOM4kwWTE5MjlDlDIxQvUIM6AAJh8Az8eaMw==
update
a = u ˜
a + (1 u) a
AAACpXichVHLShxBFD22muho4qgbwc3goDEEh2oVFFEZdBE3gq8ZBcc01T2lKewX3TUD2swP+AMuXCmoBD/DjQu3Cn6CuFRw48I7PR1CIklu01Wnzr3n1qkq07dlqBi7a9KaW1rfvW9rT3V0fvjYle7uKYZeJbBEwfJsL9gweShs6YqCksoWG34guGPaYt3cna/n16siCKXnrqk9X2w5fMeV29Liiigjneffomk1W5spfeWOw41KySt7qqSkXRYRrzWSX4b1kZ/5z3FBJpaN6LM1I51lORZH5i3QE5BFEkte+hwllOHBQgUOBFwowjY4Qvo2oYPBJ24LEXEBIRnnBWpIkbZCVYIqOLG7NO7QajNhXVrXe4ax2qJdbPoDUmYwyG7YD/bIrtgFu2cvf+0VxT3qXvZoNhta4RtdB32rz/9VOTQrfP+l+qdnhW1Mxl4lefdjpn4Kq6Gv7h8+rk6tDEZD7IQ9kP9jdscu6QRu9ck6XRYrR0jRA+h/XvdbUBzN6WO50eXxbH4ueYo29GMAw3TfE8hjAUso0L5nuMYNbrVP2qK2phUbpVpTounFb6EZrwtmomA=
‐
ཁૉ͝ͱͷੵ
ˎຊ౰͸Ξοϓσʔτήʔτ͚ͩͰͳ͘ɺϦηοτήʔτͱ͍͏ผͷήʔτ΋͋Δ
ˎ-45.΋ήʔτͷछྨͱ਺͕ҧ͏͚ͩͰࣅͨΑ͏ͳߏ଄ ೖྗήʔτɺग़ྗήʔτɺ๨٫ήʔτʣ
<>

View Slide

24
ʜ ʜ
x<1>
x<2>
x
y
y<1>
y<2>
y<1>

‣ ⼊⼒⽂を RNN (LSTM/GRU) で抽象的な"コンテキスト”にエンコードし、それからデコードして出⼒⽂を⽣成
‣ 機械翻訳や対話などのタスクでよく⽤いられる
c
AAACj3ichVFNS+NQFD1GHbXqWHUjuAkWRWQIN6226kKKbnTnV1WopSTxWYNpEpK0oKF/wJU7F64URMS9Wwdm4x9w4U8QlwpuXHibtsgsnLkh7913zj33nfee7lqmHxA9tUntHZ0/urp7Yr19/T8H4oNDW75T8QyRMxzL8XZ0zReWaYtcYAaW2HE9oZV1S2zrh0t1frsqPN907M3gyBWFslayzX3T0AKGivGpcDdqkvdKeiEkJZ3KUGbuFykpUtWZJCezM3NpNV0zasV4osXLLV5u8bKqUBQJNGPViV9jF3twYKCCMgRsBJxb0ODzl4cKgstYASFjHmdmxAvUEGNthasEV2iMHvJY4lW+idq8rvf0I7XBu1j8e6yUMU6PdEOv9EC39Ewf3/YKox51L0c86w2tcIsDJyMb7/9VlXkOcPCl+qfnAPuYjbya7N2NkPopjIa+enz2ujG/Ph5O0CW9sP8LeqI/fAK7+mZcrYn1c8T4AVq3LH+fbCUVNaUk16YT2cXmU3RjFGOY5PvOIItlrCLH+57iDvf4LQ1KGWlByjZKpbamZhh/hbTyCdkBloE=
TFRTFR &ODPEFS%FDPEFSϞσϧ
<>

View Slide

‣ ⼊⼒⽂を RNN (LSTM/GRU) で抽象的な”コンテキスト”にエンコードし、それからデコードして出⼒⽂を⽣成
‣ 機械翻訳や対話などのタスクでよく⽤いられる
‣ ただし、⼊⼒⽂が⻑くなると”コンテキスト”に⼊⼒⽂の情報を⼗分に記憶できなくなってしまう
25
TFRTFR &ODPEFS%FDPEFSϞσϧ
<>
ʜ ʜ
x<1>
x<2>
x
y
y<1>
y<2>
y<1>

c
AAACj3ichVFNS+NQFD1GHbXqWHUjuAkWRWQIN6226kKKbnTnV1WopSTxWYNpEpK0oKF/wJU7F64URMS9Wwdm4x9w4U8QlwpuXHibtsgsnLkh7913zj33nfee7lqmHxA9tUntHZ0/urp7Yr19/T8H4oNDW75T8QyRMxzL8XZ0zReWaYtcYAaW2HE9oZV1S2zrh0t1frsqPN907M3gyBWFslayzX3T0AKGivGpcDdqkvdKeiEkJZ3KUGbuFykpUtWZJCezM3NpNV0zasV4osXLLV5u8bKqUBQJNGPViV9jF3twYKCCMgRsBJxb0ODzl4cKgstYASFjHmdmxAvUEGNthasEV2iMHvJY4lW+idq8rvf0I7XBu1j8e6yUMU6PdEOv9EC39Ewf3/YKox51L0c86w2tcIsDJyMb7/9VlXkOcPCl+qfnAPuYjbya7N2NkPopjIa+enz2ujG/Ph5O0CW9sP8LeqI/fAK7+mZcrYn1c8T4AVq3LH+fbCUVNaUk16YT2cXmU3RjFGOY5PvOIItlrCLH+57iDvf4LQ1KGWlByjZKpbamZhh/hbTyCdkBloE=
‎"UUFOUJPO ஫ҙ஫ࢹ஫໨
ʣػߏͷಋೖ< >

View Slide

26
"UUFOUJPOػߏͷΠϝʔδ
%PHT "SF $VUFS 5IBO $BUT

View Slide

27
ݘ

View Slide

28
ݘ ͸

View Slide

29
ݘ ͸ ೣ

View Slide

30
ݘ ͸ ೣ ΑΓ

View Slide

31
ݘ ͸ ೣ ΑΓ ͔Θ͍͍
ਓ͕ؒӳจ຋༁͢Δͷͱಉ͡Α͏ͳ஫໨ͷ࢓ํ

View Slide

32
"UUFOUJPOͷܭࢉࣜʢ਺͋Δఆٛͷͻͱͭʣ
IUUQEFFQMFBSOJOHIBUFOBCMPHDPNFOUSZUSBOTGPSNFS
EFDPEFSͱFODPEFSͷதؒঢ়ଶͷྨࣅ౓Λܭࢉ
ɹɹɹɹ‑
TPGUNBYͰ߹ܭͷॏΈʹม׵‎

View Slide

33
4FMG"UUFOUJPO
IUUQTRJJUBDPN$IBOHLFOHPJUFNTCBGFBCEDFE

View Slide

34
3//࢖Θͣʹ4FMG"UUFOUJPO͚ͩͰ΋จ຺දݱΛநग़Մೳ
"UUFOUJPOJT"MM:PV/FFE 5SBOTGPSNFS
<>IUUQTBSYJWPSHBCT
Figure 1: The Transformer - model architecture.
1 Encoder and Decoder Stacks
ncoder: The encoder is composed of a stack of N = 6 identical layers. Each layer has two
ub-layers. The ﬁrst is a multi-head self-attention mechanism, and the second is a simple, position-
ise fully connected feed-forward network. We employ a residual connection [11] around each of

View Slide

35
Figure 1: The Transformer - model architecture.
3.1 Encoder and Decoder Stacks
‣ Bidirectional Encoder Representations from Transformers
- 双⽅向のTransformerを⽤いて⾔語表現をエンコードするモデル
#&35<>

View Slide

36
#&35Λදݱநग़ثͱͯ͠ར༻
ਆాͷَۚ๮ͷϥʔϝϯ͸ඒຯ͍͠
⼤規模データセットで訓練済みのBERT
英語・中国語・多⾔語対応版、⽇本語版*
機械翻訳モデル
l3BNFOOPPEMFTBU,JLBOCPJO,BOEBJTEFMJDJPVTz
対話モデル
ʮ΁͑ɺ͜ͷޙߦ͜͏͔ͳʂʯ
感情推定モデル
ϙδςΟϒ
*1GPUだと30⽇分の訓練

View Slide

37
ଟ਺ͷϕϯνϚʔΫλεΫͰ4UBUF0G5IF"SUͷੑೳ
IUUQTUXJUUFSDPN@3ZPCPUTUBUVT
SWAG (The Situations With Adversarial Generations dataset)
A girl is going across a set up monkey bars. She
（少⼥が雲梯を進んでいます。彼⼥は）
(i) jumps up across the monkey bars.
（雲梯で跳び上がります）
(ii) struggles onto the bars to grab her head.
（頭を掴むために雲梯でもがいています）
(iii) gets to the end and stands on a wooden plank.
（終わりまで⾏き、⽊の板の上に⽴ちます）
(iv) jumps up and does a back ﬂip.
（跳び上がり、宙返りをします）
ࢀߟʣIUUQTJUNJOUDPNEBUBTFUTPGEFFQMFBSOJOHCFSUIUNM

View Slide

‣ [CLS]: 隣接⽂クイズ開始の印、[MASK]: 単語の置き換え、[SEP]: センテンスの末尾、##: 接尾辞
38
݀ຒΊΫΠζͱྡ઀จΫΠζΛղֶ͍ͯश
Input A: [CLS] the man went to [MASK] store [SEP]
Input B: he bought a gallon [MASK] milk [SEP]
Label: IsNext
-
Input A: [CLS] the man went to [MASK] store [SEP]
Input B: penguin [MASK] are ﬂight ##less birds [SEP]
Label: NotNext

View Slide

39
BERT
E
[CLS]
E
1
E
[SEP]
... E
N
E
1
’ ... E
M
’
C T
1
T
[SEP]
... T
N
T
1
’ ... T
M
’
[CLS]
Tok
1
[SEP]
... Tok
N
Tok
1
... Tok
M
Question Paragraph
BERT
E
[CLS]
E
1
E
2
E
N
C T
1
T
2
T
N
Single Sentence
...
...
BERT
Tok 1 Tok 2 Tok N
...
[CLS]
E
[CLS]
E
1
E
2
E
N
C T
1
T
2
T
N
Single Sentence
B-PER
O O
...
...
E
[CLS]
E
1
E
[SEP]
Class
Label
... E
N
E
1
’ ... E
M
’
C T
1
T
[SEP]
... T
N
T
1
’ ... T
M
’
Start/End Span
Class
Label
BERT
Tok 1 Tok 2 Tok N
...
[CLS] Tok 1
[CLS]
[CLS]
Tok
1
[SEP]
... Tok
N
Tok
1
... Tok
M
Sentence 1
...
Sentence 2
Figure 3: Our task speciﬁc models are formed by incorporating BERT with one additional output layer, so a
minimal number of parameters need to be learned from scratch. Among the tasks, (a) and (b) are sequence-level
ಛఆͷλεΫ༻ʹγϯϓϧͳग़ྗ૚Λ௥Ճ͢Δ

View Slide

40
‣ 時系列データには全結合よりリカレント
‣ ゲートを追加して単純なRNNの⽋点を解消
- LSTM, GRUなど
‣ ⽂脈を⼀箇所にまとめずAttentionで時間の関数に
‣ Self-AttentionはRNNなしの強⼒な⽂脈表現抽出器に
‣ BERTは⾃然⾔語処理のドーピング剤（︖）
·ͱΊ

View Slide

41

View Slide

42
10 20 30 40 50 60 70
10
15
20
25
Sent Lengths
BLEU
ours, no attn (BLEU 13.9)
ours, local−p attn (BLEU 20.9)
ours, best system (BLEU 23.0)
WMT’14 best (BLEU 20.7)
Jeans et al., 2015 (BLEU 21.6)
Figure 6: Length Analysis – translation qualities
Table 6:
on the R

View Slide

43
"UUFOUJPOͷղऍ

View Slide

44
BERT
E
[CLS]
E
1
E
[SEP]
... E
N
E
1
’ ... E
M
’
C T
1
T
[SEP]
... T
N
T
1
’ ... T
M
’
[CLS]
Tok
1
[SEP]
... Tok
N
Tok
1
... Tok
M
Question Paragraph
BERT
E
[CLS]
E
1
E
2
E
N
C T
1
T
2
T
N
Single Sentence
...
...
BERT
Tok 1 Tok 2 Tok N
...
[CLS]
E
[CLS]
E
1
E
2
E
N
C T
1
T
2
T
N
Single Sentence
B-PER
O O
...
...
E
[CLS]
E
1
E
[SEP]
Class
Label
... E
N
E
1
’ ... E
M
’
C T
1
T
[SEP]
... T
N
T
1
’ ... T
M
’
Start/End Span
Class
Label
BERT
Tok 1 Tok 2 Tok N
...
[CLS] Tok 1
[CLS]
[CLS]
Tok
1
[SEP]
... Tok
N
Tok
1
... Tok
M
Sentence 1
...
Sentence 2
Figure 3: Our task specific models are formed by incorporating BERT with one additional output layer, so a
minimal number of parameters need to be learned from scratch. Among the tasks, (a) and (b) are sequence-level
tasks while (c) and (d) are token-level tasks. In the figure, E represents the input embedding, Ti
represents the
contextual representation of token i, [CLS] is the special symbol for classification output, and [SEP] is the special
symbol to separate non-consecutive token sequences.
QNLI Question Natural Language Inference is the goal is to predict whether an English sentence

View Slide

20190515 RNNと自然言語処理 RNNの基礎からAttention〜BERTの概要まで

20190515 RNNと自然言語処理 RNNの基礎からAttention〜BERTの概要まで

kmotohas

More Decks by kmotohas

Other Decks in Technology

Featured

Transcript