Tweet
Tweet

Slide 1

Slide 1 text

BERT /CREST 19/03/13

Slide 2

Slide 2 text

(1/2) • End-to-end ? 2 Figure 1: The Transformer - model architecture. NMT [Vaswani+ 17] #*('& )$ !(% + " " "

Slide 3

Slide 3 text

(1/2) • End-to-end ? 3 Figure 1: The Transformer - model architecture. NMT [Vaswani+ 17] #*('& )$ !(% + " " "

Slide 4

Slide 4 text

+$,* (2/2) • &-&-/*"210 – 3 4 [+ 2017] #*('& )$ !(% + " " " /'/%.#) (!.

Slide 5

Slide 5 text

1*20 (2/2) • ,3,35%0($876 – 9 " 5 [+ 2017] #*('& )$ !(% + " " " #! 5$-5+4)/&.'4 : 90% 1 : 5060%

Slide 6

Slide 6 text

BERT [Devlin+ 2018] 6 2. Fine-tuning 1. Pre-training $PILRH;<-"@ 8:UJRX#H /FMLJ (: ')2 %6FD0@VQXHNW[ SYK • Transformer [Vaswani+ 17]H &(%@ • A2LPOT@DE-.?MLJ= ? + A [MASK] … Q[M [MASK] … 7A + B 95 @ 3GF … “!” negative ZZZ ZZZ ZZZ ZZZ ZZZ ZZZ ZZZ ZZZ “A” A B “0C,”H%4 A, B2* 1>01H)

Slide 7

Slide 7 text

0 • BERT%BK:HJ=) /6+#8&,9( – ;H?CI00 : • 14)12/ (8" (RTE.-) • BK

Slide 8

Slide 8 text

BERTDKM(& 8 $* F + E $* 1 M2F 0 1 L49I fgh D '- 8? 32= rpq n Xkdi o P : M 6A @ … … Pre-training _m^ ( Wikipedia ("1,800) [CLS] $* F + E $* 1M2F

Slide 9

Slide 9 text

Fine-tuning 9 [CLS] ## [SEP] [ROOT] BERT X training/test head selection [Zhang+ 17] Phead(tj |ti, S) = exp(s(tj, ti)) k exp(s(tk, ti)) AAACq3ichVFNSxtBGH6yba2mfkS9CL0shkgCEiZSaBUEqQd7TIwxUleW3XWiY/aL3UmorvkD/QMePCmUUkp/hRfv0oPQ/oDSo0IvPfTdzUJRUd9lZ555nvd5550Z07dFKBm7zChPnj4beD44lH0xPDI6lhufWA+9TmDxhuXZXrBhGiG3hcsbUkibb/gBNxzT5k2zvRzrzS4PQuG5a3Lf51uOseOKlrAMSZSeW6nq0S43tntFqe8dSl3M1kvqoqq1AsOKNP7BL4axMktKqdSLtLDj6G01FtREaaeKnsuzMktCvQsqKcgjjaqX+wwN2/BgoQMHHC4kYRsGQvo2UQGDT9wWIuICQiLROXrIkrdDWZwyDGLbNO7QajNlXVrHNcPEbdEuNv0BOVUU2Hf2hV2xc/aV/WJ/760VJTXiXvZpNvte7utjH6fqfx51OTRL7P53PdizRAtvkl4F9e4nTHwKq+/vHhxd1RdWC9EMO2W/qf8TdsnO6ARu99r6VOOrx8jSA1RuX/dd0Jgrz5crtVf5pbfpSwziJaZRpOt+jSW8QxUN2vYbLvADP5Wysqa8V7R+qpJJPZO4EQr/B1NjpCc= AAACq3ichVFNSxtBGH6yba2mfkS9CL0shkgCEiZSaBUEqQd7TIwxUleW3XWiY/aL3UmorvkD/QMePCmUUkp/hRfv0oPQ/oDSo0IvPfTdzUJRUd9lZ555nvd5550Z07dFKBm7zChPnj4beD44lH0xPDI6lhufWA+9TmDxhuXZXrBhGiG3hcsbUkibb/gBNxzT5k2zvRzrzS4PQuG5a3Lf51uOseOKlrAMSZSeW6nq0S43tntFqe8dSl3M1kvqoqq1AsOKNP7BL4axMktKqdSLtLDj6G01FtREaaeKnsuzMktCvQsqKcgjjaqX+wwN2/BgoQMHHC4kYRsGQvo2UQGDT9wWIuICQiLROXrIkrdDWZwyDGLbNO7QajNlXVrHNcPEbdEuNv0BOVUU2Hf2hV2xc/aV/WJ/760VJTXiXvZpNvte7utjH6fqfx51OTRL7P53PdizRAtvkl4F9e4nTHwKq+/vHhxd1RdWC9EMO2W/qf8TdsnO6ARu99r6VOOrx8jSA1RuX/dd0Jgrz5crtVf5pbfpSwziJaZRpOt+jSW8QxUN2vYbLvADP5Wysqa8V7R+qpJJPZO4EQr/B1NjpCc= AAACq3ichVFNSxtBGH6yba2mfkS9CL0shkgCEiZSaBUEqQd7TIwxUleW3XWiY/aL3UmorvkD/QMePCmUUkp/hRfv0oPQ/oDSo0IvPfTdzUJRUd9lZ555nvd5550Z07dFKBm7zChPnj4beD44lH0xPDI6lhufWA+9TmDxhuXZXrBhGiG3hcsbUkibb/gBNxzT5k2zvRzrzS4PQuG5a3Lf51uOseOKlrAMSZSeW6nq0S43tntFqe8dSl3M1kvqoqq1AsOKNP7BL4axMktKqdSLtLDj6G01FtREaaeKnsuzMktCvQsqKcgjjaqX+wwN2/BgoQMHHC4kYRsGQvo2UQGDT9wWIuICQiLROXrIkrdDWZwyDGLbNO7QajNlXVrHNcPEbdEuNv0BOVUU2Hf2hV2xc/aV/WJ/760VJTXiXvZpNvte7utjH6fqfx51OTRL7P53PdizRAtvkl4F9e4nTHwKq+/vHhxd1RdWC9EMO2W/qf8TdsnO6ARu99r6VOOrx8jSA1RuX/dd0Jgrz5crtVf5pbfpSwziJaZRpOt+jSW8QxUN2vYbLvADP5Wysqa8V7R+qpJJPZO4EQr/B1NjpCc= AAACq3ichVFNSxtBGH6yba2mfkS9CL0shkgCEiZSaBUEqQd7TIwxUleW3XWiY/aL3UmorvkD/QMePCmUUkp/hRfv0oPQ/oDSo0IvPfTdzUJRUd9lZ555nvd5550Z07dFKBm7zChPnj4beD44lH0xPDI6lhufWA+9TmDxhuXZXrBhGiG3hcsbUkibb/gBNxzT5k2zvRzrzS4PQuG5a3Lf51uOseOKlrAMSZSeW6nq0S43tntFqe8dSl3M1kvqoqq1AsOKNP7BL4axMktKqdSLtLDj6G01FtREaaeKnsuzMktCvQsqKcgjjaqX+wwN2/BgoQMHHC4kYRsGQvo2UQGDT9wWIuICQiLROXrIkrdDWZwyDGLbNO7QajNlXVrHNcPEbdEuNv0BOVUU2Hf2hV2xc/aV/WJ/760VJTXiXvZpNvte7utjH6fqfx51OTRL7P53PdizRAtvkl4F9e4nTHwKq+/vHhxd1RdWC9EMO2W/qf8TdsnO6ARu99r6VOOrx8jSA1RuX/dd0Jgrz5crtVf5pbfpSwziJaZRpOt+jSW8QxUN2vYbLvADP5Wysqa8V7R+qpJJPZO4EQr/B1NjpCc= vT h tanh(Uh tj + Wh ti ) AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw== AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw== AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw== AAACVHichVG7TgJBFD2sL1wfrFYmNkSCsSIXGx+ViY0lDxESJGR3GXDDsrvZXUiQ+AO0FhZWmhhj/Awbf8CC3sZYYmJj4WUhMUrUO5mZM2fm3DlzR3NMw/OJeiFpYnJqeiY8K8/NywuLEWX+yLObri5yum3abkFTPWEalsj5hm+KguMKtaGZIq/V9wf7+ZZwPcO2Dv22I0oNtWYZVUNXfaZSZSVGCQoiOg6SIxDDKGzlFseowIaOJhoQsOAzNqHC41ZEEgSHuRI6zLmMjGBf4Awya5t8SvAJldk6jzVeFUesxetBTi9Q63yLyd1lZRRxeqI76tMj3dMLffyaqxPkGHhp86wNtcIpR7or2fd/VQ2efZx8qf707KOK7cCrwd6dgBm8Qh/qW6cX/exuJt5Zp2t6Zf9X1KMHfoHVetNv0iJzCZnrn/xZ7XGQ20zsJJJpQhirWMMGV3kLezhACjm+rYIuzkPPkiwpw2+SQqP/WsK3kJY/AXjih4w= AAACvHichVHNStxQFP6MbbWjraOrQjeharEUhjPdVF0Jboor/9IRjIYkXs3V/JHcCWqYF+gLdOFKQUR8DDf2AbrwEcRFFxaq4MKTzEBpB9sTcu853znfud+9x4l9mSqiyx6t98nTZ339zysDgy9eDlWHBz+nUTNxheFGfpSsOHYqfBkKQ0nli5U4EXbg+KLh7MwW+UYmklRG4bLai8VaYG+FclO6tmLIqrpm4ES7uelE/kZgKy8fy8ZaLctbN4soCfLllm4qO/QmDMvTu4pVUbytv9cbj2blO6s6SjUqTe926h1nFB2bj6onMLGBCC6aCCAQQrHvw0bK3yrqIMSMrSFnLGFPlnmBFirMbXKV4Aqb0R1etzha7aAhx0XPtGS7fIrPf8JMHeP0nU7phi7ojK7o/tFeedmj0LLHu9Pmitga+vJq6dd/WQHvCt5v1j81K2xistQqWXtcIsUt3DY/2/96szS9OJ6/pSO6Zv2HdEnnfIMw++keL4jFA1R4APW/n7vbMT7Upmr1BUI/XuMNJviVP2IGnzAPg0/7hh+4xZ02p8Va1p6U1tMZ2Qj+MG33AcrasJw= AAACvHichVHNStxQFP6MbbWjraOrQjeharEUhjPdVF0Jboor/9IRjIYkXs3V/JHcCWqYF+gLdOFKQUR8DDf2AbrwEcRFFxaq4MKTzEBpB9sTcu853znfud+9x4l9mSqiyx6t98nTZ339zysDgy9eDlWHBz+nUTNxheFGfpSsOHYqfBkKQ0nli5U4EXbg+KLh7MwW+UYmklRG4bLai8VaYG+FclO6tmLIqrpm4ES7uelE/kZgKy8fy8ZaLctbN4soCfLllm4qO/QmDMvTu4pVUbytv9cbj2blO6s6SjUqTe926h1nFB2bj6onMLGBCC6aCCAQQrHvw0bK3yrqIMSMrSFnLGFPlnmBFirMbXKV4Aqb0R1etzha7aAhx0XPtGS7fIrPf8JMHeP0nU7phi7ojK7o/tFeedmj0LLHu9Pmitga+vJq6dd/WQHvCt5v1j81K2xistQqWXtcIsUt3DY/2/96szS9OJ6/pSO6Zv2HdEnnfIMw++keL4jFA1R4APW/n7vbMT7Upmr1BUI/XuMNJviVP2IGnzAPg0/7hh+4xZ02p8Va1p6U1tMZ2Qj+MG33AcrasJw= AAACx3ichVFNTxRBEH2MKLh+sMrFxMvEBYMx2dR6QTgRuBBPfI1LwuBkZmiYlvnKTO8EmOyBK3+AgydNjDH+DC/4AzzwEwwHDpAAiQdrZicxukGr091Vr+pVv+52Yl+miuh4QLsxePPW0PDt2p279+6P1B88fJ1GncQVhhv5UbLq2KnwZSgMJZUvVuNE2IHji7azPVfk25lIUhmFK2o3FuuBvRXKTenaiiGr7pqBE+3kphP5G4GtvHwsG+t2Le+NWURJkK90dVPZoTdhWJ7eV6yK4rf6c719bVY+s+oNalJper/TqpwGKluI6p9gYgMRXHQQQCCEYt+HjZTHGlogxIytI2csYU+WeYEuasztcJXgCpvRbV63OFqr0JDjomdasl0+xeeZMFPHOH2nz3RGR/SFftDPa3vlZY9Cyy7vTo8rYmvk4NHyxX9ZAe8K3m/WPzUrbOJlqVWy9rhEilu4PX62d3i2PL00nj+lD3TC+t/TMX3lG4TZuftxUSy9Q40/oPX3c/c7xovmVLO1SI2Z2eonhvEYTzDBzz2JGcxjAQYf+w2nuMSV9kqLtUzb6ZVqAxVnFH+Ytv8LbIixyw== AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw== AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw== AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw== AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw== AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw== AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw== tj AAACe3ichVFNLwNBGH66vuurOJC4NNqKiDRTIT5OwsWRUpq0TbO7HSz7ld1pg03/gD/g4ISICH6Fiz/g4CeII4mLiLfbTYQG72Rmnnnmfd55Zkaxdc0VjD2GpKbmlta29o5wZ1d3T2+kr3/DtcqOyjOqpVtOVpFdrmsmzwhN6DxrO1w2FJ1vKntLtf3NCndczTLXxYHNC4a8bWpbmioLooqRwbyhWPteXrH0kiGLHS8u4tVqcbcYibEk8yPaCFIBiCGIFStyiTxKsKCiDAMcJgRhHTJcajmkwGATV4BHnENI8/c5qgiTtkxZnDJkYvdo3KZVLmBNWtdqur5apVN06g4po0iwB3bFXtg9u2ZP7P3XWp5fo+blgGalruV2sfdoaO3tX5VBs8DOl+pPzwJbmPW9auTd9pnaLdS6vnJ4/LI2n054o+yMPZP/U/bI7ugGZuVVvVjl6ROE6QNSP5+7EWQmk3PJ1OpUbGEx+Il2DGMEY/TcM1jAMlaQ8Y89xw1uQx9SXBqXJuqpUijQDOBbSNOf8PCTjw== AAACe3ichVFNLwNBGH66vuurOJC4NNqKiDRTIT5OwsWRUpq0TbO7HSz7ld1pg03/gD/g4ISICH6Fiz/g4CeII4mLiLfbTYQG72Rmnnnmfd55Zkaxdc0VjD2GpKbmlta29o5wZ1d3T2+kr3/DtcqOyjOqpVtOVpFdrmsmzwhN6DxrO1w2FJ1vKntLtf3NCndczTLXxYHNC4a8bWpbmioLooqRwbyhWPteXrH0kiGLHS8u4tVqcbcYibEk8yPaCFIBiCGIFStyiTxKsKCiDAMcJgRhHTJcajmkwGATV4BHnENI8/c5qgiTtkxZnDJkYvdo3KZVLmBNWtdqur5apVN06g4po0iwB3bFXtg9u2ZP7P3XWp5fo+blgGalruV2sfdoaO3tX5VBs8DOl+pPzwJbmPW9auTd9pnaLdS6vnJ4/LI2n054o+yMPZP/U/bI7ugGZuVVvVjl6ROE6QNSP5+7EWQmk3PJ1OpUbGEx+Il2DGMEY/TcM1jAMlaQ8Y89xw1uQx9SXBqXJuqpUijQDOBbSNOf8PCTjw== AAACe3ichVFNLwNBGH66vuurOJC4NNqKiDRTIT5OwsWRUpq0TbO7HSz7ld1pg03/gD/g4ISICH6Fiz/g4CeII4mLiLfbTYQG72Rmnnnmfd55Zkaxdc0VjD2GpKbmlta29o5wZ1d3T2+kr3/DtcqOyjOqpVtOVpFdrmsmzwhN6DxrO1w2FJ1vKntLtf3NCndczTLXxYHNC4a8bWpbmioLooqRwbyhWPteXrH0kiGLHS8u4tVqcbcYibEk8yPaCFIBiCGIFStyiTxKsKCiDAMcJgRhHTJcajmkwGATV4BHnENI8/c5qgiTtkxZnDJkYvdo3KZVLmBNWtdqur5apVN06g4po0iwB3bFXtg9u2ZP7P3XWp5fo+blgGalruV2sfdoaO3tX5VBs8DOl+pPzwJbmPW9auTd9pnaLdS6vnJ4/LI2n054o+yMPZP/U/bI7ugGZuVVvVjl6ROE6QNSP5+7EWQmk3PJ1OpUbGEx+Il2DGMEY/TcM1jAMlaQ8Y89xw1uQx9SXBqXJuqpUijQDOBbSNOf8PCTjw== AAACe3ichVFNLwNBGH66vuurOJC4NNqKiDRTIT5OwsWRUpq0TbO7HSz7ld1pg03/gD/g4ISICH6Fiz/g4CeII4mLiLfbTYQG72Rmnnnmfd55Zkaxdc0VjD2GpKbmlta29o5wZ1d3T2+kr3/DtcqOyjOqpVtOVpFdrmsmzwhN6DxrO1w2FJ1vKntLtf3NCndczTLXxYHNC4a8bWpbmioLooqRwbyhWPteXrH0kiGLHS8u4tVqcbcYibEk8yPaCFIBiCGIFStyiTxKsKCiDAMcJgRhHTJcajmkwGATV4BHnENI8/c5qgiTtkxZnDJkYvdo3KZVLmBNWtdqur5apVN06g4po0iwB3bFXtg9u2ZP7P3XWp5fo+blgGalruV2sfdoaO3tX5VBs8DOl+pPzwJbmPW9auTd9pnaLdS6vnJ4/LI2n054o+yMPZP/U/bI7ugGZuVVvVjl6ROE6QNSP5+7EWQmk3PJ1OpUbGEx+Il2DGMEY/TcM1jAMlaQ8Y89xw1uQx9SXBqXJuqpUijQDOBbSNOf8PCTjw== ti AAACe3ichVHNLkNBGD29/uuvWJDYNFoiIs1UiJ9Vw8ayRZGoNPdeUybuX+6dNrjpC3gBCytERPAUNl7AwiOIJYmNiK+3NxEE32Rmzpz5zjdnZjTHEJ5k7CGiNDQ2Nbe0tkXbOzq7umM9vaueXXZ1ntdtw3bXNdXjhrB4Xgpp8HXH5aqpGXxN212o7a9VuOsJ21qR+w7fNNVtS5SErkqiirH+gqnZe35Bs40tU5U7flImq9WiKMYSLMWCiP8E6RAkEEbWjl2ggC3Y0FGGCQ4LkrABFR61DaTB4BC3CZ84l5AI9jmqiJK2TFmcMlRid2ncptVGyFq0rtX0ArVOpxjUXVLGMczu2SV7Znfsij2yt19r+UGNmpd9mrW6ljvF7sOB5dd/VSbNEjufqj89S5QwE3gV5N0JmNot9Lq+cnD0vDy3NOyPsFP2RP5P2AO7pRtYlRf9PMeXjhGlD0h/f+6fID+Rmk2lc5OJzHz4E60YxBBG6bmnkcEissgHx57hGjeRdyWpjCnj9VQlEmr68CWUqQ/u8ZOO AAACe3ichVHNLkNBGD29/uuvWJDYNFoiIs1UiJ9Vw8ayRZGoNPdeUybuX+6dNrjpC3gBCytERPAUNl7AwiOIJYmNiK+3NxEE32Rmzpz5zjdnZjTHEJ5k7CGiNDQ2Nbe0tkXbOzq7umM9vaueXXZ1ntdtw3bXNdXjhrB4Xgpp8HXH5aqpGXxN212o7a9VuOsJ21qR+w7fNNVtS5SErkqiirH+gqnZe35Bs40tU5U7flImq9WiKMYSLMWCiP8E6RAkEEbWjl2ggC3Y0FGGCQ4LkrABFR61DaTB4BC3CZ84l5AI9jmqiJK2TFmcMlRid2ncptVGyFq0rtX0ArVOpxjUXVLGMczu2SV7Znfsij2yt19r+UGNmpd9mrW6ljvF7sOB5dd/VSbNEjufqj89S5QwE3gV5N0JmNot9Lq+cnD0vDy3NOyPsFP2RP5P2AO7pRtYlRf9PMeXjhGlD0h/f+6fID+Rmk2lc5OJzHz4E60YxBBG6bmnkcEissgHx57hGjeRdyWpjCnj9VQlEmr68CWUqQ/u8ZOO AAACe3ichVHNLkNBGD29/uuvWJDYNFoiIs1UiJ9Vw8ayRZGoNPdeUybuX+6dNrjpC3gBCytERPAUNl7AwiOIJYmNiK+3NxEE32Rmzpz5zjdnZjTHEJ5k7CGiNDQ2Nbe0tkXbOzq7umM9vaueXXZ1ntdtw3bXNdXjhrB4Xgpp8HXH5aqpGXxN212o7a9VuOsJ21qR+w7fNNVtS5SErkqiirH+gqnZe35Bs40tU5U7flImq9WiKMYSLMWCiP8E6RAkEEbWjl2ggC3Y0FGGCQ4LkrABFR61DaTB4BC3CZ84l5AI9jmqiJK2TFmcMlRid2ncptVGyFq0rtX0ArVOpxjUXVLGMczu2SV7Znfsij2yt19r+UGNmpd9mrW6ljvF7sOB5dd/VSbNEjufqj89S5QwE3gV5N0JmNot9Lq+cnD0vDy3NOyPsFP2RP5P2AO7pRtYlRf9PMeXjhGlD0h/f+6fID+Rmk2lc5OJzHz4E60YxBBG6bmnkcEissgHx57hGjeRdyWpjCnj9VQlEmr68CWUqQ/u8ZOO AAACe3ichVHNLkNBGD29/uuvWJDYNFoiIs1UiJ9Vw8ayRZGoNPdeUybuX+6dNrjpC3gBCytERPAUNl7AwiOIJYmNiK+3NxEE32Rmzpz5zjdnZjTHEJ5k7CGiNDQ2Nbe0tkXbOzq7umM9vaueXXZ1ntdtw3bXNdXjhrB4Xgpp8HXH5aqpGXxN212o7a9VuOsJ21qR+w7fNNVtS5SErkqiirH+gqnZe35Bs40tU5U7flImq9WiKMYSLMWCiP8E6RAkEEbWjl2ggC3Y0FGGCQ4LkrABFR61DaTB4BC3CZ84l5AI9jmqiJK2TFmcMlRid2ncptVGyFq0rtX0ArVOpxjUXVLGMczu2SV7Znfsij2yt19r+UGNmpd9mrW6ljvF7sOB5dd/VSbNEjufqj89S5QwE3gV5N0JmNot9Lq+cnD0vDy3NOyPsFP2RP5P2AO7pRtYlRf9PMeXjhGlD0h/f+6fID+Rmk2lc5OJzHz4E60YxBBG6bmnkcEissgHx57hGjeRdyWpjCnj9VQlEmr68CWUqQ/u8ZOO ti AAACZXichVHLSgMxFD0d31VrfSCCC4ul4qqkIvhYFd24bK21hVrKzJjW4HRmmEkLWvwBwa1duFIQET/DjT/gwj9QXFZw48Lb6YBoUW9IcnJyz81JotmGcCVjTwGlp7evf2BwKDg8MhoaC49P7LpWzdF5VrcMy8lrqssNYfKsFNLgedvhalUzeE473Gzv5+rccYVl7sgjmxerasUUZaGrkqiMLIlSOMrizItIN0j4IAo/Ulb4BnvYhwUdNVTBYUISNqDCpVZAAgw2cUU0iHMICW+f4wRB0tYoi1OGSuwhjRVaFXzWpHW7puupdTrFoO6QMoIYe2S3rMUe2B17YR+/1mp4NdpejmjWOlpul8ZOZzLv/6qqNEscfKn+9CxRxqrnVZB322Pat9A7+vpxs5VZ3441FtgVeyX/l+yJ3dMNzPqbfp3m2xcI0gckfj53N8guxdfiifRyNLnh/8QgZjGPRXruFSSxhRSydGwFZzhHM/CshJQpZbqTqgR8zSS+hTL3CTmHiqQ= AAACZXichVHLSgMxFD0d31VrfSCCC4ul4qqkIvhYFd24bK21hVrKzJjW4HRmmEkLWvwBwa1duFIQET/DjT/gwj9QXFZw48Lb6YBoUW9IcnJyz81JotmGcCVjTwGlp7evf2BwKDg8MhoaC49P7LpWzdF5VrcMy8lrqssNYfKsFNLgedvhalUzeE473Gzv5+rccYVl7sgjmxerasUUZaGrkqiMLIlSOMrizItIN0j4IAo/Ulb4BnvYhwUdNVTBYUISNqDCpVZAAgw2cUU0iHMICW+f4wRB0tYoi1OGSuwhjRVaFXzWpHW7puupdTrFoO6QMoIYe2S3rMUe2B17YR+/1mp4NdpejmjWOlpul8ZOZzLv/6qqNEscfKn+9CxRxqrnVZB322Pat9A7+vpxs5VZ3441FtgVeyX/l+yJ3dMNzPqbfp3m2xcI0gckfj53N8guxdfiifRyNLnh/8QgZjGPRXruFSSxhRSydGwFZzhHM/CshJQpZbqTqgR8zSS+hTL3CTmHiqQ= AAACZXichVHLSgMxFD0d31VrfSCCC4ul4qqkIvhYFd24bK21hVrKzJjW4HRmmEkLWvwBwa1duFIQET/DjT/gwj9QXFZw48Lb6YBoUW9IcnJyz81JotmGcCVjTwGlp7evf2BwKDg8MhoaC49P7LpWzdF5VrcMy8lrqssNYfKsFNLgedvhalUzeE473Gzv5+rccYVl7sgjmxerasUUZaGrkqiMLIlSOMrizItIN0j4IAo/Ulb4BnvYhwUdNVTBYUISNqDCpVZAAgw2cUU0iHMICW+f4wRB0tYoi1OGSuwhjRVaFXzWpHW7puupdTrFoO6QMoIYe2S3rMUe2B17YR+/1mp4NdpejmjWOlpul8ZOZzLv/6qqNEscfKn+9CxRxqrnVZB322Pat9A7+vpxs5VZ3441FtgVeyX/l+yJ3dMNzPqbfp3m2xcI0gckfj53N8guxdfiifRyNLnh/8QgZjGPRXruFSSxhRSydGwFZzhHM/CshJQpZbqTqgR8zSS+hTL3CTmHiqQ= AAACZXichVHLSgMxFD0d31VrfSCCC4ul4qqkIvhYFd24bK21hVrKzJjW4HRmmEkLWvwBwa1duFIQET/DjT/gwj9QXFZw48Lb6YBoUW9IcnJyz81JotmGcCVjTwGlp7evf2BwKDg8MhoaC49P7LpWzdF5VrcMy8lrqssNYfKsFNLgedvhalUzeE473Gzv5+rccYVl7sgjmxerasUUZaGrkqiMLIlSOMrizItIN0j4IAo/Ulb4BnvYhwUdNVTBYUISNqDCpVZAAgw2cUU0iHMICW+f4wRB0tYoi1OGSuwhjRVaFXzWpHW7puupdTrFoO6QMoIYe2S3rMUe2B17YR+/1mp4NdpejmjWOlpul8ZOZzLv/6qqNEscfKn+9CxRxqrnVZB322Pat9A7+vpxs5VZ3441FtgVeyX/l+yJ3dMNzPqbfp3m2xcI0gckfj53N8guxdfiifRyNLnh/8QgZjGPRXruFSSxhRSydGwFZzhHM/CshJQpZbqTqgR8zSS+hTL3CTmHiqQ= tj AAACZXichVHLSsNAFD2Nr1ofrQ9EcGGxVFyVqQg+VkU3LtUaW6ilJHFaY9MkJNNCLf6A4FYXrhRExM9w4w+46B8oLiu4ceFtGhAt6h1m5syZe+6cmVFtQ3cFY82A1NPb1z8QHAwNDY+MhiNj43uuVXU0LmuWYTlZVXG5oZtcFroweNZ2uFJRDZ5Ryxvt/UyNO65umbuibvN8RSmZelHXFEFUWhSOCpEYSzAvot0g6YMY/NiyIrfYxwEsaKiiAg4TgrABBS61HJJgsInLo0GcQ0j39jlOECJtlbI4ZSjElmks0Srnsyat2zVdT63RKQZ1h5RRxNkTu2Mt9sju2Qv7+LVWw6vR9lKnWe1ouV0In06n3/9VVWgWOPxS/elZoIgVz6tO3m2Pad9C6+hrxxet9NpOvDHPrtkr+b9iTfZANzBrb9rNNt+5RIg+IPnzubuBvJhYTSS3l2Kpdf8ngpjBHBbouZeRwia2INOxJZzhHBeBZ2lUmpSmOqlSwNdM4FtIs587hoql AAACZXichVHLSsNAFD2Nr1ofrQ9EcGGxVFyVqQg+VkU3LtUaW6ilJHFaY9MkJNNCLf6A4FYXrhRExM9w4w+46B8oLiu4ceFtGhAt6h1m5syZe+6cmVFtQ3cFY82A1NPb1z8QHAwNDY+MhiNj43uuVXU0LmuWYTlZVXG5oZtcFroweNZ2uFJRDZ5Ryxvt/UyNO65umbuibvN8RSmZelHXFEFUWhSOCpEYSzAvot0g6YMY/NiyIrfYxwEsaKiiAg4TgrABBS61HJJgsInLo0GcQ0j39jlOECJtlbI4ZSjElmks0Srnsyat2zVdT63RKQZ1h5RRxNkTu2Mt9sju2Qv7+LVWw6vR9lKnWe1ouV0In06n3/9VVWgWOPxS/elZoIgVz6tO3m2Pad9C6+hrxxet9NpOvDHPrtkr+b9iTfZANzBrb9rNNt+5RIg+IPnzubuBvJhYTSS3l2Kpdf8ngpjBHBbouZeRwia2INOxJZzhHBeBZ2lUmpSmOqlSwNdM4FtIs587hoql AAACZXichVHLSsNAFD2Nr1ofrQ9EcGGxVFyVqQg+VkU3LtUaW6ilJHFaY9MkJNNCLf6A4FYXrhRExM9w4w+46B8oLiu4ceFtGhAt6h1m5syZe+6cmVFtQ3cFY82A1NPb1z8QHAwNDY+MhiNj43uuVXU0LmuWYTlZVXG5oZtcFroweNZ2uFJRDZ5Ryxvt/UyNO65umbuibvN8RSmZelHXFEFUWhSOCpEYSzAvot0g6YMY/NiyIrfYxwEsaKiiAg4TgrABBS61HJJgsInLo0GcQ0j39jlOECJtlbI4ZSjElmks0Srnsyat2zVdT63RKQZ1h5RRxNkTu2Mt9sju2Qv7+LVWw6vR9lKnWe1ouV0In06n3/9VVWgWOPxS/elZoIgVz6tO3m2Pad9C6+hrxxet9NpOvDHPrtkr+b9iTfZANzBrb9rNNt+5RIg+IPnzubuBvJhYTSS3l2Kpdf8ngpjBHBbouZeRwia2INOxJZzhHBeBZ2lUmpSmOqlSwNdM4FtIs587hoql AAACZXichVHLSsNAFD2Nr1ofrQ9EcGGxVFyVqQg+VkU3LtUaW6ilJHFaY9MkJNNCLf6A4FYXrhRExM9w4w+46B8oLiu4ceFtGhAt6h1m5syZe+6cmVFtQ3cFY82A1NPb1z8QHAwNDY+MhiNj43uuVXU0LmuWYTlZVXG5oZtcFroweNZ2uFJRDZ5Ryxvt/UyNO65umbuibvN8RSmZelHXFEFUWhSOCpEYSzAvot0g6YMY/NiyIrfYxwEsaKiiAg4TgrABBS61HJJgsInLo0GcQ0j39jlOECJtlbI4ZSjElmks0Srnsyat2zVdT63RKQZ1h5RRxNkTu2Mt9sju2Qv7+LVWw6vR9lKnWe1ouV0In06n3/9VVWgWOPxS/elZoIgVz6tO3m2Pad9C6+hrxxet9NpOvDHPrtkr+b9iTfZANzBrb9rNNt+5RIg+IPnzubuBvJhYTSS3l2Kpdf8ngpjBHBbouZeRwia2INOxJZzhHBeBZ2lUmpSmOqlSwNdM4FtIs587hoql

Slide 10

Slide 10 text

40 • /+Q1IRNJ – 3KHJLIRNJ (,): (4 – 3FGO!PRMIRNJ (FGO): (1.5 – /+: ,=FGOE&, 1: ,QFGO5> • 1 – ).#Juman++>AC- .#*$E > 8C – 2B6E"B6>7< 1 • ": 1;?-'2=0 ?2 • ?%.#=D9C:@ 10

Slide 11

Slide 11 text

Pre-Training • 32K, BERTBASE (L=12, A=12, H=768) 11 1GPU 1epoch1

Slide 12

Slide 12 text

12 +1.85 90.22 91.12 91.22 91.84 93.69 88.44 89.32 90.07 90.2 92.57 85 86 87 88 89 90 91 92 93 94 95 KNP Cabocha J.depP BiLSTM BERT +2.37

Slide 13

Slide 13 text

: 13 BiLSTM BERT Δ BiLSTM BERT Δ → 84.7 87.3 +2.6 2,050 86.0 90.5 +4.5 1,367 → 90.0 92.2 +2.2 1,367 92.2 95.1 +2.9 1,093 → 90.9 93.7 +2.8 3,653 90.0 92.2 +2.2 2,372 → 90.9 94.0 +3.1 6,300 91.0 94.2 +3.2 3,777 91.6 94.7 +3.1 790 93.9 95.9 +2.0 393 ! 97.7 99.0 +1.3 1,154 98.0 99.2 +1.2 605 94.8 96.4 +1.6 1,114 94.9 97.0 +2.1 788 88.1 90.8 +2.7 1,284 89.2 90.9 +1.7 861 83.2 89.9 +6.7 328 75.5 88.7 +13.2 151 …

Slide 14

Slide 14 text

( ) 14

Slide 15

Slide 15 text

: 15 Pre-trainingepoch

Slide 16

Slide 16 text

: 16 Pre-training 3/4 1/2 3/4 1/2

Slide 17

Slide 17 text

: 17 Transformer L=6 L=6 L=12 L=12

Slide 18

Slide 18 text

95:57%+ • BERT>$"6) • > ./<05-4.2 • 0=947 ,>3< – FDGA?C;pre-training7@IEBA?C> ./1<054- – #H ! • 7%+ – &7 – '$*("H"87) 18