NLP2019_BERT_Parsing_Shibata.pdf

 NLP2019_BERT_Parsing_Shibata.pdf

BERTによる日本語構文解析の精度向上
柴田知秀, 河原大輔, 黒橋禎夫 (京大)
NLP2019

Fc659c8d5c6aa37f099dc273751271f0?s=128

tomohideshibata

March 13, 2019
Tweet

Transcript

  1. BERT         

     /CREST 19/03/13
  2.  (1/2) • End-to-end      

    ? 2 Figure 1: The Transformer - model architecture. NMT [Vaswani+ 17] #*('&     )$     !(% +      " " "     
  3.  (1/2) • End-to-end      

    ? 3 Figure 1: The Transformer - model architecture. NMT [Vaswani+ 17] #*('&     )$     !(% +      " " "      
  4. +$,* (2/2) • &-&-/*"210 – 3  4  [+

    2017] #*('&     )$     !(% +      " " "     /'/ %.#) (!.
  5. 1*20 (2/2) • ,3,35%0($876 – 9 " 5 [+ 2017]

    #*('&     )$     !(% +      " " "  #! 5$- 5+4)/&.'4  : 90% 1 : 5060%
  6. BERT [Devlin+ 2018] 6 2. Fine-tuning 1. Pre-training $PILRH;<-"@ 8:UJRX#H

    /FMLJ (: ')2 %6FD0@VQXHNW[ SYK • Transformer [Vaswani+ 17]H &(%@  • A2LPOT@DE-.?MLJ= ? + A [MASK] … Q[M  [MASK] … 7A + B 95 @ 3GF … “!” negative ZZZ ZZZ ZZZ ZZZ ZZZ ZZZ ZZZ ZZZ  “A” A B “0C ,”H%4 A, B2* 1>01H)
  7. 0 • BERT%BK<J0 (@>:HJ=) /6+#8&,9( – ;H?CI00 : • 14)12/

    (8" (RTE.-) • BK<J/ (8GEHJ= (NER) • @DJ (SQuAD 0QA) • Google%'*$8FAI0 0$/!#7 (23 ) → 05+Pre-Training 7 BERT E [CLS] E 1 E [SEP] ... E N E 1 ’ ... E M ’ C T 1 T [SEP] ... T N T 1 ’ ... T M ’ [CLS] Tok 1 [SEP] ... Tok N Tok 1 ... Tok M Question Paragraph BERT E [CLS] E 1 E 2 E N C T 1 T 2 T N Single Sentence ... ... BERT Tok 1 Tok 2 Tok N ... [CLS] E [CLS] E 1 E 2 E N C T 1 T 2 T N Single Sentence B-PER O O ... ... E [CLS] E 1 E [SEP] Class Label ... E N E 1 ’ ... E M ’ C T 1 T [SEP] ... T N T 1 ’ ... T M ’ Start/End Span Class Label BERT Tok 1 Tok 2 Tok N ... [CLS] Tok 1 [CLS] [CLS] Tok 1 [SEP] ... Tok N Tok 1 ... Tok M Sentence 1 ... Sentence 2 Figure 3: Our task specific models are formed by incorporating BERT with one additional output layer, s minimal number of parameters need to be learned from scratch. Among the tasks, (a) and (b) are sequence-le tasks while (c) and (d) are token-level tasks. In the figure, E represents the input embedding, Ti represents contextual representation of token i, [CLS] is the special symbol for classification output, and [SEP] is the spec symbol to separate non-consecutive token sequences. QNLI Question Natural Language Inference is a version of the Stanford Question Answering Dataset (Rajpurkar et al., 2016) which has been converted to a binary classification task (Wang et al., 2018). The positive examples are (ques- tion, sentence) pairs which do contain the correct answer, and the negative examples are (question, sentence) from the same paragraph which do not contain the answer. SST-2 The Stanford Sentiment Treebank is a binary single-sentence classification task consist- ing of sentences extracted from movie reviews with human annotations of their sentiment (Socher et al., 2013). CoLA The Corpus of Linguistic Acceptability is a binary single-sentence classification task, where the goal is to predict whether an English senten is linguistically “acceptable” or not (Warst et al., 2018). STS-B The Semantic Textual Similarity Benc mark is a collection of sentence pairs drawn fro news headlines and other sources (Cer et 2017). They were annotated with a score from to 5 denoting how similar the two sentences are terms of semantic meaning. MRPC Microsoft Research Paraphrase Corp consists of sentence pairs automatically extrac from online news sources, with human annotatio for whether the sentences in the pair are seman cally equivalent (Dolan and Brockett, 2005).
  8. BERTDKM(& 8 $*  F + E $* 1 M2F

    <N D ;M * D KM  D 8? D  7 N / Um`aS J <N D ;M * E  D 8? F Um`aS  J gm^m XSVi jm\ CB A G NM 0$*  FM  E … … rpq . n cm ##Zm ##Te 3 Q ##5Q o F / WkbV` _R\V CB @ O NM bi\ ! ) % P 2= _[^i ##Zk]X SYm E .  E HA > 0 1 L49I fgh D '- 8? 32= rpq  n Xkdi o P  : M 6A @ … … Pre-training _m^ ( Wikipedia ("1,800) [CLS] $*  F + E $* 1M2F <N D ;M * D KM  D 8? D  7 N / Um`aS [MASK] … G NM 0 [SEP] $*  F M [MASK] E… [SEP] Label: IsNext Pre-training Fine-tuning #&lBPEP, lll lll lll lll
  9.  Fine-tuning 9 [CLS]      ##

           [SEP]  [ROOT] BERT X training/test   head selection [Zhang+ 17] Phead(tj |ti, S) = exp(s(tj, ti)) k exp(s(tk, ti)) <latexit sha1_base64="yIYC5ozZHCUA9ARdZwTtSL7S/r8=">AAACq3ichVFNSxtBGH6yba2mfkS9CL0shkgCEiZSaBUEqQd7TIwxUleW3XWiY/aL3UmorvkD/QMePCmUUkp/hRfv0oPQ/oDSo0IvPfTdzUJRUd9lZ555nvd5550Z07dFKBm7zChPnj4beD44lH0xPDI6lhufWA+9TmDxhuXZXrBhGiG3hcsbUkibb/gBNxzT5k2zvRzrzS4PQuG5a3Lf51uOseOKlrAMSZSeW6nq0S43tntFqe8dSl3M1kvqoqq1AsOKNP7BL4axMktKqdSLtLDj6G01FtREaaeKnsuzMktCvQsqKcgjjaqX+wwN2/BgoQMHHC4kYRsGQvo2UQGDT9wWIuICQiLROXrIkrdDWZwyDGLbNO7QajNlXVrHNcPEbdEuNv0BOVUU2Hf2hV2xc/aV/WJ/760VJTXiXvZpNvte7utjH6fqfx51OTRL7P53PdizRAtvkl4F9e4nTHwKq+/vHhxd1RdWC9EMO2W/qf8TdsnO6ARu99r6VOOrx8jSA1RuX/dd0Jgrz5crtVf5pbfpSwziJaZRpOt+jSW8QxUN2vYbLvADP5Wysqa8V7R+qpJJPZO4EQr/B1NjpCc=</latexit> <latexit sha1_base64="yIYC5ozZHCUA9ARdZwTtSL7S/r8=">AAACq3ichVFNSxtBGH6yba2mfkS9CL0shkgCEiZSaBUEqQd7TIwxUleW3XWiY/aL3UmorvkD/QMePCmUUkp/hRfv0oPQ/oDSo0IvPfTdzUJRUd9lZ555nvd5550Z07dFKBm7zChPnj4beD44lH0xPDI6lhufWA+9TmDxhuXZXrBhGiG3hcsbUkibb/gBNxzT5k2zvRzrzS4PQuG5a3Lf51uOseOKlrAMSZSeW6nq0S43tntFqe8dSl3M1kvqoqq1AsOKNP7BL4axMktKqdSLtLDj6G01FtREaaeKnsuzMktCvQsqKcgjjaqX+wwN2/BgoQMHHC4kYRsGQvo2UQGDT9wWIuICQiLROXrIkrdDWZwyDGLbNO7QajNlXVrHNcPEbdEuNv0BOVUU2Hf2hV2xc/aV/WJ/760VJTXiXvZpNvte7utjH6fqfx51OTRL7P53PdizRAtvkl4F9e4nTHwKq+/vHhxd1RdWC9EMO2W/qf8TdsnO6ARu99r6VOOrx8jSA1RuX/dd0Jgrz5crtVf5pbfpSwziJaZRpOt+jSW8QxUN2vYbLvADP5Wysqa8V7R+qpJJPZO4EQr/B1NjpCc=</latexit> <latexit sha1_base64="yIYC5ozZHCUA9ARdZwTtSL7S/r8=">AAACq3ichVFNSxtBGH6yba2mfkS9CL0shkgCEiZSaBUEqQd7TIwxUleW3XWiY/aL3UmorvkD/QMePCmUUkp/hRfv0oPQ/oDSo0IvPfTdzUJRUd9lZ555nvd5550Z07dFKBm7zChPnj4beD44lH0xPDI6lhufWA+9TmDxhuXZXrBhGiG3hcsbUkibb/gBNxzT5k2zvRzrzS4PQuG5a3Lf51uOseOKlrAMSZSeW6nq0S43tntFqe8dSl3M1kvqoqq1AsOKNP7BL4axMktKqdSLtLDj6G01FtREaaeKnsuzMktCvQsqKcgjjaqX+wwN2/BgoQMHHC4kYRsGQvo2UQGDT9wWIuICQiLROXrIkrdDWZwyDGLbNO7QajNlXVrHNcPEbdEuNv0BOVUU2Hf2hV2xc/aV/WJ/760VJTXiXvZpNvte7utjH6fqfx51OTRL7P53PdizRAtvkl4F9e4nTHwKq+/vHhxd1RdWC9EMO2W/qf8TdsnO6ARu99r6VOOrx8jSA1RuX/dd0Jgrz5crtVf5pbfpSwziJaZRpOt+jSW8QxUN2vYbLvADP5Wysqa8V7R+qpJJPZO4EQr/B1NjpCc=</latexit> <latexit sha1_base64="yIYC5ozZHCUA9ARdZwTtSL7S/r8=">AAACq3ichVFNSxtBGH6yba2mfkS9CL0shkgCEiZSaBUEqQd7TIwxUleW3XWiY/aL3UmorvkD/QMePCmUUkp/hRfv0oPQ/oDSo0IvPfTdzUJRUd9lZ555nvd5550Z07dFKBm7zChPnj4beD44lH0xPDI6lhufWA+9TmDxhuXZXrBhGiG3hcsbUkibb/gBNxzT5k2zvRzrzS4PQuG5a3Lf51uOseOKlrAMSZSeW6nq0S43tntFqe8dSl3M1kvqoqq1AsOKNP7BL4axMktKqdSLtLDj6G01FtREaaeKnsuzMktCvQsqKcgjjaqX+wwN2/BgoQMHHC4kYRsGQvo2UQGDT9wWIuICQiLROXrIkrdDWZwyDGLbNO7QajNlXVrHNcPEbdEuNv0BOVUU2Hf2hV2xc/aV/WJ/760VJTXiXvZpNvte7utjH6fqfx51OTRL7P53PdizRAtvkl4F9e4nTHwKq+/vHhxd1RdWC9EMO2W/qf8TdsnO6ARu99r6VOOrx8jSA1RuX/dd0Jgrz5crtVf5pbfpSwziJaZRpOt+jSW8QxUN2vYbLvADP5Wysqa8V7R+qpJJPZO4EQr/B1NjpCc=</latexit> vT h tanh(Uh tj + Wh ti ) <latexit sha1_base64="t6EyU+MW6LS8WJOnyX+52LoX6HM=">AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw==</latexit> <latexit sha1_base64="t6EyU+MW6LS8WJOnyX+52LoX6HM=">AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw==</latexit> <latexit sha1_base64="t6EyU+MW6LS8WJOnyX+52LoX6HM=">AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw==</latexit> <latexit sha1_base64="kyXeYanE6VoW6jlj+lQX846h/OU=">AAACVHichVG7TgJBFD2sL1wfrFYmNkSCsSIXGx+ViY0lDxESJGR3GXDDsrvZXUiQ+AO0FhZWmhhj/Awbf8CC3sZYYmJj4WUhMUrUO5mZM2fm3DlzR3NMw/OJeiFpYnJqeiY8K8/NywuLEWX+yLObri5yum3abkFTPWEalsj5hm+KguMKtaGZIq/V9wf7+ZZwPcO2Dv22I0oNtWYZVUNXfaZSZSVGCQoiOg6SIxDDKGzlFseowIaOJhoQsOAzNqHC41ZEEgSHuRI6zLmMjGBf4Awya5t8SvAJldk6jzVeFUesxetBTi9Q63yLyd1lZRRxeqI76tMj3dMLffyaqxPkGHhp86wNtcIpR7or2fd/VQ2efZx8qf707KOK7cCrwd6dgBm8Qh/qW6cX/exuJt5Zp2t6Zf9X1KMHfoHVetNv0iJzCZnrn/xZ7XGQ20zsJJJpQhirWMMGV3kLezhACjm+rYIuzkPPkiwpw2+SQqP/WsK3kJY/AXjih4w=</latexit> <latexit sha1_base64="l25vOI9I0KHFgLCUm47hC+B3SPM=">AAACvHichVHNStxQFP6MbbWjraOrQjeharEUhjPdVF0Jboor/9IRjIYkXs3V/JHcCWqYF+gLdOFKQUR8DDf2AbrwEcRFFxaq4MKTzEBpB9sTcu853znfud+9x4l9mSqiyx6t98nTZ339zysDgy9eDlWHBz+nUTNxheFGfpSsOHYqfBkKQ0nli5U4EXbg+KLh7MwW+UYmklRG4bLai8VaYG+FclO6tmLIqrpm4ES7uelE/kZgKy8fy8ZaLctbN4soCfLllm4qO/QmDMvTu4pVUbytv9cbj2blO6s6SjUqTe926h1nFB2bj6onMLGBCC6aCCAQQrHvw0bK3yrqIMSMrSFnLGFPlnmBFirMbXKV4Aqb0R1etzha7aAhx0XPtGS7fIrPf8JMHeP0nU7phi7ojK7o/tFeedmj0LLHu9Pmitga+vJq6dd/WQHvCt5v1j81K2xistQqWXtcIsUt3DY/2/96szS9OJ6/pSO6Zv2HdEnnfIMw++keL4jFA1R4APW/n7vbMT7Upmr1BUI/XuMNJviVP2IGnzAPg0/7hh+4xZ02p8Va1p6U1tMZ2Qj+MG33AcrasJw=</latexit> <latexit sha1_base64="l25vOI9I0KHFgLCUm47hC+B3SPM=">AAACvHichVHNStxQFP6MbbWjraOrQjeharEUhjPdVF0Jboor/9IRjIYkXs3V/JHcCWqYF+gLdOFKQUR8DDf2AbrwEcRFFxaq4MKTzEBpB9sTcu853znfud+9x4l9mSqiyx6t98nTZ339zysDgy9eDlWHBz+nUTNxheFGfpSsOHYqfBkKQ0nli5U4EXbg+KLh7MwW+UYmklRG4bLai8VaYG+FclO6tmLIqrpm4ES7uelE/kZgKy8fy8ZaLctbN4soCfLllm4qO/QmDMvTu4pVUbytv9cbj2blO6s6SjUqTe926h1nFB2bj6onMLGBCC6aCCAQQrHvw0bK3yrqIMSMrSFnLGFPlnmBFirMbXKV4Aqb0R1etzha7aAhx0XPtGS7fIrPf8JMHeP0nU7phi7ojK7o/tFeedmj0LLHu9Pmitga+vJq6dd/WQHvCt5v1j81K2xistQqWXtcIsUt3DY/2/96szS9OJ6/pSO6Zv2HdEnnfIMw++keL4jFA1R4APW/n7vbMT7Upmr1BUI/XuMNJviVP2IGnzAPg0/7hh+4xZ02p8Va1p6U1tMZ2Qj+MG33AcrasJw=</latexit> <latexit sha1_base64="lkKpDh64gA5rjnE8UGWoMI+QDaw=">AAACx3ichVFNTxRBEH2MKLh+sMrFxMvEBYMx2dR6QTgRuBBPfI1LwuBkZmiYlvnKTO8EmOyBK3+AgydNjDH+DC/4AzzwEwwHDpAAiQdrZicxukGr091Vr+pVv+52Yl+miuh4QLsxePPW0PDt2p279+6P1B88fJ1GncQVhhv5UbLq2KnwZSgMJZUvVuNE2IHji7azPVfk25lIUhmFK2o3FuuBvRXKTenaiiGr7pqBE+3kphP5G4GtvHwsG+t2Le+NWURJkK90dVPZoTdhWJ7eV6yK4rf6c719bVY+s+oNalJper/TqpwGKluI6p9gYgMRXHQQQCCEYt+HjZTHGlogxIytI2csYU+WeYEuasztcJXgCpvRbV63OFqr0JDjomdasl0+xeeZMFPHOH2nz3RGR/SFftDPa3vlZY9Cyy7vTo8rYmvk4NHyxX9ZAe8K3m/WPzUrbOJlqVWy9rhEilu4PX62d3i2PL00nj+lD3TC+t/TMX3lG4TZuftxUSy9Q40/oPX3c/c7xovmVLO1SI2Z2eonhvEYTzDBzz2JGcxjAQYf+w2nuMSV9kqLtUzb6ZVqAxVnFH+Ytv8LbIixyw==</latexit> <latexit sha1_base64="t6EyU+MW6LS8WJOnyX+52LoX6HM=">AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw==</latexit> <latexit sha1_base64="t6EyU+MW6LS8WJOnyX+52LoX6HM=">AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw==</latexit> <latexit sha1_base64="t6EyU+MW6LS8WJOnyX+52LoX6HM=">AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw==</latexit> <latexit sha1_base64="t6EyU+MW6LS8WJOnyX+52LoX6HM=">AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw==</latexit> <latexit sha1_base64="t6EyU+MW6LS8WJOnyX+52LoX6HM=">AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw==</latexit> <latexit sha1_base64="t6EyU+MW6LS8WJOnyX+52LoX6HM=">AAACx3ichVFNSxxBEH2OiZo16mougVyGrIoiLL0SiHoSvUhOfo0rODrMjK3TOl/M9A6aYQ9e/QMePBkIIeRn5JL8gBz8CSGHHBRUyCE1swOii1pNd1e9qlf9utsKXRFLxs47lM5nz7u6e16Uel/29Q+UB4fW4qAR2VyzAzeI1i0z5q7wuSaFdPl6GHHTs1xet/bns3w94VEsAn9VHoZ80zN3fbEjbFMSZJRt3bOCg1S3AnfbM6WTDifDzabhbOlZFHnpalPVpek7Y5rhqG3FMiveUyfU+oNZMW6UK6zKclPbnVrhVFDYYlD+Ah3bCGCjAQ8cPiT5LkzENDZQA0NI2CZSwiLyRJ7naKJE3AZVcaowCd2ndZeijQL1Kc56xjnbplNcmhExVYywX+wru2A/2Df2m/17sFea98i0HNJutbg8NAaOX69cPcnyaJdwblmPapbYwVSuVZD2MEeyW9gtfvLx5GJlZnkkHWWf2B/Sf8bO2Xe6gZ9c2p+X+PIpSvQBtfvP3e5ok9Xpam3pXWV2rviJHrzBW4zRc7/HLBawCI2O/Ym/uMaN8kEJlUQ5aJUqHQXnFe6YcvQfbcixzw==</latexit> tj <latexit sha1_base64="jFGY8j1O+gMoB62K5/9EIEi0W98=">AAACe3ichVFNLwNBGH66vuurOJC4NNqKiDRTIT5OwsWRUpq0TbO7HSz7ld1pg03/gD/g4ISICH6Fiz/g4CeII4mLiLfbTYQG72Rmnnnmfd55Zkaxdc0VjD2GpKbmlta29o5wZ1d3T2+kr3/DtcqOyjOqpVtOVpFdrmsmzwhN6DxrO1w2FJ1vKntLtf3NCndczTLXxYHNC4a8bWpbmioLooqRwbyhWPteXrH0kiGLHS8u4tVqcbcYibEk8yPaCFIBiCGIFStyiTxKsKCiDAMcJgRhHTJcajmkwGATV4BHnENI8/c5qgiTtkxZnDJkYvdo3KZVLmBNWtdqur5apVN06g4po0iwB3bFXtg9u2ZP7P3XWp5fo+blgGalruV2sfdoaO3tX5VBs8DOl+pPzwJbmPW9auTd9pnaLdS6vnJ4/LI2n054o+yMPZP/U/bI7ugGZuVVvVjl6ROE6QNSP5+7EWQmk3PJ1OpUbGEx+Il2DGMEY/TcM1jAMlaQ8Y89xw1uQx9SXBqXJuqpUijQDOBbSNOf8PCTjw==</latexit> <latexit sha1_base64="jFGY8j1O+gMoB62K5/9EIEi0W98=">AAACe3ichVFNLwNBGH66vuurOJC4NNqKiDRTIT5OwsWRUpq0TbO7HSz7ld1pg03/gD/g4ISICH6Fiz/g4CeII4mLiLfbTYQG72Rmnnnmfd55Zkaxdc0VjD2GpKbmlta29o5wZ1d3T2+kr3/DtcqOyjOqpVtOVpFdrmsmzwhN6DxrO1w2FJ1vKntLtf3NCndczTLXxYHNC4a8bWpbmioLooqRwbyhWPteXrH0kiGLHS8u4tVqcbcYibEk8yPaCFIBiCGIFStyiTxKsKCiDAMcJgRhHTJcajmkwGATV4BHnENI8/c5qgiTtkxZnDJkYvdo3KZVLmBNWtdqur5apVN06g4po0iwB3bFXtg9u2ZP7P3XWp5fo+blgGalruV2sfdoaO3tX5VBs8DOl+pPzwJbmPW9auTd9pnaLdS6vnJ4/LI2n054o+yMPZP/U/bI7ugGZuVVvVjl6ROE6QNSP5+7EWQmk3PJ1OpUbGEx+Il2DGMEY/TcM1jAMlaQ8Y89xw1uQx9SXBqXJuqpUijQDOBbSNOf8PCTjw==</latexit> <latexit sha1_base64="jFGY8j1O+gMoB62K5/9EIEi0W98=">AAACe3ichVFNLwNBGH66vuurOJC4NNqKiDRTIT5OwsWRUpq0TbO7HSz7ld1pg03/gD/g4ISICH6Fiz/g4CeII4mLiLfbTYQG72Rmnnnmfd55Zkaxdc0VjD2GpKbmlta29o5wZ1d3T2+kr3/DtcqOyjOqpVtOVpFdrmsmzwhN6DxrO1w2FJ1vKntLtf3NCndczTLXxYHNC4a8bWpbmioLooqRwbyhWPteXrH0kiGLHS8u4tVqcbcYibEk8yPaCFIBiCGIFStyiTxKsKCiDAMcJgRhHTJcajmkwGATV4BHnENI8/c5qgiTtkxZnDJkYvdo3KZVLmBNWtdqur5apVN06g4po0iwB3bFXtg9u2ZP7P3XWp5fo+blgGalruV2sfdoaO3tX5VBs8DOl+pPzwJbmPW9auTd9pnaLdS6vnJ4/LI2n054o+yMPZP/U/bI7ugGZuVVvVjl6ROE6QNSP5+7EWQmk3PJ1OpUbGEx+Il2DGMEY/TcM1jAMlaQ8Y89xw1uQx9SXBqXJuqpUijQDOBbSNOf8PCTjw==</latexit> <latexit sha1_base64="jFGY8j1O+gMoB62K5/9EIEi0W98=">AAACe3ichVFNLwNBGH66vuurOJC4NNqKiDRTIT5OwsWRUpq0TbO7HSz7ld1pg03/gD/g4ISICH6Fiz/g4CeII4mLiLfbTYQG72Rmnnnmfd55Zkaxdc0VjD2GpKbmlta29o5wZ1d3T2+kr3/DtcqOyjOqpVtOVpFdrmsmzwhN6DxrO1w2FJ1vKntLtf3NCndczTLXxYHNC4a8bWpbmioLooqRwbyhWPteXrH0kiGLHS8u4tVqcbcYibEk8yPaCFIBiCGIFStyiTxKsKCiDAMcJgRhHTJcajmkwGATV4BHnENI8/c5qgiTtkxZnDJkYvdo3KZVLmBNWtdqur5apVN06g4po0iwB3bFXtg9u2ZP7P3XWp5fo+blgGalruV2sfdoaO3tX5VBs8DOl+pPzwJbmPW9auTd9pnaLdS6vnJ4/LI2n054o+yMPZP/U/bI7ugGZuVVvVjl6ROE6QNSP5+7EWQmk3PJ1OpUbGEx+Il2DGMEY/TcM1jAMlaQ8Y89xw1uQx9SXBqXJuqpUijQDOBbSNOf8PCTjw==</latexit> ti <latexit sha1_base64="ezfbbcdfITbMMO9qroMQqEOSuc8=">AAACe3ichVHNLkNBGD29/uuvWJDYNFoiIs1UiJ9Vw8ayRZGoNPdeUybuX+6dNrjpC3gBCytERPAUNl7AwiOIJYmNiK+3NxEE32Rmzpz5zjdnZjTHEJ5k7CGiNDQ2Nbe0tkXbOzq7umM9vaueXXZ1ntdtw3bXNdXjhrB4Xgpp8HXH5aqpGXxN212o7a9VuOsJ21qR+w7fNNVtS5SErkqiirH+gqnZe35Bs40tU5U7flImq9WiKMYSLMWCiP8E6RAkEEbWjl2ggC3Y0FGGCQ4LkrABFR61DaTB4BC3CZ84l5AI9jmqiJK2TFmcMlRid2ncptVGyFq0rtX0ArVOpxjUXVLGMczu2SV7Znfsij2yt19r+UGNmpd9mrW6ljvF7sOB5dd/VSbNEjufqj89S5QwE3gV5N0JmNot9Lq+cnD0vDy3NOyPsFP2RP5P2AO7pRtYlRf9PMeXjhGlD0h/f+6fID+Rmk2lc5OJzHz4E60YxBBG6bmnkcEissgHx57hGjeRdyWpjCnj9VQlEmr68CWUqQ/u8ZOO</latexit> <latexit sha1_base64="ezfbbcdfITbMMO9qroMQqEOSuc8=">AAACe3ichVHNLkNBGD29/uuvWJDYNFoiIs1UiJ9Vw8ayRZGoNPdeUybuX+6dNrjpC3gBCytERPAUNl7AwiOIJYmNiK+3NxEE32Rmzpz5zjdnZjTHEJ5k7CGiNDQ2Nbe0tkXbOzq7umM9vaueXXZ1ntdtw3bXNdXjhrB4Xgpp8HXH5aqpGXxN212o7a9VuOsJ21qR+w7fNNVtS5SErkqiirH+gqnZe35Bs40tU5U7flImq9WiKMYSLMWCiP8E6RAkEEbWjl2ggC3Y0FGGCQ4LkrABFR61DaTB4BC3CZ84l5AI9jmqiJK2TFmcMlRid2ncptVGyFq0rtX0ArVOpxjUXVLGMczu2SV7Znfsij2yt19r+UGNmpd9mrW6ljvF7sOB5dd/VSbNEjufqj89S5QwE3gV5N0JmNot9Lq+cnD0vDy3NOyPsFP2RP5P2AO7pRtYlRf9PMeXjhGlD0h/f+6fID+Rmk2lc5OJzHz4E60YxBBG6bmnkcEissgHx57hGjeRdyWpjCnj9VQlEmr68CWUqQ/u8ZOO</latexit> <latexit sha1_base64="ezfbbcdfITbMMO9qroMQqEOSuc8=">AAACe3ichVHNLkNBGD29/uuvWJDYNFoiIs1UiJ9Vw8ayRZGoNPdeUybuX+6dNrjpC3gBCytERPAUNl7AwiOIJYmNiK+3NxEE32Rmzpz5zjdnZjTHEJ5k7CGiNDQ2Nbe0tkXbOzq7umM9vaueXXZ1ntdtw3bXNdXjhrB4Xgpp8HXH5aqpGXxN212o7a9VuOsJ21qR+w7fNNVtS5SErkqiirH+gqnZe35Bs40tU5U7flImq9WiKMYSLMWCiP8E6RAkEEbWjl2ggC3Y0FGGCQ4LkrABFR61DaTB4BC3CZ84l5AI9jmqiJK2TFmcMlRid2ncptVGyFq0rtX0ArVOpxjUXVLGMczu2SV7Znfsij2yt19r+UGNmpd9mrW6ljvF7sOB5dd/VSbNEjufqj89S5QwE3gV5N0JmNot9Lq+cnD0vDy3NOyPsFP2RP5P2AO7pRtYlRf9PMeXjhGlD0h/f+6fID+Rmk2lc5OJzHz4E60YxBBG6bmnkcEissgHx57hGjeRdyWpjCnj9VQlEmr68CWUqQ/u8ZOO</latexit> <latexit sha1_base64="ezfbbcdfITbMMO9qroMQqEOSuc8=">AAACe3ichVHNLkNBGD29/uuvWJDYNFoiIs1UiJ9Vw8ayRZGoNPdeUybuX+6dNrjpC3gBCytERPAUNl7AwiOIJYmNiK+3NxEE32Rmzpz5zjdnZjTHEJ5k7CGiNDQ2Nbe0tkXbOzq7umM9vaueXXZ1ntdtw3bXNdXjhrB4Xgpp8HXH5aqpGXxN212o7a9VuOsJ21qR+w7fNNVtS5SErkqiirH+gqnZe35Bs40tU5U7flImq9WiKMYSLMWCiP8E6RAkEEbWjl2ggC3Y0FGGCQ4LkrABFR61DaTB4BC3CZ84l5AI9jmqiJK2TFmcMlRid2ncptVGyFq0rtX0ArVOpxjUXVLGMczu2SV7Znfsij2yt19r+UGNmpd9mrW6ljvF7sOB5dd/VSbNEjufqj89S5QwE3gV5N0JmNot9Lq+cnD0vDy3NOyPsFP2RP5P2AO7pRtYlRf9PMeXjhGlD0h/f+6fID+Rmk2lc5OJzHz4E60YxBBG6bmnkcEissgHx57hGjeRdyWpjCnj9VQlEmr68CWUqQ/u8ZOO</latexit> ti <latexit sha1_base64="tJBUolgxH+Y4AhwAUb8db68d0EU=">AAACZXichVHLSgMxFD0d31VrfSCCC4ul4qqkIvhYFd24bK21hVrKzJjW4HRmmEkLWvwBwa1duFIQET/DjT/gwj9QXFZw48Lb6YBoUW9IcnJyz81JotmGcCVjTwGlp7evf2BwKDg8MhoaC49P7LpWzdF5VrcMy8lrqssNYfKsFNLgedvhalUzeE473Gzv5+rccYVl7sgjmxerasUUZaGrkqiMLIlSOMrizItIN0j4IAo/Ulb4BnvYhwUdNVTBYUISNqDCpVZAAgw2cUU0iHMICW+f4wRB0tYoi1OGSuwhjRVaFXzWpHW7puupdTrFoO6QMoIYe2S3rMUe2B17YR+/1mp4NdpejmjWOlpul8ZOZzLv/6qqNEscfKn+9CxRxqrnVZB322Pat9A7+vpxs5VZ3441FtgVeyX/l+yJ3dMNzPqbfp3m2xcI0gckfj53N8guxdfiifRyNLnh/8QgZjGPRXruFSSxhRSydGwFZzhHM/CshJQpZbqTqgR8zSS+hTL3CTmHiqQ=</latexit> <latexit sha1_base64="tJBUolgxH+Y4AhwAUb8db68d0EU=">AAACZXichVHLSgMxFD0d31VrfSCCC4ul4qqkIvhYFd24bK21hVrKzJjW4HRmmEkLWvwBwa1duFIQET/DjT/gwj9QXFZw48Lb6YBoUW9IcnJyz81JotmGcCVjTwGlp7evf2BwKDg8MhoaC49P7LpWzdF5VrcMy8lrqssNYfKsFNLgedvhalUzeE473Gzv5+rccYVl7sgjmxerasUUZaGrkqiMLIlSOMrizItIN0j4IAo/Ulb4BnvYhwUdNVTBYUISNqDCpVZAAgw2cUU0iHMICW+f4wRB0tYoi1OGSuwhjRVaFXzWpHW7puupdTrFoO6QMoIYe2S3rMUe2B17YR+/1mp4NdpejmjWOlpul8ZOZzLv/6qqNEscfKn+9CxRxqrnVZB322Pat9A7+vpxs5VZ3441FtgVeyX/l+yJ3dMNzPqbfp3m2xcI0gckfj53N8guxdfiifRyNLnh/8QgZjGPRXruFSSxhRSydGwFZzhHM/CshJQpZbqTqgR8zSS+hTL3CTmHiqQ=</latexit> <latexit sha1_base64="tJBUolgxH+Y4AhwAUb8db68d0EU=">AAACZXichVHLSgMxFD0d31VrfSCCC4ul4qqkIvhYFd24bK21hVrKzJjW4HRmmEkLWvwBwa1duFIQET/DjT/gwj9QXFZw48Lb6YBoUW9IcnJyz81JotmGcCVjTwGlp7evf2BwKDg8MhoaC49P7LpWzdF5VrcMy8lrqssNYfKsFNLgedvhalUzeE473Gzv5+rccYVl7sgjmxerasUUZaGrkqiMLIlSOMrizItIN0j4IAo/Ulb4BnvYhwUdNVTBYUISNqDCpVZAAgw2cUU0iHMICW+f4wRB0tYoi1OGSuwhjRVaFXzWpHW7puupdTrFoO6QMoIYe2S3rMUe2B17YR+/1mp4NdpejmjWOlpul8ZOZzLv/6qqNEscfKn+9CxRxqrnVZB322Pat9A7+vpxs5VZ3441FtgVeyX/l+yJ3dMNzPqbfp3m2xcI0gckfj53N8guxdfiifRyNLnh/8QgZjGPRXruFSSxhRSydGwFZzhHM/CshJQpZbqTqgR8zSS+hTL3CTmHiqQ=</latexit> <latexit sha1_base64="tJBUolgxH+Y4AhwAUb8db68d0EU=">AAACZXichVHLSgMxFD0d31VrfSCCC4ul4qqkIvhYFd24bK21hVrKzJjW4HRmmEkLWvwBwa1duFIQET/DjT/gwj9QXFZw48Lb6YBoUW9IcnJyz81JotmGcCVjTwGlp7evf2BwKDg8MhoaC49P7LpWzdF5VrcMy8lrqssNYfKsFNLgedvhalUzeE473Gzv5+rccYVl7sgjmxerasUUZaGrkqiMLIlSOMrizItIN0j4IAo/Ulb4BnvYhwUdNVTBYUISNqDCpVZAAgw2cUU0iHMICW+f4wRB0tYoi1OGSuwhjRVaFXzWpHW7puupdTrFoO6QMoIYe2S3rMUe2B17YR+/1mp4NdpejmjWOlpul8ZOZzLv/6qqNEscfKn+9CxRxqrnVZB322Pat9A7+vpxs5VZ3441FtgVeyX/l+yJ3dMNzPqbfp3m2xcI0gckfj53N8guxdfiifRyNLnh/8QgZjGPRXruFSSxhRSydGwFZzhHM/CshJQpZbqTqgR8zSS+hTL3CTmHiqQ=</latexit> tj <latexit sha1_base64="1DJK0DnO2FA/Sia3qFYpAKYn30Y=">AAACZXichVHLSsNAFD2Nr1ofrQ9EcGGxVFyVqQg+VkU3LtUaW6ilJHFaY9MkJNNCLf6A4FYXrhRExM9w4w+46B8oLiu4ceFtGhAt6h1m5syZe+6cmVFtQ3cFY82A1NPb1z8QHAwNDY+MhiNj43uuVXU0LmuWYTlZVXG5oZtcFroweNZ2uFJRDZ5Ryxvt/UyNO65umbuibvN8RSmZelHXFEFUWhSOCpEYSzAvot0g6YMY/NiyIrfYxwEsaKiiAg4TgrABBS61HJJgsInLo0GcQ0j39jlOECJtlbI4ZSjElmks0Srnsyat2zVdT63RKQZ1h5RRxNkTu2Mt9sju2Qv7+LVWw6vR9lKnWe1ouV0In06n3/9VVWgWOPxS/elZoIgVz6tO3m2Pad9C6+hrxxet9NpOvDHPrtkr+b9iTfZANzBrb9rNNt+5RIg+IPnzubuBvJhYTSS3l2Kpdf8ngpjBHBbouZeRwia2INOxJZzhHBeBZ2lUmpSmOqlSwNdM4FtIs587hoql</latexit> <latexit sha1_base64="1DJK0DnO2FA/Sia3qFYpAKYn30Y=">AAACZXichVHLSsNAFD2Nr1ofrQ9EcGGxVFyVqQg+VkU3LtUaW6ilJHFaY9MkJNNCLf6A4FYXrhRExM9w4w+46B8oLiu4ceFtGhAt6h1m5syZe+6cmVFtQ3cFY82A1NPb1z8QHAwNDY+MhiNj43uuVXU0LmuWYTlZVXG5oZtcFroweNZ2uFJRDZ5Ryxvt/UyNO65umbuibvN8RSmZelHXFEFUWhSOCpEYSzAvot0g6YMY/NiyIrfYxwEsaKiiAg4TgrABBS61HJJgsInLo0GcQ0j39jlOECJtlbI4ZSjElmks0Srnsyat2zVdT63RKQZ1h5RRxNkTu2Mt9sju2Qv7+LVWw6vR9lKnWe1ouV0In06n3/9VVWgWOPxS/elZoIgVz6tO3m2Pad9C6+hrxxet9NpOvDHPrtkr+b9iTfZANzBrb9rNNt+5RIg+IPnzubuBvJhYTSS3l2Kpdf8ngpjBHBbouZeRwia2INOxJZzhHBeBZ2lUmpSmOqlSwNdM4FtIs587hoql</latexit> <latexit sha1_base64="1DJK0DnO2FA/Sia3qFYpAKYn30Y=">AAACZXichVHLSsNAFD2Nr1ofrQ9EcGGxVFyVqQg+VkU3LtUaW6ilJHFaY9MkJNNCLf6A4FYXrhRExM9w4w+46B8oLiu4ceFtGhAt6h1m5syZe+6cmVFtQ3cFY82A1NPb1z8QHAwNDY+MhiNj43uuVXU0LmuWYTlZVXG5oZtcFroweNZ2uFJRDZ5Ryxvt/UyNO65umbuibvN8RSmZelHXFEFUWhSOCpEYSzAvot0g6YMY/NiyIrfYxwEsaKiiAg4TgrABBS61HJJgsInLo0GcQ0j39jlOECJtlbI4ZSjElmks0Srnsyat2zVdT63RKQZ1h5RRxNkTu2Mt9sju2Qv7+LVWw6vR9lKnWe1ouV0In06n3/9VVWgWOPxS/elZoIgVz6tO3m2Pad9C6+hrxxet9NpOvDHPrtkr+b9iTfZANzBrb9rNNt+5RIg+IPnzubuBvJhYTSS3l2Kpdf8ngpjBHBbouZeRwia2INOxJZzhHBeBZ2lUmpSmOqlSwNdM4FtIs587hoql</latexit> <latexit sha1_base64="1DJK0DnO2FA/Sia3qFYpAKYn30Y=">AAACZXichVHLSsNAFD2Nr1ofrQ9EcGGxVFyVqQg+VkU3LtUaW6ilJHFaY9MkJNNCLf6A4FYXrhRExM9w4w+46B8oLiu4ceFtGhAt6h1m5syZe+6cmVFtQ3cFY82A1NPb1z8QHAwNDY+MhiNj43uuVXU0LmuWYTlZVXG5oZtcFroweNZ2uFJRDZ5Ryxvt/UyNO65umbuibvN8RSmZelHXFEFUWhSOCpEYSzAvot0g6YMY/NiyIrfYxwEsaKiiAg4TgrABBS61HJJgsInLo0GcQ0j39jlOECJtlbI4ZSjElmks0Srnsyat2zVdT63RKQZ1h5RRxNkTu2Mt9sju2Qv7+LVWw6vR9lKnWe1ouV0In06n3/9VVWgWOPxS/elZoIgVz6tO3m2Pad9C6+hrxxet9NpOvDHPrtkr+b9iTfZANzBrb9rNNt+5RIg+IPnzubuBvJhYTSS3l2Kpdf8ngpjBHBbouZeRwia2INOxJZzhHBeBZ2lUmpSmOqlSwNdM4FtIs587hoql</latexit>
  10. 40 • /+Q1IRNJ – 3KHJLIRNJ (,): (4 – 3FGO!PRMIRNJ (FGO):

    (1.5  – /+: ,=FGOE&, 1: ,QFGO 5> • 1 – ).#Juman++>AC- .#*$E > 8C – 2B6E"B6>7< 1 • ": 1;?-'2=0 ?2 • ?%.#=D9C:@ 10
  11. Pre-Training •  32K, BERTBASE (L=12, A=12, H=768) 11 1GPU

    1epoch1     
  12.  12 +1.85 90.22 91.12 91.22 91.84 93.69 88.44 89.32

    90.07 90.2 92.57 85 86 87 88 89 90 91 92 93 94 95 KNP Cabocha J.depP BiLSTM BERT   +2.37
  13.  :    13   BiLSTM BERT

    Δ  BiLSTM BERT Δ   →  84.7 87.3 +2.6 2,050 86.0 90.5 +4.5 1,367  →  90.0 92.2 +2.2 1,367 92.2 95.1 +2.9 1,093  →  90.9 93.7 +2.8 3,653 90.0 92.2 +2.2 2,372  →  90.9 94.0 +3.1 6,300 91.0 94.2 +3.2 3,777  91.6 94.7 +3.1 790 93.9 95.9 +2.0 393 ! 97.7 99.0 +1.3 1,154 98.0 99.2 +1.2 605  94.8 96.4 +1.6 1,114 94.9 97.0 +2.1 788 88.1 90.8 +2.7 1,284 89.2 90.9 +1.7 861 83.2 89.9 +6.7 328 75.5 88.7 +13.2 151 … 
  14.  ( ) 14    

  15. :     15 Pre-trainingepoch

  16. :     16 Pre-training  3/4 1/2

    3/4 1/2
  17. :     17 Transformer L=6 L=6 L=12

    L=12
  18. 95:57%+ • BERT>$"6) • > ./ <05-4.2 • 0=947 ,>3<

    – FDGA?C;pre-training7@IEBA?C> ./1<054- – #H  ! • 7%+ – &7 – '$*("H"87) 18