機械読解の活躍が期待されるシーンの代表例︓ スマートスピーカー 10 https://softwareengineeringdaily.com/2018/05/18/alexa-voice- design-with-paul-cutsinger/ What was the final score of Super Bowl 50? Broncos beat Panthers 24-10 in Super Bowl 50. Knowledge Base
2018年10⽉︓ BERTの衝撃 • BERTが登場。読解に特化したニューラルネット構造を持 たずに、⼈間のスコアを部分⼀致でも⼤きく超える 完全⼀致 部分⼀致 ⼈間のスコア BERTのスコア Jacob Devlin et al.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR abs/1810.04805 (2018) https://rajpurkar.github.io/SQuAD-explorer/ 18
Copyright@2019 NTT corp. All Rights Reserved. SQuAD: 100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang Stanford Univ. EMNLP 2016 (arXiv: 2016/06) https://arxiv.org/abs/1606.05250 (2019/02/15)
Copyright@2019 NTT corp. All Rights Reserved. 読解モデルの紹介(1) Bidirectional Attention Flow for Machine Comprehension Minjoon Seo1 Aniruddha Kembhavi2 Ali Farhadi1,2 Hannaneh Hajishirzi2 1 Univ. Washington 2 Allen Institute for Artificial Intelligence ICLR 2017 (arXiv: 2016/11) https://arxiv.org/abs/1611.01603 (2019/02/15)
学習とテスト • 学習︓損失関数は始点と終点の正解位置%, &を⽤いて負 の対数尤度により最適化 • テスト︓' %( &が最⼤となる(k,l)の範囲（k≦l）を選択 Super Bowl 50 was an American football game to determine … 回答範囲として抽出
Copyright@2019 NTT corp. All Rights Reserved. 読解モデルの紹介(2) QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension Adams Wei Yu1, David Dohan2, Minh-Thang Luong2, Rui Zhao2, Kai Chen2, Mohammad Norouzi2, Quoc V. Le2 1 Carnegie Mellon University 2 Google Brain ICLR 2018 (arXiv: 2018/04) https://arxiv.org/abs/1804.09541 (2019/2/15)
Copyright@2019 NTT corp. All Rights Reserved. 読解モデルの紹介(3) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova Google AI Language arXiv 2018/10 https://arxiv.org/abs/1810.04805 (2019/2/15) (2019/3/9)
BERTの特徴(2) 次⽂章予測 • NLPのタスクは2つの⽂の関係性理解が多い • 隣接する⽂/ランダムな2⽂をセパレータで繋げて、隣接す る⽂か否かの判定を⽳埋め予測と同時に学習 [CLS] my dog is cute [SEP] he likes play [MASK] [SEP] 2⽂⽬ 1⽂⽬ IsNext my dog is cute [SEP] he likes play piano [SEP] 分類 ⽳埋め BERT 75
BERTの他タスクでの評価 • GLUE [1] – ⽂ペア、単⽂⼊⼒の⾔語理解タスク9種。 – 含意、⾔い換え、⽂類似性、分類。 • CoNLL 2003 NER [2] – 固有表現抽出 • SWAG [3] – 次⽂予測 [1] Alex Wang et al.: GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. BlackboxNLP@EMNLP 2018: 353-355 [2] Erik F. Tjong Kim Sang, Fien De Meulder: Introduction to the CoNLL-2003 Shared Task: Language- Independent Named Entity Recognition. CoNLL 2003: 142-147 [3] Rowan Zellers et al.: SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference. EMNLP 2018: 93-104 79
質問の僅かな違いが 読解エラーになる例 Q. Who was the co-MVP of Super Bowl XII? Q. Who was the MVP of Super Bowl XII? co-MVPをMVPに するだけで間違った https://demo.allennlp.org/machine-comprehension/ 84
• 4択問題 • 質問もパッセージも⼈⼿作 成だが、データ数が少ない （660テキスト×4質問） – パッセージは⼩学⽣向 けの作り話としてクラ ウドソーシングで作成 • 2018/10の⼿法で 80-82%程度の正解率[2] [1] Matthew Richardson et al.: MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text. EMNLP 2013: 193-203 [2] Kai Sun et al. : Improving Machine Reading Comprehension with General Reading Strategies. CoRR abs/1810.13441 (2018) http://aclweb.org/anthology/D13-1020 MCTest [Richardson(Microsoft)+, EMNLP13] (2019/2/15) 99
CNN/Daily Mail Reading [Hermann (DeepMind)+, NIPS15] • ニュース⾒出しの⽳埋め問題（本⽂を読んで正しいエンティティを⾒ つける） • エンティティは匿名化されるので世界知識は使えない http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend Karl Moritz Hermann et al. : Teaching Machines to Read and Comprehend. NIPS 2015: 1693-1701 (2019/2/15) 100
Children Book Test (CBT) [Hill (Facebook)+, ICLR16] …………………………. …………………………. • 20個の連続した⽂を⾒て、次の⽂に出てくる単語の⽳埋め問題を解く • エンティティ以外にも動詞や共通名詞が対象 • 近年は⾔語モデルの評価セットとして使われるケースあり[1][2] https://arxiv.org/pdf/1511.02301.pdf Felix Hill et al.: The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations. ICLR 2016 [1] Zihang Dai et al. : Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. CoRR abs/1901.02860 (2019) [2] Alex Radford et al.: Language Models are Unsupervised Multitask Learners, OpenAI Tech. Report, 2019. (2019/2/15) 101
回答を⽣成できるか︖ MS MARCO • 質問・コンテキスト︓Bingの実クエリ (オープンドメイン)・検索結果 • 回答タイプ︓⽣成 – パッセージの範囲に限定されない（評価はRouge, BLEU） • Tri Nguyen et al.: MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. CoCo@NIPS 2016 • Payal Bajaj et al.: MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv 2018 The majority of pharmacy techs work in drug stores and hospitals, where the average annual salary was $28,940 and $34,410, respectively**. However, a higher salary can be had if you can find employment with outpatient care centers or physicians' offices, where the annual pay is in the $37,000-$39,000 range. ⼊⼒︓質問クエリ ⼊⼒︓Bing検索結果テキスト(10件): average pharmacy tech salary The majority of pharmacy techs work in drug stores and hospitals, where the average annual salary was $28,940 and $34,410, respectively**. However, a higher salary can be had if you can find employment with outpatient care centers or physicians' offices, where the annual pay is in the $37,000-$39,000 range. What can a pharmacy technician really expect to earn in today's economy? According to Salary.com, pharmacy technicians make anywhere from $26,000 to $39,000 a year, though most make around $32,000 annually. California has the highest average pharmacy technician wage, at $34,317, according to Open Farm Tech's website. システム出⼒︓ 回答⽂ [NLG] The average salary for a pharmacy technician is $26,000 to $39,000 in a year. [Q&A] From $26,000 to $39,000 a year 質問内容を含めて⽣成するNLGと、 抽出に近いQ&Aの2タスク V1 [Nguyen (Microsoft)+,CoCo@NIPS16] V2 [Bajaj (Microsoft)+,arXiv18] (2019/02/15) 109
複数の⽂章に跨る推論は可能か︖⼤規模な知識を扱えるか︖ OpenBookQA [Mihaylov (AllenAI)+, EMNLP18] • 質問︓科学の初等レベルの 問題 • コンテキスト︓ – 質問に紐付いたものは無し。 – 2287件のWorldTree corpus[2]の知識（Science Facts） – 1400万件の科学関連の⽂章 コーパス（Common Knowledge） • 回答タイプ︓4択 [1] Todor Mihaylov, Peter Clark, Tushar Khot, Ashish Sabharwal: Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. EMNLP 2018: 2381-2391 [2] Peter A. Jansen, Elizabeth Wainwright, Steven Marmorstein, Clayton T. Morrison: WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-hop Inference. LREC 2018 http://aclweb.org/anthology/D18-1260 113
kyoun
.png){kind=icon}
pharma_x_tech
unclejoe
oracle4engineer
kenta_hi
edwardkenfox
sansantech
antimon2
bengo4com
konifar
egmc
yoshiitaka
morganepeng
mthomps
chriscoyier
jacobian
orderedlist
chrisshort
holman
marktimemedia
aleyda
garrettdimon
dotmariusz