Upgrade to Pro — share decks privately, control downloads, hide ads and more …

論文紹介 / Knowledgeable Reader

論文紹介 / Knowledgeable Reader

第10回最先端NLP勉強会 発表資料

Kyosuke Nishida

July 31, 2018
Tweet

More Decks by Kyosuke Nishida

Other Decks in Technology

Transcript

  1. 勉強会資料: Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense

    Knowledge Todor Mihaylov and Anette Frank Heidelberg University 第10回最先端NLP勉強会 2018/08/03 紹介者: 西田京介 @kyoun(NTT)
  2. 概要 • 外部知識(ConceptNet)を用いる穴埋め式機械読解 – テキストを読み、質問文中のplaceholderに入る単語を選択肢か ら選ぶタスク • 新モデルとしてKnowledgeable Readerを提案: –

    数十万件規模のtriple形式の知識ベースを検索するRetriever、 検索結果を格納するMemory、 Memoryの内容を参照して読解するReaderから構成 • Children Book Testデータセットにて、外部知識を利用 することの効果を示す – SOTAではないが、読解分野に大きな貢献
  3. 機械読解とは • 「テキストを読んで、そのテキストの質問に答える」タ スクの総称 • 様々な回答方式のデータセットが公開されている – 穴埋め: CNN/Daily Mail

    [Hermann+, NIPS15], CBT [Hill+, arxiv15] – 選択: RACE [Lai+, EMNLP17] – 抽出: SQuAD [Rajpurkar+, EMNLP16][Rajpurkar+, ACL18] – 生成: MS MARCO [Nguyen+, CoCo@NIPS16] • 本研究は、穴埋め式の機械読解に関する研究 – 実験ではCBTを利用
  4. 参考: ACL'18における機械読解論文15本 Long • Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension

    with External Commonsense Knowledge • Denoising Distantly Supervised Open-Domain Question Answering • Multi-Relational Question Answering from Narratives: Machine Reading and Reasoning in Simulated Worlds • DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension • Stochastic Answer Networks for Machine Reading Comprehension • Efficient and Robust Question Answering from Minimal Context over Documents • Multi-Passage Machine Reading Comprehension with Cross-Passage Answer Verification • Joint Training of Candidate Extraction and Answer Selection for Reading Comprehension Short • Know What You Don’t Know: Unanswerable Questions for SQuAD (Best) • CNN for Text-Based Multiple Choice Question Answering • A Co-Matching Model for Multi-choice Reading Comprehension • Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering • Simple and Effective Multi-Paragraph Reading Comprehension TACL • The NarrativeQA Reading Comprehension Challenge • Constructing Datasets for Multi-hop Reading Comprehension Across Documents
  5. 参考: データセットから見る読解研究の方向性 MCTest[13.10] CNN/DailyMail [15.06] CBT [15.11] SQuAD v1[16.06] NewsQA[16.11]

    Entityの穴埋め 1文書中のSpanで回答 複数文書から読解 SearchQA [17.04] Quasar [17.07] TriviaQA [17.05] 外部知識: 常識や専門知識 MedQA [18.02] ARC [18.03] 大規模テキスト集合から 読解 RACE [17.04] NarrativeQA [17.12] MCScript [18.03] CliCR [18.03] 文章間にまたがる読解 QAngaroo [17.10] MultiRC [18.06] Multiparty dialogue [18.06 解答可能性付き読解 データセット[18.03] MS MARCO v2[18.03] SQuAD v2[18.06] 回答無しの理解 MS MARCO v1 [16.10] 自然言語で回答 DuoRC [18.05] 2014 2015 2016 TQA [17.07] マルチモーダル 2018 2017 MovieQA [15.12]
  6. 本研究のタスク: Children Book Test (CBT) [Hill, 2015] D Q A

    a • 文章D、(ブランクを含む)質問Q、回答候補Aから、ブランクとして 適切な回答を選択。回答候補は、一般名詞 or 固有名詞の2条件
  7. 提案モデルの構成 1. Knowledge Base: triple形式の知識を格納 2. Retriever: 入力情報に関連する知識を検索 3. Neural

    model: メモリ付Attention sum reader [Kadlec+, 2016] Subj Rel Obj bow IsUsedFor hunt animal Memory Reader Retriever Input: 質問(穴付) 文章 回答候補 Output: 回答 Knowledge Base Neural model
  8. 1. Knowledge base • ConceptNetに含まれるOpen Mind Common Sense (2002)の63万件を利用 •

    各知識fは、(subject, relation, object)の形式 • sub/objは複数単語からなるフレーズ • relは1単語として扱う – 例: ("bow", IsUsedFor, ”hunt animals") Subj Rel Obj bow IsUsedFor hunt animal
  9. 2. Retriever • 文章D、質問Q、回答候補Aのトークンのlemmaを含む 知識を検索し、上位P知識を返却 – Aに重み4, Qに重み3, Dに重み2を付与(heuristic) –

    Pは、50 or 100 or 200 • 各回答候補に対して上限を持たせる – たとえば、P=100、|A|=10の場合、各回答候補をsubj or obj に含む知識を10件ずつ検索する
  10. 3. Neural model • Attention sum reader (ASR) [Kadlec+, 2016]を知識メモリ付に改良

    I. 基本モデル: ASR II. Knowledgeable Reader a. 知識のエンコーディングおよびメモリへの格納 b. メモリによる文書、質問のコンテキスト表現の更新 c. 複数種類のアテンションのensemble
  11. 3-I. 基本モデル document question Word embedding BiGRU Word embedding BiGRU

    He mounted his XXXX and rode away. Placeholder pickup dot-product attention The prince was on his white horse , with a sword in his hand … answer ℝ×512 ℝ×100 ℝ512 ℝ×100 ℝ×512 ℝ10 sum+softmax ℝ Last stateではなく、 XXXXの位置のベクトルの み利用 answer candidateの出現 位置に応じたアテンション 分布の値を合計→softmax
  12. 3-II. Knowledgeable Reader document query Word embedding BiGRU Word embedding

    BiGRU He mounted his XXXX and rode away. Placeholder pickup dot-product attention The prince was on his white horse , with a sword in his hand … answer ℝ×512 ℝ×100 ℝ512 ℝ×100 ℝ×512 ℝ10 sum+softmax ℝ Knowledge emb. Knowledge emb. dot-product attention (multi-head) dot- product attention ℝ×512 ℝ×512 b. 各単語表現ごとに kvメモリを使って更新 c. 知識の利用有無で 4種類のアテンション (multi-head) dot- product attention Key-value Memory Retrieval results a. Triple知識を メモリへ格納 BiGRU
  13. b. メモリによる文書・質問の コンテキスト表現の更新 Knowledge emb. Dot-product Attention Key-value Memory (size:

    P) Key: Mk Value: Mv Weighted sum weights • 文書、質問についてそれぞれBi-GRUが出力したコンテ キスト表現の各単語でKVメモリをattention-pooling Document or question
  14. c. 複数アテンションのensemble document query Word embedding BiGRU Word embedding BiGRU

    Placeholder pickup dot-product attention ℝ×512 ℝ×100 ℝ512 ℝ×100 ℝ×512 ℝ Knowledge emb. Knowledge emb. dot-product attention (multi-head) dot- product attention ℝ×512 ℝ×512 (multi-head) dot- product attention • 知識の有無で4種類 アテンションが可 能になるので、線 形和を取って利用 • Multi-head attention(Vaswan i+, 2017)と同様
  15. 知識の利用状況 の可視化 • 質問および文書の各単語か らメモリへアテンションを 掛けた結果を可視化 • 知識なしだと"legs"と回答 していたのが、知識ありだ と"head"と正しく正解

    • 文書からは、(head, partOf, animal)の知識に強 く注意が掛り、正解の発見 に貢献 メモリ上の 知識への アテンション 従来手法: 回答候補 分布 提案手法: 回答候補 分布
  16. データセット 参考文献 (~2016) • [MCTest] Matthew Richardson, Christopher J. C.

    Burges, Erin Renshaw: MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text. EMNLP 2013: 193-203 • [CNN/DailyMail] Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, Phil Blunsom: Teaching Machines to Read and Comprehend. NIPS 2015: 1693-1701 • [CBT] Felix Hill, Antoine Bordes, Sumit Chopra, Jason Weston:The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations. ICLR 2016. • [SQuAD] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang: SQuAD: 100, 000+ Questions for Machine Comprehension of Text. EMNLP 2016: 2383-2392 • [NewsQA] Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, Kaheer Suleman: NewsQA: A Machine Comprehension Dataset. Rep4NLP@ACL 2017: 191-200 • [MovieQA] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, Sanja Fidler: MovieQA: Understanding Stories in Movies through Question- Answering. CVPR 2016: 4631-4640 • [MS MARCO] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng: MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. CoCo@NIPS 2016
  17. データセット 参考文献 (2017) • [SearchQA] Matthew Dunn, Levent Sagun, Mike

    Higgins, V. Ugur Güney, Volkan Cirik, Kyunghyun Cho: SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine. CoRR abs/1704.05179 (2017) • [TriviaQA] Mandar Joshi, Eunsol Choi, Daniel S. Weld, Luke Zettlemoyer: TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. ACL (1) 2017: 1601-1611 • [Quasar] Bhuwan Dhingra, Kathryn Mazaitis, William W. Cohen: Quasar: Datasets for Question Answering by Search and Reading. CoRR abs/1707.03904 (2017) • [RACE] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, Eduard H. Hovy: RACE: Large- scale ReAding Comprehension Dataset From Examinations. EMNLP 2017: 785-794 • [TQA] Aniruddha Kembhavi, Min Joon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, Hannaneh Hajishirzi: Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension. CVPR 2017: 5376-5384 • [QAngaroo] Johannes Welbl, Pontus Stenetorp, Sebastian Riedel: Constructing Datasets for Multi-hop Reading Comprehension Across Documents. TACL 6: 287-302 (2018) • [NarrativeQA] Tomás Kociský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, Edward Grefenstette: The NarrativeQA Reading Comprehension Challenge. TACL 6: 317-328 (2018)
  18. データセット 参考文献 (2018) • [MedQA] Xiao Zhang, Ji Wu, Zhiyang

    He, Xien Liu, Ying Su: Medical Exam Question Answering with Large-scale Reading Comprehension. AAAI 2018 • [ARC] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord: Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. CoRR abs/1803.05457 (2018) • [解答可能性付き読解] 鈴木正敏, 松田耕史, 岡崎直観, 乾健太郎. 読解による解答可能性を付与し た質問応答データセットの構築. NLP 2018: C4-5. • [MS MARCO v2] https://github.com/dfcf93/MSMARCOV2 • [MCScript] Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, Manfred Pinkal: MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge. LREC 2018 • [MultiRC] Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, Dan Roth: Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences. NAACL-HLT 2018: 252-262 • [CliCR] Simon Suster, Walter Daelemans: CliCR: a Dataset of Clinical Case Reports for Machine Reading Comprehension. NAACL-HLT 2018: 1551-1563 • [Multiparty Dialogue] Kaixin Ma, Tomasz Jurczyk, Jinho D. Choi: Challenging Reading Comprehension on Daily Conversation: Passage Completion on Multiparty Dialog. NAACL-HLT 2018: 2039-2048 • [DuoRC] Mitesh M. Khapra, Amrita Saha, Karthik Sankaranarayanan, Rahul Aralikatte: DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension. ACL (1) 2018: 1683-1693 • [SQuAD v2] Pranav Rajpurkar, Robin Jia, Percy Liang: Know What You Don't Know: Unanswerable Questions for SQuAD. ACL (2) 2018: 784-789