Вопросно-ответный поиск: краткий обзор

Вопросно-ответный поиск: краткий обзор

Павел Браславский
Доцент Санкт-Петербургского филиала Высшей Школы Экономики и исследователь JetBrains Research

14 мая 2020
Online Data Science Meetup

Павел расскажет о вопросно-ответном поиске (question answering) — популярной области исследований на пересечении информационного поиска и обработки естественного языка.
Вопросно-ответный поиск занимается разработкой методов, которые возвращают краткий и точный ответ на вопрос пользователя. Павел сделает обзор различных видов вопросно-ответного поиска, методов, а также доступных данных.
Доклад будет вам полезен, если вы интересуетесь современными методами и приложениями обработки естественного языка (natural language processing).

Видео: https://youtu.be/F4_EtkEvs5U

Подписывайтесь на Exactpro в социальных сетях:

LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/

Подписывайтесь на YouTube канал Exactpro https://www.youtube.com/c/exactprosystems



May 14, 2020


  1. Question Answering: a Brief Overview Pavel Braslavski 14.05.2020

  2. About myself • Research/academia: JetBrains Research/ Higher School of Economics

    SPb/Ural Federal University • Past industrial experience: Yandex/SKB Kontur • Recent research interests: question answering, fiction analysis, computational humor Homepage: http://kansas.ru/pb/ 2
  3. What is QA? 3

  4. 4

  5. 5

  6. 6

  7. IBM Watson @ Jeopartdy! (2011) 7

  8. 8

  9. 9

  10. 10

  11. Brief History of QA • Early experiments – 1960s •

    TREC QA – 1999 • New era: large datasets + NNs • SQuAD (2016) • BERT (2018) 11
  12. QA types 12

  13. IR-based QA 13 Document Document Document Docume nt Docume nt

    Docume nt Docume nt Docume nt Question Processing Passage Retrieval Query Formulation Answer Type Detection Question Passage Retrieval Document Retrieval Answer Processing Answer passages Indexing Relevant Docs Document Document Document [Dan Jurafsky]
  14. Full NLP QA: LCC (Harabagiu/Moldovan) Question Parse Semantic Transformation Recognition

    of Expected Answer Type (for NER) Keyword Extraction Factoid Question List Question Named Entity Recognition (CICERO LITE) Answer Type Hierarchy (WordNet) Question Processing Question Parse Pattern Matching Keyword Extraction Question Processing Definition Question Definition Answer Answer Extraction Pattern Matching Definition Answer Processing Answer Extraction Threshold Cutoff List Answer Processing List Answer Answer Extraction (NER) Answer Justification (alignment, relations) Answer Reranking (~ Theorem Prover) Factoid Answer Processing Axiomatic Knowledge Base Factoid Answer Multiple Definition Passages Pattern Repository Single Factoid Passages Multiple List Passages Passage Retrieval Document Processing Document Index Document Collection 14 [Manning]
  15. Open Domain QA • Step1: Retrieval of relevant documents/paragraphs. •

    Step2: Finding an answer in the document/paragraph (Reading Comprehension, RC). 15
  16. Knowledge Base QA 16

  17. 17

  18. Datasets 18

  19. TREC QA • 1999-2007 • Factoid questions • Facts, lists,

    definitions • Hundreds of questions • Ranked list of fragments (50/250 characters)/single answer + confidence score • News collection (~1M documents, 3Gb) • Mean Reciprocal Rank (MRR) http://trec.nist.gov/data/qamain.html 19
  20. TREC QA questions • Who is the author of the

    book, "The Iron Lady: A Biography of Margaret Thatcher"? • What was the monetary value of the Nobel Peace Prize in 1989? • What does the Peugeot company manufacture? • How much did Mercury spend on advertising in 1993? • What is the name of the managing director of Apricot Computer? • Why did David Koresh ask the FBI for a word processor? • What debts did Qintex group leave? • What is the name of the rare neurological disease with symptoms such as: involuntary movements (tics), swearing, and incoherent vocalizations (grunts, shouts, etc.)? 20
  21. SQuAD SQuAD 1.1 (2016) • ~500 popular Wikipedia pages →

    ~23,000 paragraphs (80/10/10) • Crowdsourcing: 5 questions to each paragraph + answer span →100,000+ questions Evaluation: exact match, word-level F1 SQuAD 2.0 (2018) +50,000 unanswerable questions • F1: 86%→ 66% 21
  22. SQuAD example Super Bowl 50 was an American football game

    to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24--10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the "golden anniversary" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as "Super Bowl L"), so that the logo could prominently feature the Arabic numerals 50. Which NFL team represented the AFC at Super Bowl 50? Which NFL team represented the NFC at Super Bowl 50? Where did Super Bowl 50 take place? 22
  23. 23

  24. 24

  25. 25

  26. QA datasets • TriviaQA • QuAC, CoQA • MS MARCO

    • Natural Questions • … 26
  27. SimpleQuestions (2015) • 108,442 questions + freebase triples https://research.fb.com/downloads/babi/ 27

  28. NN4QA 28

  29. Document Reader (Chen, et al. ACL 2017) 29

  30. 30

  31. KBQA methods • NER/entity linking/relation detection → SPARQL • Question

    →SPARQL (end2end) 31
  32. Multilingual QA

  33. SberQuAD (2017) 33 https://arxiv.org/abs/1912.09723

  34. SberQuAD/XQuAD/MLQA/TyDi QA 34 MLQA

  35. TyDi QA 35

  36. Multilingual transfer 36

  37. KBQA datasets 37

  38. RuBQ • 1,500 Russian questions + SPARQL queries to Wikidata

    • https://github.com/vladislavneon/RuBQ 38
  39. Questions? 39