Question Generation 4 Generated Question Input Context Answer-aware Text Factoid Q Generated Question: What Shakespeare scholar is currently on the faculty? Input Context: Current faculty include the anthropologist Marshall Sahlins and the Shakespeare scholar David Bevington. Answer: David Bevington
Methodology for Question Generation 5 q Rule-based Methods q Neural Methods • Apply linguistic rules to transform a declarative sentence into a question. • Fill out pre-defined question templates. • Sequence-to-Sequence Model • Encoder: encode the input passage and answer • Decoder: decode question token by token (Image Credit: R. Zhang et al. 2021)
(Marzieh et al., EMNLP 2018) Chatbot Applications of Question Generation 11 Generate clarification questions in dialogue I am working for an employer in Canada. Do I need to carry on paying UK National Insurance Have you been working abroad 52 weeks or less?
(Duan et al., EMNLP 2017) (Lewis et al., ACL 2019) (Puri et al., EMNLP 2020) (Yue et al., EMNLP 2021) Question Answering Applications of Question Generation 12 Generate training data for question answering
17 Unsupervised Multi-hop Question Answering by Question Generation [NAACL 2021] Pan et al: Unsupervised Multi-hop Question Answering by Question Generation Liangming Pan, Wenhu Chen, Wenhan Xiong, Min-Yen Kan, William Yang Wang
19 Human-written QA Pairs QA Model Train Training a Multi-hop QA System • 78K questions • 13K HITs • $2.3 USD/HIT • 2 min per sample • Total Cost: >30,000 USD • 112K questions Time-consuming and costly to create training data.
20 Multi-hop QA via Question Generation Generated QA Pairs Generator Input Sources Tables Documents Could we use Question Generation (QG) to automatically generate high-quality training data? QA Model Train
21 Challenges Lack of training data We lack human-annotated (source, multi-hop question, answer) as supervision. Composing sub-questions into multi-hop questions Quality of the generated data How to ensure the generated (source, question, answer) pair is valid ? o The question performs multi-hop reasoning over the input source. o The generated answer is correct. Defining an interpretable process that mimics human reasoning
Answer Multi-hop Questions 22 When did the rock band that sang "All Join Hands" rise to prominence? Multi-hop Question Which rock band sang "All Join Hands"? Paragraph A: All Join Hands "All Join Hands" is a song by the British rock band Slade, released in 1984 as the lead single from the band's twelfth studio album "Rogues Gallery". Paragraph B: Slade Slade are an English glam rock band from Wolverhampton. They rose to prominence during the early 1970s with 17 consecutive top 20 hits and six number ones on the UK Singles Chart. When did Slade rise to prominence? Ans = Slade Ans = Early 1970s
Generate Multi-hop Questions 23 When did the rock band that sang "All Join Hands" rise to prominence? Multi-hop Question Which rock band sang "All Join Hands"? Paragraph A: All Join Hands "All Join Hands" is a song by the British rock band Slade, released in 1984 as the lead single from the band's twelfth studio album "Rogues Gallery". Paragraph B: Slade Slade are an English glam rock band from Wolverhampton. They rose to prominence during the early 1970s with 17 consecutive top 20 hits and six number ones on the UK Singles Chart. When did Slade rise to prominence? Ans = Slade Ans = Early 1970s
Generate Multi-hop Questions 24 When did the rock band that sang "All Join Hands" rise to prominence? Multi-hop Question Which rock band sang "All Join Hands"? Paragraph A: All Join Hands "All Join Hands" is a song by the British rock band Slade, released in 1984 as the lead single from the band's twelfth studio album "Rogues Gallery". Paragraph B: Slade Slade are an English glam rock band from Wolverhampton. They rose to prominence during the early 1970s with 17 consecutive top 20 hits and six number ones on the UK Singles Chart. When did Slade rise to prominence? Ans = Slade Composing sub-questions into multi-hop questions An interpretable process that mimics human reasoning
26 Operators • Retrieve / generate relevant information from a single input source • Aggregate information from two sources. Reasoning Graphs • Each corresponds to one type of multi- hop question • Formulated as a computation graph built upon the operators. General Framework
27 Group Operator Inputs Output Description Selection 𝐹𝑖𝑛𝑑𝐵𝑟𝑖𝑑𝑔𝑒 (Table + Text) or (Text + Text) Bridge Entity Select an entity to serve as the bridge between two texts (or between table and text) 𝐹𝑖𝑛𝑑𝐶𝑜𝑚𝐸𝑛𝑡 Text Comparative Entities Select potential comparative entities from the given text Generation 𝑄𝐺𝑤𝑖𝑡ℎ𝐴𝑛𝑠 Text + Answer Question Generate a question with a given answer from the input text 𝑄𝐺𝑤𝑖𝑡ℎ𝐸𝑛𝑡 Text + Entity Question Generate a question which contains the given entity from the input text 𝐷𝑒𝑠𝑐𝑟𝑖𝑏𝑒𝐸𝑛𝑡 Table + Entity Sentence Generate a sentence that describes the given entity based on the information in the table Q𝑢𝑒𝑠𝑇𝑜𝑆𝑒𝑛𝑡 Question Sentence Convert a question into a declarative sentence Fusion 𝐵𝑟𝑖𝑑𝑔𝑒𝐵𝑙𝑒𝑛𝑑 Question + Sentence + Bridge Multi-hop Question Generate a bridge-type multi-hop question 𝐶𝑜𝑚𝑝𝑎𝑟𝑒𝐵𝑙𝑒𝑛𝑑 Question + Question Multi-hop Question Generate a comparative-type multi-hop question Operators
28 𝑄𝐺𝑤𝑖𝑡ℎ𝐸𝑛𝑡 #𝟏 Slade When did the rock band that sang "All Join Hands" rise to prominence? Answer: Early 1970s Paragraph A: All Join Hands "All Join Hands" is a song by the British rock band Slade, released in 1984 ⋯ ⋯ Paragraph B: Slade Slade are an English glam rock band from Wolverhampton. They rose to prominence during the early 1970s with ⋯ ⋯ ⋯ #𝟐 When did the rock band Slade rose to prominence? Answer: Early 1970s #𝟑 What rock band sang “All Join Hands”? Answer: Slade #𝟒 Slade sang “All Join Hands”. #𝟓 𝐹𝑖𝑛𝑑𝐵𝑟𝑖𝑑𝑔𝑒 𝑄𝐺𝑤𝑖𝑡ℎ𝐴𝑛𝑠 𝐵𝑟𝑖𝑑𝑔𝑒𝐵𝑙𝑒𝑛𝑑 𝑄𝑢𝑒𝑠𝑇𝑜𝑆𝑒𝑛𝑡
29 Kirsten Carlijn Wild (born 15 October 1982) is a Dutch professional racing cyclist, ⋯ ⋯ ⋯. Wild competed in two track cycling events at the 2012 Summer Olympics. Kirsten Wild Kirsten Wild of Netherlands won the bronze medal in the 2011 Apeldoorn. What is the birthdate of Kirsten Wild? Answer: 15 October 1982 𝐵𝑟𝑖𝑑𝑔𝑒𝐵𝑙𝑒𝑛𝑑 #𝟒 What is the birthdate of the athlete that of Netherlands won the bronze medal in the 2011 Apeldoorn? Answer: 15 October 1982 Medal Championship Name Event Silver 2010 Pruszkow Tim Veldt Men’s omnium Bronze 2011 Apeldoorn Kristen Wild Women’s omnium Gold 2013 Apeldoorn Elis Ligtlee Women’s keirin 𝐹𝑖𝑛𝑑𝐵𝑟𝑖𝑑𝑔𝑒 #𝟏 𝐷𝑒𝑠𝑐𝑟𝑖𝑏𝑒𝐸𝑛𝑡 #𝟐 𝑄𝐺𝑤𝑖𝑡ℎ𝐸𝑛𝑡 #𝟑
35 Group Operator Inputs Output Description Selection Generation 𝑄𝐺𝑤𝑖𝑡ℎ𝐴𝑛𝑠 Text + Answer Question Generate a question with a given answer from the input text 𝑄𝐺𝑤𝑖𝑡ℎ𝐸𝑛𝑡 Text + Entity Question Generate a question which contains the given entity from the input text 𝐷𝑒𝑠𝑐𝑟𝑖𝑏𝑒𝐸𝑛𝑡 Table + Entity Sentence Generate a sentence that describes the given entity based on the information in the table Fusion 𝐵𝑟𝑖𝑑𝑔𝑒𝐵𝑙𝑒𝑛𝑑 Question + Sentence + Bridge Multi-hop Question Generate a bridge-type multi-hop question Operators
1. 𝑄𝐺𝑤𝑖𝑡ℎ𝐴𝑛𝑠 and 𝑄𝐺𝑤𝑖𝑡ℎ𝐸𝑛𝑡 • Github Link: https://github.com/patil-suraj/question_generation • An Google-T5 model jointly trained on three tasks based on the SQuAD dataset. • Answer Prediction, Question Generation, Question Answering extract answer: 42 is the answer to life, the universe and everything.
generate question: 42 is the answer to life, the universe and everything. question: What is the answer to life? context: 42 is the answer to life, the universe and everything. 42 What is the answer to life, the universe and everything? 42 Google T5 36
2. 𝐷𝑒𝑠𝑐𝑟𝑖𝑏𝑒𝐸𝑛𝑡 (Table-to-Text) 37 • An GPT-2 model finetuned on the ToTTo dataset. Medal Championship Name Event Silver 2010 Pruszkow Tim Veldt Men’s omnium Bronze 2011 Apeldoorn Kristen Wild Women’s omnium Gold 2013 Apeldoorn Elis Ligtlee Women’s keirin Gold 2013 Apeldoorn Elis Ligtlee Women’s sprint Input Table + Target Entity Netherlands at the European Track Championships The table title is Netherlands at the European Track Championships . The Medal is Bronze . The Championship is 2011 Apeldoorn . The Name is Kirsten Wild . The Event is Women's omnium . Start describing Kirsten Wild : Kirsten Wild of Netherlands won the bronze medal in the 2011 Apeldoorn. Table Templatization Pretrained GPT-2
40 !: Kirsten Wild ": What is the birthdate of Kirsten Wild? Answer: 15 October 1982 #: Kirsten Wild of Netherlands won the bronze medal in the 2011 Apeldoorn. What is the birthdate of the _____ that of Netherlands won the bronze medal in the 2011 Apeldoorn? What is the birthdate of the athlete that of Netherlands won the bronze medal in the 2011 Apeldoorn? Answer: 15 October 1982 BERT 3. 𝐵𝑟𝑖𝑑𝑔𝑒𝐵𝑙𝑒𝑛𝑑 (Combining sub-parts)
45 Few-shot QA Performance q HotpotQA q HybridQA The F1 score for progressively larger training dataset sizes for finetuning. • 100K generated data + N human-labeled data • N human-labeled data
47 Ablation Study Single-hop questions performs bad when they are used to train the multi-hop QA model. We need data that performs multi-hop reasoning.
48 Model trained with a single reasoning chain only performs well on the corresponding question type We need data that performs diverse reasoning. Ablation Study
49 Examples of Generated Questions Type Question Answer Table-Text When did the one that won the Eurovision Song Contest in 1966 join Gals and Pals? 1963 How many students attend the teams that played in the Dryden Township Conference? 1900 Text-Table What album did the Oak Ridge Boys release in 1989? American Dreams When was the name that is the name of the bridge that crosses Youngs Bay completed? Summer Text-Text Which Canadian cinematographer is best known for his work on Fargo? Craig Wrobleski What is illegal in the country that is Bashar Hafez al - Assad ’s father? Cannabis Comparison Who was born first, Terry Southern or Neal Town Stephenson? Terry Southern Are Beth Ditto and Mary Beth Patterson of the same nationality? Yes
50 Limitations Naturalness of the generated question We rely on template-based method + BERT to compose questions. This sometimes makes the generated questions look unnatural. o Generated: What is illegal in the country that is Bashar’s father? o Human: What is illegal in Bashar’s father’s country? Hard to generate “less apparent” multi-hop questions For some multi-hop questions, the decomposition into sub-parts are not evident from the question itself. o Question: Did Aristotle use a laptop? o Evidence 1: Aristotle was died in 322BC. o Evidence 2: The first laptop was invented in 1980.
51 Summary When did the rock band that sang "All Join Hands" rise to prominence? Multi-hop Question Which rock band sang "All Join Hands"? Paragraph A: All Join Hands "All Join Hands" is a song by the British rock band Slade, released in 1984 as the lead single from the band's twelfth studio album "Rogues Gallery". Paragraph B: Slade Slade are an English glam rock band from Wolverhampton. They rose to prominence during the early 1970s with 17 consecutive top 20 hits and six number ones on the UK Singles Chart. When did Slade rise to prominence? Ans = Slade Composing simple questions into complex questions An interpretable process that mimics human reasoning Generator Tables Documents QA Model Train
53 Zero-shot Fact Verification with Claim Generation [ACL 2021] Pan et al: Zero-shot Fact Verification with Claim Generation Liangming Pan, Wenhu Chen, Wenhan Xiong, Min-Yen Kan, William Yang Wang
Fact Checking 55 v Labels: Supports, Refutes, Not Enough Info v Pipeline v Document Retrieval v Sentence Retrieval v Claim Verification “Immigrants are a drain on the economy”
57 Training a Fact Verification System Evidence Claim Fact Checking Model Human-labeled (Evidence, Claim, Label) Training SUPPORT REFUTE NEI Label Claim: Penguin Books is a publishing house founded in 1930. Evidence: Penguin Books was founded in 1935 by Sir Allen Lane as a line of the publishers The Bodley Head, only becoming a separate company the following year. Label: REFUTES
58 Training a Fact Verification System Evidence Claim Fact Checking Model Human-labeled (Evidence, Claim, Label) Training SUPPORT REFUTE NEI Label Time-consuming and costly to create training data • 50 annotators • Write ~185,000 claims • Data validation
62 Claim Generation with QG Evidence (𝓟) 1992 Los Angeles riots The 1992 Los Angeles riots, also known as the Rodney King riots were a series of riots, lootings, arsons, and civil disturbances that occurred in Los Angeles County, California in April and May 1992. By the time the riots ended, 63 people had been killed. Extra Contexts (𝓟𝒆𝒙𝒕 ) ⋯ ⋯ ⋯ ⋯ ⋯ ⋯ Q: Where did the Rodney King riots happen? A: Los Angeles County Q: How many people were killed in the Rodney King riots? A: 63 Question Generator Q: Where did the Rodney King riots happen? A: San Francisco County Answer Replacement The Rodney King riots took place in Los Angeles County. The Rodney King riots took place in San Francisco County. 63 people were killed in the Rodney King riots. SUPPORTED REFUTED NOT ENOUGH INFO QA-to-Claim Model
Key Components q Question Generator o BART model finetuned on the SQuAD dataset. q QA-to-Claim o BART model finetuned on the QA2D dataset (Demszky et al., 2018) 63 Who called Taylor? [SEP] Liz Liz called Taylor. BART generate question: 42 is the answer to life, the universe and everything. What is the answer to life, the universe and everything? BART
Answer Replacement 64 Evidence: Homeland is an American spy thriller television series developed by Howard Gordon and Alex Gansa based on the Israeli series Prisoners of War, which was created by Gideon Raff . Local Replacement v Replace the answer with an entity with the same type within the evidence. Ques: Who developed Homeland? Ans: Howard Gordon→ Gideon Raff Claim: Gideon Raff developed the Homeland. Global Replacement v Replace the answer with an entity that is close in semantics. v We use the sense2vec (Trask et. al, 2015) to find the closest phrases. Ques: Who developed Homeland? Ans: Howard Gordon→Claire Danes Claim: Claire Danes developed the Homeland.
76 Survey for Neural Question Generation • https://arxiv.org/pdf/1905.08949.pdf Know more about QG Question Generation Paper List • https://github.com/teacherpeterpan /Question-Generation-Paper-List
wingnus
mickey_kubo
yukizenimoto
tmats
ryou0634
yudai00
stktu
elix
yowata
mhangyo
hiroyaonoe
cheerfularge
palkan
philhawksworth
hursman
denniskardys
bcantrill
chriscoyier
aki_iinuma
cherdarchuk
lynnandtonic
paulrobertlloyd
marcduiker
aarron