Q Generated Question: What Shakespeare scholar is currently on the faculty? Input Context: Current faculty include the anthropologist Marshall Sahlins and the Shakespeare scholar David Bevington. Answer: David Bevington
Methods • Apply linguistic rules to transform a declarative sentence into a question. • Fill out pre-defined question templates. • Sequence-to-Sequence Model • Encoder: encode the input passage and answer • Decoder: decode question token by token (Image Credit: R. Zhang et al. 2021)
11 Generate clarification questions in dialogue I am working for an employer in Canada. Do I need to carry on paying UK National Insurance Have you been working abroad 52 weeks or less?
(Puri et al., EMNLP 2020) (Yue et al., EMNLP 2021) Question Answering Applications of Question Generation 12 Generate training data for question answering
QA System • 78K questions • 13K HITs • $2.3 USD/HIT • 2 min per sample • Total Cost: >30,000 USD • 112K questions Time-consuming and costly to create training data.
multi-hop question, answer) as supervision. Composing sub-questions into multi-hop questions Quality of the generated data How to ensure the generated (source, question, answer) pair is valid ? o The question performs multi-hop reasoning over the input source. o The generated answer is correct. Defining an interpretable process that mimics human reasoning
sang "All Join Hands" rise to prominence? Multi-hop Question Which rock band sang "All Join Hands"? Paragraph A: All Join Hands "All Join Hands" is a song by the British rock band Slade, released in 1984 as the lead single from the band's twelfth studio album "Rogues Gallery". Paragraph B: Slade Slade are an English glam rock band from Wolverhampton. They rose to prominence during the early 1970s with 17 consecutive top 20 hits and six number ones on the UK Singles Chart. When did Slade rise to prominence? Ans = Slade Ans = Early 1970s
sang "All Join Hands" rise to prominence? Multi-hop Question Which rock band sang "All Join Hands"? Paragraph A: All Join Hands "All Join Hands" is a song by the British rock band Slade, released in 1984 as the lead single from the band's twelfth studio album "Rogues Gallery". Paragraph B: Slade Slade are an English glam rock band from Wolverhampton. They rose to prominence during the early 1970s with 17 consecutive top 20 hits and six number ones on the UK Singles Chart. When did Slade rise to prominence? Ans = Slade Ans = Early 1970s
sang "All Join Hands" rise to prominence? Multi-hop Question Which rock band sang "All Join Hands"? Paragraph A: All Join Hands "All Join Hands" is a song by the British rock band Slade, released in 1984 as the lead single from the band's twelfth studio album "Rogues Gallery". Paragraph B: Slade Slade are an English glam rock band from Wolverhampton. They rose to prominence during the early 1970s with 17 consecutive top 20 hits and six number ones on the UK Singles Chart. When did Slade rise to prominence? Ans = Slade Composing sub-questions into multi-hop questions An interpretable process that mimics human reasoning
single input source • Aggregate information from two sources. Reasoning Graphs • Each corresponds to one type of multi- hop question • Formulated as a computation graph built upon the operators. General Framework
Text) or (Text + Text) Bridge Entity Select an entity to serve as the bridge between two texts (or between table and text) 𝐹𝑖𝑛𝑑𝐶𝑜𝑚𝐸𝑛𝑡 Text Comparative Entities Select potential comparative entities from the given text Generation 𝑄𝐺𝑤𝑖𝑡ℎ𝐴𝑛𝑠 Text + Answer Question Generate a question with a given answer from the input text 𝑄𝐺𝑤𝑖𝑡ℎ𝐸𝑛𝑡 Text + Entity Question Generate a question which contains the given entity from the input text 𝐷𝑒𝑠𝑐𝑟𝑖𝑏𝑒𝐸𝑛𝑡 Table + Entity Sentence Generate a sentence that describes the given entity based on the information in the table Q𝑢𝑒𝑠𝑇𝑜𝑆𝑒𝑛𝑡 Question Sentence Convert a question into a declarative sentence Fusion 𝐵𝑟𝑖𝑑𝑔𝑒𝐵𝑙𝑒𝑛𝑑 Question + Sentence + Bridge Multi-hop Question Generate a bridge-type multi-hop question 𝐶𝑜𝑚𝑝𝑎𝑟𝑒𝐵𝑙𝑒𝑛𝑑 Question + Question Multi-hop Question Generate a comparative-type multi-hop question Operators
sang "All Join Hands" rise to prominence? Answer: Early 1970s Paragraph A: All Join Hands "All Join Hands" is a song by the British rock band Slade, released in 1984 ⋯ ⋯ Paragraph B: Slade Slade are an English glam rock band from Wolverhampton. They rose to prominence during the early 1970s with ⋯ ⋯ ⋯ #𝟐 When did the rock band Slade rose to prominence? Answer: Early 1970s #𝟑 What rock band sang “All Join Hands”? Answer: Slade #𝟒 Slade sang “All Join Hands”. #𝟓 𝐹𝑖𝑛𝑑𝐵𝑟𝑖𝑑𝑔𝑒 𝑄𝐺𝑤𝑖𝑡ℎ𝐴𝑛𝑠 𝐵𝑟𝑖𝑑𝑔𝑒𝐵𝑙𝑒𝑛𝑑 𝑄𝑢𝑒𝑠𝑇𝑜𝑆𝑒𝑛𝑡
Dutch professional racing cyclist, ⋯ ⋯ ⋯. Wild competed in two track cycling events at the 2012 Summer Olympics. Kirsten Wild Kirsten Wild of Netherlands won the bronze medal in the 2011 Apeldoorn. What is the birthdate of Kirsten Wild? Answer: 15 October 1982 𝐵𝑟𝑖𝑑𝑔𝑒𝐵𝑙𝑒𝑛𝑑 #𝟒 What is the birthdate of the athlete that of Netherlands won the bronze medal in the 2011 Apeldoorn? Answer: 15 October 1982 Medal Championship Name Event Silver 2010 Pruszkow Tim Veldt Men’s omnium Bronze 2011 Apeldoorn Kristen Wild Women’s omnium Gold 2013 Apeldoorn Elis Ligtlee Women’s keirin 𝐹𝑖𝑛𝑑𝐵𝑟𝑖𝑑𝑔𝑒 #𝟏 𝐷𝑒𝑠𝑐𝑟𝑖𝑏𝑒𝐸𝑛𝑡 #𝟐 𝑄𝐺𝑤𝑖𝑡ℎ𝐸𝑛𝑡 #𝟑
+ Answer Question Generate a question with a given answer from the input text 𝑄𝐺𝑤𝑖𝑡ℎ𝐸𝑛𝑡 Text + Entity Question Generate a question which contains the given entity from the input text 𝐷𝑒𝑠𝑐𝑟𝑖𝑏𝑒𝐸𝑛𝑡 Table + Entity Sentence Generate a sentence that describes the given entity based on the information in the table Fusion 𝐵𝑟𝑖𝑑𝑔𝑒𝐵𝑙𝑒𝑛𝑑 Question + Sentence + Bridge Multi-hop Question Generate a bridge-type multi-hop question Operators
Google-T5 model jointly trained on three tasks based on the SQuAD dataset. • Answer Prediction, Question Generation, Question Answering extract answer: <hl> 42 is the answer to life, the universe and everything. <hl> generate question: <hl> 42 <hl> is the answer to life, the universe and everything. question: What is the answer to life? context: 42 is the answer to life, the universe and everything. 42 What is the answer to life, the universe and everything? 42 Google T5 36
the ToTTo dataset. Medal Championship Name Event Silver 2010 Pruszkow Tim Veldt Men’s omnium Bronze 2011 Apeldoorn Kristen Wild Women’s omnium Gold 2013 Apeldoorn Elis Ligtlee Women’s keirin Gold 2013 Apeldoorn Elis Ligtlee Women’s sprint Input Table + Target Entity Netherlands at the European Track Championships The table title is Netherlands at the European Track Championships . The Medal is Bronze . The Championship is 2011 Apeldoorn . The Name is Kirsten Wild . The Event is Women's omnium . Start describing Kirsten Wild : Kirsten Wild of Netherlands won the bronze medal in the 2011 Apeldoorn. Table Templatization Pretrained GPT-2
Kirsten Wild? Answer: 15 October 1982 #: Kirsten Wild of Netherlands won the bronze medal in the 2011 Apeldoorn. What is the birthdate of the _____ that of Netherlands won the bronze medal in the 2011 Apeldoorn? What is the birthdate of the athlete that of Netherlands won the bronze medal in the 2011 Apeldoorn? Answer: 15 October 1982 BERT 3. 𝐵𝑟𝑖𝑑𝑔𝑒𝐵𝑙𝑒𝑛𝑑 (Combining sub-parts)
did the one that won the Eurovision Song Contest in 1966 join Gals and Pals? 1963 How many students attend the teams that played in the Dryden Township Conference? 1900 Text-Table What album did the Oak Ridge Boys release in 1989? American Dreams When was the name that is the name of the bridge that crosses Youngs Bay completed? Summer Text-Text Which Canadian cinematographer is best known for his work on Fargo? Craig Wrobleski What is illegal in the country that is Bashar Hafez al - Assad ’s father? Cannabis Comparison Who was born first, Terry Southern or Neal Town Stephenson? Terry Southern Are Beth Ditto and Mary Beth Patterson of the same nationality? Yes
template-based method + BERT to compose questions. This sometimes makes the generated questions look unnatural. o Generated: What is illegal in the country that is Bashar’s father? o Human: What is illegal in Bashar’s father’s country? Hard to generate “less apparent” multi-hop questions For some multi-hop questions, the decomposition into sub-parts are not evident from the question itself. o Question: Did Aristotle use a laptop? o Evidence 1: Aristotle was died in 322BC. o Evidence 2: The first laptop was invented in 1980.
Join Hands" rise to prominence? Multi-hop Question Which rock band sang "All Join Hands"? Paragraph A: All Join Hands "All Join Hands" is a song by the British rock band Slade, released in 1984 as the lead single from the band's twelfth studio album "Rogues Gallery". Paragraph B: Slade Slade are an English glam rock band from Wolverhampton. They rose to prominence during the early 1970s with 17 consecutive top 20 hits and six number ones on the UK Singles Chart. When did Slade rise to prominence? Ans = Slade Composing simple questions into complex questions An interpretable process that mimics human reasoning Generator Tables Documents QA Model Train
Model Human-labeled (Evidence, Claim, Label) Training SUPPORT REFUTE NEI Label Claim: Penguin Books is a publishing house founded in 1930. Evidence: Penguin Books was founded in 1935 by Sir Allen Lane as a line of the publishers The Bodley Head, only becoming a separate company the following year. Label: REFUTES
Model Human-labeled (Evidence, Claim, Label) Training SUPPORT REFUTE NEI Label Time-consuming and costly to create training data • 50 annotators • Write ~185,000 claims • Data validation
riots The 1992 Los Angeles riots, also known as the Rodney King riots were a series of riots, lootings, arsons, and civil disturbances that occurred in Los Angeles County, California in April and May 1992. By the time the riots ended, 63 people had been killed. Extra Contexts (𝓟𝒆𝒙𝒕 ) ⋯ ⋯ ⋯ ⋯ ⋯ ⋯ Q: Where did the Rodney King riots happen? A: Los Angeles County Q: How many people were killed in the Rodney King riots? A: 63 Question Generator Q: Where did the Rodney King riots happen? A: San Francisco County Answer Replacement The Rodney King riots took place in Los Angeles County. The Rodney King riots took place in San Francisco County. 63 people were killed in the Rodney King riots. SUPPORTED REFUTED NOT ENOUGH INFO QA-to-Claim Model
the SQuAD dataset. q QA-to-Claim o BART model finetuned on the QA2D dataset (Demszky et al., 2018) 63 Who called Taylor? [SEP] Liz Liz called Taylor. BART generate question: <hl> 42 <hl> is the answer to life, the universe and everything. What is the answer to life, the universe and everything? BART
television series developed by Howard Gordon and Alex Gansa based on the Israeli series Prisoners of War, which was created by Gideon Raff . Local Replacement v Replace the answer with an entity with the same type within the evidence. Ques: Who developed Homeland? Ans: Howard Gordon→ Gideon Raff Claim: Gideon Raff developed the Homeland. Global Replacement v Replace the answer with an entity that is close in semantics. v We use the sense2vec (Trask et. al, 2015) to find the closest phrases. Ques: Who developed Homeland? Ans: Howard Gordon→Claire Danes Claim: Claire Danes developed the Homeland.
into complex questions. q Benefit Fact Verification by connecting claim generation with question generation. How can question generation benefit downstream NLP applications?