Natural Language Inference for Humans

Valeria de Paiva

Talk starting the series
Women+@DCS Sheffield
July 2020

Transcript

1. 1/23 Introduction NL Inference Women in Computer Science Natural Language Inference for Humans Valeria de Paiva Women+@DCS Sheffield July 2020 Valeria de Paiva Women+@DCS
2. 2/23 Introduction NL Inference Women in Computer Science Thanks, Aline! Valeria de Paiva Women+@DCS
3. 3/23 Introduction NL Inference Women in Computer Science Personal stories I’m a logician, a proof-theorist, a computational semanticist and a category theorist. I work in industry in Silicon Valley, have done so for the last 20 years, applying the purest of pure mathematics, in surprising ways. Valeria de Paiva Women+@DCS
4. 4/23 Introduction NL Inference Women in Computer Science Personal stories Valeria de Paiva Women+@DCS
5. 5/23 Introduction NL Inference Women in Computer Science PARC, XLE and Bridge Valeria de Paiva Women+@DCS
6. 6/23 Introduction NL Inference Women in Computer Science Powerset, Cuil and Nuance Valeria de Paiva Women+@DCS
7. 7/23 Introduction NL Inference Women in Computer Science Natural Language Inference (NLI) A shock when the work of almost a decade at PARC was out of reach when I left in 2008 I gave a talk at SRI proposing to redo it all, open source (de Paiva 2010 Bridges) Pleased to report that almost all of it is now available open-source Most work with/by Katerina Kalouli, PhD student at Konstanz Valeria de Paiva Women+@DCS
8. 8/23 Introduction NL Inference Women in Computer Science Natural Language Inference: why? In May 2016 Google announced Parsey McParseface, the world’s most accurate parser1: 94% accuracy In 2014 Marelli et al launched the SICK corpus at SemEval 2014: an easy (no named entities, no temporal phenomena, limited vocabulary, etc..), linguist curated corpus to test compositional knowledge Can we use SyntaxNet to process SICK with off-the-shelf tools such as WordNet and SUMO? It’s complicated! Five papers and counting! 1ai.googleblog.com/2016/0/announcing-syntaxnet-worlds-most. html Valeria de Paiva Women+@DCS
9. 9/23 Introduction NL Inference Women in Computer Science Natural Language Inference: what? Examples from SNLI dataset at Stanford Valeria de Paiva Women+@DCS
10. 10/23 Introduction NL Inference Women in Computer Science NLI for Humans Easier to detect inference than to decide on “good”semantic representations Data-driven NLU need large, diverse, high-quality corpora annotated to learn inference relations: entails, contradicts, neutral Can we trust the corpora we have? Are they really learning logical inferences? Are the findings on the big corpora available SNLI, MNLI, SciTail, etc transferable and generalizable? (Plenty of recent work showing no, systems learn biases of the corpora, cannot be redeployed) Valeria de Paiva Women+@DCS
11. 11/23 Introduction NL Inference Women in Computer Science NLI for SICK Explaining Simple Natural Language Inference ACL2019 Textual Inference: getting logic from humans IWCS2017 Correcting Contradictions, CONLI 2017 Graph Knowledge Representations for SICK, NLCS2018 WordNet for “Easy” Textual Inferences LREC2018 Valeria de Paiva Women+@DCS
12. 12/23 Introduction NL Inference Women in Computer Science NLI for SICK Are the annotations in SICK logical? Can we trust them? Several problems: lack of guidelines on co-reference, how to annotate contradictions, ungrammatical and non-sensical sentences, noisy data, etc.. This meant contradictions in SICK are not symmetric and they need to be Contradictions require alignment between entities and events, which need to be ”close enough” how to decide when things are close enough? Can we do simpler case where sentences are ”one-word-apart”using WordNet? More guidelines necessary for SICK annotation? Valeria de Paiva Women+@DCS
13. 13/23 Introduction NL Inference Women in Computer Science NLI for SICK https://logic-forall.blogspot.com/2020/03/ sick-dataset-in-these-trying-times.html Valeria de Paiva Women+@DCS
14. 14/23 Introduction NL Inference Women in Computer Science Are we there yet? Manning: Computational Linguistics and Deep Learning, 2015 ”NLP is kind of like a rabbit in the headlights of the Deep Learning machine, waiting to be flattened.” Hinton 2015: ”I will be disappointed if in five years’ time we do not have something that can watch a YouTube video and tell a story about what happened.” [not totally flattened, yet] Valeria de Paiva Women+@DCS
15. 15/23 Introduction NL Inference Women in Computer Science Conclusions so far Working for division of semantic labor between symbolic/structural and distributional approaches Have fledgling proposal GKR with strict separation of conceptual and contextual structures Also concrete proposal for injecting distributionality in GKR: promising results (COLING submission) Further Work: Still working to produce a ‘correct’ SICK Working on annotations and theorem provers test GKR with further datasets, further distributional architectures Valeria de Paiva Women+@DCS
16. 16/23 Introduction NL Inference Women in Computer Science 4th Workshop Women in Logic 2020 Valeria de Paiva Women+@DCS
17. 17/23 Introduction NL Inference Women in Computer Science Women in Computer Science I grew up believing most of the gender wars had been fought by our grandmothers, suffragettes or not. that the law allowed me to get into colleges and work places. that I could always apply for scholarships and grants. I had plenty of women teachers. I thought my job was to work hard and show people I could do the job as well as any man I knew the numbers were bad both in Computing and in Maths, but I thought they’re bad as usual, not particularly bad. That time would be on our side, that things were going to get more equal as time went by Valeria de Paiva Women+@DCS
18. 18/23 Introduction NL Inference Women in Computer Science Women in Computer Science Valeria de Paiva Women+@DCS
19. 19/23 Introduction NL Inference Women in Computer Science Women in Computer Science Valeria de Paiva Women+@DCS
20. 20/23 Introduction NL Inference Women in Computer Science Women in Computer Science When Nat Shankar asked me if I wanted to say a few words about Logic in Computer Science, in its 30th birthday, I warned him that he might not like the few words. Then we launched the Workshop Women in Logic, the facebook group Women in Logic and the blog. Valeria de Paiva Women+@DCS
21. 21/23 Introduction NL Inference Women in Computer Science Women in Computer Science Workshops in Iceland, UK, Canada and this year Paris, France. Funding for scholarships from SIGLOG, VCLA (Vienna Center for Logic and Algorithms), and ILLC (institute for Language, Logic, and Computation), Amsterdam, Netherlands. Valeria de Paiva Women+@DCS
22. 22/23 Introduction NL Inference Women in Computer Science Women in Computer Science Data We have a spreadsheet of women logicians, editable by everyone, since 2012. A collection of spreadsheets checking numbers of female Invited Speakers in many of the theoretical Computer Science main conferences. Careful work on number of women invited speakers for the ASL meetings (thanks Johanna Franklin!) Have a mailing list and many plans. Join us! Valeria de Paiva Women+@DCS
23. 23/23 Introduction NL Inference Women in Computer Science More information GKR Demo: http://lap0973.sprachwiss.uni-konstanz.de: 8080/sem.mapper/ GKR source code: https://github.com/kkalouli/GKR_semantic_parser Ask KAterina questions! Play with it and tells us all the other things we haven’t done, yet! Thanks! Valeria de Paiva Women+@DCS