[VJAI seminar] Language_models_2018

Presentation of language model breakthroughs in 2018 at VJAI seminar.

Transcript

2. 
      • : 2 : , 2 : ? • 2 2 2 ? : ? : :: : • 2 : : ? : • 2 , 2 2 : • : : ? • : ? 2
3. 
    • • •
4. • ) /0 0 - - 1 08 0 08 0 • /0 • 8 2 • - 0/ 8 2 • 0 80 82 20 /0 () • - 0/ 0 0 08 8 1 8 / 0 1 / 08 8 • 0 80 0 8 0 / - 0/ 0 0 08 8 • 0 0 - 0 / 0 0 / /0 /00 08 08 08 0
5. • B F 2A • 1 , 0A2 B A A 0 • 0 A A2 B A 2A 8 A ,0 E2A F 2B B 8 A2 2 F
6. 
   Task Description ELMo OpenAI Transformer BERT_base GLUE A set of 8 tasks of language understanding 1. MNLI Predict entailment/contradiction/neutral of a sentence pair 2. QQP Determine if two questions asked on Quora are semantically equivalent 3. QNLI Binary classification version of SQuAD: whether sentence in (question, sentence) pair contains answer to question 4. SST-2 Binary single-sentence classification of movie reviews 5.CoLA Binary single-sentence classification of linguistical acceptability 6. STS-B Score 1 to 5 for semantic similarity of sentence pairs from news headlines and other resources 7. MRPC Binary classification for semantic similarity of sentence pairs from online news 8. RTE Binary entailment task similar to MNLI, but with much less training data SQuAD Find answer for a question in a text span from Wikipedia CoNLL NER Named entity recognition SWAG Predict sentence entailment from multiple choice question 81.x Coref Clustering mentions in text that refer to the same underlying real world entities (OntoNotes coreference from CoNLL 2012 shared task) SRL "Who did what to whom" tagging SNLI Predict entailment/contradiction/neutral of a sentence pair SciTail Multiple choice question-answering given candidate knowledge sentences RACE Multiple choice for sentence completion after reading comprehension ROCStories Binary choice question-answering after reading comprehension of short stories COPA Commonsense binary choice question-answering
8. 
   
9. • - - - • - • !
10. • http://u.cs.biu.ac.il/~yogo/blackbox2018.pdf
11. • 1 C :B ? C : C? ( • ? : C B B B: BB: : C:? B: C? C B: : :BC: C B ? B C • ,? B -) : : C : )?0 BC/ B ): ./, B ? - C • ): ./, :B BC ? • )?0 :B BC: ?? B :C B: L S N MLW C C V LT
12. • - • ? >
13. • • - EMNLP 2018 ? ? >
14. • 0 ? - 0 - ? 0 / ? : • A/ / : ? A? 0 • ,? - : 0 A / : - ? • :- / -: : ? A22 -/ / : 22 / / : ? :? :2 A : 0? : 0 / : : A: -0 : 0 / : -: 0 :
15. • ? / 0 /0 ? 5 50 = /0. • ? / 0 0 = -0=?00 0 / . =0 = • . =0 = ? / ? 0 - /0 = 50 • A 0 :0 : 0 =A .3 0 ?30 / : : . =0 = 0: .0 3 = 0 =3 / : - = 0 0.= ? / =3 = ::0 A ? A 0 -A / : = 0= 0.= 0 =3 / : .= / : .= 0.= 0
16. • )( ( ) ( • )( ( ( ) • ) ( () ) • ( ( ( (
17. 
     
18. • . •
19. • - • - • # • - -
20. • • http://aclweb.org/anthology/P18-1167
21. • - : • • : • : : : • - - - • •
22. + • • + • + + • + + • +
23. + . . + . +2 2 1 2 + 2 + . .2 2 2 2 1 2 + 1 2 2 2+ A 2 + 1 2 • BERT objective: • total_loss = masked_lm_loss + next_sentence_loss
24. + . . + . +2 2 1 2 + 2 + . .2 2 2 2 1 2 + 1 2 2 2+ A 2 + 1 2 • BERT objective: • total_loss = masked_lm_loss + next_sentence_loss
25. • • • • •
27. • + + • -
28. • - / : : , / , • : : : : , : : : + - : , : / - , : + , :
29. • • - -
30. • • : -
31. 
     • , , / , , • , , , /, - / /, , , - - , , /, • , , , - , , / , , • - , , / /, , / , , , , • , , , , ,- , , -- - , - ,
32. • - 2/ 1 / 0 2 0 • 1 1 : 0 / / : 2 • 1 / 1 0/ 1 0 18E 1/ /0 • - / : . / / : 2 / : 2 2 1 • / / / 2 2 /: 1 1